Compare commits

...
Author SHA1 Message Date
Andrew Beveridge ☄️ 4fcd899f1b Update pyproject.toml 2026-01-06 01:26:16 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 b345135be1 feat: Unify GCE encoding with LocalEncodingService (single code path) (#214)
## Summary
- Unifies GCE encoding worker with LocalEncodingService (same code as
local CLI)
- Removes all fallback/simplified encoding logic
- Produces proper filenames: `Artist - Title (Final Karaoke Lossless
4k).mp4`
- Includes concatenated title + karaoke + end screens
- CI uploads wheel to GCS for hot code updates without restart

## Problem
Cloud/remote karaoke generation was using simplified, duplicated
encoding logic that:
- Used generic filenames like `output_4k_lossless.mp4`
- Did NOT concatenate title + karaoke + end screens
- Was missing MKV format for YouTube uploads

## Solution
GCE worker now:
1. Downloads latest karaoke-gen wheel from GCS at job start
2. Imports and uses `LocalEncodingService` (same as local CLI)
3. Produces identical output to local karaoke-gen

## Test plan
- [ ] Run a cloud generation job after merging
- [ ] Verify output files have proper names
- [ ] Verify videos include title/end screens
- [ ] Verify Dropbox/GDrive uploads work correctly

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 01:25:45 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 1114bd5554 fix: Use FFmpeg filter escaping instead of shell escaping for paths (#213)
## Summary
- Fixes video generation for songs with apostrophes in titles (e.g.,
"I'm With You")
- PR #212 introduced shell-style escaping (`'\''`) which doesn't work
with subprocess command lists
- Changed to FFmpeg's native filter escaping rules

## Test plan
- [x] All 22 video.py unit tests pass
- [x] All 68 backend tests pass  
- [x] End-to-end FFmpeg test with apostrophe in path succeeds
- [ ] Manual test with "Avril Lavigne - I'm With You"

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 23:45:27 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 60af90de31 fix: Escape FFmpeg filter paths using single-quote wrapping (#212)
## Summary
- Fix FFmpeg failing on paths with apostrophes (e.g., "I'm With You")
- Use single-quote wrapping with `'\''` pattern for embedded quotes
- Update both `video.py` and `video_background_processor.py`

## Test plan
- [x] Unit tests updated and passing (1006 tests)
- [x] E2E tests passing (68 tests)
- [x] Manually tested with "Avril Lavigne - I'm With You" - video
generates successfully

## Technical details
FFmpeg filter syntax interprets apostrophes as special characters.
Simple backslash escaping (`\'`) doesn't work because FFmpeg's filter
parser consumes the backslash. The solution is to wrap paths in single
quotes and escape embedded quotes using the `'\''` pattern (end quote,
literal quote, start quote).

Example: `I'm With You` becomes `'I'\''m With You'`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 03:50:24 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 2110d0f84b perf: Optimize agentic correction with parallel gap processing (~5-6x speedup) (#211)
## Summary

- Fix anti-pattern of creating new LLM model instance per gap (caused
repeated 2s warm-up overhead)
- Create agentic corrector once and reuse for all gaps
- Process gaps in parallel using ThreadPoolExecutor (default 5 workers)
- Remove unnecessary warm-up code from langchain_bridge.py
- Add `AGENTIC_MAX_PARALLEL_GAPS` env var to configure parallelism

**Performance improvement**: ~5-6x speedup (20 gaps: ~55s vs ~5+ minutes
previously)

## Test plan

- [x] Ran local test with 20 gaps - completed in ~55 seconds
- [x] Verified parallel processing logs show concurrent gap processing
- [x] Verified model initialization only happens once at start
- [x] CodeRabbit review passed locally

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 22:43:13 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 4b0dd6169f ci: Propagate base image to all runners after rebuild (#210)
## Summary
- Adds new `propagate-base-image` job that runs after base image rebuild
- Spawns 20 parallel jobs to pull the new base image to all self-hosted
runners
- Eliminates ~8 minute pull delay on subsequent app builds

## Problem
When the base image is rebuilt (due to `poetry.lock` or
`Dockerfile.base` changes), other runners still have the old image
cached. The first app build on each runner must pull the new ~5GB image,
adding ~8 minutes to deployment time.

Example:
https://github.com/nomadkaraoke/karaoke-gen/actions/runs/20735879195/job/59532914508

## Solution
After base image is pushed, trigger 20 parallel jobs that pull the new
image to all runners immediately. Future app builds will already have
the latest base image cached.

## Implementation
- Added `base_rebuilt` output to `deploy-backend` job
- New `propagate-base-image` job with matrix strategy (20 parallel jobs)
- `fail-fast: false` so one offline runner doesn't block others
- Uses same Workload Identity auth as other deployment jobs

## Test plan
- [ ] Verify workflow syntax is valid (YAML parses correctly)
- [ ] Merge and observe next base image rebuild triggers propagation
- [ ] Confirm subsequent app builds don't need to pull base image

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 21:54:55 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 8e3e3e4713 fix: Check base image hash from registry instead of stale local cache (#209)
## Summary
- Fix GitHub Actions base image cache check to query registry directly
instead of trusting potentially stale local Docker cache
- Prevent unnecessary base image rebuilds caused by runners with
outdated cached images
- Ensure app image is always built on the correct base image

## Changes
- Fetch image hash label via lightweight registry API (~10KB config
blob) instead of pulling full image
- Add step to verify local cache matches registry before building app
image
- Compare config digests consistently for reliable image matching

## Problem
GitHub Actions runners can retain Docker images between runs. When the
runner's local cache has an old base image, the previous logic would:
1. Check local cache first → find old image with wrong hash
2. Conclude base image needs rebuild → unnecessary ~10 min rebuild
3. Or worse: build app image on stale base if hashes happened to match
older state

## Testing
- [x] Verified registry API returns correct hash label
- [x] CodeRabbit CLI review completed locally

## Review
- [x] Local CodeRabbit review completed
- [x] Review feedback addressed (fixed digest comparison types)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 21:27:02 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 fef9a45739 docs: Add Pulumi CI skip-preview lesson learned (#208)
## Summary
- Add Pulumi CI skip-preview lesson to `LESSONS-LEARNED.md`
- Update `CLAUDE.md` to emphasize infrastructure changes require PRs
- Add detailed Infrastructure section to `DEVELOPMENT.md` documenting:
  - What's managed by Pulumi
  - The PR-based workflow
  - Why not to run `pulumi up` locally

## Changes
| File | Change |
|------|--------|
| `docs/LESSONS-LEARNED.md` | Added "Pulumi CI Deployments Need
Skip-Preview" section |
| `CLAUDE.md` | Strengthened Infrastructure section with PR workflow |
| `docs/DEVELOPMENT.md` | Added comprehensive Infrastructure Changes
section |

## Test plan
- [x] Documentation only - no code changes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 21:09:45 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 69b87c232a fix: Add IAM permissions for Pulumi to refresh state in CI (#207)
## Summary
- Add `roles/resourcemanager.projectIamAdmin` for IAM binding state
refresh
- Add `roles/secretmanager.admin` for secret state refresh
- Use `--skip-preview` instead of `--refresh` to avoid needing read
permissions on all GCP resource types

The service account would need many additional roles (Cloud Tasks,
Firestore, Monitoring, etc.) for Pulumi to refresh all resources. Using
`--skip-preview` is the standard approach for CI/CD where Pulumi state
is authoritative.

## Test plan
- [ ] CI passes on this PR
- [ ] After merge, Pulumi deployment succeeds

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 18:57:10 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 0530e5f226 fix: Use correct Pulumi stack name in CI infrastructure deploy (#206)
## Summary
- Fixed Pulumi stack reference from `nomadkaraoke/karaoke-gen/prod` to
`nomadkaraoke/karaoke-gen-infrastructure/prod` (matches project name in
`Pulumi.yaml`)
- Added GCP Workload Identity authentication for Pulumi deployment step

## Test plan
- [ ] CI passes on this PR
- [ ] After merge, infrastructure deployment job succeeds

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 15:34:00 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 d8710bc3d4 ci: Add infrastructure deployment job for GCE encoding worker (#205)
## Summary
Adds a Pulumi deployment job that runs on main branch pushes to
auto-deploy infrastructure changes.

## Problem
Infrastructure changes (like GCE encoding worker code) weren't being
deployed automatically. This caused issues where the backend was
deployed with new code but the GCE worker still had old code.

## Solution
New `deploy-infrastructure` job that:
- Runs Pulumi to deploy infrastructure changes
- Restarts the encoding worker service to pick up new code

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 15:07:41 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 c83ac4376c fix: Handle cached preview responses from GCE worker (#204)
## Summary
- Fixes issue where subsequent preview requests hang indefinitely
- GCE worker now returns "cached" status for already-encoded previews
instead of 409 Conflict
- Backend handles cached response by returning immediately without
polling

## Problem
When users clicked preview multiple times without changing lyrics:
1. First preview worked correctly
2. Subsequent previews hung forever because GCE returned 409 Conflict
3. Backend raised an error instead of handling the cached case

## Solution
- GCE `/encode-preview` endpoint now returns `{"status": "cached", ...}`
for completed jobs
- Backend `encode_preview_video()` checks submit response and returns
immediately for cached
- Also handles "in_progress" status when another request is already
encoding

## Test plan
- [x] First preview generates correctly
- [ ] Subsequent previews (unchanged lyrics) return cached video
instantly
- [ ] Changed lyrics generate new preview video

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 19:47:35 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 e70ce2b558 chore: Enable USE_GCE_PREVIEW_ENCODING in production CI (#203)
## Summary
Enable the GCE preview encoding feature in production CI deployments.

## Changes
- Add `USE_GCE_PREVIEW_ENCODING=true` to Cloud Run environment variables
in CI deploy

## Note
To enable immediately (before next deploy), run:
```bash
gcloud run services update karaoke-backend \
  --region us-central1 \
  --project nomadkaraoke \
  --update-env-vars "USE_GCE_PREVIEW_ENCODING=true"
```

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 18:17:40 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 506db844c2 fix: Double-escape backslashes in GCE embedded Python code (#202)
## Summary
Fix syntax error in GCE encoding worker startup script caused by
incorrect backslash escaping.

## Problem
The `escape_ffmpeg_filter_path` function in the GCE worker startup
script had a syntax error:
```
SyntaxError: unexpected character after line continuation character
```

The issue was that backslashes in the embedded Python code weren't
properly escaped for the triple-quoted string context in Pulumi.

## Solution
Double the backslashes in the Pulumi source:
- 4 backslashes in source → 2 in string value → 2 on VM (valid Python)

## Testing
- GCE encoding worker now starts successfully
- Health endpoint responds correctly
- Already deployed via `pulumi up` and verified working

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 02:03:33 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 be04d3efaf feat: Offload preview video encoding to GCE for faster generation (#200)
## Summary
- Move preview video encoding from Cloud Run to GCE encoding worker
- Reduce preview generation time from 60+ seconds to ~15-20 seconds
- Improve UX during lyrics review by dramatically speeding up preview
generation

## Changes
- **GCE Worker** (`infrastructure/__main__.py`): Add `/encode-preview`
endpoint with `EncodePreviewRequest` model, FFmpeg command for 480x270
preview encoding, and FFmpeg filter path escaping
- **EncodingService** (`backend/services/encoding_service.py`): Add
`is_preview_enabled` property, `submit_preview_encoding_job()`, and
`encode_preview_video()` methods
- **Config** (`backend/config.py`): Add `USE_GCE_PREVIEW_ENCODING`
setting (default: false)
- **Review Route** (`backend/api/routes/review.py`): Modify
`generate_preview_video()` to use GCE with fallback to local encoding
- **LyricsTranscriber** (`operations.py`, `generator.py`): Add
`ass_only` parameter to generate ASS subtitles without video encoding
- **Documentation**: Update ARCHITECTURE.md and LESSONS-LEARNED.md

## Technical Approach
The optimization splits preview generation into two phases:
1. **ASS Generation (Cloud Run)** - Fast text processing (~2s)
2. **Video Encoding (GCE)** - Offloaded to C4-standard-8 VM (~15-20s)

If GCE encoding fails or is disabled, falls back to local Cloud Run
encoding.

## Testing
- [x] Unit tests pass (1004 tests)
- [x] Emulator tests pass (68 tests)
- [x] CodeRabbit CLI review completed locally

## Deployment
1. Deploy infrastructure changes (`pulumi up` or restart GCE worker)
2. Deploy backend to Cloud Run
3. Enable via `USE_GCE_PREVIEW_ENCODING=true` in Cloud Run env

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 01:04:35 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 f2639754a9 fix: Download GCE-encoded files locally for YouTube upload (#201)
## Summary
- Fix YouTube upload failing when using GCE encoding backend
- GCE encoding returns GCS blob paths, but YouTube upload expected local
files
- Added `_download_gce_encoded_files()` to download encoded videos from
GCS before distribution

## Root Cause
Job `62a95e01` completed but didn't upload to YouTube. Investigation
showed:
1. GCE encoding stores files in GCS and returns blob paths like
`jobs/{id}/finals/output_4k_lossless.mp4`
2. The orchestrator stored these paths directly without downloading
3. YouTube upload checks `os.path.isfile()` which returns `False` for
GCS paths
4. Result: "No video file available for YouTube upload" warning

## Changes
| File | Change |
|------|--------|
| `backend/workers/video_worker_orchestrator.py` | Added
`_download_gce_encoded_files()` method called after GCE encoding |
| `backend/tests/test_video_worker_orchestrator.py` | Added 2 tests for
GCE download behavior |
| `docs/LESSONS-LEARNED.md` | Documented the issue and fix |

## Test plan
- [x] All 30 orchestrator tests pass
- [x] New test `test_run_encoding_gce_downloads_files` verifies GCE
paths are downloaded
- [x] New test `test_run_encoding_local_does_not_download` verifies
local encoding skips download
- [ ] Manual test: Create job via web UI and verify YouTube upload works

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 05:51:52 +00:00
Andrew Beveridge ☄️ 94182277fa Update pyproject.toml (#199) 2026-01-05 04:14:37 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 e16da0fa5f fix: Allow explicit override of YouTube upload server default (#198)
## Summary
- Changed `enable_youtube_upload` from `bool=False` to
`Optional[bool]=None` in request models
- Allows explicit `false` to override server default when
`DEFAULT_ENABLE_YOUTUBE_UPLOAD=true`
- E2E tests now intercept requests to disable YouTube upload for test
jobs

## Changes
- **Request models**: Changed to `Optional[bool] = None` to support
three states (None/True/False)
- **API routes**: All job creation endpoints now compute effective value
using `if is not None else` pattern
- **E2E test**: Added Playwright request interception to set
`enable_youtube_upload=false` for test jobs
- **Tests**: Updated to expect `None` instead of `False` for request
model defaults
- **Docs**: Added lesson learned about Python `or` operator gotcha with
boolean defaults

## Testing
- [x] Backend tests pass (`make test`)
- [x] No TypeScript errors in frontend changes

## Review
- [x] CodeRabbit CLI review completed locally
- [x] No issues found

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 23:04:06 -05:00
Andrew Beveridge ☄️andClaude b8e15b2af5 fix: Ensure user data is fetched after client-side navigation to /app (#197)
## Summary

- Fix E2E happy path test failure at Step 3 (Verify Credits)
- After beta enrollment, client-side navigation to `/app` didn't fetch
user data
- Credits weren't displayed because zustand auth store had `user: null`

## Changes

- **frontend/app/app/page.tsx**: Added `useEffect` that calls
`fetchUser()` when there's a token but no user data loaded. This ensures
user data is fetched regardless of how the user navigated to the page
(page reload vs client-side navigation).

- **frontend/e2e/regression/authentication.spec.ts**: Added new test
`'beta enrollment shows credits after redirect to app'` that
specifically verifies credits are displayed after the beta enrollment
flow.

- **docs/LESSONS-LEARNED.md**: Documented the root cause (zustand
module-level initialization only runs once, not on client-side
navigation) for future reference.

## Root Cause

The auth module's initialization code (`fetchUser()` call) only runs
once when the module is first imported. After beta enrollment,
`router.push('/app')` performs client-side navigation which doesn't
reload modules. The user data was never fetched, leaving `user: null` in
the store.

## Testing

- [x] New regression test passes locally (9/9 auth tests pass)
- [x] Build succeeds
- [x] CodeRabbit CLI review completed locally

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 21:33:01 -05:00
Andrew Beveridge ☄️ 2ee3cabc92 Added admin API script (#196) 2026-01-05 01:33:17 +00:00
Andrew Beveridge ☄️andClaude ae85ec0a09 feat: Move worker logs to Firestore subcollection to avoid 1MB limit (#195)
## Summary

Move `worker_logs` from embedded array in job documents to a Firestore
subcollection (`jobs/{job_id}/logs`) to avoid the 1MB document size
limit. This fixes job failures when logs exceed the limit during
long-running operations like video encoding.

**Problem:** Job `501258e1` failed when logs reached 1.26MB (2,547 log
entries), representing 98.6% of the maximum document size.

## Changes

### New Files
- `backend/models/worker_log.py` - WorkerLogEntry dataclass with factory
methods, serialization (to_dict, to_legacy_dict, from_dict)
- `backend/tests/test_worker_log_subcollection.py` - 24 unit tests
- `backend/tests/emulator/test_worker_logs_subcollection.py` - 11
integration tests using Firestore emulator

### Modified Files
- `backend/services/firestore_service.py` - Added subcollection methods
(append, get, count, delete)
- `backend/services/job_manager.py` - Use subcollection with fallback to
legacy embedded arrays
- `backend/api/routes/jobs.py` - Updated logs endpoint to use new
methods
- `backend/config.py` - Added `USE_LOG_SUBCOLLECTION` feature flag
(default: true)
- `infrastructure/__main__.py` - Added TTL policy (30 days) and
composite index for logs subcollection
- `pyproject.toml` - Version bump 0.89.6 → 0.90.0

### Documentation
- `docs/README.md` - Added to Recent Changes
- `docs/ARCHITECTURE.md` - Added logs subcollection to Firestore
Collections table
- `docs/LESSONS-LEARNED.md` - Added "Firestore Document 1MB Limit with
Embedded Arrays" section

## Testing

- [x] 24 unit tests pass
(`backend/tests/test_worker_log_subcollection.py`)
- [x] 11 emulator integration tests pass
(`backend/tests/emulator/test_worker_logs_subcollection.py`)
- [x] All 68 existing emulator tests pass

## Review

- [x] CodeRabbit CLI review completed locally
- [x] Review feedback addressed (timestamp formatting fix)

## Rollback Plan

Set `USE_LOG_SUBCOLLECTION=false` environment variable and redeploy. New
logs will go back to embedded array; subcollection logs will remain but
won't be read.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 20:14:25 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 a2282357da docs: Add lessons learned for GCE encoding response format handling (#194)
## Summary
Document lessons learned from debugging GCE encoding response format
issues.

## Changes
- Add "External Service Response Format Mismatches" section to
LESSONS-LEARNED.md
- Document defensive type checking patterns for API integration
- Add recent changes entry for GCE encoding fixes (PRs #186-193)
- Reference the worker logs rearchitecture plan

## Context
This session fixed multiple response format bugs in the GCE encoding
integration:
- Status endpoint returning list instead of dict
- output_files as list of paths instead of dict with format keys
- Created plan for worker logs to use Firestore subcollection (avoid 1MB
limit)

These lessons are valuable for future debugging of external service
integrations.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 00:29:23 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 44c44da95e fix: Handle GCE worker output_files as list of paths (#193)
## Summary
Fix the actual root cause of `'list' object has no attribute 'get'`
error in GCE encoding.

## Problem
The GCE encoding worker returns `output_files` as a **list of paths**:
```json
"output_files": [
    "jobs/501258e1/finals/output_4k_lossless.mp4",
    "jobs/501258e1/finals/output_4k_lossy.mp4",
    "jobs/501258e1/finals/output_720p.mp4"
]
```

But the code expected a **dict** with format keys:
```json
{"mp4_4k_lossless": "path/...", "mp4_4k_lossy": "path/...", ...}
```

This caused the error when calling `.get()` on the list.

## Solution
Added conversion logic to map the list of paths to the expected dict
format by matching filename patterns (e.g., `output_4k_lossless.mp4` ->
`mp4_4k_lossless`).

## Testing
- [x] Verified GCE worker response format
- [ ] Deploy and test with job 501258e1

## Notes
PR #191 fixed a similar issue in `wait_for_completion` where the entire
`status` response could be a list. This PR fixes a different issue where
`output_files` within the response is a list.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 23:23:21 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 16dc451bdf docs: Add API auth guide and worker logs rearchitecture plan (#192)
## Summary
- Add API authentication documentation to CLAUDE.md for future agents
- Add comprehensive plan for rearchitecting worker logs storage

## Changes

### CLAUDE.md
- New "API Authentication (for Agents)" section covering:
  - Internal worker endpoints with X-Admin-Token
  - Firestore direct access
  - GCE encoding worker SSH commands
  - Common pitfalls to avoid

### .claude/plans/worker-logs-rearchitecture.plan.md
- Plan to move worker_logs from embedded array to Firestore
subcollection
- Fixes 1MB document size limit issue (job 501258e1 hit 1.26 MB)
- Includes TTL policy for automatic cleanup (30 days)

## Testing
- [x] Documentation review
- No code changes

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 22:37:17 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 278690b373 fix: Handle list response in encoding service wait_for_completion (#191)
## Summary
- Fix `'list' object has no attribute 'get'` error in GCE encoding
wait_for_completion
- The GCE encoding worker sometimes returns a list instead of a dict
from the /status endpoint
- PR #186 added handling in encode(), but same issue occurs in
wait_for_completion()

## Changes
- Added defensive handling to convert list responses to dicts in
wait_for_completion()

## Testing
- [x] Code review
- [ ] Deploy and test with job 501258e1

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 22:35:55 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 0802e79261 fix: Pass job_id in encoding options for GCE backend (#190)
## Summary
- Pass `job_id` in the encoding options for GCE backend
- Fixes "Encoding job unknown already exists" error on retry

## Root Cause
The `EncodingInput.options` dict was missing `job_id`, causing it to
fall back to "unknown".

## Testing
- [x] Unit tests pass
- [ ] Test with job `501258e1` after deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 20:40:44 +00:00
Andrew Beveridge ☄️ 504d1159ec chore: force backend deployment (#189)
Trigger deployment for admin endpoint

@coderabbitai ignore
2026-01-04 20:12:12 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 7acc2c9f17 feat: Add admin endpoint to reset stale worker state (#188)
## Summary
- Add `/api/admin/jobs/{id}/reset-worker-state` endpoint
- Allows admins to reset stale worker progress when jobs get stuck

## Context
When a job fails, the worker progress (e.g., `video_progress.stage =
'running'`)
can get stuck, preventing retries. This endpoint resets all worker
progress to 'pending'.

## Testing
- [ ] Test with job `501258e1` after deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 20:00:21 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 a3f5c68660 fix: Reset worker progress state on job retry (#187)
## Summary
- Fix job retry getting stuck because worker progress state wasn't reset
- When retrying, idempotency check saw `video_progress.stage =
'running'` from previous failed run
- Now properly resets progress state to 'pending' for video, render, and
screens workers

## Changes
- `backend/api/routes/jobs.py` - Add `update_state_data` calls to reset
progress

## Testing
- [x] Encoding tests pass locally
- [ ] Test with job `501258e1` after deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 19:24:02 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 27495fb4fe fix: Handle list response from GCE encoding worker (#186)
## Summary
- Fix "'list' object has no attribute 'get'" error when GCE worker
returns a list instead of dict
- Add defensive handling in `GCEEncodingBackend.encode()` to extract
first dict from list responses
- Add local tests to catch encoding integration issues earlier

## Changes
- `backend/services/encoding_interface.py` - Handle list/non-dict
responses from GCE worker
- `backend/tests/test_encoding_interface.py` - Add tests for list
response handling and backend options
- `backend/tests/test_video_worker_orchestrator.py` - Add test for GCS
paths structure
- `backend/tests/conftest.py` - Move env var setup to top before imports

## Testing
- [x] All 57 tests pass locally (`make test`)
- [x] `test_encode_handles_list_result` validates fix
- [ ] Deploy and test with existing job at
`awaiting_instrumental_selection`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 18:48:22 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 b32014a9a0 fix: Pass GCS paths to encoding backend for GCE encoding (#185)
## Summary
- Fixed `GCE encoding requires input_gcs_path and output_gcs_path in
options` error
- Jobs were failing at video generation because orchestrator wasn't
passing required GCS paths

## Changes
- Added GCS path construction in
`VideoWorkerOrchestrator._run_encoding()`
- Paths are now passed in `EncodingInput.options` dict

## Testing
- [x] All 27 orchestrator tests pass
- [x] Ready for quick deploy and test with existing job at
`awaiting_instrumental_selection`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 17:34:35 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 bb19a6e1fc fix: Add dry_run parameter to GCEEncodingBackend (#184)
## Summary
- Fixed `GCEEncodingBackend.__init__() got an unexpected keyword
argument 'dry_run'` error
- Jobs were failing during video generation when using GCE encoding

## Changes
- Added `dry_run` parameter to `GCEEncodingBackend.__init__()` for
interface consistency with `LocalEncodingBackend`
- Parameter is stored but not used since GCE is a remote service that
doesn't support dry run mode

## Testing
- [x] All 23 encoding interface tests pass
- [x] No breaking changes to existing functionality

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 11:54:27 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 1b0521cb83 docs: Add video worker orchestrator documentation (#183)
## Summary
- Add entry to README.md Recent Changes section for orchestrator
refactor
- Add Video Worker Orchestrator section to ARCHITECTURE.md with diagram
- Add "Alternative Code Paths Must Implement All Features" lesson to
LESSONS-LEARNED.md
- Create archive entry documenting the refactor

## Changes
- `docs/README.md` - Added recent change entry
- `docs/ARCHITECTURE.md` - Added orchestrator section with stage diagram
- `docs/LESSONS-LEARNED.md` - Added lesson about code path divergence
- `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` - Full
documentation

## Context
Documentation for PR #182 which unified the video worker pipeline.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 11:07:16 -05:00
Andrew Beveridge ☄️ c1588992ce refactor: Unify video worker pipeline with new orchestrator (#182)
Resolves GCE vs KaraokeFinalise code path divergence that caused features like YouTube upload, Discord notifications, and CDG/TXT packaging to be missing from the GCE encoding path.

- New services: youtube_upload_service, discord_service, packaging_service, local_encoding_service, encoding_interface
- New VideoWorkerOrchestrator coordinates all stages (packaging, encoding, distribution, notifications)
- Works with any encoding backend (GCE or local)
- 139 new tests, 976 total tests passing
- Feature flag USE_NEW_ORCHESTRATOR (default: true) with rollback capability
2026-01-04 00:25:23 -05:00
Andrew Beveridge ☄️andClaude b3045c5487 fix: Add Noto fonts for full Unicode support in video rendering (#181)
## Summary

- Fixes rendering of musical symbols (♪) in INTRO/INSTRUMENTAL/OUTRO
screens
- Adds comprehensive international font support for CJK and other
scripts
- Changes default karaoke font from Arial to Noto Sans

## Changes

- **backend/Dockerfile.base**: Install Noto font packages (core, CJK,
extra, color-emoji) with fontconfig
- **karaoke_gen/style_loader.py**: Change default font from "Arial" to
"Noto Sans"
- **backend/tests/test_routes_review.py**: Update test assertions to
match new default font
- **docs/LESSONS-LEARNED.md**: Add "Fonts in Docker for Video Rendering"
section
- **docs/README.md**: Add "Full Unicode Font Support" to Recent Changes

## Testing

- [x] Tests pass locally (`make test`)
- [x] Manual review of changes completed

## Notes

- Adds ~200-300MB to Docker base image for comprehensive font coverage
- Color emoji has limited support in FFmpeg/libass (renders as
monochrome)
- Requires base image rebuild to take effect

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 23:11:30 -05:00
Andrew Beveridge ☄️andClaude 8a7cbb4a8b feat: Integrate flacfetch cache management API (#180)
## Summary

- Integrate karaoke-gen with flacfetch's new cache management API
(v0.15.4)
- When admins clear a job's cache, both Firestore AND flacfetch GCS
cache are now cleared
- Add bulk cache management (clear all, view stats) to admin UI

## Changes

### Backend
- **FlacfetchClient** (`backend/services/flacfetch_client.py`): Add
cache management methods
  - `clear_search_cache(artist, title)` - clears specific cache entry
  - `clear_all_cache()` - clears entire flacfetch cache  
  - `get_cache_stats()` - returns cache statistics

- **Admin API** (`backend/api/routes/admin.py`): 
- Update `POST /api/admin/audio-searches/{job_id}/clear-cache` to also
call flacfetch cache API
  - Add `DELETE /api/admin/cache` - clear entire flacfetch cache
  - Add `GET /api/admin/cache/stats` - get cache statistics

### Frontend
- **API Client** (`frontend/lib/api.ts`): Add `clearAllCache()` and
`getCacheStats()` methods
- **Admin Searches Page** (`frontend/app/admin/searches/page.tsx`):
  - Add "Clear All Cache" button with confirmation dialog
  - Display cache stats (entry count, size, newest entry)

### Documentation
- Update API.md with new endpoints
- Update README.md with recent changes
- Update LESSONS-LEARNED.md with two-layer caching insight

## Architecture

```
User clicks "Clear Cache" on job
    ↓
karaoke-gen clears Firestore state_data (existing)
    ↓
karaoke-gen calls flacfetch DELETE /cache/search (NEW)
    ↓
flacfetch deletes GCS cache entry
    ↓
Next search hits trackers fresh ✓
```

## Testing

- [x] Backend tests pass (`make test` - 57 tests)
- [x] Frontend tests pass (`npm run test:unit` - 128 tests)
- [x] Frontend builds successfully
- [x] CodeRabbit CLI review completed locally

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 22:48:29 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 3ffed23880 feat: Add customer support link to app header (#179)
## Summary
- Add "Need help?" mailto link visible in header on large screens
- Add support contact option in user dropdown menu for mobile/tablet
users
- Links to andrew@nomadkaraoke.com

## Test plan
- [ ] Verify link appears in header on desktop (lg breakpoint and above)
- [ ] Verify link appears in user dropdown menu
- [ ] Verify clicking opens email client with correct address

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 21:01:47 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 ac39cc5ff7 fix: Add admin link to AuthStatus component dropdown (#178)
## Summary
- The `/app` page uses the `AuthStatus` component for its header
dropdown menu, not the `app-header.tsx` component
- This adds the admin link for users with admin role or
@nomadkaraoke.com email to the correct component

## Test plan
- [ ] Log in as admin user at https://gen.nomadkaraoke.com/app
- [ ] Click user dropdown in header
- [ ] Verify "Admin Dashboard" link appears between "Buy More Credits"
and "Sign Out"

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 20:27:43 -05:00
Andrew Beveridge ☄️andClaude d47634016d feat: Add admin link to header for admin users (#177)
## Summary

- Add "Admin" link to header navigation for users with admin access
- Add "Admin Dashboard" option to user dropdown menu
- Uses same access logic as admin layout (role=admin OR
@nomadkaraoke.com email)

## Changes

- `frontend/components/app-header.tsx` - Added admin link to nav and
dropdown menu with Shield icon

## Testing

- [x] TypeScript compiles
- [ ] Manual testing needed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 20:20:04 -05:00
Andrew Beveridge ☄️andClaude 68188e9a28 feat: Add admin UI for audio search cache management (#176)
## Summary

- Add admin UI at `/admin/searches` to view and manage cached audio
search results
- Allow admins to clear stale caches when flacfetch is updated with new
providers
- Fixes issue where jobs showed only YouTube results when lossless
sources should be available

## Changes

- **Backend**: New admin endpoints `GET /api/admin/audio-searches` and
`POST /api/admin/audio-searches/{job_id}/clear-cache`
- **Frontend**: New admin page with search results table, status
indicators, and cache clear action
- **Safety**: Status validation prevents clearing cache on
active/complete jobs
- **Docs**: Updated API.md, LESSONS-LEARNED.md, and README.md

## Testing

- [x] Backend module imports successfully
- [x] Frontend TypeScript compiles without errors
- [x] CodeRabbit CLI review completed and issues addressed

## Review

- [x] CodeRabbit CLI review completed locally (3 cycles)
- [x] Fixed: `has_lossless` now computed from ALL results, not just
first 10
- [x] Fixed: Added status validation before cache clear
- [x] Noted (not fixed): Scalability optimization for large-scale -
acceptable for admin tool

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 20:08:43 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 cbf3b815dd docs: Add gen_users migration documentation (#174)
## Summary
Adds documentation for the gen_users migration completed in PR #173.

- **README.md**: Add entry in Recent Changes about user database
separation
- **ARCHITECTURE.md**: Add Firestore Collections section documenting
`gen_users` and other collections
- **LESSONS-LEARNED.md**: Add lesson about separate collections for
multi-app projects

## Test plan
- [x] Documentation-only change, no testing required

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 18:09:57 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 6aa8f150df docs: Add GCS path lessons and update E2E spec for testmail.app (#175)
## Summary

- Add "GCS Blob Paths vs Full URIs" section to LESSONS-LEARNED.md
documenting the path format confusion bug and double-slash gotcha that
was fixed in PR #168
- Update E2E-HAPPY-PATH-TEST-SPEC.md secrets section to correctly
reflect testmail.app migration (was referencing MAILSLURP)

## Test plan

- [ ] Documentation review - verify changes are accurate and helpful

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 18:08:09 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 925dd3c262 fix: Complete mailslurp to testmail.app migration (#168)
## Summary
- Complete the migration from MailSlurp to testmail.app for E2E testing
- Update all workflow files to use `TESTMAIL_API_KEY` and
`TESTMAIL_NAMESPACE` secrets
- Update documentation and test comments

## Changes
- `.github/workflows/e2e-happy-path.yml` - Use testmail.app secrets
- `.github/workflows/ci.yml` - Use testmail.app secrets  
- `docs/E2E-HAPPY-PATH-TEST-SPEC.md` - Update documentation
- `frontend/e2e/production/happy-path-real-user.spec.ts` - Update
comments

## Test plan
- [x] Verify no remaining mailslurp references in active files (archives
excluded)
- [ ] E2E Happy Path test should run (not skip) with testmail.app
secrets

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 17:32:53 -05:00
80 changed files with 17278 additions and 5519 deletions

No files matched your search

@@ -0,0 +1,401 @@
# Plan: Video Worker Stage Refactor
**Created:** 2026-01-03
**Branch:** feat/sess-20260103-2250-investigate-youtube-upload
**Status:** COMPLETE - Ready for PR
**Last Updated:** 2026-01-04
## Overview
Refactor `video_worker.py` and `KaraokeFinalise` into separate, composable stages following SOLID principles. This eliminates the GCE vs KaraokeFinalise code path divergence that causes features like YouTube upload to be missing from the GCE encoding path.
## Three Execution Modes
Understanding the three ways this system runs is critical to the refactor:
### 1. Local CLI (`karaoke-gen`)
- Runs entirely on user's machine
- KaraokeFinalise.process() orchestrates everything
- Writes directly to local Dropbox sync folder (e.g., `~/Dropbox/Karaoke/Tracks-Organized/`)
- **Produces full output structure** with stems/, lyrics/, visualizations/, etc.
- **This is the source of truth for output format**
### 2. Remote CLI (`karaoke-gen-remote`)
- User's CLI talks to cloud backend (api.nomadkaraoke.com)
- Cloud does ALL processing (audio, lyrics, video encoding)
- CLI just monitors job status and downloads final outputs
- With Dropbox configured: cloud uploads full structure to user's Dropbox
- Without Dropbox: user downloads individual files
### 3. Web Frontend (`gen.nomadkaraoke.com`)
- Same cloud backend as remote CLI
- User sees:
- YouTube video link
- Dropbox folder link
- Direct download: final karaoke video (720p)
- Direct download: 4K video
- Direct download: "with vocals" video
- Direct download: CDG zip (if enabled)
## Problem Statement
Currently, `video_worker.py` has two completely divergent code paths:
```python
if use_gce_encoding:
# GCE PATH: Only encoding, missing YouTube/Discord/CDG/TXT
result = await _encode_via_gce(...)
# result DOES NOT include youtube_url!
else:
# STANDARD PATH: KaraokeFinalise does everything
finalise = KaraokeFinalise(...)
result = finalise.process()
# result includes youtube_url from upload
```
### What GCE Encoding Broke
| Feature | KaraokeFinalise Path | GCE Path |
|---------|---------------------|----------|
| Video encoding | ✅ 4 formats | ✅ 4 formats |
| YouTube upload | ✅ | ❌ **BROKEN** |
| Discord notification | ✅ | ❌ **BROKEN** |
| CDG/TXT packaging | ✅ | ❌ **BROKEN** |
| Brand code generation | ✅ | ✅ (via Dropbox service) |
| Dropbox upload | ✅ (rclone) | ✅ (native SDK) |
| Google Drive upload | ❌ | ✅ (native SDK) |
### Root Cause
`KaraokeFinalise.process()` is a monolith that handles:
1. CDG/TXT packaging
2. Video encoding (the CPU-intensive part)
3. YouTube upload
4. Discord notifications
5. Folder organization
The GCE encoding was added as a bypass around the entire class, rather than replacing just the encoding step.
## Requirements
### Functional Requirements
- [ ] YouTube upload works regardless of encoding backend (GCE or local)
- [ ] Discord notifications work regardless of encoding backend
- [ ] CDG/TXT packaging works regardless of encoding backend
- [ ] Brand code generation works consistently
- [ ] Dropbox/Google Drive upload works (already works in both paths)
- [ ] All existing CLI functionality continues to work
- [ ] Output structure matches local CLI exactly
### Output Structure Parity
Cloud backend must produce identical output to local CLI:
```
NOMAD-XXXX - Artist - Title/
├── Artist - Title (Title).mov
├── Artist - Title (Title).jpg
├── Artist - Title (End).mov
├── Artist - Title (End).jpg
├── Artist - Title (Final Karaoke Lossless 4k).mp4
├── Artist - Title (Final Karaoke Lossless 4k).mkv
├── Artist - Title (Final Karaoke Lossy 4k).mp4
├── Artist - Title (Final Karaoke Lossy 720p).mp4
├── Artist - Title (Final Karaoke CDG).zip
├── Artist - Title (Final Karaoke TXT).zip
├── Artist - Title (Instrumental {model}).flac
├── Artist - Title (Instrumental +BV {model}).flac
├── Artist - Title (Karaoke).mp4
├── Artist - Title (Karaoke).cdg
├── Artist - Title (Karaoke).mp3
├── Artist - Title (Karaoke).lrc
├── Artist - Title (With Vocals).mp4
├── stems/
│ ├── Artist - Title (Instrumental {model}).flac
│ ├── Artist - Title (Vocals {model}).flac
│ ├── Artist - Title (Lead Vocals {model}).flac
│ ├── Artist - Title (Backing Vocals {model}).flac
│ └── ... (other stems)
├── lyrics/
│ ├── Artist - Title (Karaoke).lrc
│ ├── Artist - Title (Karaoke).ass
│ ├── Artist - Title (Karaoke).srt
│ └── ... (corrections JSONs)
├── previews/
├── visualizations/
└── styles_assets/
```
### Non-Functional Requirements
- [ ] Single Responsibility: Each stage has one job
- [ ] Open/Closed: Easy to add new encoding backends without modifying stages
- [ ] Testable: Each stage can be unit tested in isolation
- [ ] No duplicate code between paths
- [ ] Backwards compatible with existing KaraokeFinalise CLI usage
## Technical Approach
### Architecture: Stage-Based Pipeline
```
video_worker.py (VideoWorkerOrchestrator)
│
├── 1. EncodingStage (interface)
│ ├── LocalEncodingService (wraps KaraokeFinalise encoding methods)
│ └── GCEEncodingService (existing encoding_service.py)
│
├── 2. PackagingStage
│ ├── CDGPackager (extracts from KaraokeFinalise)
│ └── TXTPackager (extracts from KaraokeFinalise)
│
├── 3. DistributionStage
│ ├── YouTubeUploadService (extracts from KaraokeFinalise)
│ ├── DropboxService (existing)
│ ├── GoogleDriveService (existing)
│ └── DiscordService (extracts from KaraokeFinalise)
│
└── 4. OrganizationStage
└── BrandCodeService (extracts from KaraokeFinalise + DropboxService)
```
### Key Design Decisions
1. **Extract, Don't Rewrite**: Pull methods out of KaraokeFinalise into services, don't rewrite from scratch
2. **Keep KaraokeFinalise for CLI**: Local CLI still uses KaraokeFinalise.process() - we extract services that it can also use
3. **Interface-Based Encoding**: EncodingStage interface allows swapping GCE/local without changing orchestration
4. **Existing Service Pattern**: Follow singleton + factory pattern already used in codebase
5. **Local CLI stays sync**: Local karaoke-gen CLI remains synchronous as it is today
### Data Flow
```
JobConfig (from Firestore)
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ VideoWorkerOrchestrator │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 1. Setup: Download files, prepare directories ││
│ │ - Download title.mov, end.mov, with_vocals.mkv ││
│ │ - Download instrumental audio ││
│ │ - Prepare temp directory ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 2. Packaging: CDG/TXT (if enabled) - runs BEFORE encoding ││
│ │ Reason: CDG depends on LRC file which is available ││
│ │ Input: LRC file, instrumental audio ││
│ │ Output: CDG.zip, TXT.zip, .cdg, .mp3 ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 3. Encoding: GCE or Local (via interface) ││
│ │ Input: title.mov, karaoke.mov, end.mov ││
│ │ Output: 4K lossless, 4K lossy, 720p, MKV ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 4. Organization: Brand code, folder structure ││
│ │ - Generate brand code (NOMAD-XXXX) ││
│ │ - Prepare full output directory structure ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 5. Distribution: Upload to destinations ││
│ │ - YouTube upload (if enabled) ││
│ │ - Dropbox upload (if configured) ││
│ │ - Google Drive upload (if configured) ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 6. Notification: Discord, Email ││
│ │ Input: YouTube URL, brand code, Dropbox link ││
│ └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘
│
▼
Result (youtube_url, brand_code, dropbox_link, file_paths, etc.)
```
## Implementation Steps
### Phase 1: Extract Services from KaraokeFinalise (Non-Breaking) ✅ COMPLETE
These services can be used by both the cloud orchestrator AND KaraokeFinalise.
1. [x] **Create `backend/services/youtube_upload_service.py`** ✅
- Extract YouTube upload logic from KaraokeFinalise
- Methods: `upload_video()`, `check_duplicate()`, `delete_video()`, `truncate_title()`
- Supports pre-loaded credentials (server-side) or client secrets file (CLI)
- 23 unit tests with mocked YouTube API
2. [x] **Create `backend/services/discord_service.py`** ✅
- Extract Discord notification from KaraokeFinalise
- Methods: `post_message()`, `post_video_notification()`, `is_enabled()`
- Simple HTTP POST to webhook with validation
- 20 unit tests with mocked requests
3. [x] **Create `backend/services/packaging_service.py`** ✅
- Extract CDG/TXT generation from KaraokeFinalise
- Methods: `create_cdg_package()`, `create_txt_package()`
- Uses lyrics_converter and CDGGenerator (existing dependencies)
- 18 unit tests with temp files
4. [x] **Create `backend/services/local_encoding_service.py`** ✅
- Extract encoding methods from KaraokeFinalise
- Methods: `encode_all_formats()`, individual encoding methods
- Hardware acceleration detection with software fallback
- 28 unit tests with mocked subprocess
### Phase 2: Create Encoding Interface ✅ COMPLETE
5. [x] **Create `backend/services/encoding_interface.py`** ✅
- Define ABC `EncodingBackend` with methods: `encode()`, `is_available()`, `get_status()`
- `EncodingInput` and `EncodingOutput` dataclasses for config/results
- Factory function `get_encoding_backend()` with "local", "gce", "auto" options
6. [x] **`LocalEncodingBackend`** ✅
- Implements `EncodingBackend` interface
- Wraps `LocalEncodingService` with `asyncio.to_thread()`
- 17 unit tests
7. [x] **`GCEEncodingBackend`** ✅
- Implements `EncodingBackend` interface
- Wraps existing `EncodingService` (encoding_service.py)
- Requires GCS paths in options for cloud execution
- 6 additional unit tests
### Phase 3: Create Orchestrator ✅ COMPLETE
8. [x] **Create `backend/workers/video_worker_orchestrator.py`** ✅
- New class that coordinates all stages
- Replaces the branching logic in `generate_video()`
- Methods:
- `__init__(config, job_manager, storage, job_logger)`
- `async run() -> OrchestratorResult`
- `_run_packaging()` - CDG/TXT
- `_run_encoding()` - GCE or local via EncodingBackend interface
- `_run_organization()` - Brand code, folder structure
- `_run_distribution()` - YouTube, Dropbox, GDrive
- `_run_notifications()` - Discord
- 27 unit tests (test_video_worker_orchestrator.py)
9. [x] **Update `backend/workers/video_worker.py`** ✅
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: true)
- `generate_video()` routes to orchestrator or legacy path
- `generate_video_orchestrated()` - new unified pipeline
- `generate_video_legacy()` - original code for rollback
- Legacy GCE and KaraokeFinalise paths preserved for safety
### Phase 4: Update KaraokeFinalise to Use Services (OPTIONAL)
10. [ ] **Update `karaoke_gen/karaoke_finalise/karaoke_finalise.py`**
- Import and use the new services
- Replace inline YouTube/Discord/CDG code with service calls
- Keep process() working for CLI backwards compatibility
- Services become the single source of truth
- **NOTE**: This is optional for MVP - the orchestrator already uses the extracted services directly
### Phase 5: Testing & Validation ✅ COMPLETE
11. [x] **Add unit tests for each service** ✅
- test_youtube_upload_service.py (23 tests)
- test_discord_service.py (20 tests)
- test_packaging_service.py (18 tests)
- test_local_encoding_service.py (28 tests)
- test_encoding_interface.py (23 tests)
- test_video_worker_orchestrator.py (27 tests)
- **Total: 139 new tests**
12. [x] **Full test suite passes** ✅
- 976 tests passed (86 skipped - emulator/integration tests)
- No regressions from changes
13. [ ] **Manual E2E testing** (After deployment)
- [ ] GCE encoding job with YouTube upload enabled
- [ ] Local encoding job with YouTube upload enabled
- [ ] CLI karaoke-gen with all features
- [ ] Job with CDG/TXT enabled
- [ ] Verify output structure matches local CLI
## Files Created/Modified
| File | Action | Description | Status |
|------|--------|-------------|--------|
| `backend/services/youtube_upload_service.py` | Create | YouTube upload logic (~446 lines) | ✅ |
| `backend/services/discord_service.py` | Create | Discord notification service (~167 lines) | ✅ |
| `backend/services/packaging_service.py` | Create | CDG/TXT package generation (~266 lines) | ✅ |
| `backend/services/local_encoding_service.py` | Create | Local FFmpeg encoding service (~512 lines) | ✅ |
| `backend/services/encoding_interface.py` | Create | Abstract interface + backends (~418 lines) | ✅ |
| `backend/workers/video_worker_orchestrator.py` | Create | Stage orchestration logic (~550 lines) | ✅ |
| `backend/workers/video_worker.py` | Modify | Added orchestrator path + feature flag | ✅ |
| `backend/tests/test_youtube_upload_service.py` | Create | 23 unit tests | ✅ |
| `backend/tests/test_discord_service.py` | Create | 20 unit tests | ✅ |
| `backend/tests/test_packaging_service.py` | Create | 18 unit tests | ✅ |
| `backend/tests/test_local_encoding_service.py` | Create | 28 unit tests | ✅ |
| `backend/tests/test_encoding_interface.py` | Create | 23 unit tests | ✅ |
| `backend/tests/test_video_worker_orchestrator.py` | Create | 27 unit tests | ✅ |
| `karaoke_gen/karaoke_finalise/karaoke_finalise.py` | Modify | Use extracted services (optional) | Deferred |
## Testing Strategy
### Unit Tests
- **youtube_upload_service**: Mock YouTube API, test upload/delete/duplicate check
- **discord_service**: Mock requests, test webhook POST
- **packaging_service**: Use temp files, test CDG/TXT generation
- **local_encoding_service**: Mock subprocess, test FFmpeg command construction
### Integration Tests
- **video_worker_orchestrator**: Mock all services, test stage sequencing
- **video_worker**: Test with mocked orchestrator
### E2E Verification
- [ ] GCE encoding job with YouTube upload enabled → YouTube link appears
- [ ] Local encoding job with YouTube upload enabled → Same behavior
- [ ] CLI `karaoke-gen` with all features → No regression
- [ ] Job with CDG/TXT enabled → Packages generated
- [ ] Dropbox output structure → Matches local CLI exactly
## Open Questions
- [x] Should we keep KaraokeFinalise.process() or deprecate it?
- **Decision**: Keep it for CLI backwards compatibility. It will use the new services internally.
- [x] Should local_encoding_service run FFmpeg sync or async?
- **Decision**: Sync implementation, async wrapper via `asyncio.to_thread()` for interface compatibility.
- [x] Order of packaging vs encoding?
- **Decision**: CDG/TXT runs BEFORE encoding (depends on LRC file, not videos). KaraokeFinalise does it this way too.
## Rollback Plan
1. The refactor is additive - new services don't replace old code until orchestrator is wired up
2. Feature flag: `USE_NEW_ORCHESTRATOR=true` to enable new code path
3. If issues found in production:
- Set `USE_NEW_ORCHESTRATOR=false` to revert to old behavior
- Old `generate_video()` code preserved behind feature flag
4. After 1 week stable, remove feature flag and old code
## Success Criteria
1. Job `5b78ad46` scenario (GCE + YouTube) would upload to YouTube
2. All 803+ existing tests pass
3. E2E happy path test passes
4. No regression in local CLI functionality
5. Output structure matches local CLI exactly
6. Code coverage maintained or improved
## Estimated Scope
- **Phase 1** (Extract Services): 4 new service files + tests (~400-600 lines each)
- **Phase 2** (Interface): 1 new file, 2 modifications (~100 lines)
- **Phase 3** (Orchestrator): 2 files (~300-400 lines)
- **Phase 4** (KaraokeFinalise): 1 modification (~100 lines of changes)
- **Phase 5** (Testing): Integration tests + manual verification
This is a medium-sized refactor that can be done incrementally. Each phase produces working, tested code.
@@ -0,0 +1,222 @@
# Plan: Worker Logs Rearchitecture
**Created:** 2026-01-04
**Branch:** `feat/sess-20260104-1931-logs-research`
**Status:** Implementation Complete - Pending PR Merge & Deployment
## Overview
Move `worker_logs` from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).
## Problem Statement
- Firestore documents have a 1 MB size limit
- `worker_logs` is stored as an array field within job documents
- Long-running jobs (especially video encoding) can generate thousands of log entries
- When the limit is exceeded, job updates fail with: `Document cannot be written because its size exceeds the maximum allowed size`
## Requirements
- [x] Logs must remain associated with their job
- [x] Logs should be queryable by job_id, worker type, timestamp
- [x] Old logs should auto-cleanup (TTL: 30 days)
- [x] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
- [x] Frontend must continue to display logs without changes
- [x] Migration path for existing jobs (optional - can just start fresh)
## Technical Approach
### Option A: Subcollection (Recommended)
Store logs in `jobs/{job_id}/logs/{log_id}` subcollection.
**Pros:**
- Natural association with job (delete job = delete logs with cascading)
- Each log entry is its own document (no size limit concerns)
- Can add TTL via Firestore TTL policies
- Efficient queries with composite indexes
**Cons:**
- More Firestore read operations for paginated log fetching
- Slightly more complex write logic
### Option B: Separate Collection
Store logs in `job_logs` collection with `job_id` field.
**Pros:**
- Can query across all jobs
- Simpler TTL management
**Cons:**
- Orphaned logs if job deleted (need cleanup logic)
- Extra index needed for job_id queries
### Decision: Option A (Subcollection)
## Implementation Steps
### Phase 1: Backend Changes
1. [x] **Create log entry model** - `backend/models/worker_log.py`
- `WorkerLogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
- Factory method `create()` for easy instantiation
- `to_dict()` for Firestore storage
- `to_legacy_dict()` for API compatibility
- `from_dict()` for Firestore retrieval
2. [x] **Update FirestoreService** - `backend/services/firestore_service.py`
- Added `append_log_to_subcollection(job_id, log_entry)` method
- Added `get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset)` method
- Added `get_logs_count_from_subcollection(job_id)` method
- Added `delete_logs_subcollection(job_id, batch_size)` method
- Old `append_worker_log` marked as deprecated
3. [x] **Update JobManager** - `backend/services/job_manager.py`
- Modified `append_worker_log()` to use subcollection when `USE_LOG_SUBCOLLECTION=true`
- Modified `get_worker_logs()` to read from subcollection with fallback to embedded array
- Added `get_worker_logs_count()` for total count
- Updated `delete_job()` to clean up logs subcollection
4. [x] **Update jobs API** - `backend/api/routes/jobs.py`
- `GET /api/jobs/{id}/logs` - Updated to use new methods
- Response format stays the same (backward compatible)
- Fixed `next_index` calculation
5. [x] **Add feature flag** - `backend/config.py`
- Added `USE_LOG_SUBCOLLECTION` environment variable (default: true)
6. [x] **Add TTL configuration** - `infrastructure/__main__.py`
- Added Firestore TTL policy for logs subcollection (30 days)
- Added composite index for worker + timestamp queries
### Phase 2: Testing
7. [x] **Unit tests** - `backend/tests/test_worker_log_subcollection.py`
- 24 tests covering:
- WorkerLogEntry model creation and conversion
- FirestoreService subcollection methods
- JobManager logging with feature flag
- Edge cases (unicode, long messages, empty messages)
8. [x] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
- 11 tests using real Firestore emulator:
- Single and sequential writes
- TTL field verification
- Concurrent writes (no race conditions)
- Large volume (500 logs)
- Worker filtering
- Timestamp ordering
- Subcollection deletion
- Concurrent workers interleaved
- Job deletion cleanup
- Comparison with embedded array
### Phase 3: Deployment & Migration
9. [ ] **Deploy with feature flag**
- Add `USE_LOG_SUBCOLLECTION=true` env var (default is true)
- Deploy to production
10. [ ] **Monitor**
- Watch for errors in Cloud Logging
- Verify logs appear in frontend
11. [x] **Backward compatibility**
- Old jobs with embedded arrays still work (fallback in get_worker_logs)
- No migration needed - new jobs use subcollection, old jobs use array
### Phase 4: Cleanup (Future)
12. [ ] **Remove old code path** (after confidence period)
- Remove `worker_logs` field from job model
- Remove old `append_worker_log` method
- Remove feature flag
## Files Created/Modified
| File | Action | Description |
|------|--------|-------------|
| `backend/models/worker_log.py` | Created | WorkerLogEntry dataclass |
| `backend/services/firestore_service.py` | Modified | Added subcollection methods |
| `backend/services/job_manager.py` | Modified | Switch to subcollection with fallback |
| `backend/api/routes/jobs.py` | Modified | Updated logs endpoint |
| `backend/config.py` | Modified | Added USE_LOG_SUBCOLLECTION setting |
| `backend/tests/test_worker_log_subcollection.py` | Created | Unit tests (24 tests) |
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Created | Integration tests (11 tests) |
| `infrastructure/__main__.py` | Modified | Added TTL policy and index |
## Log Entry Schema
```python
@dataclass
class WorkerLogEntry:
timestamp: datetime
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
worker: str # "audio", "lyrics", "video", "render", "screens", "distribution"
message: str
id: str # Auto-generated UUID
job_id: str
ttl_expiry: datetime # timestamp + 30 days
metadata: Optional[Dict[str, Any]] = None
```
## Firestore Structure
```
jobs/
{job_id}/
... (job fields, worker_logs still present for old jobs)
logs/ # subcollection for new jobs
{log_id}/
id: string
job_id: string
timestamp: datetime
level: string
worker: string
message: string
metadata: map (optional)
ttl_expiry: datetime
```
## API Response (Unchanged)
```json
{
"logs": [
{
"timestamp": "2026-01-04T12:00:00Z",
"level": "INFO",
"worker": "video",
"message": "Starting video generation"
}
],
"next_index": 42,
"total_logs": 150
}
```
## Testing Results
All tests pass:
- 24 unit tests in `test_worker_log_subcollection.py`
- 11 emulator integration tests in `test_worker_logs_subcollection.py`
- 68 total tests in emulator suite pass
## Rollback Plan
1. Set `USE_LOG_SUBCOLLECTION=false` env var
2. Redeploy backend
3. New logs will go back to embedded array
4. Subcollection logs will remain but won't be read
## Answers to Open Questions
- [x] **Should we paginate logs in the API?** - Yes, added `offset` parameter support
- [x] **What's the optimal TTL period?** - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
- [x] **Should we index by timestamp for efficient range queries?** - Yes, added composite index (worker + timestamp)
## References
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
- [Pulumi Firestore Field](https://www.pulumi.com/registry/packages/gcp/api-docs/firestore/field/)
+199 -22
View File
@@ -225,7 +225,8 @@ jobs:
- name: Run Happy Path Production Test
working-directory: frontend
env:
MAILSLURP_API_KEY: ${{ secrets.MAILSLURP_API_KEY }}
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ secrets.E2E_TEST_TOKEN }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
run: |
@@ -902,11 +903,14 @@ jobs:
deploy-backend:
name: "Deploy - Backend (Cloud Run)"
needs: [package-build]
runs-on: [self-hosted, linux, gcp]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
outputs:
base_rebuilt: ${{ steps.check-base.outputs.needs_rebuild }}
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
@@ -964,6 +968,32 @@ jobs:
with:
version: "latest"
# ============================================
# Upload wheel to GCS for GCE encoding worker
# The GCE worker installs this wheel at job start for hot code updates
# ============================================
- name: Download wheel artifact
uses: actions/download-artifact@v4
with:
name: package-dist
path: dist/
- name: Upload wheel to GCS for GCE worker
run: |
echo "📦 Uploading wheel to GCS for GCE encoding worker..."
WHEEL_FILE=$(ls dist/*.whl | head -1)
if [ -z "$WHEEL_FILE" ]; then
echo "::error::No wheel file found in dist/"
exit 1
fi
echo "Found wheel: $WHEEL_FILE"
# Upload to GCS (GCE worker downloads latest at job start)
gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/
echo "✅ Wheel uploaded - GCE worker will pick up changes on next job"
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
@@ -989,32 +1019,32 @@ jobs:
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
# Try to get the existing base image's hash label
# First check if image exists locally (fast, no network call)
# Only pull from registry if not cached locally
# Fetch hash label directly from registry (10KB config blob, not full image)
# This avoids issues with stale local Docker cache on GitHub runners
EXISTING_HASH=""
if docker image inspect $BASE_IMAGE >/dev/null 2>&1; then
echo "✅ Base image found in local cache (skipping slow registry pull)"
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
if [ -z "$EXISTING_HASH" ]; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
fi
echo "Existing base image hash: $EXISTING_HASH"
elif docker pull $BASE_IMAGE 2>/dev/null; then
echo "📥 Base image pulled from registry"
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
if [ -z "$EXISTING_HASH" ]; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
fi
echo "Existing base image hash: $EXISTING_HASH"
# Get the manifest (small JSON, ~1KB) - includes config digest
MANIFEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null || echo "")
CONFIG_DIGEST=$(echo "$MANIFEST" | jq -r '.config.digest // ""' 2>/dev/null || echo "")
if [ -n "$CONFIG_DIGEST" ] && [ "$CONFIG_DIGEST" != "null" ] && [ "$CONFIG_DIGEST" != "" ]; then
echo "📋 Found manifest, config digest: $CONFIG_DIGEST"
# Fetch just the config blob (~10KB) to get labels - much faster than pulling full image
TOKEN=$(gcloud auth print-access-token)
CONFIG_JSON=$(curl -sL -H "Authorization: Bearer $TOKEN" \
"https://${{ env.REGISTRY }}/v2/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}/blobs/$CONFIG_DIGEST" 2>/dev/null)
EXISTING_HASH=$(echo "$CONFIG_JSON" | jq -r '.config.Labels["base.content.hash"] // .config.Labels["poetry.lock.hash"] // ""' 2>/dev/null || echo "")
echo "📋 Registry base image hash: $EXISTING_HASH"
else
echo "⚠️ No existing base image found - will build"
echo "⚠️ No existing base image found in registry - will build"
fi
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
echo " Current: ${{ steps.base-hash.outputs.hash }}"
echo " Registry: $EXISTING_HASH"
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
else
echo "✅ Base image is up to date"
@@ -1042,6 +1072,31 @@ jobs:
# Provenance disabled for smaller image and faster push
provenance: false
- name: Ensure local base image is current (if not rebuilt)
if: steps.check-base.outputs.needs_rebuild == 'false'
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
# Get the config digest from registry manifest (identifies the image content)
# This is the same value as Docker's Image ID for the same image
REGISTRY_CONFIG_DIGEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null | jq -r '.config.digest' || echo "")
# Get local image's ID (which is the config digest)
LOCAL_IMAGE_ID=$(docker image inspect $BASE_IMAGE --format='{{.Id}}' 2>/dev/null || echo "")
echo "Registry config digest: $REGISTRY_CONFIG_DIGEST"
echo "Local image ID: $LOCAL_IMAGE_ID"
if [ -z "$LOCAL_IMAGE_ID" ]; then
echo "📥 Base image not in local cache - pulling from registry"
docker pull $BASE_IMAGE
elif [ "$LOCAL_IMAGE_ID" != "$REGISTRY_CONFIG_DIGEST" ]; then
echo "🔄 Local cache has different image - pulling latest"
docker pull $BASE_IMAGE
else
echo "✅ Local cache matches registry - skipping pull"
fi
- name: Build and push app image
uses: docker/build-push-action@v6
with:
@@ -1084,7 +1139,7 @@ jobs:
--concurrency 1 \
--max-instances 50 \
--min-instances 4 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true" \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest"
- name: Verify deployment and version
@@ -1138,3 +1193,125 @@ jobs:
echo ""
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
# ============================================
# PROPAGATE BASE IMAGE TO ALL RUNNERS
# When base image is rebuilt, pull it to all 20 runners
# so subsequent app builds don't wait 8 min for the pull
#
# NOTE: This is best-effort distribution. GitHub Actions doesn't
# support targeting specific runners by name, so we spawn 20 jobs
# and rely on the scheduler to distribute them across idle runners.
# In practice, with all runners idle after deploy-backend, distribution
# is good. Runners that miss this pull will get the image on their
# next app build (with the 8 min delay).
# ============================================
propagate-base-image:
name: "Propagate Base Image (${{ matrix.runner_index }})"
needs: deploy-backend
if: github.ref == 'refs/heads/main' && github.event_name == 'push' && needs.deploy-backend.outputs.base_rebuilt == 'true'
runs-on: [self-hosted, linux, gcp]
permissions:
contents: read
id-token: write
strategy:
matrix:
# 20 parallel jobs - GitHub distributes across available runners
runner_index: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
max-parallel: 20
fail-fast: false # Don't cancel other runners if one fails
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
REPOSITORY: karaoke-repo
BASE_IMAGE_NAME: karaoke-backend-base
steps:
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
- name: Pull latest base image
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
RUNNER_NAME=$(hostname)
echo "📥 Pulling latest base image on runner $RUNNER_NAME..."
docker pull $BASE_IMAGE
echo "✅ Base image cached on $RUNNER_NAME"
deploy-infrastructure:
name: "Deploy - Infrastructure (Pulumi)"
runs-on: [self-hosted, linux, gcp]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: '3.13'
- name: Install Pulumi
run: |
curl -fsSL https://get.pulumi.com | sh
echo "$HOME/.pulumi/bin" >> $GITHUB_PATH
- name: Install Python dependencies
working-directory: infrastructure
run: |
pip install pulumi pulumi-gcp pulumi-docker
- name: Configure Pulumi
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
pulumi login
- name: Deploy Infrastructure
working-directory: infrastructure
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
echo "🚀 Deploying infrastructure changes..."
# Skip preview to avoid needing broad read permissions on all GCP resources
# Pulumi state is authoritative since infrastructure is only managed via Pulumi
pulumi up --yes --skip-preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
- name: Restart GCE Encoding Worker
run: |
echo "🔄 Restarting encoding worker to pick up new code..."
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker" || true
echo "✅ Encoding worker restart triggered"
- name: Summary
if: success()
run: |
echo "🎉 Infrastructure deployment successful!"
echo "Commit: ${{ github.sha }}"
echo "Branch: ${{ github.ref_name }}"
+3 -2
View File
@@ -70,8 +70,9 @@ jobs:
- name: Run E2E Happy Path Test
working-directory: frontend
env:
MAILSLURP_API_KEY: ${{ secrets.MAILSLURP_API_KEY }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token == 'true' && secrets.E2E_TEST_TOKEN || '' }}
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
+50 -2
View File
@@ -48,10 +48,58 @@ cd frontend && npm run test:all 2>&1 | tail -n 200
- Skip for docs-only changes
### Infrastructure
- All GCP changes via Pulumi in `infrastructure/`
- `gcloud` CLI for reading/debugging only
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
- Never modify GCP resources directly via console or `gcloud` CLI
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
- Stop and notify user on auth issues
## API Authentication (for Agents)
When you need to call backend APIs programmatically:
### Internal Worker Endpoints
Use the `X-Admin-Token` header with the token from Secret Manager:
```bash
# Trigger video worker
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
-H "Content-Type: application/json" \
-d '{"job_id": "YOUR_JOB_ID"}'
# Other internal endpoints follow the same pattern
```
### Firestore Direct Access
```bash
# Set the correct project
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
# Use Python with google-cloud-firestore
python3 << 'EOF'
import os
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
from google.cloud import firestore
db = firestore.Client(project='nomadkaraoke')
# Query/update documents...
EOF
```
### GCE Encoding Worker
```bash
# SSH to restart service (clears in-memory job queue)
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
# Check health
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
```
### What Doesn't Work
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
## Documentation Maintenance
### Structure
+17
View File
@@ -18,6 +18,23 @@ RUN apt-get update && apt-get install -y \
xz-utils \
&& rm -rf /var/lib/apt/lists/*
# Install comprehensive font support for international lyrics and symbols
# Noto fonts provide full Unicode coverage including:
# - Latin, Greek, Cyrillic (core)
# - CJK: Chinese, Japanese, Korean (~150MB)
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
# - Color emoji (limited support in libass, but available)
# - Musical symbols (♪ ♫ etc.)
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
RUN apt-get update && apt-get install -y \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
# Install optimized static FFmpeg build (7.0.2) instead of Debian package (5.1.8)
# John Van Sickle's builds are compiled with better optimizations for x86_64
# This provides ~20-40% performance improvement for encoding tasks
+385 -2
View File
@@ -5,18 +5,20 @@ Handles:
- Dashboard overview statistics
- System-wide metrics
- Admin-only operations
- Audio search cache management
"""
import logging
from datetime import datetime, timedelta
from typing import Tuple
from typing import Tuple, List, Optional, Any, Dict
from fastapi import APIRouter, Depends
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from backend.api.dependencies import require_admin
from backend.services.auth_service import UserType
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
from backend.services.job_manager import JobManager
from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchServiceError
from backend.models.job import JobStatus
@@ -197,3 +199,384 @@ async def get_admin_stats_overview(
total_credits_issued_30d=total_credits_issued_30d,
total_beta_testers=total_beta_testers,
)
# =============================================================================
# Audio Search Management Models
# =============================================================================
class AudioSearchResultSummary(BaseModel):
"""Summary of a single audio search result."""
index: int
provider: str
artist: str
title: str
is_lossless: bool
quality: Optional[str] = None
seeders: Optional[int] = None
class AudioSearchJobSummary(BaseModel):
"""Summary of a job with audio search results."""
job_id: str
status: str
user_email: Optional[str] = None
audio_search_artist: Optional[str] = None
audio_search_title: Optional[str] = None
created_at: Optional[datetime] = None
results_count: int
results_summary: List[AudioSearchResultSummary]
has_lossless: bool
providers: List[str]
class AudioSearchListResponse(BaseModel):
"""Response for listing audio search jobs."""
jobs: List[AudioSearchJobSummary]
total: int
class ClearSearchCacheResponse(BaseModel):
"""Response for clearing search cache."""
status: str
job_id: str
message: str
previous_status: str
new_status: str
results_cleared: int
flacfetch_cache_cleared: bool = False
flacfetch_error: Optional[str] = None
class ClearAllCacheResponse(BaseModel):
"""Response for clearing all flacfetch cache."""
status: str
message: str
deleted_count: int
class CacheStatsResponse(BaseModel):
"""Response for cache statistics."""
count: int
total_size_bytes: int
oldest_entry: Optional[str] = None
newest_entry: Optional[str] = None
configured: bool
# =============================================================================
# Audio Search Management Endpoints
# =============================================================================
@router.get("/audio-searches", response_model=AudioSearchListResponse)
async def list_audio_searches(
limit: int = 50,
status_filter: Optional[str] = None,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
"""
List jobs with audio search results.
Returns jobs that have cached audio search results, useful for:
- Monitoring search activity
- Identifying stale cached results
- Clearing cache for specific jobs
Args:
limit: Maximum number of jobs to return (default 50)
status_filter: Optional filter by job status (e.g., 'awaiting_audio_selection')
"""
from google.cloud.firestore_v1 import FieldFilter
db = user_service.db
jobs_collection = db.collection("jobs")
# Query jobs - we'll filter for those with audio_search_results in Python
# since Firestore can't query for existence of nested fields efficiently
query = jobs_collection.order_by("created_at", direction="DESCENDING").limit(500)
if status_filter:
query = jobs_collection.where(
filter=FieldFilter("status", "==", status_filter)
).order_by("created_at", direction="DESCENDING").limit(500)
jobs_with_searches = []
for doc in query.stream():
data = doc.to_dict()
state_data = data.get("state_data", {})
audio_results = state_data.get("audio_search_results", [])
if not audio_results:
continue
# Compute has_lossless and providers from ALL results (not just first 10)
has_lossless = any(r.get("is_lossless", False) for r in audio_results)
providers = {r.get("provider", "Unknown") for r in audio_results}
# Build summary from first 10 results only
results_summary = []
for r in audio_results[:10]:
results_summary.append(AudioSearchResultSummary(
index=r.get("index", 0),
provider=r.get("provider", "Unknown"),
artist=r.get("artist", ""),
title=r.get("title", ""),
is_lossless=r.get("is_lossless", False),
quality=r.get("quality"),
seeders=r.get("seeders"),
))
jobs_with_searches.append(AudioSearchJobSummary(
job_id=doc.id,
status=data.get("status", "unknown"),
user_email=data.get("user_email"),
audio_search_artist=data.get("audio_search_artist"),
audio_search_title=data.get("audio_search_title"),
created_at=data.get("created_at"),
results_count=len(audio_results),
results_summary=results_summary,
has_lossless=has_lossless,
providers=sorted(providers),
))
if len(jobs_with_searches) >= limit:
break
return AudioSearchListResponse(
jobs=jobs_with_searches,
total=len(jobs_with_searches),
)
@router.post("/audio-searches/{job_id}/clear-cache", response_model=ClearSearchCacheResponse)
async def clear_audio_search_cache(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
"""
Clear the cached audio search results for a job.
This will:
1. Remove the cached search results from job.state_data
2. Reset the job status to 'pending' so a new search can be performed
3. Clear the flacfetch GCS cache for this artist/title (if available)
Use this when:
- Cached results are stale (e.g., flacfetch was updated)
- User wants to search again with different terms
- Results appear incomplete or incorrect
"""
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Get current state
state_data = job.state_data or {}
audio_results = state_data.get("audio_search_results", [])
results_count = len(audio_results)
previous_status = job.status
if not audio_results:
raise HTTPException(
status_code=400,
detail=f"Job {job_id} has no cached audio search results"
)
# Validate job status - only allow cache clear for appropriate states
# Don't allow clearing cache for jobs that are actively processing or complete
forbidden_statuses = {
"downloading", "downloading_audio", "searching_audio",
"separating_stage1", "separating_stage2", "transcribing", "correcting",
"generating_screens", "applying_padding", "rendering_video",
"generating_video", "encoding", "packaging", "uploading",
"complete", "prep_complete",
}
if previous_status in forbidden_statuses:
raise HTTPException(
status_code=400,
detail=f"Cannot clear cache for job in '{previous_status}' state. "
f"Only jobs in pending, awaiting_audio_selection, failed, or cancelled states can have cache cleared."
)
# Clear the cache by removing the keys
db = user_service.db
job_ref = db.collection("jobs").document(job_id)
from google.cloud.firestore_v1 import DELETE_FIELD
# Update job: clear cache and reset status
job_ref.update({
"state_data.audio_search_results": DELETE_FIELD,
"state_data.audio_search_count": DELETE_FIELD,
"state_data.remote_search_id": DELETE_FIELD,
"status": "pending",
"progress": 0,
"message": "Audio search cache cleared by admin. Ready for new search.",
"updated_at": datetime.utcnow(),
})
# Also clear flacfetch's GCS cache if we have artist/title
flacfetch_cache_cleared = False
flacfetch_error = None
artist = job.audio_search_artist
title = job.audio_search_title
if artist and title:
flacfetch_client = get_flacfetch_client()
if flacfetch_client:
try:
flacfetch_cache_cleared = await flacfetch_client.clear_search_cache(artist, title)
logger.info(
f"Cleared flacfetch cache for '{artist}' - '{title}': "
f"{'deleted' if flacfetch_cache_cleared else 'no entry found'}"
)
except FlacfetchServiceError as e:
flacfetch_error = str(e)
logger.warning(f"Failed to clear flacfetch cache: {e}")
else:
flacfetch_error = "flacfetch client not configured"
logger.debug("Skipping flacfetch cache clear - client not configured")
else:
flacfetch_error = "missing artist or title"
logger.debug(f"Skipping flacfetch cache clear - missing artist ({artist}) or title ({title})")
logger.info(
f"Admin {auth_data[0]} cleared audio search cache for job {job_id}. "
f"Cleared {results_count} results. Status changed from {previous_status} to pending. "
f"Flacfetch cache cleared: {flacfetch_cache_cleared}"
)
message = f"Cleared {results_count} cached search results. Job reset to pending."
if flacfetch_cache_cleared:
message += " Flacfetch cache also cleared."
elif flacfetch_error:
message += f" Note: flacfetch cache not cleared ({flacfetch_error})."
return ClearSearchCacheResponse(
status="success",
job_id=job_id,
message=message,
previous_status=previous_status,
new_status="pending",
results_cleared=results_count,
flacfetch_cache_cleared=flacfetch_cache_cleared,
flacfetch_error=flacfetch_error,
)
@router.post("/jobs/{job_id}/reset-worker-state")
async def reset_worker_state(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Reset stale worker progress state for a job.
This clears the video_progress, render_progress, and screens_progress
from state_data, allowing workers to be re-triggered.
Use this when a job is stuck because worker progress shows 'running'
from a previous failed attempt.
"""
from backend.services.job_manager import JobManager
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Reset worker progress states
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
logger.info(f"Admin {auth_data[0]} reset worker state for job {job_id}")
return {
"status": "success",
"job_id": job_id,
"message": "Worker progress states reset to pending"
}
@router.delete("/cache", response_model=ClearAllCacheResponse)
async def clear_all_flacfetch_cache(
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Clear the entire flacfetch search cache.
This will delete all cached search results from flacfetch's GCS cache.
Use with caution - this will cause all subsequent searches to hit
the trackers fresh.
Note: This does NOT clear Firestore job.state_data caches, only the
flacfetch-side GCS cache.
"""
flacfetch_client = get_flacfetch_client()
if not flacfetch_client:
raise HTTPException(
status_code=503,
detail="flacfetch client not configured"
)
try:
deleted_count = await flacfetch_client.clear_all_cache()
logger.info(
f"Admin {auth_data[0]} cleared all flacfetch cache. "
f"Deleted {deleted_count} entries."
)
return ClearAllCacheResponse(
status="success",
message=f"Cleared {deleted_count} cache entries from flacfetch.",
deleted_count=deleted_count,
)
except FlacfetchServiceError as e:
logger.error(f"Failed to clear all flacfetch cache: {e}")
raise HTTPException(
status_code=502,
detail=f"Failed to clear flacfetch cache: {e}"
)
@router.get("/cache/stats", response_model=CacheStatsResponse)
async def get_flacfetch_cache_stats(
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Get statistics about the flacfetch search cache.
Returns information about:
- Number of cached entries
- Total size in bytes
- Oldest and newest cache entries
- Whether cache is configured
"""
flacfetch_client = get_flacfetch_client()
if not flacfetch_client:
raise HTTPException(
status_code=503,
detail="flacfetch client not configured"
)
try:
stats = await flacfetch_client.get_cache_stats()
return CacheStatsResponse(
count=stats.get("count", 0),
total_size_bytes=stats.get("total_size_bytes", 0),
oldest_entry=stats.get("oldest_entry"),
newest_entry=stats.get("newest_entry"),
configured=stats.get("configured", True),
)
except FlacfetchServiceError as e:
logger.error(f"Failed to get flacfetch cache stats: {e}")
raise HTTPException(
status_code=502,
detail=f"Failed to get cache stats: {e}"
)
# Trigger deployment
+3 -2
View File
@@ -95,7 +95,7 @@ class AudioSearchRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
@@ -485,7 +485,8 @@ async def search_audio(
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
# Apply defaults for YouTube/Dropbox distribution (for web service)
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
# Use explicit value if provided, otherwise fall back to server default
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
effective_youtube_description = body.youtube_description or settings.default_youtube_description
+36 -13
View File
@@ -69,7 +69,7 @@ class CreateJobFromUrlRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
@@ -123,7 +123,7 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
@@ -406,7 +406,7 @@ async def upload_and_create_job(
enable_txt: Optional[bool] = Form(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise"),
# Finalisation options
brand_prefix: Optional[str] = Form(None, description="Brand code prefix (e.g., NOMAD)"),
enable_youtube_upload: bool = Form(False, description="Upload to YouTube"),
enable_youtube_upload: Optional[bool] = Form(None, description="Upload to YouTube. None = use server default"),
youtube_description: Optional[str] = Form(None, description="YouTube video description text"),
discord_webhook_url: Optional[str] = Form(None, description="Discord webhook URL for notifications"),
webhook_url: Optional[str] = Form(None, description="Generic webhook URL"),
@@ -510,12 +510,19 @@ async def upload_and_create_job(
if dist.brand_prefix and not brand_prefix:
logger.info(f"Using default brand_prefix: {dist.brand_prefix}")
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload
if effective_enable_youtube_upload and enable_youtube_upload is None:
logger.info("Using default enable_youtube_upload: True (from env)")
# Validate credentials for requested distribution services (including defaults)
# This prevents accepting jobs that will fail later due to missing credentials
invalid_services = []
credential_manager = get_credential_manager()
if enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -594,7 +601,7 @@ async def upload_and_create_job(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=youtube_description,
discord_webhook_url=dist.discord_webhook_url,
webhook_url=webhook_url,
@@ -815,11 +822,12 @@ async def upload_and_create_job(
"using_default": gdrive_folder_id is None,
}
if enable_youtube_upload:
if effective_enable_youtube_upload:
youtube_result = credential_manager.check_youtube_credentials()
distribution_services["youtube"] = {
"enabled": True,
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
"using_default": enable_youtube_upload is None,
}
if dist.discord_webhook_url:
@@ -908,7 +916,7 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code from folder name")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
@@ -1048,11 +1056,16 @@ async def create_job_with_upload_urls(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1105,7 +1118,7 @@ async def create_job_with_upload_urls(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
youtube_description_template=body.youtube_description, # video_worker reads this field
discord_webhook_url=dist.discord_webhook_url,
@@ -1466,11 +1479,16 @@ async def create_job_from_url(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1522,7 +1540,7 @@ async def create_job_from_url(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
discord_webhook_url=dist.discord_webhook_url,
webhook_url=body.webhook_url,
@@ -1712,11 +1730,16 @@ async def create_finalise_only_job(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate distribution credentials if services are requested
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1766,7 +1789,7 @@ async def create_finalise_only_job(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
discord_webhook_url=dist.discord_webhook_url,
dropbox_path=dist.dropbox_path,
+24 -12
View File
@@ -87,6 +87,11 @@ async def create_job(
user_email = request.user_email
logger.info(f"Admin {auth_result.user_email} creating job on behalf of {user_email}")
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = request.enable_youtube_upload if request.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Create job with all preferences
job_create = JobCreate(
url=str(request.url),
@@ -94,7 +99,7 @@ async def create_job(
title=request.title,
enable_cdg=request.enable_cdg,
enable_txt=request.enable_txt,
enable_youtube_upload=request.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=request.youtube_description,
webhook_url=request.webhook_url,
user_email=user_email
@@ -1291,12 +1296,13 @@ async def retry_job(
logger.info(f"Job {job_id}: Has rendered video and instrumental selection, retrying video generation")
# Clear error state
# Clear error state and reset worker progress for idempotency
job_manager.update_job(job_id, {
'error_message': None,
'error_details': None,
})
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
# Reset to INSTRUMENTAL_SELECTED and trigger video worker
if not job_manager.transition_to_state(
job_id=job_id,
@@ -1324,13 +1330,14 @@ async def retry_job(
file_urls.get('screens', {}).get('title')):
logger.info(f"Job {job_id}: Has corrections and screens, retrying from render stage")
# Clear error state
# Clear error state and reset worker progress for idempotency
job_manager.update_job(job_id, {
'error_message': None,
'error_details': None,
})
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
# Reset to REVIEW_COMPLETE and trigger render worker
if not job_manager.transition_to_state(
job_id=job_id,
@@ -1358,13 +1365,14 @@ async def retry_job(
file_urls.get('lyrics', {}).get('corrections')):
logger.info(f"Job {job_id}: Has stems and corrections, retrying from screens stage")
# Clear error state
# Clear error state and reset worker progress for idempotency
job_manager.update_job(job_id, {
'error_message': None,
'error_details': None,
})
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
# Reset to a state before screens and trigger screens worker
if not job_manager.transition_to_state(
job_id=job_id,
@@ -1449,10 +1457,14 @@ async def get_worker_logs(
This endpoint returns worker logs stored in Firestore.
Use `since_index` for efficient polling (returns only new logs).
Logs are stored in a subcollection (jobs/{job_id}/logs) to avoid
the 1MB document size limit. Older jobs may have logs in an embedded
array (worker_logs field) - this endpoint handles both transparently.
Args:
job_id: Job ID
since_index: Return only logs after this index (for pagination/polling)
worker: Filter by worker name (audio, lyrics, screens, video, render)
worker: Filter by worker name (audio, lyrics, screens, video, render, distribution)
Returns:
{
@@ -1470,11 +1482,11 @@ async def get_worker_logs(
raise HTTPException(status_code=403, detail="You don't have permission to access logs for this job")
logs = job_manager.get_worker_logs(job_id, since_index=since_index, worker=worker)
total = len(job.worker_logs) if job.worker_logs else 0
total = job_manager.get_worker_logs_count(job_id)
return {
"logs": logs,
"next_index": total,
"next_index": since_index + len(logs),
"total_logs": total
}
+122 -40
View File
@@ -27,8 +27,10 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_logging import job_log_context, JobLogger
from backend.services.tracing import create_span, add_span_attribute, add_span_event
from backend.services.encoding_service import get_encoding_service
from backend.api.dependencies import require_auth, require_review_auth
from backend.services.auth_service import UserType
from backend.config import get_settings
# LyricsTranscriber imports for preview generation
from lyrics_transcriber.types import CorrectionResult
@@ -385,28 +387,35 @@ async def generate_preview_video(
):
"""
Generate a preview video with the current corrections.
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
video with the user's current corrections applied.
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
the high-performance GCE worker for faster generation (15-20s vs 60+s).
"""
job_manager = JobManager()
storage = StorageService()
settings = get_settings()
encoding_service = get_encoding_service()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Job must be in review state to generate preview
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
logger.info(f"Job {job_id}: Generating preview video")
# Check if GCE preview encoding is enabled
use_gce_preview = encoding_service.is_preview_enabled
logger.info(f"Job {job_id}: Generating preview video (GCE preview: {use_gce_preview})")
# Use tracing and job_log_context for full observability
with create_span("generate-preview-video", {"job_id": job_id}) as span:
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
with job_log_context(job_id, worker="preview"):
try:
# Create temp directory for this preview operation
@@ -416,19 +425,19 @@ async def generate_preview_video(
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
corrections_path = os.path.join(temp_dir, "corrections.json")
storage.download_file(corrections_gcs, corrections_path)
with open(corrections_path, 'r', encoding='utf-8') as f:
original_data = json.load(f)
# 2. Download input audio
audio_path = os.path.join(temp_dir, "audio.flac")
storage.download_file(job.input_media_gcs_path, audio_path)
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
# 3. Load original as CorrectionResult
correction_result = CorrectionResult.from_dict(original_data)
add_span_event("corrections_loaded")
# 4. Get or create styles file for preview using unified style loader
with create_span("load-styles") as styles_span:
styles_path, _ = load_styles_from_gcs(
@@ -439,54 +448,127 @@ async def generate_preview_video(
logger=logger,
)
styles_span.set_attribute("styles_path", styles_path)
# 5. Set up output config for preview
output_dir = os.path.join(temp_dir, "output")
cache_dir = os.path.join(temp_dir, "cache")
os.makedirs(output_dir, exist_ok=True)
os.makedirs(cache_dir, exist_ok=True)
output_config = OutputConfig(
output_styles_json=styles_path,
output_dir=output_dir,
cache_dir=cache_dir,
video_resolution="360p",
)
# 6. Generate preview video using CorrectionOperations (heavy operation)
with create_span("render-preview-video") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger
)
add_span_event("render_complete")
# 7. Upload preview video to GCS for serving
preview_hash = result["preview_hash"]
video_path = result["video_path"]
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# 6. Generate preview (ASS-only if using GCE, or full video if local)
preview_gcs_path = None
if use_gce_preview:
# GCE path: Generate ASS only, then offload encoding to GCE
try:
with create_span("generate-ass-subtitles") as ass_span:
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger,
ass_only=True, # Only generate ASS, skip video encoding
)
preview_hash = result["preview_hash"]
ass_path = result["ass_path"]
ass_span.set_attribute("ass_path", ass_path)
add_span_event("ass_generated")
# Upload ASS to GCS
with create_span("upload-ass-to-gcs") as upload_ass_span:
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
storage.upload_file(ass_path, ass_gcs_path)
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
# Call GCE encoding service
with create_span("gce-preview-encoding") as gce_span:
bucket_name = settings.gcs_bucket_name
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
# Get background image and font from style assets if available
style_assets = job.style_assets or {}
background_image_gcs_path = None
for key in ["karaoke_background", "style_karaoke_background"]:
if key in style_assets:
background_image_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
gce_span.set_attribute("background_image", background_image_gcs_path)
break
font_gcs_path = None
for key in ["font", "style_font"]:
if key in style_assets:
font_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
gce_span.set_attribute("font", font_gcs_path)
break
gce_result = await encoding_service.encode_preview_video(
job_id=f"preview_{job_id}_{preview_hash}",
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
background_color="black",
background_image_gcs_path=background_image_gcs_path,
font_gcs_path=font_gcs_path,
)
gce_span.set_attribute("gce_status", gce_result.get("status"))
add_span_event("gce_encoding_complete")
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
except Exception as gce_error:
# Fall back to local encoding if GCE fails
logger.warning(
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
)
span.set_attribute("gce_fallback", True)
use_gce_preview = False # Fall through to local encoding below
if not use_gce_preview:
# Local path: Generate full preview video locally
with create_span("render-preview-video-local") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger,
ass_only=False, # Generate full video locally
)
preview_hash = result["preview_hash"]
video_path = result["video_path"]
add_span_event("render_complete")
# Upload preview video to GCS
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# Store the GCS path for serving
if job_id not in _preview_videos:
_preview_videos[job_id] = {}
_preview_videos[job_id][preview_hash] = preview_gcs_path
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
span.set_attribute("preview_hash", preview_hash)
span.set_attribute("success", True)
return {"status": "success", "preview_hash": preview_hash}
except Exception as e:
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
span.set_attribute("error", str(e))
+12
View File
@@ -68,9 +68,21 @@ class Settings(BaseSettings):
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
# GCE Preview Encoding (for faster preview video generation)
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
# Worker logs storage mode
# When enabled, worker logs are stored in a Firestore subcollection (jobs/{job_id}/logs)
# instead of an embedded array. This avoids the 1MB document size limit.
# Default is true for new deployments.
use_log_subcollection: bool = os.getenv("USE_LOG_SUBCOLLECTION", "true").lower() in ("true", "1", "yes")
# Flacfetch remote service (for torrent downloads)
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
+2 -2
View File
@@ -14,7 +14,7 @@ class URLSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
enable_youtube_upload: Optional[bool] = None # None = use server default
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
@@ -28,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
enable_youtube_upload: Optional[bool] = None # None = use server default
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
+164
View File
@@ -0,0 +1,164 @@
"""
Worker log entry model for subcollection storage.
This module defines the LogEntry model for storing worker logs in a Firestore
subcollection instead of an embedded array. This avoids the 1MB document size
limit that caused job 501258e1 to fail when logs reached 1.26 MB.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone, timedelta
from typing import Optional, Dict, Any
import uuid
# Default TTL for log entries (30 days)
DEFAULT_LOG_TTL_DAYS = 30
@dataclass
class WorkerLogEntry:
"""
Worker log entry for Firestore subcollection storage.
Stored at: jobs/{job_id}/logs/{log_id}
This model is separate from the legacy LogEntry (in job.py) which is
stored as an embedded array in the job document. This new model supports
the subcollection approach with TTL and richer metadata.
"""
# Core fields (required)
timestamp: datetime
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
worker: str # "audio", "lyrics", "screens", "video", "render", "distribution"
message: str
# Identifiers
id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = "" # Set when writing to Firestore
# TTL for automatic cleanup
ttl_expiry: datetime = field(
default_factory=lambda: datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
)
# Optional metadata for debugging
metadata: Optional[Dict[str, Any]] = None
@classmethod
def create(
cls,
job_id: str,
worker: str,
level: str,
message: str,
metadata: Optional[Dict[str, Any]] = None,
ttl_days: int = DEFAULT_LOG_TTL_DAYS
) -> "WorkerLogEntry":
"""
Factory method to create a new log entry.
Args:
job_id: Job ID this log belongs to
worker: Worker name (audio, lyrics, screens, video, render, distribution)
level: Log level (DEBUG, INFO, WARNING, ERROR)
message: Log message (truncated to 1000 chars)
metadata: Optional additional metadata
ttl_days: Days until log expires (default 30)
Returns:
New WorkerLogEntry instance
"""
now = datetime.now(timezone.utc)
return cls(
id=str(uuid.uuid4()),
job_id=job_id,
timestamp=now,
level=level.upper(),
worker=worker,
message=message[:1000], # Truncate long messages
metadata=metadata,
ttl_expiry=now + timedelta(days=ttl_days)
)
def to_dict(self) -> Dict[str, Any]:
"""
Convert to dictionary for Firestore storage.
Returns:
Dictionary representation for Firestore
"""
result = {
"id": self.id,
"job_id": self.job_id,
"timestamp": self.timestamp,
"level": self.level,
"worker": self.worker,
"message": self.message,
"ttl_expiry": self.ttl_expiry
}
if self.metadata:
result["metadata"] = self.metadata
return result
def to_legacy_dict(self) -> Dict[str, Any]:
"""
Convert to legacy format for backward compatibility with API responses.
The existing API returns logs in this format:
{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}
Returns:
Dictionary in legacy format
"""
# Format timestamp: use "Z" suffix for UTC, isoformat() for others
if self.timestamp.tzinfo is None:
# Naive datetime - use as-is
timestamp_str = self.timestamp.isoformat()
elif self.timestamp.utcoffset() == timedelta(0):
# UTC datetime - replace +00:00 with Z for cleaner format
timestamp_str = self.timestamp.replace(tzinfo=None).isoformat() + "Z"
else:
# Non-UTC timezone-aware - use full isoformat with offset
timestamp_str = self.timestamp.isoformat()
return {
"timestamp": timestamp_str,
"level": self.level,
"worker": self.worker,
"message": self.message
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "WorkerLogEntry":
"""
Create from Firestore document data.
Args:
data: Dictionary from Firestore document
Returns:
WorkerLogEntry instance
"""
# Handle timestamp conversion (Firestore returns datetime objects)
timestamp = data.get("timestamp")
if isinstance(timestamp, str):
# Parse ISO format string
timestamp = datetime.fromisoformat(timestamp.rstrip("Z")).replace(tzinfo=timezone.utc)
ttl_expiry = data.get("ttl_expiry")
if isinstance(ttl_expiry, str):
ttl_expiry = datetime.fromisoformat(ttl_expiry.rstrip("Z")).replace(tzinfo=timezone.utc)
elif ttl_expiry is None:
# Default TTL if not set
ttl_expiry = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
return cls(
id=data.get("id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
timestamp=timestamp or datetime.now(timezone.utc),
level=data.get("level", "INFO"),
worker=data.get("worker", "unknown"),
message=data.get("message", ""),
metadata=data.get("metadata"),
ttl_expiry=ttl_expiry
)
+172
View File
@@ -0,0 +1,172 @@
"""
Discord Notification Service.
Provides Discord webhook notification functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- Posting messages to Discord webhooks
- Video upload notifications
- Validation of webhook URLs
"""
import logging
import re
from typing import Optional
import requests
logger = logging.getLogger(__name__)
class DiscordNotificationService:
"""
Service for sending Discord webhook notifications.
Supports posting messages to Discord channels via webhooks,
commonly used to notify when new karaoke videos are uploaded.
"""
# Discord webhook URL pattern
WEBHOOK_URL_PATTERN = re.compile(
r"^https://discord\.com/api/webhooks/\d+/[a-zA-Z0-9_-]+$"
)
def __init__(
self,
webhook_url: Optional[str] = None,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the Discord notification service.
Args:
webhook_url: Discord webhook URL for posting notifications
dry_run: If True, log actions without performing them
logger: Optional logger instance
"""
self.webhook_url = webhook_url
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
# Validate webhook URL if provided
if webhook_url:
self._validate_webhook_url(webhook_url)
def _validate_webhook_url(self, url: str) -> None:
"""
Validate that the webhook URL is a valid Discord webhook URL.
Args:
url: URL to validate
Raises:
ValueError: If URL is not a valid Discord webhook URL
"""
url = url.strip()
if not url.startswith("https://discord.com/api/webhooks/"):
raise ValueError(
f"Invalid Discord webhook URL: {url}. "
"URL must start with 'https://discord.com/api/webhooks/'"
)
def post_message(
self,
message: str,
webhook_url: Optional[str] = None,
) -> bool:
"""
Post a message to a Discord channel via webhook.
Args:
message: The message content to post
webhook_url: Optional webhook URL (uses instance URL if not provided)
Returns:
True if message was posted successfully, False otherwise
Raises:
ValueError: If no webhook URL is available
requests.HTTPError: If the webhook request fails
"""
url = webhook_url or self.webhook_url
if not url:
raise ValueError("No Discord webhook URL provided")
url = url.strip()
self._validate_webhook_url(url)
if self.dry_run:
self.logger.info(
f"DRY RUN: Would post Discord message: '{message}' to {url}"
)
return True
self.logger.info(f"Posting message to Discord webhook...")
data = {"content": message}
response = requests.post(url, json=data, timeout=30)
response.raise_for_status()
self.logger.info("Message posted to Discord successfully")
return True
def post_video_notification(
self,
youtube_url: str,
webhook_url: Optional[str] = None,
) -> bool:
"""
Post a notification about a new video upload.
Args:
youtube_url: The YouTube URL of the uploaded video
webhook_url: Optional webhook URL (uses instance URL if not provided)
Returns:
True if notification was posted successfully, False otherwise
"""
if not youtube_url:
self.logger.info("Skipping Discord notification - no YouTube URL available")
return False
message = f"New upload: {youtube_url}"
return self.post_message(message, webhook_url)
def is_enabled(self) -> bool:
"""
Check if Discord notifications are enabled.
Returns:
True if a valid webhook URL is configured
"""
return bool(self.webhook_url)
# Singleton instance and factory function (following existing service pattern)
_discord_notification_service: Optional[DiscordNotificationService] = None
def get_discord_notification_service(
webhook_url: Optional[str] = None,
**kwargs
) -> DiscordNotificationService:
"""
Get a Discord notification service instance.
Args:
webhook_url: Discord webhook URL for notifications
**kwargs: Additional arguments passed to DiscordNotificationService
Returns:
DiscordNotificationService instance
"""
global _discord_notification_service
# Create new instance if webhook URL changed or not yet created
if _discord_notification_service is None or webhook_url:
_discord_notification_service = DiscordNotificationService(
webhook_url=webhook_url,
**kwargs
)
return _discord_notification_service
+454
View File
@@ -0,0 +1,454 @@
"""
Encoding Interface.
Defines the abstract interface for video encoding backends, allowing
the video worker orchestrator to use either GCE or local encoding
interchangeably.
This follows the Strategy pattern - different encoding implementations
can be swapped without changing the orchestration logic.
"""
import logging
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Any
logger = logging.getLogger(__name__)
@dataclass
class EncodingInput:
"""
Input configuration for video encoding.
Contains all the paths and metadata needed to encode a karaoke video.
"""
# Required input files
title_video_path: str # Title card video (MOV)
karaoke_video_path: str # Main karaoke video with vocals (MOV/MKV)
instrumental_audio_path: str # Instrumental audio track (FLAC)
# Optional input files
end_video_path: Optional[str] = None # End credits video (MOV)
# Metadata for output naming
artist: str = ""
title: str = ""
brand_code: Optional[str] = None
# Output directory
output_dir: str = ""
# Additional options
options: Dict[str, Any] = field(default_factory=dict)
@dataclass
class EncodingOutput:
"""
Output from video encoding.
Contains paths to all generated video files and status information.
"""
success: bool
error_message: Optional[str] = None
# Output file paths (relative to output_dir or absolute)
karaoke_mp4_path: Optional[str] = None # Karaoke video with instrumental audio
with_vocals_mp4_path: Optional[str] = None # With vocals MP4 version
lossless_4k_mp4_path: Optional[str] = None # Final lossless 4K MP4
lossy_4k_mp4_path: Optional[str] = None # Final lossy 4K MP4
lossless_mkv_path: Optional[str] = None # MKV with FLAC (for YouTube)
lossy_720p_mp4_path: Optional[str] = None # 720p web version
# All output files as a dict for convenience
output_files: Dict[str, str] = field(default_factory=dict)
# Encoding metadata
encoding_time_seconds: Optional[float] = None
encoding_backend: Optional[str] = None # "gce" or "local"
class EncodingBackend(ABC):
"""
Abstract base class for video encoding backends.
Implementations:
- GCEEncodingBackend: Cloud-based encoding using Google Compute Engine
- LocalEncodingBackend: Local FFmpeg-based encoding
"""
@property
@abstractmethod
def name(self) -> str:
"""Return the name of this encoding backend."""
pass
@abstractmethod
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video files according to the input configuration.
This method should:
1. Create karaoke video with instrumental audio
2. Convert with-vocals video to MP4 if needed
3. Encode lossless 4K MP4 (concatenating title + karaoke + end)
4. Encode lossy 4K MP4 with AAC audio
5. Create MKV with FLAC audio for YouTube
6. Encode 720p version for web
Args:
input_config: EncodingInput with all required paths and options
Returns:
EncodingOutput with paths to generated files and status
"""
pass
@abstractmethod
async def is_available(self) -> bool:
"""
Check if this encoding backend is available and configured.
Returns:
True if the backend can be used, False otherwise
"""
pass
@abstractmethod
async def get_status(self) -> Dict[str, Any]:
"""
Get the current status of the encoding backend.
Returns:
Dict with status information (availability, queue length, etc.)
"""
pass
class LocalEncodingBackend(EncodingBackend):
"""
Local FFmpeg-based encoding backend.
Wraps the LocalEncodingService to implement the EncodingBackend interface.
Uses asyncio.to_thread() to run synchronous FFmpeg operations asynchronously.
"""
def __init__(
self,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the local encoding backend.
Args:
dry_run: If True, log operations without executing
logger: Optional logger instance
"""
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
self._service = None
@property
def name(self) -> str:
return "local"
def _get_service(self):
"""Lazy-load the local encoding service."""
if self._service is None:
from backend.services.local_encoding_service import LocalEncodingService
self._service = LocalEncodingService(
dry_run=self.dry_run,
logger=self.logger
)
return self._service
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video using local FFmpeg.
Args:
input_config: Encoding input configuration
Returns:
EncodingOutput with results
"""
import asyncio
import time
from backend.services.local_encoding_service import EncodingConfig
start_time = time.time()
# Build output file paths
base_name = f"{input_config.artist} - {input_config.title}"
output_dir = input_config.output_dir or "."
import os
config = EncodingConfig(
title_video=input_config.title_video_path,
karaoke_video=input_config.karaoke_video_path,
instrumental_audio=input_config.instrumental_audio_path,
end_video=input_config.end_video_path,
output_karaoke_mp4=os.path.join(output_dir, f"{base_name} (Karaoke).mp4"),
output_with_vocals_mp4=os.path.join(output_dir, f"{base_name} (With Vocals).mp4"),
output_lossless_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mp4"),
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
)
# Run encoding in thread pool to avoid blocking
service = self._get_service()
result = await asyncio.to_thread(service.encode_all_formats, config)
encoding_time = time.time() - start_time
if result.success:
return EncodingOutput(
success=True,
karaoke_mp4_path=config.output_karaoke_mp4,
with_vocals_mp4_path=config.output_with_vocals_mp4,
lossless_4k_mp4_path=config.output_lossless_4k_mp4,
lossy_4k_mp4_path=config.output_lossy_4k_mp4,
lossless_mkv_path=config.output_lossless_mkv,
lossy_720p_mp4_path=config.output_720p_mp4,
output_files=result.output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
else:
return EncodingOutput(
success=False,
error_message=result.error,
output_files=result.output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
async def is_available(self) -> bool:
"""Check if local encoding is available (FFmpeg installed)."""
import subprocess
import asyncio
try:
result = await asyncio.to_thread(
subprocess.run,
["ffmpeg", "-version"],
capture_output=True,
timeout=10
)
return result.returncode == 0
except (FileNotFoundError, subprocess.TimeoutExpired):
return False
async def get_status(self) -> Dict[str, Any]:
"""Get local encoding status."""
available = await self.is_available()
service = self._get_service()
return {
"backend": self.name,
"available": available,
"hwaccel_available": service.hwaccel_available if available else False,
"video_encoder": service.video_encoder if available else None,
}
class GCEEncodingBackend(EncodingBackend):
"""
GCE-based encoding backend.
Wraps the existing EncodingService to implement the EncodingBackend interface.
Submits jobs to a remote GCE worker for high-performance encoding.
"""
def __init__(
self,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the GCE encoding backend.
Args:
dry_run: Ignored for GCE backend (remote service doesn't support dry run)
logger: Optional logger instance
"""
self.dry_run = dry_run # Stored but not used (GCE is remote)
self.logger = logger or logging.getLogger(__name__)
self._service = None
@property
def name(self) -> str:
return "gce"
def _get_service(self):
"""Lazy-load the GCE encoding service."""
if self._service is None:
from backend.services.encoding_service import get_encoding_service
self._service = get_encoding_service()
return self._service
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video using GCE worker.
Note: GCE encoding requires files to be in GCS, so this method
expects input_config.options to contain 'input_gcs_path' and
'output_gcs_path'.
Args:
input_config: Encoding input configuration
Returns:
EncodingOutput with results
"""
import time
start_time = time.time()
service = self._get_service()
# GCE requires GCS paths
input_gcs_path = input_config.options.get("input_gcs_path")
output_gcs_path = input_config.options.get("output_gcs_path")
job_id = input_config.options.get("job_id", input_config.brand_code or "unknown")
if not input_gcs_path or not output_gcs_path:
return EncodingOutput(
success=False,
error_message="GCE encoding requires input_gcs_path and output_gcs_path in options",
encoding_backend=self.name
)
try:
# Build encoding config
encoding_config = {
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
"artist": input_config.artist,
"title": input_config.title,
}
# Submit and wait for completion
result = await service.encode_videos(
job_id=job_id,
input_gcs_path=input_gcs_path,
output_gcs_path=output_gcs_path,
encoding_config=encoding_config,
)
encoding_time = time.time() - start_time
# Extract output file paths from result
# Handle case where GCE worker returns a list or unexpected format
if isinstance(result, list):
# If result is a list, try to find the output_files in the first dict
self.logger.warning(f"GCE returned list instead of dict: {result}")
result = result[0] if result and isinstance(result[0], dict) else {}
if not isinstance(result, dict):
self.logger.error(f"Unexpected GCE result type: {type(result)}")
result = {}
raw_output_files = result.get("output_files", {})
# Convert output_files from list of paths to dict
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
if isinstance(raw_output_files, list):
output_files = {}
for path in raw_output_files:
if not isinstance(path, str):
continue
filename = path.split("/")[-1] if "/" in path else path
filename_lower = filename.lower()
# Map filename patterns to output format keys
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
if "lossless 4k" in filename_lower:
if filename.endswith(".mkv"):
output_files["mkv_4k"] = path
else:
output_files["mp4_4k_lossless"] = path
elif "lossy 4k" in filename_lower:
output_files["mp4_4k_lossy"] = path
elif "720p" in filename_lower:
output_files["mp4_720p"] = path
self.logger.info(f"Converted output_files list to dict: {output_files}")
else:
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
return EncodingOutput(
success=True,
lossless_4k_mp4_path=output_files.get("mp4_4k_lossless"),
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
lossless_mkv_path=output_files.get("mkv_4k"),
lossy_720p_mp4_path=output_files.get("mp4_720p"),
output_files=output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
except Exception as e:
encoding_time = time.time() - start_time
self.logger.error(f"GCE encoding failed: {e}")
return EncodingOutput(
success=False,
error_message=str(e),
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
async def is_available(self) -> bool:
"""Check if GCE encoding is available and configured."""
service = self._get_service()
return service.is_enabled
async def get_status(self) -> Dict[str, Any]:
"""Get GCE encoding status."""
service = self._get_service()
status = {
"backend": self.name,
"available": service.is_enabled,
"configured": service.is_configured,
}
if service.is_configured:
try:
health = await service.health_check()
status["health"] = health
except Exception as e:
status["health_error"] = str(e)
return status
# Factory function to get an encoding backend
def get_encoding_backend(
backend_type: str = "auto",
**kwargs
) -> EncodingBackend:
"""
Get an encoding backend instance.
Args:
backend_type: Type of backend - "local", "gce", or "auto"
**kwargs: Additional arguments for the backend
Returns:
EncodingBackend instance
Raises:
ValueError: If backend_type is unknown
"""
if backend_type == "local":
return LocalEncodingBackend(**kwargs)
elif backend_type == "gce":
return GCEEncodingBackend(**kwargs)
elif backend_type == "auto":
# Check if GCE is available, otherwise use local
gce_backend = GCEEncodingBackend(**kwargs)
if gce_backend._get_service().is_enabled:
return gce_backend
return LocalEncodingBackend(**kwargs)
else:
raise ValueError(f"Unknown encoding backend type: {backend_type}")
+135
View File
@@ -63,6 +63,11 @@ class EncodingService:
"""Check if GCE encoding is enabled and configured."""
return self.settings.use_gce_encoding and self.is_configured
@property
def is_preview_enabled(self) -> bool:
"""Check if GCE preview encoding is enabled and configured."""
return self.settings.use_gce_preview_encoding and self.is_configured
async def submit_encoding_job(
self,
job_id: str,
@@ -177,6 +182,15 @@ class EncodingService:
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
status = await self.get_job_status(job_id)
# Handle case where GCE worker returns a list instead of dict
if isinstance(status, list):
logger.warning(f"[job:{job_id}] GCE returned list instead of dict: {status}")
status = status[0] if status and isinstance(status[0], dict) else {}
if not isinstance(status, dict):
logger.error(f"[job:{job_id}] Unexpected status type: {type(status)}")
status = {}
job_status = status.get("status", "unknown")
progress = status.get("progress", 0)
@@ -233,6 +247,127 @@ class EncodingService:
job_id, progress_callback=progress_callback
)
async def submit_preview_encoding_job(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
font_gcs_path: Optional[str] = None,
) -> Dict[str, Any]:
"""
Submit a preview video encoding job to the GCE worker.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
font_gcs_path: Optional GCS path to custom font file
Returns:
Response from the encoding worker
Raises:
Exception: If submission fails
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._url}/encode-preview"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
"ass_gcs_path": ass_gcs_path,
"audio_gcs_path": audio_gcs_path,
"output_gcs_path": output_gcs_path,
"background_color": background_color,
}
if background_image_gcs_path:
payload["background_image_gcs_path"] = background_image_gcs_path
if font_gcs_path:
payload["font_gcs_path"] = font_gcs_path
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
async with aiohttp.ClientSession() as session:
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
if resp.status == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp.status == 409:
raise RuntimeError(f"Preview encoding job {job_id} already exists")
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"Failed to submit preview encoding job: {resp.status} - {text}")
return await resp.json()
async def encode_preview_video(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
font_gcs_path: Optional[str] = None,
timeout: float = 90.0,
poll_interval: float = 2.0,
) -> Dict[str, Any]:
"""
Submit preview encoding job and wait for completion.
This is a convenience method that combines submit + wait with shorter
timeouts suitable for preview videos.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
font_gcs_path: Optional GCS path to custom font file
timeout: Maximum time to wait (default 90s for preview)
poll_interval: Seconds between status checks (default 2s)
Returns:
Final job status with output files
"""
# Submit the job
submit_result = await self.submit_preview_encoding_job(
job_id=job_id,
ass_gcs_path=ass_gcs_path,
audio_gcs_path=audio_gcs_path,
output_gcs_path=output_gcs_path,
background_color=background_color,
background_image_gcs_path=background_image_gcs_path,
font_gcs_path=font_gcs_path,
)
# If cached, return immediately - video already exists in GCS
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
return {"status": "complete", "output_path": submit_result.get("output_path")}
# If in_progress, another request is encoding it - just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
# Wait for completion with shorter timeout
return await self.wait_for_completion(
job_id=job_id,
poll_interval=poll_interval,
timeout=timeout,
)
async def health_check(self) -> Dict[str, Any]:
"""
Check the health of the encoding worker.
+165 -2
View File
@@ -9,6 +9,7 @@ from google.cloud.firestore_v1 import FieldFilter
from backend.config import settings
from backend.models.job import Job, JobStatus, TimelineEvent
from backend.models.worker_log import WorkerLogEntry
logger = logging.getLogger(__name__)
@@ -245,10 +246,13 @@ class FirestoreService:
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
"""
Atomically append a log entry to worker_logs using ArrayUnion.
This avoids the race condition of read-modify-write when multiple
workers are logging concurrently.
DEPRECATED: Use append_log_to_subcollection() instead to avoid
the 1MB document size limit.
Args:
job_id: Job ID
log_entry: Log entry dict with timestamp, level, worker, message
@@ -263,6 +267,165 @@ class FirestoreService:
except Exception as e:
# Log but don't raise - logging shouldn't break workers
logger.debug(f"Error appending worker log for job {job_id}: {e}")
# ============================================
# Worker Log Subcollection Methods
# ============================================
# These methods store logs in a subcollection (jobs/{job_id}/logs)
# instead of an embedded array to avoid the 1MB document size limit.
def append_log_to_subcollection(self, job_id: str, log_entry: WorkerLogEntry) -> None:
"""
Append a log entry to the logs subcollection.
Stores logs at: jobs/{job_id}/logs/{log_id}
This approach avoids the 1MB document size limit by storing each
log entry as a separate document in a subcollection.
Args:
job_id: Job ID
log_entry: WorkerLogEntry instance
"""
try:
# Ensure job_id is set on the log entry
log_entry.job_id = job_id
# Get subcollection reference
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
# Add document with auto-generated ID or use log_entry.id
doc_ref = logs_ref.document(log_entry.id)
doc_ref.set(log_entry.to_dict())
# Don't log every append - too spammy
except Exception as e:
# Log but don't raise - logging shouldn't break workers
logger.debug(f"Error appending log to subcollection for job {job_id}: {e}")
def get_logs_from_subcollection(
self,
job_id: str,
limit: int = 500,
since_timestamp: Optional[datetime] = None,
worker: Optional[str] = None,
offset: int = 0
) -> List[WorkerLogEntry]:
"""
Get log entries from the logs subcollection.
Args:
job_id: Job ID
limit: Maximum number of logs to return (default 500)
since_timestamp: Return only logs after this timestamp
worker: Filter by worker name (optional)
offset: Number of logs to skip (for pagination)
Returns:
List of WorkerLogEntry instances, ordered by timestamp ascending
"""
try:
# Get subcollection reference
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
# Build query
query = logs_ref.order_by("timestamp", direction=firestore.Query.ASCENDING)
if since_timestamp:
query = query.where(filter=FieldFilter("timestamp", ">", since_timestamp))
if worker:
query = query.where(filter=FieldFilter("worker", "==", worker))
# Apply offset and limit
if offset > 0:
query = query.offset(offset)
query = query.limit(limit)
# Execute query
docs = query.stream()
logs = [WorkerLogEntry.from_dict(doc.to_dict()) for doc in docs]
return logs
except Exception as e:
logger.error(f"Error getting logs from subcollection for job {job_id}: {e}")
return []
def get_logs_count_from_subcollection(self, job_id: str) -> int:
"""
Get the total count of log entries in the subcollection.
Args:
job_id: Job ID
Returns:
Total count of log entries
"""
try:
# Get subcollection reference
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
# Use aggregation query for efficient counting
count_query = logs_ref.count()
result = count_query.get()
# Result is a list of AggregationResult, we want the first one's count
if result and len(result) > 0:
return result[0][0].value
return 0
except Exception as e:
logger.error(f"Error counting logs for job {job_id}: {e}")
return 0
def delete_logs_subcollection(self, job_id: str, batch_size: int = 500) -> int:
"""
Delete all log entries in the logs subcollection.
This is used when deleting a job to clean up its logs.
Args:
job_id: Job ID
batch_size: Number of documents to delete per batch
Returns:
Number of logs deleted
"""
try:
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
deleted_count = 0
while True:
# Get a batch of documents
docs = logs_ref.limit(batch_size).stream()
deleted_in_batch = 0
# Delete in a batch
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_in_batch += 1
if deleted_in_batch == 0:
break
batch.commit()
deleted_count += deleted_in_batch
# If we deleted less than batch_size, we're done
if deleted_in_batch < batch_size:
break
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} logs for job {job_id}")
return deleted_count
except Exception as e:
logger.error(f"Error deleting logs subcollection for job {job_id}: {e}")
return 0
# ============================================
# Token Management Methods
+103 -3
View File
@@ -408,11 +408,11 @@ class FlacfetchClient:
) -> Dict[str, Any]:
"""
Trigger disk cleanup.
Args:
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
target_free_gb: Target free space to achieve
Returns:
Cleanup response with removed count and freed space
"""
@@ -429,12 +429,112 @@ class FlacfetchClient:
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
# =========================================================================
# Cache Management
# =========================================================================
async def clear_search_cache(self, artist: str, title: str) -> bool:
"""
Clear a specific cached search result by artist and title.
Args:
artist: Artist name
title: Track title
Returns:
True if a cache entry was deleted, False if no entry existed
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.request(
"DELETE",
f"{self.base_url}/cache/search",
headers=self._headers(),
json={"artist": artist, "title": title},
timeout=30,
)
resp.raise_for_status()
data = resp.json()
deleted = data.get("deleted", False)
logger.info(
f"Cleared flacfetch cache for '{artist}' - '{title}': "
f"{'deleted' if deleted else 'no entry found'}"
)
return deleted
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
)
async def clear_all_cache(self) -> int:
"""
Clear all cached search results.
Returns:
Number of cache entries deleted
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.delete(
f"{self.base_url}/cache",
headers=self._headers(),
timeout=60,
)
resp.raise_for_status()
data = resp.json()
deleted_count = data.get("deleted_count", 0)
logger.info(f"Cleared all flacfetch cache: {deleted_count} entries deleted")
return deleted_count
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
)
async def get_cache_stats(self) -> Dict[str, Any]:
"""
Get statistics about the cache.
Returns:
Dict with count, total_size_bytes, oldest_entry, newest_entry, configured
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/cache/stats",
headers=self._headers(),
timeout=30,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
)
# Singleton client instance
_client: Optional[FlacfetchClient] = None
+90 -28
View File
@@ -13,7 +13,9 @@ import uuid
from datetime import datetime
from typing import Optional, Dict, Any, List
from backend.config import settings
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
from backend.models.worker_log import WorkerLogEntry
from backend.services.firestore_service import FirestoreService
from backend.services.storage_service import StorageService
@@ -224,7 +226,7 @@ class JobManager:
return urls
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
"""Delete a job and optionally its files."""
"""Delete a job, its files, and its logs subcollection."""
if delete_files:
job = self.get_job(job_id)
if job and job.output_files:
@@ -233,7 +235,15 @@ class JobManager:
self.storage.delete_file(gcs_path)
except Exception as e:
logger.error(f"Error deleting file {gcs_path}: {e}")
# Delete logs subcollection first (must be done before deleting parent doc)
try:
deleted_logs = self.firestore.delete_logs_subcollection(job_id)
if deleted_logs > 0:
logger.info(f"Deleted {deleted_logs} log entries for job {job_id}")
except Exception as e:
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
self.firestore.delete_job(job_id)
logger.info(f"Deleted job {job_id}")
@@ -579,33 +589,45 @@ class JobManager:
worker: str,
level: str,
message: str,
max_logs: int = 500
max_logs: int = 500,
metadata: Optional[Dict[str, Any]] = None
) -> None:
"""
Append a log entry to the job's worker_logs atomically.
Uses Firestore ArrayUnion for atomic append to avoid race conditions
when multiple workers (audio + lyrics) are logging concurrently.
Note: max_logs is not enforced per-append (would require transactions).
Logs may grow beyond max_logs; trim periodically if needed.
Append a log entry to the job's logs.
By default (USE_LOG_SUBCOLLECTION=true), logs are stored in a Firestore
subcollection (jobs/{job_id}/logs) to avoid the 1MB document size limit.
For backward compatibility (USE_LOG_SUBCOLLECTION=false), logs are stored
in the embedded worker_logs array using atomic ArrayUnion.
Args:
job_id: Job ID
worker: Worker name (audio, lyrics, screens, video, render)
worker: Worker name (audio, lyrics, screens, video, render, distribution)
level: Log level (DEBUG, INFO, WARNING, ERROR)
message: Log message
max_logs: Not used (kept for API compatibility)
metadata: Optional additional metadata dict
"""
log_entry = {
'timestamp': datetime.utcnow().isoformat() + 'Z',
'level': level,
'worker': worker,
'message': message[:1000] # Truncate long messages
}
# Use atomic ArrayUnion to avoid race conditions
self.firestore.append_worker_log(job_id, log_entry)
if settings.use_log_subcollection:
# New subcollection approach - avoids 1MB limit
log_entry = WorkerLogEntry.create(
job_id=job_id,
worker=worker,
level=level,
message=message,
metadata=metadata
)
self.firestore.append_log_to_subcollection(job_id, log_entry)
else:
# Legacy embedded array approach
log_entry = {
'timestamp': datetime.utcnow().isoformat() + 'Z',
'level': level,
'worker': worker,
'message': message[:1000] # Truncate long messages
}
self.firestore.append_worker_log(job_id, log_entry)
def get_worker_logs(
self,
@@ -615,28 +637,68 @@ class JobManager:
) -> List[Dict[str, Any]]:
"""
Get worker logs for a job, optionally filtered by worker and index.
By default (USE_LOG_SUBCOLLECTION=true), logs are read from the
subcollection. Falls back to embedded array for older jobs.
Args:
job_id: Job ID
since_index: Return only logs after this index (for pagination)
worker: Filter by worker name (optional)
Returns:
List of log entries as dicts
List of log entries as dicts (in legacy format for API compatibility)
"""
if settings.use_log_subcollection:
# Try subcollection first
subcollection_logs = self.firestore.get_logs_from_subcollection(
job_id=job_id,
offset=since_index,
worker=worker,
limit=500
)
if subcollection_logs:
# Convert to legacy format for API compatibility
return [log.to_legacy_dict() for log in subcollection_logs]
# Fall through to check embedded array for older jobs
# Embedded array approach (legacy jobs or fallback)
job = self.get_job(job_id)
if not job or not job.worker_logs:
return []
logs = [log.dict() if hasattr(log, 'dict') else log for log in job.worker_logs]
# Filter by index
if since_index > 0:
logs = logs[since_index:]
# Filter by worker
if worker:
logs = [log for log in logs if log.get('worker') == worker]
return logs
def get_worker_logs_count(self, job_id: str) -> int:
"""
Get the total count of worker logs for a job.
Args:
job_id: Job ID
Returns:
Total count of logs
"""
if settings.use_log_subcollection:
# Try subcollection first
count = self.firestore.get_logs_count_from_subcollection(job_id)
if count > 0:
return count
# Fall through to check embedded array
# Embedded array (legacy jobs or fallback)
job = self.get_job(job_id)
if not job or not job.worker_logs:
return 0
return len(job.worker_logs)
+590
View File
@@ -0,0 +1,590 @@
"""
Local Encoding Service.
Provides local FFmpeg-based video encoding functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker as a fallback) and local CLI.
This service handles:
- Concatenating title, karaoke, and end videos
- Encoding to multiple output formats (4K lossless, 4K lossy, 720p, MKV)
- Hardware acceleration detection and fallback to software encoding
- Remuxing video with instrumental audio
"""
import logging
import os
import shlex
import subprocess
import sys
from dataclasses import dataclass
from typing import Optional, List, Dict, Any, Tuple
logger = logging.getLogger(__name__)
@dataclass
class EncodingConfig:
"""Configuration for video encoding."""
title_video: str # Path to title video
karaoke_video: str # Path to karaoke video (with vocals)
instrumental_audio: str # Path to instrumental audio
end_video: Optional[str] = None # Optional path to end credits video
# Output paths
output_karaoke_mp4: Optional[str] = None
output_with_vocals_mp4: Optional[str] = None
output_lossless_4k_mp4: Optional[str] = None
output_lossy_4k_mp4: Optional[str] = None
output_lossless_mkv: Optional[str] = None
output_720p_mp4: Optional[str] = None
@dataclass
class EncodingResult:
"""Result of video encoding operation."""
success: bool
output_files: Dict[str, str]
error: Optional[str] = None
class LocalEncodingService:
"""
Service for local FFmpeg-based video encoding.
Supports hardware acceleration (NVENC) with automatic fallback
to software encoding (libx264) when unavailable.
"""
# MP4 flags for better compatibility
MP4_FLAGS = "-movflags +faststart"
def __init__(
self,
dry_run: bool = False,
log_level: int = logging.INFO,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the local encoding service.
Args:
dry_run: If True, log commands without executing them
log_level: Logging level (affects FFmpeg verbosity)
logger: Optional logger instance
"""
self.dry_run = dry_run
self.log_level = log_level
self.logger = logger or logging.getLogger(__name__)
# Hardware acceleration settings (detected on first use)
self._hwaccel_available: Optional[bool] = None
self._video_encoder: Optional[str] = None
self._scale_filter: Optional[str] = None
self._hwaccel_decode_flags: Optional[str] = None
self._aac_codec: Optional[str] = None
# Build FFmpeg base command
self._ffmpeg_base_command = self._build_ffmpeg_base_command()
def _build_ffmpeg_base_command(self) -> str:
"""Build the FFmpeg base command with appropriate flags."""
# Use bundled FFmpeg for frozen builds
ffmpeg_path = os.path.join(sys._MEIPASS, "ffmpeg.exe") if getattr(sys, "frozen", False) else "ffmpeg"
base_cmd = f"{ffmpeg_path} -hide_banner -nostats -y"
if self.log_level == logging.DEBUG:
base_cmd += " -loglevel verbose"
else:
base_cmd += " -loglevel fatal"
return base_cmd
def _detect_hardware_acceleration(self) -> Tuple[bool, str, str, str, str]:
"""
Detect available hardware acceleration.
Returns:
Tuple of (hwaccel_available, video_encoder, scale_filter,
hwaccel_decode_flags, aac_codec)
"""
self.logger.info("Detecting hardware acceleration capabilities...")
# Try NVENC (NVIDIA)
try:
test_cmd = f"{self._ffmpeg_base_command} -hide_banner -loglevel error " \
f"-f lavfi -i testsrc=duration=1:size=320x240:rate=1 " \
f"-c:v h264_nvenc -f null -"
subprocess.run(
test_cmd, shell=True, check=True,
capture_output=True, timeout=30
)
self.logger.info("NVIDIA NVENC hardware acceleration available")
return (
True,
"h264_nvenc",
"scale_cuda",
"-hwaccel cuda -hwaccel_output_format cuda",
"aac"
)
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
pass
# No hardware acceleration available
self.logger.info("No hardware acceleration available, using software encoding")
return (False, "libx264", "scale", "", "aac")
@property
def hwaccel_available(self) -> bool:
"""Check if hardware acceleration is available."""
if self._hwaccel_available is None:
self._detect_and_set_hwaccel()
return self._hwaccel_available
@property
def video_encoder(self) -> str:
"""Get the video encoder to use."""
if self._video_encoder is None:
self._detect_and_set_hwaccel()
return self._video_encoder
@property
def scale_filter(self) -> str:
"""Get the scale filter to use."""
if self._scale_filter is None:
self._detect_and_set_hwaccel()
return self._scale_filter
@property
def hwaccel_decode_flags(self) -> str:
"""Get hardware acceleration decode flags."""
if self._hwaccel_decode_flags is None:
self._detect_and_set_hwaccel()
return self._hwaccel_decode_flags
@property
def aac_codec(self) -> str:
"""Get the AAC codec to use."""
if self._aac_codec is None:
self._detect_and_set_hwaccel()
return self._aac_codec
def _detect_and_set_hwaccel(self) -> None:
"""Detect and set hardware acceleration settings."""
(
self._hwaccel_available,
self._video_encoder,
self._scale_filter,
self._hwaccel_decode_flags,
self._aac_codec
) = self._detect_hardware_acceleration()
def _get_nvenc_quality_settings(self, preset: str = "medium") -> str:
"""Get NVENC quality settings for different presets."""
if not self.hwaccel_available:
return ""
settings = {
"lossless": "-preset p7 -tune hq -rc vbr -cq 0 -qmin 0 -qmax 0",
"high": "-preset p7 -tune hq -rc vbr -cq 19 -b:v 0",
"medium": "-preset p4 -tune hq -rc vbr -cq 23 -b:v 0",
"fast": "-preset p2 -tune ll -rc vbr -cq 28 -b:v 0",
}
return settings.get(preset, settings["medium"])
def _execute_command(
self,
command: str,
description: str,
timeout: int = 3600, # 1 hour default
) -> bool:
"""
Execute an FFmpeg command.
Args:
command: The FFmpeg command to execute
description: Human-readable description of the operation
timeout: Command timeout in seconds
Returns:
True if successful, False otherwise
"""
self.logger.info(f"Executing: {description}")
self.logger.debug(f"Command: {command}")
if self.dry_run:
self.logger.info(f"DRY RUN: Would execute: {command}")
return True
try:
result = subprocess.run(
command,
shell=True,
check=True,
capture_output=True,
text=True,
timeout=timeout
)
self.logger.info(f"Completed: {description}")
return True
except subprocess.CalledProcessError as e:
self.logger.error(f"Failed: {description}")
self.logger.error(f"Error: {e.stderr}")
return False
except subprocess.TimeoutExpired:
self.logger.error(f"Timeout: {description}")
return False
def _execute_command_with_fallback(
self,
gpu_command: str,
cpu_command: str,
description: str,
) -> bool:
"""
Execute a command with GPU, falling back to CPU if it fails.
Args:
gpu_command: Hardware-accelerated command
cpu_command: Software fallback command
description: Human-readable description
Returns:
True if successful, False otherwise
"""
if self.hwaccel_available:
self.logger.info(f"Trying hardware-accelerated encoding for: {description}")
if self._execute_command(gpu_command, description):
return True
self.logger.warning(f"Hardware encoding failed, falling back to software")
return self._execute_command(cpu_command, description)
def remux_with_instrumental(
self,
input_video: str,
instrumental_audio: str,
output_file: str,
) -> bool:
"""
Remux video with instrumental audio track.
Args:
input_video: Path to input video file
instrumental_audio: Path to instrumental audio file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_video}" '
f'-i "{instrumental_audio}" -map 0:v -map 1:a -c copy '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command(command, "Remuxing with instrumental audio")
def convert_mov_to_mp4(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Convert MOV to MP4 format.
Args:
input_file: Path to input MOV file
output_file: Path for output MP4 file
Returns:
True if successful, False otherwise
"""
gpu_command = (
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
f'-c:v {self.video_encoder} -c:a copy {self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v libx264 -c:a copy {self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Converting MOV to MP4"
)
def encode_lossless_mp4(
self,
title_video: str,
karaoke_video: str,
output_file: str,
end_video: Optional[str] = None,
) -> bool:
"""
Create lossless 4K MP4 by concatenating title, karaoke, and optionally end videos.
Args:
title_video: Path to title video
karaoke_video: Path to karaoke video
output_file: Path for output file
end_video: Optional path to end credits video
Returns:
True if successful, False otherwise
"""
# Quote file paths
title_quoted = shlex.quote(os.path.abspath(title_video))
karaoke_quoted = shlex.quote(os.path.abspath(karaoke_video))
# Build filter and inputs for concatenation
if end_video and os.path.isfile(end_video):
end_quoted = shlex.quote(os.path.abspath(end_video))
extra_input = f"-i {end_quoted}"
concat_filter = (
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0]'
'concat=n=3:v=1:a=1[outv][outa]"'
)
else:
extra_input = ""
concat_filter = (
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0]'
'concat=n=2:v=1:a=1[outv][outa]"'
)
gpu_command = (
f"{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i {title_quoted} "
f"{self.hwaccel_decode_flags} -i {karaoke_quoted} {extra_input} "
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v {self.video_encoder} '
f'{self._get_nvenc_quality_settings("lossless")} -c:a pcm_s16le '
f'{self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f"{self._ffmpeg_base_command} -i {title_quoted} -i {karaoke_quoted} {extra_input} "
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v libx264 -c:a pcm_s16le '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Encoding lossless 4K MP4"
)
def encode_lossy_mp4(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create lossy 4K MP4 with AAC audio.
Args:
input_file: Path to input file (typically lossless MP4)
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v copy -c:a {self.aac_codec} -ar 48000 -b:a 320k '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command(command, "Encoding lossy 4K MP4 with AAC")
def encode_lossless_mkv(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create MKV with FLAC audio (for YouTube upload).
Args:
input_file: Path to input file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v copy -c:a flac "{output_file}"'
)
return self._execute_command(command, "Creating MKV with FLAC for YouTube")
def encode_720p(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create 720p MP4 with AAC audio.
Args:
input_file: Path to input file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
gpu_command = (
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
f'-c:v {self.video_encoder} -vf "{self.scale_filter}=1280:720" '
f'{self._get_nvenc_quality_settings("medium")} -b:v 2000k '
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
f'{self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v libx264 -vf "scale=1280:720" -b:v 2000k -preset medium -tune animation '
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Encoding 720p MP4"
)
def encode_all_formats(
self,
config: EncodingConfig,
) -> EncodingResult:
"""
Encode video to all output formats.
This performs the full encoding pipeline:
1. Remux with instrumental audio
2. Convert to MP4 if needed
3. Encode lossless 4K MP4 (concatenated)
4. Encode lossy 4K MP4
5. Encode lossless MKV (for YouTube)
6. Encode 720p MP4
Args:
config: Encoding configuration with input/output paths
Returns:
EncodingResult with success status and output file paths
"""
output_files = {}
try:
# Step 1: Remux with instrumental audio
if config.output_karaoke_mp4:
self.logger.info("[Step 1/6] Remuxing video with instrumental audio...")
if not self.remux_with_instrumental(
config.karaoke_video,
config.instrumental_audio,
config.output_karaoke_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to remux with instrumental audio"
)
output_files["karaoke_mp4"] = config.output_karaoke_mp4
# Step 2: Convert to MP4 if needed
if config.output_with_vocals_mp4:
if not config.karaoke_video.endswith(".mp4"):
self.logger.info("[Step 2/6] Converting karaoke video to MP4...")
if not self.convert_mov_to_mp4(
config.karaoke_video,
config.output_with_vocals_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to convert to MP4"
)
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
else:
self.logger.info("[Step 2/6] Skipped - video already MP4")
# Step 3: Encode lossless 4K MP4
if config.output_lossless_4k_mp4:
self.logger.info("[Step 3/6] Encoding lossless 4K MP4...")
karaoke_for_concat = config.output_karaoke_mp4 or config.karaoke_video
if not self.encode_lossless_mp4(
config.title_video,
karaoke_for_concat,
config.output_lossless_4k_mp4,
config.end_video
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode lossless 4K MP4"
)
output_files["lossless_4k_mp4"] = config.output_lossless_4k_mp4
# Step 4: Encode lossy 4K MP4
if config.output_lossy_4k_mp4 and config.output_lossless_4k_mp4:
self.logger.info("[Step 4/6] Encoding lossy 4K MP4...")
if not self.encode_lossy_mp4(
config.output_lossless_4k_mp4,
config.output_lossy_4k_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode lossy 4K MP4"
)
output_files["lossy_4k_mp4"] = config.output_lossy_4k_mp4
# Step 5: Create MKV with FLAC audio
if config.output_lossless_mkv and config.output_lossless_4k_mp4:
self.logger.info("[Step 5/6] Creating MKV with FLAC audio...")
if not self.encode_lossless_mkv(
config.output_lossless_4k_mp4,
config.output_lossless_mkv
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to create MKV"
)
output_files["lossless_mkv"] = config.output_lossless_mkv
# Step 6: Encode 720p version
if config.output_720p_mp4 and config.output_lossless_4k_mp4:
self.logger.info("[Step 6/6] Encoding 720p MP4...")
if not self.encode_720p(
config.output_lossless_4k_mp4,
config.output_720p_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode 720p"
)
output_files["720p_mp4"] = config.output_720p_mp4
self.logger.info("All encoding steps completed successfully")
return EncodingResult(success=True, output_files=output_files)
except Exception as e:
self.logger.error(f"Encoding failed with exception: {e}")
return EncodingResult(
success=False,
output_files=output_files,
error=str(e)
)
# Singleton instance and factory function (following existing service pattern)
_local_encoding_service: Optional[LocalEncodingService] = None
def get_local_encoding_service(**kwargs) -> LocalEncodingService:
"""
Get a local encoding service instance.
Args:
**kwargs: Arguments passed to LocalEncodingService
Returns:
LocalEncodingService instance
"""
global _local_encoding_service
if _local_encoding_service is None:
_local_encoding_service = LocalEncodingService(**kwargs)
return _local_encoding_service
+287
View File
@@ -0,0 +1,287 @@
"""
Packaging Service.
Provides CDG and TXT package generation functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- CDG (CD+G) file generation from LRC files
- TXT lyric file generation from LRC files
- ZIP packaging of CDG/MP3 and TXT/MP3 pairs
"""
import logging
import os
import zipfile
from typing import Optional, Dict, Any, Tuple
logger = logging.getLogger(__name__)
class PackagingService:
"""
Service for creating CDG and TXT karaoke packages.
CDG (CD+Graphics) is a format used by karaoke machines.
TXT packages are used by software karaoke players.
Both formats are packaged as ZIP files with an MP3 audio track.
"""
def __init__(
self,
cdg_styles: Optional[Dict[str, Any]] = None,
dry_run: bool = False,
non_interactive: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the packaging service.
Args:
cdg_styles: CDG style configuration for CDG generation
dry_run: If True, log actions without performing them
non_interactive: If True, skip interactive prompts
logger: Optional logger instance
"""
self.cdg_styles = cdg_styles
self.dry_run = dry_run
self.non_interactive = non_interactive
self.logger = logger or logging.getLogger(__name__)
def create_cdg_package(
self,
lrc_file: str,
audio_file: str,
output_zip_path: str,
artist: str,
title: str,
output_mp3_path: Optional[str] = None,
output_cdg_path: Optional[str] = None,
) -> Tuple[str, Optional[str], Optional[str]]:
"""
Create a CDG package (ZIP containing CDG and MP3 files).
Args:
lrc_file: Path to the LRC lyrics file
audio_file: Path to the instrumental audio file
output_zip_path: Path for the output ZIP file
artist: Artist name for metadata
title: Song title for metadata
output_mp3_path: Optional path for the extracted MP3 file
output_cdg_path: Optional path for the extracted CDG file
Returns:
Tuple of (zip_path, mp3_path, cdg_path)
Raises:
ValueError: If CDG styles are not configured
FileNotFoundError: If input files are missing
Exception: If CDG generation fails
"""
self.logger.info(f"Creating CDG package for {artist} - {title}")
# Validate inputs
if not os.path.isfile(lrc_file):
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
if not os.path.isfile(audio_file):
raise FileNotFoundError(f"Audio file not found: {audio_file}")
# Check if ZIP already exists
if os.path.isfile(output_zip_path):
if self.non_interactive:
self.logger.info(
f"CDG ZIP exists, will be overwritten: {output_zip_path}"
)
else:
self.logger.info(f"CDG ZIP already exists: {output_zip_path}")
# Check if individual files exist (allows skipping generation)
if output_mp3_path and output_cdg_path:
if os.path.isfile(output_mp3_path) and os.path.isfile(output_cdg_path):
self.logger.info("Found existing MP3 and CDG files, creating ZIP directly")
if not self.dry_run:
self._create_zip_from_files(
output_zip_path,
[(output_mp3_path, os.path.basename(output_mp3_path)),
(output_cdg_path, os.path.basename(output_cdg_path))]
)
return output_zip_path, output_mp3_path, output_cdg_path
if self.dry_run:
self.logger.info(
f"DRY RUN: Would generate CDG package: {output_zip_path}"
)
return output_zip_path, output_mp3_path, output_cdg_path
# Generate CDG files
if self.cdg_styles is None:
raise ValueError(
"CDG styles configuration is required for CDG generation"
)
self.logger.info("Generating CDG and MP3 files")
from lyrics_transcriber.output.cdg import CDGGenerator
output_dir = os.path.dirname(output_zip_path) or os.getcwd()
generator = CDGGenerator(output_dir=output_dir, logger=self.logger)
cdg_file, mp3_file, zip_file = generator.generate_cdg_from_lrc(
lrc_file=lrc_file,
audio_file=audio_file,
title=title,
artist=artist,
cdg_styles=self.cdg_styles,
)
# Rename ZIP to expected output path if different
if os.path.isfile(zip_file) and zip_file != output_zip_path:
os.rename(zip_file, output_zip_path)
self.logger.info(f"Renamed CDG ZIP: {zip_file} -> {output_zip_path}")
if not os.path.isfile(output_zip_path):
raise Exception(f"Failed to create CDG ZIP file: {output_zip_path}")
# Extract the ZIP to get individual files if paths provided
extracted_mp3 = None
extracted_cdg = None
if output_mp3_path or output_cdg_path:
self.logger.info(f"Extracting CDG ZIP file: {output_zip_path}")
with zipfile.ZipFile(output_zip_path, "r") as zip_ref:
zip_ref.extractall(output_dir)
# Find extracted files
if output_mp3_path and os.path.isfile(output_mp3_path):
extracted_mp3 = output_mp3_path
self.logger.info(f"Extracted MP3: {extracted_mp3}")
if output_cdg_path and os.path.isfile(output_cdg_path):
extracted_cdg = output_cdg_path
self.logger.info(f"Extracted CDG: {extracted_cdg}")
self.logger.info(f"CDG package created: {output_zip_path}")
return output_zip_path, extracted_mp3, extracted_cdg
def create_txt_package(
self,
lrc_file: str,
mp3_file: str,
output_zip_path: str,
output_txt_path: Optional[str] = None,
) -> Tuple[str, Optional[str]]:
"""
Create a TXT package (ZIP containing TXT lyrics and MP3 files).
Args:
lrc_file: Path to the LRC lyrics file
mp3_file: Path to the MP3 audio file
output_zip_path: Path for the output ZIP file
output_txt_path: Optional path for the generated TXT file
Returns:
Tuple of (zip_path, txt_path)
Raises:
FileNotFoundError: If input files are missing
Exception: If TXT generation fails
"""
self.logger.info(f"Creating TXT package from {lrc_file}")
# Validate inputs
if not os.path.isfile(lrc_file):
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
if not os.path.isfile(mp3_file):
raise FileNotFoundError(f"MP3 file not found: {mp3_file}")
# Check if ZIP already exists
if os.path.isfile(output_zip_path):
if self.non_interactive:
self.logger.info(
f"TXT ZIP exists, will be overwritten: {output_zip_path}"
)
else:
self.logger.info(f"TXT ZIP already exists: {output_zip_path}")
if self.dry_run:
self.logger.info(
f"DRY RUN: Would create TXT package: {output_zip_path}"
)
return output_zip_path, output_txt_path
# Generate TXT from LRC
self.logger.info(f"Converting LRC to TXT format: {lrc_file}")
from lyrics_converter import LyricsConverter
txt_converter = LyricsConverter(output_format="txt", filepath=lrc_file)
converted_txt = txt_converter.convert_file()
# Write TXT file
if output_txt_path is None:
# Default to same name as ZIP but with .txt extension
output_txt_path = output_zip_path.replace(".zip", ".txt")
with open(output_txt_path, "w") as txt_file:
txt_file.write(converted_txt)
self.logger.info(f"TXT file written: {output_txt_path}")
# Create ZIP containing MP3 and TXT
self.logger.info(f"Creating TXT ZIP: {output_zip_path}")
self._create_zip_from_files(
output_zip_path,
[(mp3_file, os.path.basename(mp3_file)),
(output_txt_path, os.path.basename(output_txt_path))]
)
if not os.path.isfile(output_zip_path):
raise Exception(f"Failed to create TXT ZIP file: {output_zip_path}")
self.logger.info(f"TXT package created: {output_zip_path}")
return output_zip_path, output_txt_path
def _create_zip_from_files(
self,
zip_path: str,
files: list,
) -> None:
"""
Create a ZIP file from a list of files.
Args:
zip_path: Path for the output ZIP file
files: List of (file_path, archive_name) tuples
"""
with zipfile.ZipFile(zip_path, "w") as zipf:
for file_path, archive_name in files:
if os.path.isfile(file_path):
zipf.write(file_path, archive_name)
self.logger.debug(f"Added to ZIP: {archive_name}")
else:
self.logger.warning(f"File not found for ZIP: {file_path}")
# Singleton instance and factory function (following existing service pattern)
_packaging_service: Optional[PackagingService] = None
def get_packaging_service(
cdg_styles: Optional[Dict[str, Any]] = None,
**kwargs
) -> PackagingService:
"""
Get a packaging service instance.
Args:
cdg_styles: CDG style configuration
**kwargs: Additional arguments passed to PackagingService
Returns:
PackagingService instance
"""
global _packaging_service
# Create new instance if settings changed
if _packaging_service is None or cdg_styles:
_packaging_service = PackagingService(
cdg_styles=cdg_styles,
**kwargs
)
return _packaging_service
+445
View File
@@ -0,0 +1,445 @@
"""
YouTube Upload Service.
Provides video upload functionality for YouTube, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- Video uploads with metadata (title, description, tags)
- Thumbnail uploads
- Duplicate video detection and replacement
- Authentication via pre-stored credentials or client secrets file
"""
import logging
import os
from typing import Optional, Dict, Any, Tuple
from thefuzz import fuzz
logger = logging.getLogger(__name__)
class YouTubeUploadService:
"""
Service for uploading videos to YouTube.
Supports two authentication modes:
1. Pre-stored credentials (for server-side/non-interactive use)
2. Client secrets file (for interactive CLI use with browser OAuth)
"""
YOUTUBE_URL_PREFIX = "https://www.youtube.com/watch?v="
SCOPES = ["https://www.googleapis.com/auth/youtube"]
def __init__(
self,
credentials: Optional[Dict[str, Any]] = None,
client_secrets_file: Optional[str] = None,
non_interactive: bool = False,
server_side_mode: bool = False,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the YouTube upload service.
Args:
credentials: Pre-stored OAuth credentials dict with keys:
- token: Access token
- refresh_token: Refresh token
- token_uri: Token endpoint URL
- client_id: OAuth client ID
- client_secret: OAuth client secret
- scopes: List of OAuth scopes
client_secrets_file: Path to OAuth client secrets JSON file
(for interactive authentication)
non_interactive: If True, skip interactive prompts
server_side_mode: If True, use exact matching for duplicate detection
dry_run: If True, log actions without performing them
logger: Optional logger instance
"""
self.credentials = credentials
self.client_secrets_file = client_secrets_file
self.non_interactive = non_interactive
self.server_side_mode = server_side_mode
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
self._youtube_service = None
self._channel_id = None
def _authenticate(self):
"""
Authenticate with YouTube API and return service object.
Returns:
YouTube API service object
Raises:
Exception: If authentication fails or is required in non-interactive mode
"""
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
import pickle
# Check if we have pre-stored credentials (for non-interactive mode)
if self.credentials and self.non_interactive:
try:
creds = Credentials(
token=self.credentials.get('token'),
refresh_token=self.credentials.get('refresh_token'),
token_uri=self.credentials.get('token_uri'),
client_id=self.credentials.get('client_id'),
client_secret=self.credentials.get('client_secret'),
scopes=self.credentials.get('scopes', self.SCOPES)
)
# Refresh token if needed
if creds.expired and creds.refresh_token:
creds.refresh(Request())
youtube = build('youtube', 'v3', credentials=creds)
self.logger.info("Successfully authenticated with YouTube using pre-stored credentials")
return youtube
except Exception as e:
self.logger.error(f"Failed to authenticate with pre-stored credentials: {str(e)}")
# Fall through to original authentication if pre-stored credentials fail
# For non-interactive mode without pre-stored credentials, raise error
if self.non_interactive:
raise Exception(
"YouTube authentication required but running in non-interactive mode. "
"Please pre-authenticate or disable YouTube upload."
)
# Interactive authentication using client secrets file
if not self.client_secrets_file:
raise Exception("No YouTube credentials or client secrets file provided")
# Token file stores the user's access and refresh tokens
youtube_token_file = "/tmp/karaoke-finalise-youtube-token.pickle"
creds = None
# Check if we have saved credentials
if os.path.exists(youtube_token_file):
with open(youtube_token_file, "rb") as token:
creds = pickle.load(token)
# If there are no valid credentials, let the user log in
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
self.client_secrets_file, scopes=self.SCOPES
)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open(youtube_token_file, "wb") as token:
pickle.dump(creds, token)
return build("youtube", "v3", credentials=creds)
@property
def youtube_service(self):
"""Lazy-load YouTube service on first access."""
if self._youtube_service is None:
self._youtube_service = self._authenticate()
return self._youtube_service
def get_channel_id(self) -> Optional[str]:
"""
Get the authenticated user's YouTube channel ID.
Returns:
Channel ID string, or None if not found
"""
if self._channel_id:
return self._channel_id
request = self.youtube_service.channels().list(part="snippet", mine=True)
response = request.execute()
if "items" in response and len(response["items"]) > 0:
self._channel_id = response["items"][0]["id"]
return self._channel_id
return None
def check_duplicate(
self,
title: str,
exact_match: Optional[bool] = None
) -> Tuple[bool, Optional[str], Optional[str]]:
"""
Check if a video with the given title already exists on the channel.
Args:
title: Video title to search for
exact_match: If True, require exact title match. If None, uses
server_side_mode setting (exact in server mode, fuzzy in CLI)
Returns:
Tuple of (exists: bool, video_id: Optional[str], video_url: Optional[str])
"""
channel_id = self.get_channel_id()
if not channel_id:
self.logger.warning("Could not get channel ID, skipping duplicate check")
return False, None, None
use_exact_match = exact_match if exact_match is not None else self.server_side_mode
self.logger.info(f"Searching YouTube channel {channel_id} for title: {title}")
request = self.youtube_service.search().list(
part="snippet",
channelId=channel_id,
q=title,
type="video",
maxResults=10
)
response = request.execute()
if "items" not in response or len(response["items"]) == 0:
self.logger.info(f"No matching video found with title: {title}")
return False, None, None
for item in response["items"]:
# Verify the video actually belongs to our channel
result_channel_id = item["snippet"]["channelId"]
if result_channel_id != channel_id:
self.logger.debug(
f"Skipping video from different channel: {item['snippet']['title']} "
f"(channel: {result_channel_id})"
)
continue
found_title = item["snippet"]["title"]
# Determine if this is a match
if use_exact_match:
is_match = title.lower() == found_title.lower()
similarity_score = 100 if is_match else 0
else:
similarity_score = fuzz.ratio(title.lower(), found_title.lower())
is_match = similarity_score >= 70
if is_match:
video_id = item["id"]["videoId"]
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
self.logger.info(
f"Potential match found on YouTube channel with ID: {video_id} "
f"and title: {found_title} (similarity: {similarity_score}%)"
)
# In non-interactive mode, return the match directly
if self.non_interactive:
self.logger.info("Non-interactive mode, found a match.")
return True, video_id, video_url
# Interactive confirmation
confirmation = input(
f"Is '{found_title}' the video you are finalising? (y/n): "
).strip().lower()
if confirmation == "y":
return True, video_id, video_url
self.logger.info(f"No matching video found with title: {title}")
return False, None, None
def delete_video(self, video_id: str) -> bool:
"""
Delete a YouTube video by its ID.
Args:
video_id: The YouTube video ID to delete
Returns:
True if successful, False otherwise
"""
self.logger.info(f"Deleting YouTube video with ID: {video_id}")
if self.dry_run:
self.logger.info(f"DRY RUN: Would delete YouTube video with ID: {video_id}")
return True
try:
self.youtube_service.videos().delete(id=video_id).execute()
self.logger.info(f"Successfully deleted YouTube video with ID: {video_id}")
return True
except Exception as e:
self.logger.error(f"Failed to delete YouTube video with ID {video_id}: {e}")
return False
@staticmethod
def truncate_title(title: str, max_length: int = 95) -> str:
"""
Truncate title to the nearest whole word within max_length.
Args:
title: Title to truncate
max_length: Maximum length (default 95 for YouTube)
Returns:
Truncated title with "..." if needed
"""
if len(title) <= max_length:
return title
truncated_title = title[:max_length].rsplit(" ", 1)[0]
if len(truncated_title) < max_length:
truncated_title += " ..."
return truncated_title
def upload_video(
self,
video_path: str,
title: str,
description: str,
thumbnail_path: Optional[str] = None,
tags: Optional[list] = None,
category_id: str = "10", # Music category
privacy_status: str = "public",
replace_existing: bool = False,
) -> Tuple[Optional[str], Optional[str]]:
"""
Upload a video to YouTube with metadata and optional thumbnail.
Args:
video_path: Path to the video file to upload
title: Video title (will be truncated to 95 chars if needed)
description: Video description
thumbnail_path: Optional path to thumbnail image
tags: Optional list of tags/keywords
category_id: YouTube category ID (default "10" for Music)
privacy_status: "public", "private", or "unlisted"
replace_existing: If True, delete existing video with same title
Returns:
Tuple of (video_id, video_url) or (None, None) if upload failed/skipped
"""
from googleapiclient.http import MediaFileUpload
# Truncate title if needed
youtube_title = self.truncate_title(title)
self.logger.info(f"Uploading video to YouTube: {youtube_title}")
if self.dry_run:
self.logger.info(
f"DRY RUN: Would upload {video_path} to YouTube with title: {youtube_title}"
)
return "dry_run_video_id", f"{self.YOUTUBE_URL_PREFIX}dry_run_video_id"
# Check for existing video
should_replace = True if self.server_side_mode else replace_existing
exists, existing_id, existing_url = self.check_duplicate(youtube_title)
if exists:
if should_replace:
self.logger.info(f"Video already exists on YouTube, deleting before re-upload: {existing_url}")
if self.delete_video(existing_id):
self.logger.info("Successfully deleted existing video, proceeding with upload")
else:
self.logger.error("Failed to delete existing video, aborting upload")
return None, None
else:
self.logger.warning(f"Video already exists on YouTube, skipping upload: {existing_url}")
return existing_id, existing_url
# Prepare video metadata
body = {
"snippet": {
"title": youtube_title,
"description": description,
"tags": tags or [],
"categoryId": category_id,
},
"status": {"privacyStatus": privacy_status},
}
# Determine MIME type based on file extension
ext = os.path.splitext(video_path)[1].lower()
mime_type = {
".mkv": "video/x-matroska",
".mp4": "video/mp4",
".mov": "video/quicktime",
".avi": "video/x-msvideo",
}.get(ext, "video/*")
# Upload video
self.logger.info(f"Authenticating with YouTube...")
media_file = MediaFileUpload(video_path, mimetype=mime_type, resumable=True)
self.logger.info(f"Uploading video to YouTube...")
request = self.youtube_service.videos().insert(
part="snippet,status",
body=body,
media_body=media_file
)
response = request.execute()
video_id = response.get("id")
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
self.logger.info(f"Uploaded video to YouTube: {video_url}")
# Upload thumbnail if provided
if thumbnail_path and os.path.isfile(thumbnail_path):
try:
self.logger.info(f"Uploading thumbnail from: {thumbnail_path}")
media_thumbnail = MediaFileUpload(thumbnail_path, mimetype="image/jpeg")
self.youtube_service.thumbnails().set(
videoId=video_id,
media_body=media_thumbnail
).execute()
self.logger.info(f"Uploaded thumbnail for video ID {video_id}")
except Exception as e:
self.logger.error(f"Failed to upload thumbnail: {e}")
self.logger.warning(
"Video uploaded but thumbnail not set. "
"You may need to set it manually on YouTube."
)
elif thumbnail_path:
self.logger.warning(f"Thumbnail file not found, skipping: {thumbnail_path}")
return video_id, video_url
# Singleton instance and factory function (following existing service pattern)
_youtube_upload_service: Optional[YouTubeUploadService] = None
def get_youtube_upload_service(
credentials: Optional[Dict[str, Any]] = None,
client_secrets_file: Optional[str] = None,
**kwargs
) -> YouTubeUploadService:
"""
Get a YouTube upload service instance.
For server-side use, pass credentials from YouTubeService.
For CLI use, pass client_secrets_file.
Args:
credentials: Pre-stored OAuth credentials dict
client_secrets_file: Path to OAuth client secrets JSON file
**kwargs: Additional arguments passed to YouTubeUploadService
Returns:
YouTubeUploadService instance
"""
global _youtube_upload_service
# Create new instance if credentials/settings changed
if _youtube_upload_service is None or credentials or client_secrets_file:
_youtube_upload_service = YouTubeUploadService(
credentials=credentials,
client_secrets_file=client_secrets_file,
**kwargs
)
return _youtube_upload_service
+15 -9
View File
@@ -6,23 +6,29 @@ Provides common mocks and test utilities across all test modules.
NOTE: Module-level mocks are NOT applied here to allow emulator tests to work.
Individual tests must mock dependencies as needed.
"""
# IMPORTANT: Set environment variables FIRST, before ANY imports that might
# trigger backend module loading and Settings singleton creation.
import os
import pytest
from unittest.mock import Mock, MagicMock, AsyncMock, patch
from datetime import datetime, UTC
from fastapi.testclient import TestClient
# Mock google.auth.default AND firestore for unit tests if not using emulator
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
# Set up test environment variables BEFORE importing any backend modules
# This must happen before Settings is instantiated, which occurs on first import
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
from unittest.mock import MagicMock
# Set up test environment variables BEFORE importing any backend modules
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
os.environ.setdefault('ENVIRONMENT', 'test')
import pytest
from unittest.mock import Mock, MagicMock, AsyncMock, patch
from datetime import datetime, UTC
from fastapi.testclient import TestClient
# Mock google.auth.default AND firestore for unit tests if not using emulator
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
from unittest.mock import MagicMock
# Mock google.auth.default - prevents credential errors
try:
@@ -0,0 +1,443 @@
"""
Integration tests for worker log subcollection storage.
Tests the subcollection approach using real Firestore emulator to verify:
- Logs are stored in subcollection (jobs/{job_id}/logs)
- TTL expiry field is set correctly
- Large log volumes work without 1MB limit issues
- Concurrent writes are handled correctly
- Logs can be queried efficiently
Run with: ./scripts/run-emulator-tests.sh
"""
import pytest
import time
import requests
import os
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone, timedelta
def emulators_running() -> bool:
"""Check if GCP emulators are running."""
try:
requests.get("http://127.0.0.1:8080", timeout=1)
return True
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
return False
# Skip all tests in this module if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Set up environment for emulator
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
class TestWorkerLogsSubcollectionDirect:
"""Direct Firestore tests for worker logs subcollection."""
@pytest.fixture(autouse=True)
def setup_firestore(self):
"""Set up Firestore client for each test."""
from google.cloud import firestore
self.db = firestore.Client(project="test-project")
self.collection = "test-subcollection-jobs"
yield
def _create_test_job(self):
"""Create a test job document."""
job_id = f"test-sub-{int(time.time() * 1000)}"
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.set({
"job_id": job_id,
"status": "pending",
"created_at": datetime.now(timezone.utc),
"worker_logs": [] # Legacy field - kept empty for new jobs
})
return job_id
def _append_log_to_subcollection(self, job_id: str, worker: str, message: str, ttl_days: int = 30):
"""Add log to subcollection at jobs/{job_id}/logs."""
import uuid
log_id = str(uuid.uuid4())
now = datetime.now(timezone.utc)
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
logs_ref.document(log_id).set({
"id": log_id,
"job_id": job_id,
"timestamp": now,
"level": "INFO",
"worker": worker,
"message": message,
"ttl_expiry": now + timedelta(days=ttl_days)
})
return log_id
def _get_logs_from_subcollection(self, job_id: str, worker: str = None, limit: int = 500):
"""Get logs from subcollection."""
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
query = logs_ref.order_by("timestamp")
if worker:
from google.cloud.firestore_v1 import FieldFilter
query = query.where(filter=FieldFilter("worker", "==", worker))
query = query.limit(limit)
return [doc.to_dict() for doc in query.stream()]
def _count_logs_in_subcollection(self, job_id: str) -> int:
"""Count logs in subcollection."""
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
count_query = logs_ref.count()
result = count_query.get()
if result and len(result) > 0:
return result[0][0].value
return 0
def _delete_logs_subcollection(self, job_id: str, batch_size: int = 100) -> int:
"""Delete all logs in subcollection."""
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
deleted_count = 0
while True:
docs = list(logs_ref.limit(batch_size).stream())
if not docs:
break
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_count += 1
batch.commit()
return deleted_count
def test_subcollection_single_write(self):
"""Test writing single log to subcollection."""
job_id = self._create_test_job()
log_id = self._append_log_to_subcollection(job_id, "test", "Single log message")
time.sleep(0.1)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == 1
assert logs[0]["message"] == "Single log message"
assert logs[0]["worker"] == "test"
assert logs[0]["id"] == log_id
print("Single write to subcollection works")
def test_subcollection_ttl_field_is_set(self):
"""Test TTL expiry field is set correctly."""
job_id = self._create_test_job()
self._append_log_to_subcollection(job_id, "test", "Log with TTL", ttl_days=7)
time.sleep(0.1)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == 1
ttl_expiry = logs[0]["ttl_expiry"]
assert ttl_expiry is not None
# TTL should be ~7 days from now
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
if hasattr(ttl_expiry, 'timestamp'):
# Firestore datetime object
diff = abs((ttl_expiry.replace(tzinfo=timezone.utc) - expected_ttl).total_seconds())
else:
diff = 0 # If already datetime
assert diff < 60, f"TTL expiry should be ~7 days from now, diff was {diff}s"
print("TTL field is set correctly")
def test_subcollection_sequential_writes(self):
"""Test multiple sequential writes to subcollection."""
job_id = self._create_test_job()
for i in range(20):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.2)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == 20, f"Expected 20 logs, got {len(logs)}"
print(f"Sequential writes: {len(logs)} logs created")
def test_subcollection_concurrent_writes(self):
"""Test concurrent writes to subcollection - no race conditions."""
job_id = self._create_test_job()
num_writes = 50
def write_log(index):
self._append_log_to_subcollection(job_id, "worker", f"Concurrent Log {index}")
with ThreadPoolExecutor(max_workers=10) as executor:
futures = [executor.submit(write_log, i) for i in range(num_writes)]
for future in as_completed(futures):
future.result()
time.sleep(0.5)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == num_writes, f"Expected {num_writes} logs, got {len(logs)}"
print(f"Concurrent writes: All {num_writes} logs preserved")
def test_subcollection_large_volume(self):
"""Test large volume of logs - demonstrates no 1MB limit."""
job_id = self._create_test_job()
num_logs = 500 # Would exceed 1MB in embedded array
# Write logs in batches
batch_size = 50
for batch_start in range(0, num_logs, batch_size):
with ThreadPoolExecutor(max_workers=10) as executor:
futures = []
for i in range(batch_start, min(batch_start + batch_size, num_logs)):
futures.append(executor.submit(
self._append_log_to_subcollection,
job_id,
"stress",
f"Large volume test log {i} - " + "x" * 500 # ~500 bytes per log
))
for future in as_completed(futures):
future.result()
time.sleep(1)
count = self._count_logs_in_subcollection(job_id)
assert count == num_logs, f"Expected {num_logs} logs, got {count}"
print(f"Large volume: {count} logs created (~{count * 500 / 1024:.1f}KB would exceed 1MB if embedded)")
def test_subcollection_filter_by_worker(self):
"""Test filtering logs by worker type."""
job_id = self._create_test_job()
# Add logs from different workers
for i in range(10):
self._append_log_to_subcollection(job_id, "audio", f"Audio log {i}")
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics log {i}")
self._append_log_to_subcollection(job_id, "video", f"Video log {i}")
time.sleep(0.3)
# Query by worker
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
all_logs = self._get_logs_from_subcollection(job_id)
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
assert len(all_logs) == 30, f"Expected 30 total logs, got {len(all_logs)}"
print("Worker filtering works correctly")
def test_subcollection_ordered_by_timestamp(self):
"""Test logs are returned in timestamp order."""
job_id = self._create_test_job()
# Add logs with slight delays to ensure different timestamps
for i in range(5):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.02)
logs = self._get_logs_from_subcollection(job_id)
# Verify order (timestamps should be ascending)
for i in range(len(logs) - 1):
assert logs[i]["timestamp"] <= logs[i + 1]["timestamp"], \
f"Logs not in order at index {i}"
print("Logs are ordered by timestamp")
def test_subcollection_delete_all_logs(self):
"""Test deleting all logs in subcollection."""
job_id = self._create_test_job()
# Add logs
for i in range(25):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.2)
count_before = self._count_logs_in_subcollection(job_id)
assert count_before == 25
# Delete all logs
deleted = self._delete_logs_subcollection(job_id)
time.sleep(0.2)
count_after = self._count_logs_in_subcollection(job_id)
assert deleted == 25, f"Expected to delete 25 logs, deleted {deleted}"
assert count_after == 0, f"Expected 0 logs after delete, got {count_after}"
print(f"Deleted {deleted} logs from subcollection")
def test_subcollection_concurrent_workers_interleaved(self):
"""Test simulating audio and lyrics workers writing concurrently."""
job_id = self._create_test_job()
logs_per_worker = 20
def audio_worker():
for i in range(logs_per_worker):
self._append_log_to_subcollection(job_id, "audio", f"Audio processing step {i}")
time.sleep(0.005)
def lyrics_worker():
for i in range(logs_per_worker):
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics processing step {i}")
time.sleep(0.005)
def video_worker():
for i in range(logs_per_worker):
self._append_log_to_subcollection(job_id, "video", f"Video encoding step {i}")
time.sleep(0.005)
# Start all workers
threads = [
threading.Thread(target=audio_worker),
threading.Thread(target=lyrics_worker),
threading.Thread(target=video_worker)
]
for t in threads:
t.start()
for t in threads:
t.join()
time.sleep(0.5)
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
video_logs = self._get_logs_from_subcollection(job_id, worker="video")
total = self._count_logs_in_subcollection(job_id)
assert len(audio_logs) == logs_per_worker
assert len(lyrics_logs) == logs_per_worker
assert len(video_logs) == logs_per_worker
assert total == logs_per_worker * 3
print(f"Concurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics + {len(video_logs)} video = {total}")
def test_subcollection_job_deletion_cleans_up_logs(self):
"""Test that deleting job document doesn't orphan logs."""
job_id = self._create_test_job()
# Add logs
for i in range(10):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.2)
assert self._count_logs_in_subcollection(job_id) == 10
# Delete logs first (must happen before parent doc deletion in real code)
deleted = self._delete_logs_subcollection(job_id)
assert deleted == 10
# Now delete job document
self.db.collection(self.collection).document(job_id).delete()
# Verify subcollection is empty (Firestore doesn't cascade delete subcollections)
time.sleep(0.2)
assert self._count_logs_in_subcollection(job_id) == 0
print("Job deletion with log cleanup works correctly")
class TestSubcollectionVsEmbeddedArray:
"""Compare subcollection approach with embedded array."""
@pytest.fixture(autouse=True)
def setup_firestore(self):
"""Set up Firestore client for each test."""
from google.cloud import firestore
self.db = firestore.Client(project="test-project")
self.collection = "test-comparison-jobs"
yield
def _create_test_job(self, use_array: bool):
"""Create test job."""
job_id = f"test-cmp-{int(time.time() * 1000)}-{'arr' if use_array else 'sub'}"
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.set({
"job_id": job_id,
"status": "pending",
"created_at": datetime.now(timezone.utc),
"worker_logs": []
})
return job_id
def _append_log_array(self, job_id: str, message: str):
"""Add log to embedded array."""
from google.cloud import firestore
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.update({
"worker_logs": firestore.ArrayUnion([{
"timestamp": datetime.now(timezone.utc).isoformat() + "Z",
"level": "INFO",
"worker": "test",
"message": message
}])
})
def _append_log_subcollection(self, job_id: str, message: str):
"""Add log to subcollection."""
import uuid
now = datetime.now(timezone.utc)
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
logs_ref.document(str(uuid.uuid4())).set({
"timestamp": now,
"level": "INFO",
"worker": "test",
"message": message,
"ttl_expiry": now + timedelta(days=30)
})
def test_document_size_comparison(self):
"""Compare document sizes between approaches."""
array_job_id = self._create_test_job(use_array=True)
sub_job_id = self._create_test_job(use_array=False)
num_logs = 100
large_message = "x" * 5000 # 5KB per log
# Add logs to both
print("\nAdding logs to both approaches...")
for i in range(num_logs):
self._append_log_array(array_job_id, f"Array log {i}: {large_message}")
self._append_log_subcollection(sub_job_id, f"Sub log {i}: {large_message}")
time.sleep(0.5)
# Get document sizes (approximate via raw data)
array_doc = self.db.collection(self.collection).document(array_job_id).get()
sub_doc = self.db.collection(self.collection).document(sub_job_id).get()
array_data = array_doc.to_dict()
sub_data = sub_doc.to_dict()
array_logs = array_data.get("worker_logs", [])
sub_logs = sub_data.get("worker_logs", [])
print(f"\nEmbedded array: {len(array_logs)} logs in document")
print(f"Subcollection: {len(sub_logs)} logs in document (0 expected)")
# Embedded array should have all logs in document
assert len(array_logs) == num_logs
# Subcollection should have empty array in main document
assert len(sub_logs) == 0
# Subcollection logs are in subcollection
sub_count = self.db.collection(self.collection).document(sub_job_id).collection("logs").count().get()[0][0].value
assert sub_count == num_logs
print(f"Subcollection logs stored separately: {sub_count}")
print("Subcollection approach keeps main document small")
print("Worker logs subcollection integration tests ready")
+244
View File
@@ -0,0 +1,244 @@
"""
Tests for DiscordNotificationService.
Tests cover:
- Service initialization
- Webhook URL validation
- Message posting
- Video notification posting
- Dry run mode
"""
import pytest
from unittest.mock import MagicMock, patch
from backend.services.discord_service import (
DiscordNotificationService,
get_discord_notification_service,
)
class TestDiscordNotificationServiceInit:
"""Test service initialization."""
def test_init_with_webhook_url(self):
"""Test initialization with valid webhook URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef123456"
service = DiscordNotificationService(webhook_url=url)
assert service.webhook_url == url
assert service.is_enabled() is True
def test_init_without_webhook_url(self):
"""Test initialization without webhook URL."""
service = DiscordNotificationService()
assert service.webhook_url is None
assert service.is_enabled() is False
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = DiscordNotificationService(dry_run=True)
assert service.dry_run is True
def test_init_with_invalid_webhook_url_raises(self):
"""Test that invalid webhook URL raises ValueError."""
with pytest.raises(ValueError) as exc_info:
DiscordNotificationService(webhook_url="https://example.com/webhook")
assert "Invalid Discord webhook URL" in str(exc_info.value)
def test_init_with_non_discord_url_raises(self):
"""Test that non-Discord URL raises ValueError."""
with pytest.raises(ValueError):
DiscordNotificationService(
webhook_url="https://slack.com/api/webhooks/123"
)
class TestDiscordNotificationServiceValidation:
"""Test webhook URL validation."""
def test_validate_valid_webhook_url(self):
"""Test that valid webhook URL passes validation."""
service = DiscordNotificationService()
# Should not raise
service._validate_webhook_url(
"https://discord.com/api/webhooks/123456789/abcdef"
)
def test_validate_invalid_webhook_url(self):
"""Test that invalid webhook URL raises ValueError."""
service = DiscordNotificationService()
with pytest.raises(ValueError) as exc_info:
service._validate_webhook_url("https://example.com/webhook")
assert "Invalid Discord webhook URL" in str(exc_info.value)
def test_validate_strips_whitespace(self):
"""Test that whitespace is stripped from URL."""
service = DiscordNotificationService()
# Should not raise - whitespace is stripped
service._validate_webhook_url(
" https://discord.com/api/webhooks/123456789/abcdef "
)
class TestDiscordNotificationServicePostMessage:
"""Test message posting."""
@patch("backend.services.discord_service.requests.post")
def test_post_message_success(self, mock_post):
"""Test successful message posting."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_message("Test message")
assert result is True
mock_post.assert_called_once_with(
url,
json={"content": "Test message"},
timeout=30
)
@patch("backend.services.discord_service.requests.post")
def test_post_message_with_custom_webhook(self, mock_post):
"""Test message posting with custom webhook URL."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
service = DiscordNotificationService()
custom_url = "https://discord.com/api/webhooks/987654321/fedcba"
result = service.post_message("Test message", webhook_url=custom_url)
assert result is True
mock_post.assert_called_once_with(
custom_url,
json={"content": "Test message"},
timeout=30
)
def test_post_message_no_webhook_raises(self):
"""Test that posting without webhook URL raises ValueError."""
service = DiscordNotificationService()
with pytest.raises(ValueError) as exc_info:
service.post_message("Test message")
assert "No Discord webhook URL provided" in str(exc_info.value)
def test_post_message_dry_run(self):
"""Test message posting in dry run mode."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url, dry_run=True)
result = service.post_message("Test message")
assert result is True
# In dry run mode, no actual request should be made
@patch("backend.services.discord_service.requests.post")
def test_post_message_http_error(self, mock_post):
"""Test handling of HTTP errors."""
import requests as req
mock_response = MagicMock()
mock_response.raise_for_status.side_effect = req.HTTPError("404 Not Found")
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
with pytest.raises(req.HTTPError):
service.post_message("Test message")
class TestDiscordNotificationServiceVideoNotification:
"""Test video notification posting."""
@patch("backend.services.discord_service.requests.post")
def test_post_video_notification_success(self, mock_post):
"""Test successful video notification."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification(
"https://www.youtube.com/watch?v=abc123"
)
assert result is True
mock_post.assert_called_once()
call_args = mock_post.call_args
assert "New upload:" in call_args[1]["json"]["content"]
assert "abc123" in call_args[1]["json"]["content"]
def test_post_video_notification_no_youtube_url(self):
"""Test video notification with empty YouTube URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification("")
assert result is False
def test_post_video_notification_none_youtube_url(self):
"""Test video notification with None YouTube URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification(None)
assert result is False
def test_post_video_notification_dry_run(self):
"""Test video notification in dry run mode."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url, dry_run=True)
result = service.post_video_notification(
"https://www.youtube.com/watch?v=abc123"
)
assert result is True
class TestGetDiscordNotificationService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.discord_service as module
module._discord_notification_service = None
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = get_discord_notification_service(webhook_url=url)
assert service is not None
assert isinstance(service, DiscordNotificationService)
assert service.webhook_url == url
def test_get_service_without_webhook(self):
"""Test factory function without webhook URL."""
import backend.services.discord_service as module
module._discord_notification_service = None
service = get_discord_notification_service()
assert service is not None
assert service.webhook_url is None
assert service.is_enabled() is False
def test_get_service_with_dry_run(self):
"""Test factory function with dry run mode."""
import backend.services.discord_service as module
module._discord_notification_service = None
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = get_discord_notification_service(webhook_url=url, dry_run=True)
assert service.dry_run is True
+412
View File
@@ -0,0 +1,412 @@
"""
Tests for Encoding Interface.
Tests cover:
- EncodingInput and EncodingOutput dataclasses
- LocalEncodingBackend implementation
- Backend factory function
"""
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from backend.services.encoding_interface import (
EncodingInput,
EncodingOutput,
EncodingBackend,
LocalEncodingBackend,
GCEEncodingBackend,
get_encoding_backend,
)
class TestEncodingInput:
"""Test EncodingInput dataclass."""
def test_required_fields(self):
"""Test that required fields must be provided."""
input_config = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert input_config.title_video_path == "/path/title.mov"
assert input_config.end_video_path is None
def test_all_fields(self):
"""Test all fields including optional ones."""
input_config = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
end_video_path="/path/end.mov",
artist="Test Artist",
title="Test Title",
brand_code="NOMAD-1234",
output_dir="/output",
options={"quality": "high"}
)
assert input_config.artist == "Test Artist"
assert input_config.brand_code == "NOMAD-1234"
assert input_config.options["quality"] == "high"
class TestEncodingOutput:
"""Test EncodingOutput dataclass."""
def test_success_output(self):
"""Test successful output."""
output = EncodingOutput(
success=True,
lossless_4k_mp4_path="/output/video.mp4",
encoding_backend="local"
)
assert output.success is True
assert output.error_message is None
def test_failure_output(self):
"""Test failure output."""
output = EncodingOutput(
success=False,
error_message="Encoding failed",
encoding_backend="gce"
)
assert output.success is False
assert output.error_message == "Encoding failed"
def test_output_files_dict(self):
"""Test output_files dictionary."""
output = EncodingOutput(
success=True,
output_files={
"lossless_4k_mp4": "/output/lossless.mp4",
"720p_mp4": "/output/720p.mp4"
}
)
assert "lossless_4k_mp4" in output.output_files
assert output.output_files["720p_mp4"] == "/output/720p.mp4"
class TestLocalEncodingBackend:
"""Test LocalEncodingBackend implementation."""
def test_name(self):
"""Test backend name."""
backend = LocalEncodingBackend()
assert backend.name == "local"
def test_init_with_dry_run(self):
"""Test initialization with dry run."""
backend = LocalEncodingBackend(dry_run=True)
assert backend.dry_run is True
@pytest.mark.asyncio
@patch("subprocess.run")
async def test_is_available_success(self, mock_run):
"""Test availability check when FFmpeg is installed."""
mock_run.return_value = MagicMock(returncode=0)
backend = LocalEncodingBackend()
available = await backend.is_available()
assert available is True
@pytest.mark.asyncio
@patch("subprocess.run")
async def test_is_available_not_found(self, mock_run):
"""Test availability check when FFmpeg is not installed."""
mock_run.side_effect = FileNotFoundError()
backend = LocalEncodingBackend()
available = await backend.is_available()
assert available is False
@pytest.mark.asyncio
@patch.object(LocalEncodingBackend, "is_available")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_get_status(self, mock_get_service, mock_is_available):
"""Test status retrieval."""
mock_is_available.return_value = True
mock_service = MagicMock()
mock_service.hwaccel_available = True
mock_service.video_encoder = "h264_nvenc"
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
status = await backend.get_status()
assert status["backend"] == "local"
assert status["available"] is True
@pytest.mark.asyncio
@patch("asyncio.to_thread")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_encode_success(self, mock_get_service, mock_to_thread):
"""Test successful encoding."""
from backend.services.local_encoding_service import EncodingResult
mock_result = EncodingResult(
success=True,
output_files={"key": "/path/file.mp4"}
)
mock_to_thread.return_value = mock_result
mock_service = MagicMock()
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test Artist",
title="Test Title",
output_dir="/output"
)
output = await backend.encode(input_config)
assert output.success is True
assert output.encoding_backend == "local"
assert output.encoding_time_seconds is not None
@pytest.mark.asyncio
@patch("asyncio.to_thread")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_encode_failure(self, mock_get_service, mock_to_thread):
"""Test encoding failure."""
from backend.services.local_encoding_service import EncodingResult
mock_result = EncodingResult(
success=False,
output_files={},
error="FFmpeg failed"
)
mock_to_thread.return_value = mock_result
mock_service = MagicMock()
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test",
title="Test",
)
output = await backend.encode(input_config)
assert output.success is False
assert output.error_message == "FFmpeg failed"
class TestGCEEncodingBackend:
"""Test GCEEncodingBackend implementation."""
def test_name(self):
"""Test backend name."""
backend = GCEEncodingBackend()
assert backend.name == "gce"
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_is_available_enabled(self, mock_get_service):
"""Test availability when GCE is enabled."""
mock_service = MagicMock()
mock_service.is_enabled = True
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
available = await backend.is_available()
assert available is True
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_is_available_disabled(self, mock_get_service):
"""Test availability when GCE is disabled."""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
available = await backend.is_available()
assert available is False
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_missing_gcs_paths(self, mock_get_service):
"""Test encoding fails without GCS paths."""
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
# Missing GCS paths in options
)
output = await backend.encode(input_config)
assert output.success is False
assert "gcs_path" in output.error_message.lower()
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_success(self, mock_get_service):
"""Test successful GCE encoding."""
mock_service = MagicMock()
mock_service.encode_videos = AsyncMock(return_value={
"status": "complete",
"output_files": {
"mp4_4k_lossless": "gs://bucket/output/lossless.mp4",
"mp4_720p": "gs://bucket/output/720p.mp4",
}
})
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test Artist",
title="Test Title",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
output = await backend.encode(input_config)
assert output.success is True
assert output.encoding_backend == "gce"
mock_service.encode_videos.assert_called_once()
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_failure(self, mock_get_service):
"""Test GCE encoding failure handling."""
mock_service = MagicMock()
mock_service.encode_videos = AsyncMock(side_effect=Exception("GCE worker error"))
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
output = await backend.encode(input_config)
assert output.success is False
assert "GCE worker error" in output.error_message
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_handles_list_result(self, mock_get_service):
"""Test GCE encoding handles list response gracefully.
This would have caught: 'list' object has no attribute 'get' error
when GCE worker returns a list instead of a dict.
"""
mock_service = MagicMock()
# Simulate GCE worker returning a list instead of dict
mock_service.encode_videos = AsyncMock(return_value=[
{"output_files": {"mp4_4k_lossless": "gs://bucket/output/lossless.mp4"}}
])
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
# This should not raise an error
output = await backend.encode(input_config)
# Should still succeed by extracting from the list
assert output.success is True
assert output.lossless_4k_mp4_path == "gs://bucket/output/lossless.mp4"
class TestGetEncodingBackend:
"""Test encoding backend factory function."""
def test_get_local_backend(self):
"""Test getting local backend."""
backend = get_encoding_backend("local")
assert isinstance(backend, LocalEncodingBackend)
assert backend.name == "local"
@patch.object(GCEEncodingBackend, "_get_service")
def test_get_auto_backend_gce_disabled(self, mock_get_service):
"""Test getting auto backend falls back to local when GCE disabled."""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
backend = get_encoding_backend("auto")
assert isinstance(backend, LocalEncodingBackend)
def test_get_local_backend_with_options(self):
"""Test getting local backend with options."""
backend = get_encoding_backend("local", dry_run=True)
assert backend.dry_run is True
def test_get_gce_backend(self):
"""Test getting GCE backend."""
backend = get_encoding_backend("gce")
assert isinstance(backend, GCEEncodingBackend)
assert backend.name == "gce"
def test_get_unknown_backend_raises(self):
"""Test that unknown backend raises ValueError."""
with pytest.raises(ValueError) as exc_info:
get_encoding_backend("unknown")
assert "Unknown encoding backend type" in str(exc_info.value)
def test_get_gce_backend_with_options(self):
"""Test getting GCE backend with common options like dry_run.
This ensures all backends accept the same kwargs, preventing
errors when get_encoding_backend() passes **kwargs to different backends.
"""
# This would have caught: GCEEncodingBackend.__init__() got an unexpected keyword argument 'dry_run'
backend = get_encoding_backend("gce", dry_run=True)
assert isinstance(backend, GCEEncodingBackend)
assert backend.dry_run is True
@patch.object(GCEEncodingBackend, "_get_service")
def test_all_backends_accept_dry_run(self, mock_get_service):
"""Test that all backend types accept dry_run parameter.
This is an integration test to ensure the factory function
can pass dry_run to any backend without TypeError.
"""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
for backend_type in ["local", "gce", "auto"]:
# This should not raise TypeError
backend = get_encoding_backend(backend_type, dry_run=True)
assert backend is not None
+2 -1
View File
@@ -1128,7 +1128,8 @@ class TestFinaliseOnlyModels:
assert request.enable_txt is None
assert request.brand_prefix is None
assert request.keep_brand_code is None
assert request.enable_youtube_upload is False
# YouTube upload default is None (server applies default_enable_youtube_upload)
assert request.enable_youtube_upload is None
assert request.dropbox_path is None
assert request.gdrive_folder_id is None
@@ -0,0 +1,423 @@
"""
Tests for LocalEncodingService.
Tests cover:
- Service initialization
- Hardware acceleration detection
- FFmpeg command execution
- Individual encoding methods
- Full encoding pipeline
- Dry run mode
"""
import os
import tempfile
import pytest
from unittest.mock import MagicMock, patch, call
from backend.services.local_encoding_service import (
LocalEncodingService,
EncodingConfig,
EncodingResult,
get_local_encoding_service,
)
class TestLocalEncodingServiceInit:
"""Test service initialization."""
def test_init_default_values(self):
"""Test default initialization."""
service = LocalEncodingService()
assert service.dry_run is False
assert "ffmpeg" in service._ffmpeg_base_command
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = LocalEncodingService(dry_run=True)
assert service.dry_run is True
def test_init_with_debug_logging(self):
"""Test initialization with debug logging level."""
import logging
service = LocalEncodingService(log_level=logging.DEBUG)
assert "-loglevel verbose" in service._ffmpeg_base_command
def test_init_with_info_logging(self):
"""Test initialization with info logging level."""
import logging
service = LocalEncodingService(log_level=logging.INFO)
assert "-loglevel fatal" in service._ffmpeg_base_command
class TestLocalEncodingServiceHWAccel:
"""Test hardware acceleration detection."""
@patch("subprocess.run")
def test_detect_nvenc_available(self, mock_run):
"""Test NVENC detection when available."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalEncodingService()
# Force detection
service._detect_and_set_hwaccel()
assert service._hwaccel_available is True
assert service._video_encoder == "h264_nvenc"
assert service._scale_filter == "scale_cuda"
@patch("subprocess.run")
def test_detect_no_hwaccel(self, mock_run):
"""Test fallback when no hardware acceleration available."""
import subprocess
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
service = LocalEncodingService()
# Force detection
service._detect_and_set_hwaccel()
assert service._hwaccel_available is False
assert service._video_encoder == "libx264"
assert service._scale_filter == "scale"
def test_nvenc_quality_settings(self):
"""Test NVENC quality settings for different presets."""
service = LocalEncodingService()
service._hwaccel_available = True
lossless = service._get_nvenc_quality_settings("lossless")
assert "cq 0" in lossless
medium = service._get_nvenc_quality_settings("medium")
assert "p4" in medium
def test_nvenc_quality_settings_disabled(self):
"""Test NVENC quality settings when hwaccel is disabled."""
service = LocalEncodingService()
service._hwaccel_available = False
settings = service._get_nvenc_quality_settings("lossless")
assert settings == ""
class TestLocalEncodingServiceExecuteCommand:
"""Test command execution."""
@patch("subprocess.run")
def test_execute_command_success(self, mock_run):
"""Test successful command execution."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command")
assert result is True
mock_run.assert_called_once()
@patch("subprocess.run")
def test_execute_command_failure(self, mock_run):
"""Test command execution failure."""
import subprocess
mock_run.side_effect = subprocess.CalledProcessError(
1, "test", stderr="Error message"
)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command")
assert result is False
@patch("subprocess.run")
def test_execute_command_timeout(self, mock_run):
"""Test command execution timeout."""
import subprocess
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command", timeout=30)
assert result is False
def test_execute_command_dry_run(self):
"""Test command execution in dry run mode."""
service = LocalEncodingService(dry_run=True)
result = service._execute_command("echo test", "Test command")
assert result is True
# No actual subprocess should be called in dry run mode
class TestLocalEncodingServiceEncodingMethods:
"""Test individual encoding methods."""
@patch.object(LocalEncodingService, "_execute_command")
def test_remux_with_instrumental(self, mock_execute):
"""Test remuxing with instrumental audio."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.remux_with_instrumental(
"/input/video.mov",
"/input/audio.flac",
"/output/video.mp4"
)
assert result is True
mock_execute.assert_called_once()
call_args = mock_execute.call_args[0][0]
assert "/input/video.mov" in call_args
assert "/input/audio.flac" in call_args
assert "-map 0:v -map 1:a" in call_args
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_convert_mov_to_mp4(self, mock_execute):
"""Test MOV to MP4 conversion."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.convert_mov_to_mp4(
"/input/video.mov",
"/output/video.mp4"
)
assert result is True
mock_execute.assert_called_once()
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_lossless_mp4_without_end(self, mock_execute):
"""Test lossless 4K MP4 encoding without end credits."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossless_mp4(
"/input/title.mov",
"/input/karaoke.mp4",
"/output/lossless.mp4"
)
assert result is True
mock_execute.assert_called_once()
call_args = mock_execute.call_args
assert "concat=n=2" in call_args[0][0] # GPU command
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_lossless_mp4_with_end(self, mock_execute):
"""Test lossless 4K MP4 encoding with end credits."""
mock_execute.return_value = True
with tempfile.TemporaryDirectory() as tmpdir:
end_file = os.path.join(tmpdir, "end.mov")
with open(end_file, "w") as f:
f.write("fake video")
service = LocalEncodingService()
result = service.encode_lossless_mp4(
"/input/title.mov",
"/input/karaoke.mp4",
"/output/lossless.mp4",
end_video=end_file
)
assert result is True
call_args = mock_execute.call_args
assert "concat=n=3" in call_args[0][0] # 3 videos
@patch.object(LocalEncodingService, "_execute_command")
def test_encode_lossy_mp4(self, mock_execute):
"""Test lossy 4K MP4 encoding."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossy_mp4(
"/input/lossless.mp4",
"/output/lossy.mp4"
)
assert result is True
call_args = mock_execute.call_args[0][0]
assert "-c:v copy" in call_args
assert "aac" in call_args.lower()
@patch.object(LocalEncodingService, "_execute_command")
def test_encode_lossless_mkv(self, mock_execute):
"""Test MKV encoding with FLAC audio."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossless_mkv(
"/input/lossless.mp4",
"/output/video.mkv"
)
assert result is True
call_args = mock_execute.call_args[0][0]
assert "-c:v copy" in call_args
assert "-c:a flac" in call_args
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_720p(self, mock_execute):
"""Test 720p encoding."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_720p(
"/input/lossless.mp4",
"/output/720p.mp4"
)
assert result is True
call_args = mock_execute.call_args
# Should contain scale filter
assert "1280:720" in call_args[0][0] or "1280:720" in call_args[0][1]
class TestLocalEncodingServiceFullPipeline:
"""Test full encoding pipeline."""
@patch.object(LocalEncodingService, "remux_with_instrumental")
@patch.object(LocalEncodingService, "convert_mov_to_mp4")
@patch.object(LocalEncodingService, "encode_lossless_mp4")
@patch.object(LocalEncodingService, "encode_lossy_mp4")
@patch.object(LocalEncodingService, "encode_lossless_mkv")
@patch.object(LocalEncodingService, "encode_720p")
def test_encode_all_formats_success(
self, mock_720p, mock_mkv, mock_lossy, mock_lossless, mock_convert, mock_remux
):
"""Test successful full encoding pipeline."""
mock_remux.return_value = True
mock_convert.return_value = True
mock_lossless.return_value = True
mock_lossy.return_value = True
mock_mkv.return_value = True
mock_720p.return_value = True
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_with_vocals_mp4="/output/with_vocals.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
output_lossy_4k_mp4="/output/lossy_4k.mp4",
output_lossless_mkv="/output/lossless.mkv",
output_720p_mp4="/output/720p.mp4",
)
result = service.encode_all_formats(config)
assert result.success is True
assert "karaoke_mp4" in result.output_files
assert "lossless_4k_mp4" in result.output_files
assert "720p_mp4" in result.output_files
@patch.object(LocalEncodingService, "remux_with_instrumental")
def test_encode_all_formats_failure_early(self, mock_remux):
"""Test encoding pipeline failure at early step."""
mock_remux.return_value = False
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
)
result = service.encode_all_formats(config)
assert result.success is False
assert "Failed to remux" in result.error
def test_encode_all_formats_dry_run(self):
"""Test encoding pipeline in dry run mode."""
service = LocalEncodingService(dry_run=True)
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mp4", # Already MP4
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
output_lossy_4k_mp4="/output/lossy_4k.mp4",
output_lossless_mkv="/output/lossless.mkv",
output_720p_mp4="/output/720p.mp4",
)
result = service.encode_all_formats(config)
# In dry run mode, all operations should "succeed"
assert result.success is True
class TestEncodingConfig:
"""Test EncodingConfig dataclass."""
def test_config_required_fields(self):
"""Test that required fields must be provided."""
config = EncodingConfig(
title_video="/path/title.mov",
karaoke_video="/path/karaoke.mov",
instrumental_audio="/path/audio.flac",
)
assert config.title_video == "/path/title.mov"
assert config.end_video is None # Optional field
def test_config_all_fields(self):
"""Test config with all fields."""
config = EncodingConfig(
title_video="/path/title.mov",
karaoke_video="/path/karaoke.mov",
instrumental_audio="/path/audio.flac",
end_video="/path/end.mov",
output_karaoke_mp4="/output/karaoke.mp4",
output_720p_mp4="/output/720p.mp4",
)
assert config.end_video == "/path/end.mov"
assert config.output_karaoke_mp4 == "/output/karaoke.mp4"
class TestEncodingResult:
"""Test EncodingResult dataclass."""
def test_result_success(self):
"""Test successful result."""
result = EncodingResult(
success=True,
output_files={"key": "/path/file.mp4"}
)
assert result.success is True
assert result.error is None
def test_result_failure(self):
"""Test failure result."""
result = EncodingResult(
success=False,
output_files={},
error="Something went wrong"
)
assert result.success is False
assert result.error == "Something went wrong"
class TestGetLocalEncodingService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.local_encoding_service as module
module._local_encoding_service = None
service = get_local_encoding_service()
assert service is not None
assert isinstance(service, LocalEncodingService)
def test_get_service_with_dry_run(self):
"""Test factory function with dry run option."""
import backend.services.local_encoding_service as module
module._local_encoding_service = None
service = get_local_encoding_service(dry_run=True)
assert service.dry_run is True
+382
View File
@@ -0,0 +1,382 @@
"""
Tests for PackagingService.
Tests cover:
- Service initialization
- CDG package creation
- TXT package creation
- Dry run mode
- Error handling
"""
import os
import zipfile
import tempfile
import pytest
from unittest.mock import MagicMock, patch
from backend.services.packaging_service import (
PackagingService,
get_packaging_service,
)
class TestPackagingServiceInit:
"""Test service initialization."""
def test_init_default_values(self):
"""Test default initialization."""
service = PackagingService()
assert service.cdg_styles is None
assert service.dry_run is False
assert service.non_interactive is False
def test_init_with_cdg_styles(self):
"""Test initialization with CDG styles."""
styles = {"font_size": 24, "bg_color": "black"}
service = PackagingService(cdg_styles=styles)
assert service.cdg_styles == styles
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = PackagingService(dry_run=True)
assert service.dry_run is True
def test_init_with_non_interactive(self):
"""Test initialization with non-interactive mode."""
service = PackagingService(non_interactive=True)
assert service.non_interactive is True
class TestPackagingServiceCreateZipFromFiles:
"""Test internal ZIP creation."""
def test_create_zip_from_files(self):
"""Test creating a ZIP file from multiple files."""
with tempfile.TemporaryDirectory() as tmpdir:
# Create test files
file1 = os.path.join(tmpdir, "test1.txt")
file2 = os.path.join(tmpdir, "test2.txt")
zip_path = os.path.join(tmpdir, "output.zip")
with open(file1, "w") as f:
f.write("content1")
with open(file2, "w") as f:
f.write("content2")
service = PackagingService()
service._create_zip_from_files(
zip_path,
[(file1, "test1.txt"), (file2, "test2.txt")]
)
assert os.path.isfile(zip_path)
with zipfile.ZipFile(zip_path, "r") as zf:
names = zf.namelist()
assert "test1.txt" in names
assert "test2.txt" in names
class TestPackagingServiceCDGPackage:
"""Test CDG package creation."""
def test_create_cdg_package_missing_lrc_file(self):
"""Test that missing LRC file raises FileNotFoundError."""
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_cdg_package(
lrc_file="/nonexistent/file.lrc",
audio_file="/some/audio.flac",
output_zip_path="/output/test.zip",
artist="Test Artist",
title="Test Title",
)
assert "LRC file not found" in str(exc_info.value)
def test_create_cdg_package_missing_audio_file(self):
"""Test that missing audio file raises FileNotFoundError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_cdg_package(
lrc_file=lrc_file,
audio_file="/nonexistent/audio.flac",
output_zip_path="/output/test.zip",
artist="Test Artist",
title="Test Title",
)
assert "Audio file not found" in str(exc_info.value)
def test_create_cdg_package_no_styles_raises(self):
"""Test that CDG generation without styles raises ValueError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
# Service without CDG styles
service = PackagingService(cdg_styles=None)
with pytest.raises(ValueError) as exc_info:
service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
assert "CDG styles configuration is required" in str(exc_info.value)
def test_create_cdg_package_dry_run(self):
"""Test CDG package creation in dry run mode."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
service = PackagingService(dry_run=True)
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
# In dry run, ZIP should not be created
assert not os.path.isfile(zip_path)
assert result[0] == zip_path
def test_create_cdg_package_existing_files(self):
"""Test CDG package creation with existing MP3 and CDG files."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
mp3_file = os.path.join(tmpdir, "test.mp3")
cdg_file = os.path.join(tmpdir, "test.cdg")
zip_path = os.path.join(tmpdir, "output.zip")
# Create all files
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
with open(mp3_file, "w") as f:
f.write("fake mp3")
with open(cdg_file, "w") as f:
f.write("fake cdg")
service = PackagingService()
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
output_mp3_path=mp3_file,
output_cdg_path=cdg_file,
)
# ZIP should be created from existing files
assert os.path.isfile(zip_path)
assert result[0] == zip_path
assert result[1] == mp3_file
assert result[2] == cdg_file
@patch("lyrics_transcriber.output.cdg.CDGGenerator")
def test_create_cdg_package_with_generator(self, mock_cdg_generator_class):
"""Test CDG package creation using CDGGenerator."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
generated_zip = os.path.join(tmpdir, "generated.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
# Create the zip file that would be generated
with zipfile.ZipFile(generated_zip, "w") as zf:
zf.writestr("test.mp3", "fake mp3")
zf.writestr("test.cdg", "fake cdg")
mock_generator = MagicMock()
mock_generator.generate_cdg_from_lrc.return_value = (
"test.cdg", "test.mp3", generated_zip
)
mock_cdg_generator_class.return_value = mock_generator
styles = {"font_size": 24}
service = PackagingService(cdg_styles=styles)
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
mock_generator.generate_cdg_from_lrc.assert_called_once_with(
lrc_file=lrc_file,
audio_file=audio_file,
title="Test Title",
artist="Test Artist",
cdg_styles=styles,
)
assert os.path.isfile(zip_path)
class TestPackagingServiceTXTPackage:
"""Test TXT package creation."""
def test_create_txt_package_missing_lrc_file(self):
"""Test that missing LRC file raises FileNotFoundError."""
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_txt_package(
lrc_file="/nonexistent/file.lrc",
mp3_file="/some/audio.mp3",
output_zip_path="/output/test.zip",
)
assert "LRC file not found" in str(exc_info.value)
def test_create_txt_package_missing_mp3_file(self):
"""Test that missing MP3 file raises FileNotFoundError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_txt_package(
lrc_file=lrc_file,
mp3_file="/nonexistent/audio.mp3",
output_zip_path="/output/test.zip",
)
assert "MP3 file not found" in str(exc_info.value)
def test_create_txt_package_dry_run(self):
"""Test TXT package creation in dry run mode."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
mp3_file = os.path.join(tmpdir, "test.mp3")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(mp3_file, "w") as f:
f.write("fake mp3")
service = PackagingService(dry_run=True)
result = service.create_txt_package(
lrc_file=lrc_file,
mp3_file=mp3_file,
output_zip_path=zip_path,
)
# In dry run, ZIP should not be created
assert not os.path.isfile(zip_path)
assert result[0] == zip_path
@patch("lyrics_converter.LyricsConverter")
def test_create_txt_package_success(self, mock_converter_class):
"""Test successful TXT package creation."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
mp3_file = os.path.join(tmpdir, "test.mp3")
zip_path = os.path.join(tmpdir, "output.zip")
txt_path = os.path.join(tmpdir, "output.txt")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(mp3_file, "w") as f:
f.write("fake mp3")
mock_converter = MagicMock()
mock_converter.convert_file.return_value = "Converted lyrics text"
mock_converter_class.return_value = mock_converter
service = PackagingService()
result = service.create_txt_package(
lrc_file=lrc_file,
mp3_file=mp3_file,
output_zip_path=zip_path,
output_txt_path=txt_path,
)
mock_converter_class.assert_called_once_with(
output_format="txt",
filepath=lrc_file
)
mock_converter.convert_file.assert_called_once()
assert os.path.isfile(zip_path)
assert os.path.isfile(txt_path)
assert result[0] == zip_path
assert result[1] == txt_path
# Verify ZIP contents
with zipfile.ZipFile(zip_path, "r") as zf:
names = zf.namelist()
assert "test.mp3" in names
assert "output.txt" in names
class TestGetPackagingService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.packaging_service as module
module._packaging_service = None
service = get_packaging_service()
assert service is not None
assert isinstance(service, PackagingService)
def test_get_service_with_cdg_styles(self):
"""Test factory function with CDG styles."""
import backend.services.packaging_service as module
module._packaging_service = None
styles = {"font_size": 24}
service = get_packaging_service(cdg_styles=styles)
assert service.cdg_styles == styles
def test_get_service_with_options(self):
"""Test factory function with additional options."""
import backend.services.packaging_service as module
module._packaging_service = None
service = get_packaging_service(dry_run=True, non_interactive=True)
assert service.dry_run is True
assert service.non_interactive is True
+2 -1
View File
@@ -149,7 +149,8 @@ class TestUploadSubmissionRequest:
# CDG/TXT disabled by default (requires style config)
assert request.enable_cdg is False
assert request.enable_txt is False
assert request.enable_youtube_upload is False
# YouTube upload default is None (use server default)
assert request.enable_youtube_upload is None
class TestStartReviewRequest:
+3 -3
View File
@@ -99,7 +99,7 @@ class TestStylesConfigRequirements:
"karaoke": {
"background_color": "#000000",
"font_path": "", # MUST be string, NOT None
"font": "Arial",
"font": "Noto Sans",
"ass_name": "Default", # REQUIRED
"primary_color": "112, 112, 247, 255",
"secondary_color": "255, 255, 255, 255",
@@ -241,7 +241,7 @@ class TestPreviewStyleLoading:
"background_image": "/original/path/background.png",
"font_path": "/original/path/font.ttf",
"background_color": "#000000",
"font": "Arial",
"font": "Noto Sans",
"ass_name": "Default",
}
}
@@ -308,7 +308,7 @@ class TestPreviewStyleLoading:
# Should have karaoke section with minimal/default values
assert "karaoke" in result_styles
assert result_styles["karaoke"]["background_color"] == "#000000"
assert result_styles["karaoke"]["font"] == "Arial"
assert result_styles["karaoke"]["font"] == "Noto Sans"
# Minimal styles have background_image as None (default)
assert result_styles["karaoke"].get("background_image") is None
@@ -0,0 +1,847 @@
"""
Tests for Video Worker Orchestrator.
Tests cover:
- OrchestratorConfig and OrchestratorResult dataclasses
- VideoWorkerOrchestrator initialization
- Individual stage methods
- Full pipeline execution
- Error handling and recovery
"""
import os
import tempfile
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from dataclasses import asdict
from backend.workers.video_worker_orchestrator import (
OrchestratorConfig,
OrchestratorResult,
VideoWorkerOrchestrator,
create_orchestrator_config_from_job,
)
class TestOrchestratorConfig:
"""Test OrchestratorConfig dataclass."""
def test_required_fields(self):
"""Test that required fields must be provided."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert config.job_id == "test-job"
assert config.artist == "Test Artist"
assert config.encoding_backend == "auto"
def test_default_values(self):
"""Test default values for optional fields."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert config.enable_cdg is False
assert config.enable_txt is False
assert config.enable_youtube_upload is False
assert config.dry_run is False
assert config.non_interactive is True
assert config.end_video_path is None
def test_all_fields(self):
"""Test all fields including optional ones."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
end_video_path="/path/end.mov",
enable_cdg=True,
enable_txt=True,
enable_youtube_upload=True,
brand_prefix="TEST",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
encoding_backend="gce",
)
assert config.enable_cdg is True
assert config.encoding_backend == "gce"
assert config.discord_webhook_url is not None
class TestOrchestratorResult:
"""Test OrchestratorResult dataclass."""
def test_success_result(self):
"""Test successful result."""
result = OrchestratorResult(
success=True,
final_video="/output/video.mp4",
brand_code="TEST-1234",
youtube_url="https://youtube.com/watch?v=test",
)
assert result.success is True
assert result.error_message is None
assert result.brand_code == "TEST-1234"
def test_failure_result(self):
"""Test failure result."""
result = OrchestratorResult(
success=False,
error_message="Encoding failed",
)
assert result.success is False
assert result.error_message == "Encoding failed"
assert result.final_video is None
def test_all_output_files(self):
"""Test all output file fields."""
result = OrchestratorResult(
success=True,
final_video="/output/lossless.mp4",
final_video_mkv="/output/lossless.mkv",
final_video_lossy="/output/lossy.mp4",
final_video_720p="/output/720p.mp4",
final_karaoke_cdg_zip="/output/cdg.zip",
final_karaoke_txt_zip="/output/txt.zip",
)
assert result.final_video == "/output/lossless.mp4"
assert result.final_video_mkv == "/output/lossless.mkv"
class TestVideoWorkerOrchestratorInit:
"""Test VideoWorkerOrchestrator initialization."""
def test_init_with_config(self):
"""Test initialization with config."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
assert orchestrator.config == config
assert orchestrator.result.success is False
def test_init_with_job_manager(self):
"""Test initialization with job manager."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
job_manager = MagicMock()
orchestrator = VideoWorkerOrchestrator(config, job_manager=job_manager)
assert orchestrator.job_manager == job_manager
class TestVideoWorkerOrchestratorServices:
"""Test service lazy-loading."""
def test_get_encoding_backend(self):
"""Test encoding backend lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
encoding_backend="local",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.encoding_interface.get_encoding_backend") as mock_get:
mock_backend = MagicMock()
mock_backend.name = "local"
mock_get.return_value = mock_backend
backend = orchestrator._get_encoding_backend()
assert backend == mock_backend
mock_get.assert_called_once()
def test_get_packaging_service(self):
"""Test packaging service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
cdg_styles={"background_color": "black"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.packaging_service.PackagingService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_packaging_service()
assert service == mock_service
MockService.assert_called_once()
def test_get_youtube_service(self):
"""Test YouTube service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
youtube_credentials={"token": "test"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.youtube_upload_service.YouTubeUploadService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_youtube_service()
assert service == mock_service
MockService.assert_called_once()
def test_get_discord_service(self):
"""Test Discord service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.discord_service.DiscordNotificationService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_discord_service()
assert service == mock_service
MockService.assert_called_once()
class TestVideoWorkerOrchestratorPackaging:
"""Test packaging stage."""
@pytest.mark.asyncio
async def test_run_packaging_no_lrc(self):
"""Test packaging stage with no LRC file."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=True,
lrc_file_path=None,
)
orchestrator = VideoWorkerOrchestrator(config)
# Should not raise, just skip
await orchestrator._run_packaging()
assert orchestrator.result.final_karaoke_cdg_zip is None
@pytest.mark.asyncio
async def test_run_packaging_cdg(self):
"""Test CDG packaging."""
with tempfile.TemporaryDirectory() as temp_dir:
lrc_file = os.path.join(temp_dir, "test.lrc")
audio_file = os.path.join(temp_dir, "test.flac")
# Create dummy files
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("dummy audio")
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path=audio_file,
lrc_file_path=lrc_file,
output_dir=temp_dir,
enable_cdg=True,
cdg_styles={"background_color": "black"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_packaging_service") as mock_get:
mock_service = MagicMock()
mock_service.create_cdg_package.return_value = (
f"{temp_dir}/cdg.zip",
f"{temp_dir}/test.mp3",
f"{temp_dir}/test.cdg",
)
mock_get.return_value = mock_service
await orchestrator._run_packaging()
mock_service.create_cdg_package.assert_called_once()
assert orchestrator.result.final_karaoke_cdg_zip == f"{temp_dir}/cdg.zip"
class TestVideoWorkerOrchestratorEncoding:
"""Test encoding stage."""
@pytest.mark.asyncio
async def test_run_encoding_success(self):
"""Test successful encoding."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
output_dir="/output",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
mock_backend = MagicMock()
mock_backend.name = "local"
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path="/output/lossless.mp4",
lossy_4k_mp4_path="/output/lossy.mp4",
lossy_720p_mp4_path="/output/720p.mp4",
lossless_mkv_path="/output/lossless.mkv",
encoding_time_seconds=120.5,
encoding_backend="local",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
mock_backend.encode.assert_called_once()
assert orchestrator.result.final_video == "/output/lossless.mp4"
assert orchestrator.result.final_video_720p == "/output/720p.mp4"
assert orchestrator.result.encoding_time_seconds == 120.5
def test_encoding_input_gcs_paths_pattern(self):
"""Test that EncodingInput.options contains proper GCS paths structure.
This test verifies the GCS path construction pattern that GCE encoding
requires. It tests the structure without running _run_encoding().
This would have caught: 'GCE encoding requires input_gcs_path and
output_gcs_path in options' error when the orchestrator didn't pass
the required paths for GCE encoding.
"""
from backend.services.encoding_interface import EncodingInput
# Test that EncodingInput can hold GCS paths in options
# This mirrors what the orchestrator should build
job_id = "test-job-123"
bucket = "test-bucket"
input_gcs_path = f"gs://{bucket}/jobs/{job_id}/"
output_gcs_path = f"gs://{bucket}/jobs/{job_id}/finals/"
encoding_input = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
artist="Test Artist",
title="Test Title",
brand_code="TEST-001",
output_dir="/output",
options={
"input_gcs_path": input_gcs_path,
"output_gcs_path": output_gcs_path,
},
)
# Verify the structure that GCEEncodingBackend expects
assert "input_gcs_path" in encoding_input.options, \
"EncodingInput.options must include input_gcs_path for GCE encoding"
assert "output_gcs_path" in encoding_input.options, \
"EncodingInput.options must include output_gcs_path for GCE encoding"
# Verify path format
assert encoding_input.options["input_gcs_path"].startswith("gs://")
assert encoding_input.options["output_gcs_path"].startswith("gs://")
assert job_id in encoding_input.options["input_gcs_path"]
assert job_id in encoding_input.options["output_gcs_path"]
@pytest.mark.asyncio
async def test_run_encoding_failure(self):
"""Test encoding failure."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
mock_backend = MagicMock()
mock_backend.name = "local"
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=False,
error_message="FFmpeg failed",
encoding_backend="local",
))
mock_get.return_value = mock_backend
with pytest.raises(Exception) as exc_info:
await orchestrator._run_encoding()
assert "Encoding failed" in str(exc_info.value)
@pytest.mark.asyncio
async def test_run_encoding_gce_downloads_files(self):
"""Test that GCE encoding downloads files from GCS to local directory.
This test verifies the fix for YouTube upload failure when using GCE encoding.
GCE encoding returns GCS blob paths, which need to be downloaded locally
before YouTube upload can access them.
"""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path=os.path.join(temp_dir, "title.mov"),
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
output_dir=temp_dir,
)
# Mock storage service
mock_storage = MagicMock()
mock_storage.download_file = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config,
storage=mock_storage,
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
# GCE backend returns GCS blob paths (not local paths)
mock_backend = MagicMock()
mock_backend.name = "gce" # Important: must be "gce" to trigger download
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path="jobs/test-job/finals/output_4k_lossless.mp4",
lossy_4k_mp4_path="jobs/test-job/finals/output_4k_lossy.mp4",
lossy_720p_mp4_path="jobs/test-job/finals/output_720p.mp4",
lossless_mkv_path="jobs/test-job/finals/output_4k.mkv",
encoding_time_seconds=60.0,
encoding_backend="gce",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# Verify download_file was called for each output file
assert mock_storage.download_file.call_count == 4
# Verify the result paths were updated to local paths
assert orchestrator.result.final_video.startswith(temp_dir)
assert orchestrator.result.final_video_mkv.startswith(temp_dir)
assert orchestrator.result.final_video_lossy.startswith(temp_dir)
assert orchestrator.result.final_video_720p.startswith(temp_dir)
@pytest.mark.asyncio
async def test_run_encoding_local_does_not_download(self):
"""Test that local encoding does NOT trigger GCS download.
Local encoding produces files directly in the output directory,
so no download is needed.
"""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path=os.path.join(temp_dir, "title.mov"),
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
output_dir=temp_dir,
)
# Mock storage service
mock_storage = MagicMock()
mock_storage.download_file = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config,
storage=mock_storage,
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
# Local backend returns local paths directly
mock_backend = MagicMock()
mock_backend.name = "local" # Local backend
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path=os.path.join(temp_dir, "lossless.mp4"),
lossy_4k_mp4_path=os.path.join(temp_dir, "lossy.mp4"),
lossy_720p_mp4_path=os.path.join(temp_dir, "720p.mp4"),
lossless_mkv_path=os.path.join(temp_dir, "lossless.mkv"),
encoding_time_seconds=120.0,
encoding_backend="local",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# Verify download_file was NOT called for local encoding
mock_storage.download_file.assert_not_called()
class TestVideoWorkerOrchestratorOrganization:
"""Test organization stage."""
@pytest.mark.asyncio
async def test_run_organization_keep_brand_code(self):
"""Test organization with existing brand code."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
keep_brand_code="NOMAD-9999",
)
orchestrator = VideoWorkerOrchestrator(config)
await orchestrator._run_organization()
assert orchestrator.result.brand_code == "NOMAD-9999"
@pytest.mark.asyncio
async def test_run_organization_generate_brand_code(self):
"""Test brand code generation from Dropbox."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
dropbox_path="/Karaoke/Tracks",
brand_prefix="TEST",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
mock_dropbox = MagicMock()
mock_dropbox.is_configured = True
mock_dropbox.get_next_brand_code.return_value = "TEST-0001"
mock_get.return_value = mock_dropbox
await orchestrator._run_organization()
assert orchestrator.result.brand_code == "TEST-0001"
class TestVideoWorkerOrchestratorDistribution:
"""Test distribution stage."""
@pytest.mark.asyncio
async def test_upload_to_youtube(self):
"""Test YouTube upload."""
with tempfile.TemporaryDirectory() as temp_dir:
video_file = os.path.join(temp_dir, "test.mp4")
with open(video_file, "w") as f:
f.write("dummy video")
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_youtube_upload=True,
youtube_credentials={"token": "test"},
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.final_video_lossy = video_file
with patch.object(orchestrator, "_get_youtube_service") as mock_get:
mock_service = MagicMock()
mock_service.upload_video.return_value = (
"video123",
"https://youtube.com/watch?v=video123"
)
mock_get.return_value = mock_service
await orchestrator._upload_to_youtube()
mock_service.upload_video.assert_called_once()
assert orchestrator.result.youtube_url == "https://youtube.com/watch?v=video123"
@pytest.mark.asyncio
async def test_upload_to_dropbox(self):
"""Test Dropbox upload."""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
output_dir=temp_dir,
dropbox_path="/Karaoke/Tracks",
brand_prefix="TEST",
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.brand_code = "TEST-0001"
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
mock_dropbox = MagicMock()
mock_dropbox.is_configured = True
mock_dropbox.create_shared_link.return_value = "https://dropbox.com/link"
mock_get.return_value = mock_dropbox
await orchestrator._upload_to_dropbox()
mock_dropbox.upload_folder.assert_called_once()
assert orchestrator.result.dropbox_link == "https://dropbox.com/link"
class TestVideoWorkerOrchestratorNotifications:
"""Test notifications stage."""
@pytest.mark.asyncio
async def test_run_notifications_with_youtube_url(self):
"""Test Discord notification with YouTube URL."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.youtube_url = "https://youtube.com/watch?v=test"
with patch.object(orchestrator, "_get_discord_service") as mock_get:
mock_service = MagicMock()
mock_service.post_video_notification.return_value = True
mock_get.return_value = mock_service
await orchestrator._run_notifications()
mock_service.post_video_notification.assert_called_once_with(
"https://youtube.com/watch?v=test"
)
@pytest.mark.asyncio
async def test_run_notifications_no_youtube_url(self):
"""Test notification skipped without YouTube URL."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
# No youtube_url set
with patch.object(orchestrator, "_get_discord_service") as mock_get:
mock_service = MagicMock()
mock_get.return_value = mock_service
await orchestrator._run_notifications()
mock_service.post_video_notification.assert_not_called()
class TestVideoWorkerOrchestratorFullPipeline:
"""Test full pipeline execution."""
@pytest.mark.asyncio
async def test_run_full_pipeline_success(self):
"""Test successful full pipeline."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=True, # Enable to trigger packaging stage
)
orchestrator = VideoWorkerOrchestrator(config)
# Mock all stages
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org, \
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock) as mock_dist, \
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock) as mock_notify:
result = await orchestrator.run()
assert result.success is True
mock_packaging.assert_called_once()
mock_encoding.assert_called_once()
mock_org.assert_called_once()
mock_dist.assert_called_once()
mock_notify.assert_called_once()
@pytest.mark.asyncio
async def test_run_pipeline_skips_packaging_when_disabled(self):
"""Test pipeline skips packaging when CDG/TXT disabled."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=False,
enable_txt=False,
)
orchestrator = VideoWorkerOrchestrator(config)
# Mock all stages
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock):
result = await orchestrator.run()
assert result.success is True
mock_packaging.assert_not_called()
mock_encoding.assert_called_once()
@pytest.mark.asyncio
async def test_run_full_pipeline_encoding_failure(self):
"""Test pipeline failure during encoding."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org:
mock_encoding.side_effect = Exception("Encoding failed")
result = await orchestrator.run()
assert result.success is False
assert "Encoding failed" in result.error_message
mock_org.assert_not_called()
class TestCreateOrchestratorConfigFromJob:
"""Test helper function for creating config from job."""
def test_create_config_from_job(self):
"""Test creating config from a job object."""
job = MagicMock()
job.job_id = "test-123"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {"instrumental_selection": "clean"}
job.enable_cdg = True
job.enable_txt = False
job.enable_youtube_upload = True
job.brand_prefix = "NOMAD"
job.discord_webhook_url = "https://discord.com/api/webhooks/123/abc"
job.youtube_description_template = "Test description"
job.dropbox_path = "/Karaoke"
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = None
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
youtube_credentials={"token": "test"},
cdg_styles={"background": "black"},
)
assert config.job_id == "test-123"
assert config.artist == "Test Artist"
assert config.title == "Test Title"
assert config.enable_cdg is True
assert config.enable_youtube_upload is True
assert config.title_video_path == "/tmp/test/Test Artist - Test Title (Title).mov"
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental Clean).flac"
def test_create_config_from_job_with_existing_instrumental(self):
"""Test config with user-provided instrumental."""
job = MagicMock()
job.job_id = "test-123"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {"instrumental_selection": "custom"}
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = False
job.brand_prefix = None
job.discord_webhook_url = None
job.youtube_description_template = None
job.dropbox_path = None
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = "gs://bucket/instrumental.mp3"
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
)
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental User).mp3"
@@ -0,0 +1,509 @@
"""
Unit tests for worker log subcollection functionality.
Tests the WorkerLogEntry model, FirestoreService subcollection methods,
and JobManager log operations with the feature flag.
"""
import pytest
from unittest.mock import Mock, MagicMock, patch
from datetime import datetime, timezone, timedelta
import uuid
# Mock Firestore before imports
import sys
sys.modules['google.cloud.firestore'] = MagicMock()
sys.modules['google.cloud.firestore_v1'] = MagicMock()
from backend.models.worker_log import WorkerLogEntry, DEFAULT_LOG_TTL_DAYS
class TestWorkerLogEntry:
"""Tests for WorkerLogEntry dataclass."""
def test_create_log_entry(self):
"""Test creating a log entry with factory method."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
assert entry.job_id == "job123"
assert entry.worker == "audio"
assert entry.level == "INFO"
assert entry.message == "Test message"
assert entry.id is not None
assert entry.timestamp is not None
assert entry.ttl_expiry is not None
# TTL should be ~30 days from now
expected_ttl = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
def test_create_log_entry_truncates_long_message(self):
"""Test that messages longer than 1000 chars are truncated."""
long_message = "x" * 2000
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message=long_message
)
assert len(entry.message) == 1000
def test_create_log_entry_normalizes_level(self):
"""Test that log level is normalized to uppercase."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="info",
message="Test"
)
assert entry.level == "INFO"
def test_create_log_entry_custom_ttl(self):
"""Test creating entry with custom TTL."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test",
ttl_days=7
)
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
def test_create_log_entry_with_metadata(self):
"""Test creating entry with metadata."""
metadata = {"file_size": 1024, "duration": 300}
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test",
metadata=metadata
)
assert entry.metadata == metadata
def test_to_dict(self):
"""Test converting entry to dict for Firestore."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message",
metadata={"key": "value"}
)
d = entry.to_dict()
assert d["job_id"] == "job123"
assert d["worker"] == "audio"
assert d["level"] == "INFO"
assert d["message"] == "Test message"
assert d["metadata"] == {"key": "value"}
assert "id" in d
assert "timestamp" in d
assert "ttl_expiry" in d
def test_to_dict_without_metadata(self):
"""Test to_dict doesn't include metadata when None."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test"
)
d = entry.to_dict()
assert "metadata" not in d
def test_to_legacy_dict(self):
"""Test converting to legacy format for API compatibility."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
d = entry.to_legacy_dict()
assert "timestamp" in d
assert d["level"] == "INFO"
assert d["worker"] == "audio"
assert d["message"] == "Test message"
# Should not include id, job_id, ttl_expiry, metadata
assert "id" not in d
assert "job_id" not in d
assert "ttl_expiry" not in d
def test_from_dict(self):
"""Test creating entry from Firestore document."""
timestamp = datetime.now(timezone.utc)
ttl_expiry = timestamp + timedelta(days=30)
data = {
"id": "log123",
"job_id": "job123",
"timestamp": timestamp,
"level": "WARNING",
"worker": "video",
"message": "Warning message",
"metadata": {"error_code": 500},
"ttl_expiry": ttl_expiry
}
entry = WorkerLogEntry.from_dict(data)
assert entry.id == "log123"
assert entry.job_id == "job123"
assert entry.level == "WARNING"
assert entry.worker == "video"
assert entry.message == "Warning message"
assert entry.metadata == {"error_code": 500}
def test_from_dict_with_iso_strings(self):
"""Test creating entry from dict with ISO format strings."""
data = {
"timestamp": "2026-01-04T12:00:00Z",
"level": "INFO",
"worker": "audio",
"message": "Test"
}
entry = WorkerLogEntry.from_dict(data)
assert entry.timestamp.tzinfo == timezone.utc
assert entry.message == "Test"
def test_from_dict_missing_fields(self):
"""Test creating entry with missing fields uses defaults."""
data = {
"message": "Minimal log"
}
entry = WorkerLogEntry.from_dict(data)
assert entry.message == "Minimal log"
assert entry.level == "INFO" # Default
assert entry.worker == "unknown" # Default
assert entry.id is not None # Generated
class TestFirestoreServiceSubcollection:
"""Tests for FirestoreService log subcollection methods."""
@pytest.fixture
def mock_db(self):
"""Create mock Firestore database."""
return MagicMock()
@pytest.fixture
def firestore_service(self, mock_db):
"""Create FirestoreService with mocked DB."""
with patch('backend.services.firestore_service.firestore') as mock_firestore:
mock_firestore.Client.return_value = mock_db
from backend.services.firestore_service import FirestoreService
service = FirestoreService()
service.db = mock_db
return service
def test_append_log_to_subcollection(self, firestore_service, mock_db):
"""Test appending log to subcollection."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
# Setup mock chain
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_log_doc = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_logs_ref.document.return_value = mock_log_doc
firestore_service.append_log_to_subcollection("job123", entry)
# Verify correct path: jobs/{job_id}/logs/{log_id}
mock_db.collection.assert_called_with("jobs")
mock_collection.document.assert_called_with("job123")
mock_doc.collection.assert_called_with("logs")
mock_logs_ref.document.assert_called_with(entry.id)
mock_log_doc.set.assert_called_once()
def test_get_logs_from_subcollection(self, firestore_service, mock_db):
"""Test getting logs from subcollection."""
# Setup mock chain
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_query = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_logs_ref.order_by.return_value = mock_query
mock_query.limit.return_value = mock_query
# Mock document data
mock_doc1 = MagicMock()
mock_doc1.to_dict.return_value = {
"timestamp": datetime.now(timezone.utc),
"level": "INFO",
"worker": "audio",
"message": "Log 1"
}
mock_query.stream.return_value = iter([mock_doc1])
logs = firestore_service.get_logs_from_subcollection("job123")
assert len(logs) == 1
assert logs[0].message == "Log 1"
mock_logs_ref.order_by.assert_called_once()
def test_get_logs_from_subcollection_with_worker_filter(self, firestore_service, mock_db):
"""Test filtering logs by worker."""
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_query = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_logs_ref.order_by.return_value = mock_query
mock_query.where.return_value = mock_query
mock_query.limit.return_value = mock_query
mock_query.stream.return_value = iter([])
firestore_service.get_logs_from_subcollection("job123", worker="audio")
# Verify where clause was added
mock_query.where.assert_called()
def test_delete_logs_subcollection(self, firestore_service, mock_db):
"""Test deleting all logs in subcollection."""
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_batch = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_db.batch.return_value = mock_batch
# First call returns 2 docs, second call returns empty
mock_log_doc1 = MagicMock()
mock_log_doc2 = MagicMock()
call_count = [0]
def stream_side_effect(*args, **kwargs):
call_count[0] += 1
if call_count[0] == 1:
return iter([mock_log_doc1, mock_log_doc2])
return iter([])
mock_logs_ref.limit.return_value = mock_logs_ref
mock_logs_ref.stream.side_effect = stream_side_effect
deleted = firestore_service.delete_logs_subcollection("job123")
assert deleted == 2
mock_batch.commit.assert_called()
class TestJobManagerLogging:
"""Tests for JobManager log methods with feature flag."""
@pytest.fixture
def mock_firestore_service(self):
"""Mock FirestoreService."""
with patch('backend.services.job_manager.FirestoreService') as mock:
service = MagicMock()
mock.return_value = service
yield service
@pytest.fixture
def mock_storage_service(self):
"""Mock StorageService."""
with patch('backend.services.job_manager.StorageService') as mock:
service = MagicMock()
mock.return_value = service
yield service
def test_append_worker_log_uses_subcollection_when_enabled(
self, mock_firestore_service, mock_storage_service
):
"""Test that logs go to subcollection when feature is enabled."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = True
from backend.services.job_manager import JobManager
manager = JobManager()
manager.append_worker_log(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
# Should call subcollection method
mock_firestore_service.append_log_to_subcollection.assert_called_once()
# Should NOT call legacy method
mock_firestore_service.append_worker_log.assert_not_called()
def test_append_worker_log_uses_array_when_disabled(
self, mock_firestore_service, mock_storage_service
):
"""Test that logs go to embedded array when feature is disabled."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = False
from backend.services.job_manager import JobManager
manager = JobManager()
manager.append_worker_log(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
# Should call legacy method
mock_firestore_service.append_worker_log.assert_called_once()
# Should NOT call subcollection method
mock_firestore_service.append_log_to_subcollection.assert_not_called()
def test_get_worker_logs_from_subcollection(
self, mock_firestore_service, mock_storage_service
):
"""Test getting logs from subcollection."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = True
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test"
)
mock_firestore_service.get_logs_from_subcollection.return_value = [entry]
from backend.services.job_manager import JobManager
manager = JobManager()
logs = manager.get_worker_logs("job123")
assert len(logs) == 1
assert logs[0]["message"] == "Test"
mock_firestore_service.get_logs_from_subcollection.assert_called_once()
def test_get_worker_logs_falls_back_to_array(
self, mock_firestore_service, mock_storage_service
):
"""Test fallback to embedded array for legacy jobs."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = True
# Subcollection returns empty
mock_firestore_service.get_logs_from_subcollection.return_value = []
# Mock job with legacy logs
from backend.models.job import Job, JobStatus
mock_job = Job(
job_id="job123",
status=JobStatus.PENDING,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
worker_logs=[
{"timestamp": "2026-01-04T12:00:00Z", "level": "INFO", "worker": "audio", "message": "Legacy log"}
]
)
mock_firestore_service.get_job.return_value = mock_job
from backend.services.job_manager import JobManager
manager = JobManager()
logs = manager.get_worker_logs("job123")
assert len(logs) == 1
assert logs[0]["message"] == "Legacy log"
def test_delete_job_deletes_logs_subcollection(
self, mock_firestore_service, mock_storage_service
):
"""Test that deleting a job also deletes its logs subcollection."""
mock_firestore_service.get_job.return_value = None
mock_firestore_service.delete_logs_subcollection.return_value = 5
from backend.services.job_manager import JobManager
manager = JobManager()
manager.delete_job("job123", delete_files=False)
mock_firestore_service.delete_logs_subcollection.assert_called_once_with("job123")
mock_firestore_service.delete_job.assert_called_once_with("job123")
class TestWorkerLogEntryEdgeCases:
"""Edge case tests for WorkerLogEntry."""
def test_create_with_empty_message(self):
"""Test creating entry with empty message."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message=""
)
assert entry.message == ""
def test_create_with_unicode_message(self):
"""Test creating entry with Unicode characters."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Processing song: 日本語の曲 - アーティスト"
)
assert "日本語" in entry.message
def test_create_with_newlines_in_message(self):
"""Test creating entry with newlines."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="ERROR",
message="Error occurred:\nLine 1\nLine 2"
)
assert "\n" in entry.message
def test_from_dict_handles_missing_ttl_expiry(self):
"""Test from_dict creates default TTL when missing."""
data = {
"timestamp": datetime.now(timezone.utc),
"level": "INFO",
"worker": "audio",
"message": "Test"
}
entry = WorkerLogEntry.from_dict(data)
assert entry.ttl_expiry is not None
# Should be ~30 days from now
expected = datetime.now(timezone.utc) + timedelta(days=30)
assert abs((entry.ttl_expiry - expected).total_seconds()) < 10
@@ -0,0 +1,568 @@
"""
Tests for YouTubeUploadService.
Tests cover:
- Authentication with pre-stored credentials
- Authentication with client secrets file
- Duplicate video detection (exact and fuzzy matching)
- Video deletion
- Video upload with metadata and thumbnail
- Title truncation
- Dry run mode
"""
import os
import pytest
from unittest.mock import MagicMock, patch, mock_open
from backend.services.youtube_upload_service import YouTubeUploadService, get_youtube_upload_service
class TestYouTubeUploadServiceInit:
"""Test service initialization."""
def test_init_with_credentials(self):
"""Test initialization with pre-stored credentials."""
creds = {
"token": "test_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
}
service = YouTubeUploadService(
credentials=creds,
non_interactive=True
)
assert service.credentials == creds
assert service.non_interactive is True
def test_init_with_client_secrets(self):
"""Test initialization with client secrets file."""
service = YouTubeUploadService(
client_secrets_file="/path/to/secrets.json"
)
assert service.client_secrets_file == "/path/to/secrets.json"
assert service.non_interactive is False
def test_init_default_values(self):
"""Test default values on initialization."""
service = YouTubeUploadService()
assert service.credentials is None
assert service.client_secrets_file is None
assert service.non_interactive is False
assert service.server_side_mode is False
assert service.dry_run is False
class TestYouTubeUploadServiceAuthentication:
"""Test authentication methods."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_authenticate_with_prestored_credentials(
self, mock_request, mock_credentials_class, mock_build
):
"""Test authentication using pre-stored credentials."""
# Setup mocks
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_build.return_value = mock_youtube
creds = {
"token": "test_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
"scopes": ["https://www.googleapis.com/auth/youtube"],
}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
# Access youtube_service to trigger authentication
result = service.youtube_service
assert result == mock_youtube
mock_credentials_class.assert_called_once()
mock_build.assert_called_once_with('youtube', 'v3', credentials=mock_creds)
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_authenticate_refreshes_expired_token(
self, mock_request, mock_credentials_class, mock_build
):
"""Test that expired tokens are refreshed."""
mock_creds = MagicMock()
mock_creds.expired = True
mock_creds.refresh_token = "test_refresh"
mock_credentials_class.return_value = mock_creds
creds = {
"token": "expired_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
service.youtube_service
mock_creds.refresh.assert_called_once()
def test_authenticate_non_interactive_no_credentials_raises(self):
"""Test that non-interactive mode without credentials raises error."""
service = YouTubeUploadService(non_interactive=True)
with pytest.raises(Exception) as exc_info:
service._authenticate()
assert "non-interactive mode" in str(exc_info.value).lower()
class TestYouTubeUploadServiceChannelId:
"""Test channel ID retrieval."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_get_channel_id_success(
self, mock_request, mock_credentials_class, mock_build
):
"""Test successful channel ID retrieval."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_list = MagicMock()
mock_list.execute.return_value = {
"items": [{"id": "UC123456"}]
}
mock_channels.list.return_value = mock_list
mock_youtube.channels.return_value = mock_channels
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
channel_id = service.get_channel_id()
assert channel_id == "UC123456"
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_get_channel_id_no_items(
self, mock_request, mock_credentials_class, mock_build
):
"""Test channel ID returns None when no items found."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_list = MagicMock()
mock_list.execute.return_value = {"items": []}
mock_channels.list.return_value = mock_list
mock_youtube.channels.return_value = mock_channels
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
channel_id = service.get_channel_id()
assert channel_id is None
class TestYouTubeUploadServiceDuplicateCheck:
"""Test duplicate video detection."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_exact_match_found(
self, mock_request, mock_credentials_class, mock_build
):
"""Test duplicate detection with exact match in server mode."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
# Mock channels().list() for get_channel_id
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Mock search().list() for duplicate check
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {
"items": [{
"id": {"videoId": "VIDEO123"},
"snippet": {
"channelId": "UC123456",
"title": "Test Artist - Test Song (Karaoke)"
}
}]
}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(
credentials=creds,
non_interactive=True,
server_side_mode=True
)
exists, video_id, video_url = service.check_duplicate(
"Test Artist - Test Song (Karaoke)"
)
assert exists is True
assert video_id == "VIDEO123"
assert "VIDEO123" in video_url
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_no_match(
self, mock_request, mock_credentials_class, mock_build
):
"""Test duplicate detection when no match found."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
exists, video_id, video_url = service.check_duplicate("Some Title")
assert exists is False
assert video_id is None
assert video_url is None
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_skips_other_channels(
self, mock_request, mock_credentials_class, mock_build
):
"""Test that videos from other channels are skipped."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Return video from different channel
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {
"items": [{
"id": {"videoId": "VIDEO123"},
"snippet": {
"channelId": "UC_DIFFERENT", # Different channel
"title": "Test Title"
}
}]
}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
exists, video_id, video_url = service.check_duplicate("Test Title")
assert exists is False
class TestYouTubeUploadServiceDelete:
"""Test video deletion."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_delete_video_success(
self, mock_request, mock_credentials_class, mock_build
):
"""Test successful video deletion."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_videos = MagicMock()
mock_delete = MagicMock()
mock_delete.execute.return_value = None
mock_videos.delete.return_value = mock_delete
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
result = service.delete_video("VIDEO123")
assert result is True
mock_videos.delete.assert_called_once_with(id="VIDEO123")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_delete_video_failure(
self, mock_request, mock_credentials_class, mock_build
):
"""Test video deletion failure handling."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_videos = MagicMock()
mock_delete = MagicMock()
mock_delete.execute.side_effect = Exception("API Error")
mock_videos.delete.return_value = mock_delete
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
result = service.delete_video("VIDEO123")
assert result is False
def test_delete_video_dry_run(self):
"""Test video deletion in dry run mode."""
service = YouTubeUploadService(dry_run=True)
result = service.delete_video("VIDEO123")
assert result is True
class TestYouTubeUploadServiceTitleTruncation:
"""Test title truncation."""
def test_truncate_title_short_title(self):
"""Test that short titles are not truncated."""
title = "Short Title"
result = YouTubeUploadService.truncate_title(title)
assert result == title
def test_truncate_title_exact_length(self):
"""Test title at exact max length."""
title = "A" * 95
result = YouTubeUploadService.truncate_title(title, max_length=95)
assert result == title
assert len(result) == 95
def test_truncate_title_long_title(self):
"""Test that long titles are truncated at word boundary."""
title = "This is a very long title that exceeds the maximum length and needs to be truncated properly at a word boundary"
result = YouTubeUploadService.truncate_title(title, max_length=50)
assert len(result) <= 50
assert result.endswith("...")
def test_truncate_title_no_space(self):
"""Test truncation of title without spaces."""
title = "A" * 100
result = YouTubeUploadService.truncate_title(title, max_length=50)
assert len(result) <= 50
class TestYouTubeUploadServiceUpload:
"""Test video upload."""
@patch("googleapiclient.http.MediaFileUpload")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_upload_video_success(
self, mock_request, mock_credentials_class, mock_build, mock_media_upload
):
"""Test successful video upload."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
# Mock channels for get_channel_id
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Mock search for duplicate check (no duplicates)
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
# Mock video insert
mock_videos = MagicMock()
mock_insert = MagicMock()
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
mock_videos.insert.return_value = mock_insert
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
mock_media_upload.return_value = MagicMock()
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Artist - Test Song (Karaoke)",
description="Test description",
tags=["karaoke", "test"]
)
assert video_id == "NEW_VIDEO_ID"
assert "NEW_VIDEO_ID" in video_url
def test_upload_video_dry_run(self):
"""Test video upload in dry run mode."""
service = YouTubeUploadService(dry_run=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Title",
description="Test description"
)
assert video_id == "dry_run_video_id"
assert "dry_run_video_id" in video_url
@patch("googleapiclient.http.MediaFileUpload")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
@patch("os.path.isfile")
def test_upload_video_with_thumbnail(
self, mock_isfile, mock_request, mock_credentials_class,
mock_build, mock_media_upload
):
"""Test video upload with thumbnail."""
mock_isfile.return_value = True
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_videos = MagicMock()
mock_insert = MagicMock()
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
mock_videos.insert.return_value = mock_insert
mock_youtube.videos.return_value = mock_videos
# Mock thumbnail upload
mock_thumbnails = MagicMock()
mock_set = MagicMock()
mock_set.execute.return_value = None
mock_thumbnails.set.return_value = mock_set
mock_youtube.thumbnails.return_value = mock_thumbnails
mock_build.return_value = mock_youtube
mock_media_upload.return_value = MagicMock()
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Title",
description="Test description",
thumbnail_path="/path/to/thumbnail.jpg"
)
assert video_id == "NEW_VIDEO_ID"
mock_thumbnails.set.assert_called_once()
class TestGetYouTubeUploadService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
# Reset global
import backend.services.youtube_upload_service as module
module._youtube_upload_service = None
service = get_youtube_upload_service(
credentials={"token": "test"},
non_interactive=True
)
assert service is not None
assert isinstance(service, YouTubeUploadService)
def test_get_service_with_client_secrets(self):
"""Test factory function with client secrets file."""
import backend.services.youtube_upload_service as module
module._youtube_upload_service = None
service = get_youtube_upload_service(
client_secrets_file="/path/to/secrets.json"
)
assert service is not None
assert service.client_secrets_file == "/path/to/secrets.json"
+22 -17
View File
@@ -639,40 +639,45 @@ async def upload_lyrics_results(
logger.error(f"Job {job_id}: {error_msg}")
raise Exception(error_msg)
# Upload reference lyrics if available
# Upload ALL reference lyrics files (not just first) so they're available for distribution
# Note: Source names use .title() so "lrclib" -> "Lrclib", "genius" -> "Genius"
# Use sanitized artist/title to match LyricsProcessor's file naming
reference_files = [
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "Genius"),
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "Spotify"),
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "Musixmatch"),
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "LRCLib"), # Note: source is 'lrclib', .title() = 'Lrclib'
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "genius"),
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "spotify"),
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "musixmatch"),
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "lrclib"),
]
found_reference = False
for ref_filename, source_name in reference_files:
found_references = []
for ref_filename, source_key in reference_files:
ref_path = os.path.join(lyrics_dir, ref_filename)
if os.path.exists(ref_path):
# Upload with original filename to preserve proper naming
gcs_path = f"jobs/{job_id}/lyrics/{ref_filename}"
url = storage.upload_file(ref_path, gcs_path)
# Track in job.files so video_worker can download for distribution
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
if job_log:
job_log.info(f"Found reference lyrics from {source_name}")
job_log.info(f"Found reference lyrics from {source_key}")
logger.info(f"Job {job_id}: Uploaded reference lyrics: {ref_filename}")
found_reference = True
break # Only upload first available reference
if not found_reference:
found_references.append(source_key)
if not found_references:
if job_log:
job_log.warning("No reference lyrics found from any source (Genius, Spotify, Musixmatch, LRCLib)")
logger.warning(f"Job {job_id}: No reference lyrics found from any source")
else:
logger.info(f"Job {job_id}: Found {len(found_references)} reference lyrics sources: {found_references}")
# Upload uncorrected transcription if available (use sanitized names)
uncorrected_file = os.path.join(lyrics_dir, f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt")
# Upload uncorrected transcription if available (preserve original filename for distribution)
uncorrected_filename = f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt"
uncorrected_file = os.path.join(lyrics_dir, uncorrected_filename)
if os.path.exists(uncorrected_file):
gcs_path = f"jobs/{job_id}/lyrics/uncorrected.txt"
gcs_path = f"jobs/{job_id}/lyrics/{uncorrected_filename}"
url = storage.upload_file(uncorrected_file, gcs_path)
job_manager.update_file_url(job_id, 'lyrics', 'uncorrected', url)
logger.info(f"Job {job_id}: Uploaded uncorrected transcription")
logger.info(f"Job {job_id}: Uploaded uncorrected transcription: {uncorrected_filename}")
logger.info(f"Job {job_id}: All lyrics results uploaded successfully")
+316 -41
View File
@@ -1,20 +1,21 @@
"""
Video generation and finalization worker.
This worker uses KaraokeFinalise from the karaoke_gen package to handle:
1. Video encoding (4 formats)
2. CDG/TXT package generation
3. Discord notifications
4. Brand code generation
5. All other finalisation features
This worker has two code paths controlled by USE_NEW_ORCHESTRATOR:
The worker's job is simply to:
1. Download files from GCS and set up the directory structure KaraokeFinalise expects
2. Instantiate KaraokeFinalise with parameters from the job
3. Call process()
4. Upload results back to GCS
1. NEW PATH (USE_NEW_ORCHESTRATOR=true):
Uses VideoWorkerOrchestrator which provides a unified pipeline for all
encoding backends (GCE or local). This ensures features like YouTube upload,
Discord notifications, and CDG/TXT packaging work regardless of encoding backend.
This reuses 100% of the existing karaoke_gen functionality.
2. LEGACY PATH (USE_NEW_ORCHESTRATOR=false):
Uses KaraokeFinalise from the karaoke_gen package. This path has divergent
behavior between GCE and local encoding - GCE encoding bypasses many features.
The orchestrator-based approach is preferred as it:
- Eliminates GCE vs local code path divergence
- Ensures all features work consistently
- Provides better testability and maintainability
Observability:
- All operations wrapped in tracing spans for Cloud Trace visibility
@@ -48,6 +49,12 @@ from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
logger = logging.getLogger(__name__)
# Feature flag for new orchestrator-based pipeline
# Set to True to use the new unified pipeline that works with any encoding backend
# Set to False to use the legacy KaraokeFinalise-based pipeline
USE_NEW_ORCHESTRATOR = os.environ.get("USE_NEW_ORCHESTRATOR", "true").lower() == "true"
# Loggers to capture for video worker
# Include the full module path to properly capture KaraokeFinalise logs
VIDEO_WORKER_LOGGERS = [
@@ -153,14 +160,17 @@ async def _encode_via_gce(
storage.download_file(gcs_path, local_path)
# Map to result keys expected by _upload_results
if "4K Lossless" in filename or "4k_lossless" in filename.lower():
local_files["final_video"] = local_path
elif "4K Lossy" in filename or "4k_lossy" in filename.lower():
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
filename_lower = filename.lower()
if "lossless 4k" in filename_lower:
if filename.endswith(".mkv"):
local_files["final_video_mkv"] = local_path
else:
local_files["final_video"] = local_path
elif "lossy 4k" in filename_lower:
local_files["final_video_lossy"] = local_path
elif "720p" in filename:
elif "720p" in filename_lower:
local_files["final_video_720p"] = local_path
elif filename.endswith(".mkv"):
local_files["final_video_mkv"] = local_path
job_log.info(f"Downloaded {len(local_files)} encoded files")
@@ -172,16 +182,247 @@ async def _encode_via_gce(
raise
async def generate_video(job_id: str) -> bool:
async def generate_video_orchestrated(job_id: str) -> bool:
"""
Generate final karaoke videos using KaraokeFinalise.
This sets up the directory structure and calls KaraokeFinalise.process()
which handles all encoding, packaging, and notifications.
Generate final karaoke videos using the new VideoWorkerOrchestrator.
This provides a unified pipeline that works with any encoding backend
(GCE or local), ensuring all features like YouTube upload, Discord
notifications, and CDG/TXT packaging work consistently.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
from backend.workers.video_worker_orchestrator import (
VideoWorkerOrchestrator,
create_orchestrator_config_from_job,
)
start_time = time.time()
job_manager = JobManager()
storage = StorageService()
# Create job logger for remote debugging
job_log = create_job_logger(job_id, "video")
# Log with structured markers for easy Cloud Logging queries
logger.info(f"[job:{job_id}] WORKER_START worker=video orchestrator=true")
# Set up log capture for KaraokeFinalise (still used by some services)
log_handler = setup_job_logging(job_id, "video", *VIDEO_WORKER_LOGGERS)
job = job_manager.get_job(job_id)
if not job:
logger.error(f"[job:{job_id}] Job not found")
return False
# Validate prerequisites
if not _validate_prerequisites(job):
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
return False
# Create temporary working directory
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_video_{job_id}_")
original_cwd = os.getcwd()
# Set up rclone config if needed (for legacy Dropbox upload path)
rclone_service = None
if getattr(job, 'organised_dir_rclone_root', None):
rclone_service = get_rclone_service()
if rclone_service.setup_rclone_config():
job_log.info("Rclone config loaded for Dropbox upload")
# Load YouTube credentials if needed
youtube_credentials = None
if getattr(job, 'enable_youtube_upload', False):
youtube_service = get_youtube_service()
if youtube_service.is_configured:
youtube_credentials = youtube_service.get_credentials_dict()
job_log.info("YouTube credentials loaded for video upload")
else:
job_log.warning("YouTube credentials not available - upload will be skipped")
try:
# Wrap entire worker in a tracing span
with job_span("video-worker-orchestrated", job_id, {"artist": job.artist, "title": job.title}) as root_span:
with job_logging_context(job_id):
job_log.info(f"Starting orchestrated video generation for {job.artist} - {job.title}")
logger.info(f"[job:{job_id}] Starting orchestrated video generation")
# Transition to GENERATING_VIDEO state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.GENERATING_VIDEO,
progress=70,
message="Preparing files for video generation"
)
# Download and set up files
base_name = f"{job.artist} - {job.title}"
with job_span("download-files", job_id):
job_log.info("Downloading files from GCS...")
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
job_log.info("All files downloaded successfully")
# Load style config for CDG styles
style_config = await load_style_config(job, storage, temp_dir)
cdg_styles = style_config.get_cdg_styles()
# Change to working directory
os.chdir(temp_dir)
# Create orchestrator config from job
config = create_orchestrator_config_from_job(
job=job,
temp_dir=temp_dir,
youtube_credentials=youtube_credentials,
cdg_styles=cdg_styles,
)
# Run the orchestrated pipeline
orchestrator = VideoWorkerOrchestrator(
config=config,
job_manager=job_manager,
storage=storage,
job_logger=job_log,
)
with job_span("orchestrator-run", job_id):
result = await orchestrator.run()
if not result.success:
raise Exception(result.error_message or "Orchestrator failed")
# Prepare distribution directory for native uploads
with job_span("distribution", job_id):
await _handle_native_distribution(
job_id=job_id,
job=job,
job_log=job_log,
job_manager=job_manager,
temp_dir=temp_dir,
result={
'brand_code': result.brand_code,
'youtube_url': result.youtube_url,
'final_video': result.final_video,
'final_video_lossy': result.final_video_lossy,
'final_video_720p': result.final_video_720p,
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
'dropbox_link': result.dropbox_link,
'gdrive_files': result.gdrive_files,
},
storage=storage,
)
# Upload generated files to GCS
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.PACKAGING,
progress=95,
message="Uploading final files"
)
with job_span("upload-results", job_id):
await _upload_results(job_id, job_manager, storage, temp_dir, {
'final_video': result.final_video,
'final_video_mkv': result.final_video_mkv,
'final_video_lossy': result.final_video_lossy,
'final_video_720p': result.final_video_720p,
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
'final_karaoke_txt_zip': result.final_karaoke_txt_zip,
})
# Mark job as complete
logger.info(f"[job:{job_id}] Video generation complete")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
progress=100,
message="Karaoke generation complete!"
)
# Store result metadata in job
job_manager.update_job(job_id, {
'state_data': {
**job.state_data,
'brand_code': result.brand_code,
'youtube_url': result.youtube_url,
'dropbox_link': result.dropbox_link,
'gdrive_files': result.gdrive_files,
}
})
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.brand_code or '')
logger.info(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=success duration={duration:.1f}s")
return True
except Exception as e:
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Video generation failed: {str(e)}",
error_details={"stage": "video_generation", "error": str(e)}
)
return False
finally:
# Restore original working directory
os.chdir(original_cwd)
# Remove log handler
for logger_name in VIDEO_WORKER_LOGGERS:
try:
logging.getLogger(logger_name).removeHandler(log_handler)
except Exception:
pass
# Cleanup rclone config file
if rclone_service:
rclone_service.cleanup()
# Cleanup temporary directory
if os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
logger.debug(f"Cleaned up temp directory: {temp_dir}")
async def generate_video(job_id: str) -> bool:
"""
Generate final karaoke videos.
Routes to either the new orchestrator-based pipeline or the legacy
KaraokeFinalise-based pipeline based on USE_NEW_ORCHESTRATOR flag.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
if USE_NEW_ORCHESTRATOR:
logger.info(f"[job:{job_id}] Using new orchestrator-based pipeline")
return await generate_video_orchestrated(job_id)
else:
logger.info(f"[job:{job_id}] Using legacy KaraokeFinalise pipeline")
return await generate_video_legacy(job_id)
async def generate_video_legacy(job_id: str) -> bool:
"""
Generate final karaoke videos using KaraokeFinalise (legacy path).
This is the original implementation that has divergent behavior between
GCE and local encoding paths. Kept for rollback purposes.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
@@ -870,29 +1111,63 @@ async def _prepare_distribution_directory(
log_progress(f"Downloaded {stem_key} to stems/")
# --- Download lyrics files to lyrics/ subfolder ---
# Use proper filenames matching local CLI output structure
# Common lyrics file types
lyrics_file_keys = ['lrc', 'ass', 'srt', 'txt', 'json', 'original_json', 'corrected_json']
for lyrics_key in lyrics_file_keys:
# Map job.files.lyrics keys to proper local filenames
# Format: (job_files_key, local_filename_pattern)
lyrics_file_mappings = [
# Karaoke output files
('lrc', f"{base_name} (Karaoke).lrc"),
('ass', f"{base_name} (Karaoke).ass"),
# Corrections and transcription files
('corrections', f"{base_name} (Lyrics Corrections).json"),
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
# Reference lyrics from various sources
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
('reference_spotify', f"{base_name} (Lyrics Spotify).txt"),
('reference_musixmatch', f"{base_name} (Lyrics Musixmatch).txt"),
('reference_lrclib', f"{base_name} (Lyrics Lrclib).txt"),
]
for lyrics_key, local_filename in lyrics_file_mappings:
if lyrics.get(lyrics_key):
# Determine proper extension
ext_map = {
'lrc': '.lrc',
'ass': '.ass',
'srt': '.srt',
'txt': '.txt',
'json': '.json',
'original_json': '_original.json',
'corrected_json': '_corrected.json',
}
ext = ext_map.get(lyrics_key, f'.{lyrics_key}')
dest_path = os.path.join(lyrics_dir, f"{base_name} (Karaoke){ext}")
dest_path = os.path.join(lyrics_dir, local_filename)
try:
storage.download_file(lyrics[lyrics_key], dest_path)
log_progress(f"Downloaded {lyrics_key} to lyrics/")
log_progress(f"Downloaded {lyrics_key} to lyrics/{local_filename}")
except Exception as e:
log_progress(f"Could not download {lyrics_key}: {e}")
# --- Download (With Vocals).mkv to lyrics/ subfolder ---
# Local CLI places the karaoke video with vocals in the lyrics folder
videos = job.file_urls.get('videos', {})
if videos.get('with_vocals'):
with_vocals_dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
try:
storage.download_file(videos['with_vocals'], with_vocals_dest)
log_progress(f"Downloaded with_vocals to lyrics/{base_name} (With Vocals).mkv")
except Exception as e:
log_progress(f"Could not download with_vocals to lyrics/: {e}")
# --- Download preview ASS files to lyrics/previews/ subfolder ---
# Preview files are stored in jobs/{job_id}/previews/ in GCS
previews_dir = os.path.join(lyrics_dir, "previews")
os.makedirs(previews_dir, exist_ok=True)
# List all preview files from GCS and download them
try:
preview_prefix = f"jobs/{job_id}/previews/"
preview_files = storage.list_files(preview_prefix)
for blob_name in preview_files:
if blob_name.endswith('.ass'):
filename = os.path.basename(blob_name)
dest_path = os.path.join(previews_dir, filename)
storage.download_file(blob_name, dest_path)
log_progress(f"Downloaded preview {filename} to lyrics/previews/")
except Exception as e:
log_progress(f"Could not download preview files: {e}")
log_progress("Distribution directory prepared with stems/ and lyrics/ subfolders")
@@ -0,0 +1,694 @@
"""
Video Worker Orchestrator.
Coordinates the video generation pipeline stages in a unified way,
regardless of whether encoding happens locally or on GCE.
This resolves the code path divergence where GCE encoding bypassed
features like YouTube upload, Discord notifications, and CDG/TXT packaging.
Pipeline stages:
1. Setup - Download files, prepare directories
2. Packaging - CDG/TXT generation (if enabled)
3. Encoding - GCE or Local via EncodingBackend interface
4. Organization - Brand code, folder structure
5. Distribution - YouTube, Dropbox, Google Drive uploads
6. Notification - Discord notifications
"""
import logging
import os
import time
from dataclasses import dataclass, field
from typing import Optional, Dict, Any, List
from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.tracing import job_span, add_span_event
logger = logging.getLogger(__name__)
@dataclass
class OrchestratorConfig:
"""Configuration for the video worker orchestrator."""
job_id: str
artist: str
title: str
# Input file paths (in temp_dir)
title_video_path: str
karaoke_video_path: str
instrumental_audio_path: str
end_video_path: Optional[str] = None
lrc_file_path: Optional[str] = None
title_jpg_path: Optional[str] = None
# Output directory
output_dir: str = ""
# Feature flags
enable_cdg: bool = False
enable_txt: bool = False
enable_youtube_upload: bool = False
# Service configurations
brand_prefix: Optional[str] = None
discord_webhook_url: Optional[str] = None
youtube_credentials: Optional[Dict[str, Any]] = None
youtube_description_template: Optional[str] = None
cdg_styles: Optional[Dict[str, Any]] = None
# Dropbox/GDrive configuration
dropbox_path: Optional[str] = None
gdrive_folder_id: Optional[str] = None
# Keep existing brand code (for re-processing)
keep_brand_code: Optional[str] = None
# Encoding backend preference
encoding_backend: str = "auto" # "auto", "local", "gce"
# Additional options
dry_run: bool = False
non_interactive: bool = True
@dataclass
class OrchestratorResult:
"""Result from the video worker orchestrator."""
success: bool
error_message: Optional[str] = None
# Generated files
final_video: Optional[str] = None # Lossless 4K MP4
final_video_mkv: Optional[str] = None # Lossless 4K MKV
final_video_lossy: Optional[str] = None # Lossy 4K MP4
final_video_720p: Optional[str] = None # Lossy 720p MP4
final_karaoke_cdg_zip: Optional[str] = None
final_karaoke_txt_zip: Optional[str] = None
# Organization
brand_code: Optional[str] = None
# Distribution results
youtube_url: Optional[str] = None
dropbox_link: Optional[str] = None
gdrive_files: Optional[Dict[str, str]] = field(default_factory=dict)
# Timing
encoding_time_seconds: Optional[float] = None
total_time_seconds: Optional[float] = None
class VideoWorkerOrchestrator:
"""
Orchestrates the video generation pipeline.
This class coordinates all stages of video generation in a unified way,
ensuring that features like YouTube upload and Discord notifications
work regardless of whether GCE or local encoding is used.
"""
def __init__(
self,
config: OrchestratorConfig,
job_manager: Optional[JobManager] = None,
storage: Optional[StorageService] = None,
job_logger: Optional[logging.Logger] = None,
):
"""
Initialize the orchestrator.
Args:
config: Orchestrator configuration
job_manager: Job manager for state updates (optional)
storage: Storage service for file downloads (optional)
job_logger: Job-specific logger (optional)
"""
self.config = config
self.job_manager = job_manager
self.storage = storage
self.job_log = job_logger or logging.getLogger(__name__)
# Services (lazy-loaded)
self._encoding_backend = None
self._packaging_service = None
self._youtube_service = None
self._discord_service = None
# State
self.result = OrchestratorResult(success=False)
def _get_encoding_backend(self):
"""Lazy-load the encoding backend."""
if self._encoding_backend is None:
from backend.services.encoding_interface import get_encoding_backend
self._encoding_backend = get_encoding_backend(
backend_type=self.config.encoding_backend,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._encoding_backend
def _get_packaging_service(self):
"""Lazy-load the packaging service."""
if self._packaging_service is None:
from backend.services.packaging_service import PackagingService
self._packaging_service = PackagingService(
cdg_styles=self.config.cdg_styles,
dry_run=self.config.dry_run,
non_interactive=self.config.non_interactive,
logger=self.job_log,
)
return self._packaging_service
def _get_youtube_service(self):
"""Lazy-load the YouTube upload service."""
if self._youtube_service is None:
from backend.services.youtube_upload_service import YouTubeUploadService
self._youtube_service = YouTubeUploadService(
credentials=self.config.youtube_credentials,
non_interactive=self.config.non_interactive,
server_side_mode=True,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._youtube_service
def _get_discord_service(self):
"""Lazy-load the Discord notification service."""
if self._discord_service is None:
from backend.services.discord_service import DiscordNotificationService
self._discord_service = DiscordNotificationService(
webhook_url=self.config.discord_webhook_url,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._discord_service
def _update_progress(self, status: JobStatus, progress: int, message: str):
"""Update job progress if job_manager is available."""
if self.job_manager:
self.job_manager.transition_to_state(
job_id=self.config.job_id,
new_status=status,
progress=progress,
message=message
)
async def run(self) -> OrchestratorResult:
"""
Run the full video generation pipeline.
Returns:
OrchestratorResult with generated files and metadata
"""
start_time = time.time()
try:
with job_span("orchestrator", self.config.job_id) as span:
self.job_log.info(f"Starting orchestrated video generation for {self.config.artist} - {self.config.title}")
# Stage 1: Packaging (CDG/TXT) - runs BEFORE encoding
# This depends on LRC file, not on encoded videos
if self.config.enable_cdg or self.config.enable_txt:
await self._run_packaging()
# Stage 2: Encoding
await self._run_encoding()
# Stage 3: Organization (brand code)
await self._run_organization()
# Stage 4: Distribution (YouTube, Dropbox, GDrive)
await self._run_distribution()
# Stage 5: Notifications (Discord)
await self._run_notifications()
self.result.success = True
self.result.total_time_seconds = time.time() - start_time
self.job_log.info(f"Orchestrated video generation complete in {self.result.total_time_seconds:.1f}s")
except Exception as e:
self.result.success = False
self.result.error_message = str(e)
self.result.total_time_seconds = time.time() - start_time
self.job_log.error(f"Orchestrated video generation failed: {e}")
logger.error(f"[job:{self.config.job_id}] Orchestration failed: {e}")
return self.result
async def _run_packaging(self):
"""Run the packaging stage (CDG/TXT generation)."""
self.job_log.info("Starting packaging stage (CDG/TXT)")
if not self.config.lrc_file_path or not os.path.isfile(self.config.lrc_file_path):
self.job_log.warning("No LRC file available, skipping CDG/TXT packaging")
return
base_name = f"{self.config.artist} - {self.config.title}"
packaging_service = self._get_packaging_service()
# Generate CDG package
if self.config.enable_cdg:
self.job_log.info("Generating CDG package")
try:
cdg_zip_path = os.path.join(
self.config.output_dir,
f"{base_name} (Final Karaoke CDG).zip"
)
mp3_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).mp3"
)
cdg_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).cdg"
)
zip_file, mp3_file, cdg_file = packaging_service.create_cdg_package(
lrc_file=self.config.lrc_file_path,
audio_file=self.config.instrumental_audio_path,
output_zip_path=cdg_zip_path,
artist=self.config.artist,
title=self.config.title,
output_mp3_path=mp3_path,
output_cdg_path=cdg_path,
)
self.result.final_karaoke_cdg_zip = zip_file
self.job_log.info(f"CDG package created: {zip_file}")
except Exception as e:
self.job_log.error(f"CDG generation failed: {e}")
# Don't fail the pipeline, CDG is optional
# Generate TXT package
if self.config.enable_txt:
self.job_log.info("Generating TXT package")
try:
# TXT package needs MP3 file (from CDG generation or create it)
mp3_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).mp3"
)
if not os.path.isfile(mp3_path):
self.job_log.info("MP3 not found, CDG must be enabled first for TXT")
return
txt_zip_path = os.path.join(
self.config.output_dir,
f"{base_name} (Final Karaoke TXT).zip"
)
zip_file, txt_file = packaging_service.create_txt_package(
lrc_file=self.config.lrc_file_path,
mp3_file=mp3_path,
output_zip_path=txt_zip_path,
)
self.result.final_karaoke_txt_zip = zip_file
self.job_log.info(f"TXT package created: {zip_file}")
except Exception as e:
self.job_log.error(f"TXT generation failed: {e}")
# Don't fail the pipeline, TXT is optional
async def _run_encoding(self):
"""Run the encoding stage."""
self.job_log.info("Starting encoding stage")
self._update_progress(JobStatus.ENCODING, 75, "Encoding videos")
encoding_backend = self._get_encoding_backend()
self.job_log.info(f"Using encoding backend: {encoding_backend.name}")
# Build encoding input
from backend.services.encoding_interface import EncodingInput
from backend.config import settings
# For GCE encoding, we need to provide GCS paths
gcs_bucket = settings.gcs_bucket_name
input_gcs_path = f"gs://{gcs_bucket}/jobs/{self.config.job_id}/"
output_gcs_path = f"gs://{gcs_bucket}/jobs/{self.config.job_id}/finals/"
encoding_input = EncodingInput(
title_video_path=self.config.title_video_path,
karaoke_video_path=self.config.karaoke_video_path,
instrumental_audio_path=self.config.instrumental_audio_path,
end_video_path=self.config.end_video_path,
artist=self.config.artist,
title=self.config.title,
brand_code=self.config.keep_brand_code,
output_dir=self.config.output_dir,
options={
"job_id": self.config.job_id,
"input_gcs_path": input_gcs_path,
"output_gcs_path": output_gcs_path,
},
)
# Run encoding
with job_span("encoding", self.config.job_id) as span:
add_span_event("encoding_started", {"backend": encoding_backend.name})
output = await encoding_backend.encode(encoding_input)
add_span_event("encoding_completed", {
"success": output.success,
"duration": output.encoding_time_seconds or 0
})
if not output.success:
raise Exception(f"Encoding failed: {output.error_message}")
# Store results - for GCE backend, these are GCS blob paths that need to be downloaded
self.result.final_video = output.lossless_4k_mp4_path
self.result.final_video_mkv = output.lossless_mkv_path
self.result.final_video_lossy = output.lossy_4k_mp4_path
self.result.final_video_720p = output.lossy_720p_mp4_path
self.result.encoding_time_seconds = output.encoding_time_seconds
# For GCE encoding, download the encoded files from GCS to local directory
# This is required for YouTube upload and other local file operations
if encoding_backend.name == "gce" and self.storage:
await self._download_gce_encoded_files(output)
self.job_log.info(f"Encoding complete ({encoding_backend.name}) in {output.encoding_time_seconds:.1f}s")
async def _download_gce_encoded_files(self, output):
"""
Download GCE-encoded files from GCS to the local output directory.
GCE encoding stores files in GCS and returns blob paths like:
'jobs/{job_id}/finals/Artist - Title (Final Karaoke Lossless 4k).mp4'
This method downloads those files locally so that subsequent stages
(YouTube upload, etc.) can access them as local files.
Args:
output: EncodingOutput from the GCE backend with GCS blob paths
"""
self.job_log.info("Downloading GCE-encoded files from GCS")
# Map of result attributes to download
file_mappings = [
('lossless_4k_mp4_path', 'final_video'),
('lossless_mkv_path', 'final_video_mkv'),
('lossy_4k_mp4_path', 'final_video_lossy'),
('lossy_720p_mp4_path', 'final_video_720p'),
]
downloaded_count = 0
for output_attr, result_attr in file_mappings:
gcs_path = getattr(output, output_attr, None)
if not gcs_path:
continue
# Extract filename from GCS path
filename = os.path.basename(gcs_path)
local_path = os.path.join(self.config.output_dir, filename)
try:
self.job_log.info(f"Downloading {filename} from GCS")
self.storage.download_file(gcs_path, local_path)
# Update the result to point to local file
setattr(self.result, result_attr, local_path)
downloaded_count += 1
self.job_log.info(f"Downloaded {filename} to {local_path}")
except Exception as e:
self.job_log.error(f"Failed to download {filename}: {e}")
# Clear the result attribute so downstream doesn't try to use invalid GCS path
setattr(self.result, result_attr, None)
# Don't fail - some formats might not be generated
self.job_log.info(f"Downloaded {downloaded_count} encoded files from GCS")
async def _run_organization(self):
"""Run the organization stage (brand code generation)."""
self.job_log.info("Starting organization stage")
# Use existing brand code if provided
if self.config.keep_brand_code:
self.result.brand_code = self.config.keep_brand_code
self.job_log.info(f"Using preserved brand code: {self.result.brand_code}")
return
# Generate brand code from Dropbox if configured
if self.config.dropbox_path and self.config.brand_prefix:
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if dropbox.is_configured:
brand_code = dropbox.get_next_brand_code(
self.config.dropbox_path,
self.config.brand_prefix
)
self.result.brand_code = brand_code
self.job_log.info(f"Generated brand code: {brand_code}")
else:
self.job_log.warning("Dropbox not configured, skipping brand code generation")
except Exception as e:
self.job_log.error(f"Brand code generation failed: {e}")
# Don't fail - brand code is optional
async def _run_distribution(self):
"""Run the distribution stage (YouTube, Dropbox, GDrive uploads)."""
self.job_log.info("Starting distribution stage")
self._update_progress(JobStatus.PACKAGING, 90, "Uploading files")
# YouTube upload
if self.config.enable_youtube_upload and self.config.youtube_credentials:
await self._upload_to_youtube()
# Dropbox upload
if self.config.dropbox_path and self.config.brand_prefix:
await self._upload_to_dropbox()
# Google Drive upload
if self.config.gdrive_folder_id:
await self._upload_to_gdrive()
async def _upload_to_youtube(self):
"""Upload video to YouTube."""
self.job_log.info("Uploading to YouTube")
# Find the best video file to upload (prefer MKV for FLAC audio, then lossless MP4)
video_to_upload = None
if self.result.final_video_mkv and os.path.isfile(self.result.final_video_mkv):
video_to_upload = self.result.final_video_mkv
elif self.result.final_video and os.path.isfile(self.result.final_video):
video_to_upload = self.result.final_video
elif self.result.final_video_lossy and os.path.isfile(self.result.final_video_lossy):
video_to_upload = self.result.final_video_lossy
if not video_to_upload:
self.job_log.warning("No video file available for YouTube upload")
return
try:
youtube_service = self._get_youtube_service()
# Build video title
title = f"{self.config.artist} - {self.config.title} (Karaoke)"
# Build description
description = self.config.youtube_description_template or ""
if self.result.brand_code:
description = f"{description}\n\nBrand Code: {self.result.brand_code}".strip()
# Upload
video_id, video_url = youtube_service.upload_video(
video_path=video_to_upload,
title=title,
description=description,
thumbnail_path=self.config.title_jpg_path,
tags=["karaoke", self.config.artist, self.config.title],
replace_existing=True, # Server-side always replaces
)
if video_url:
self.result.youtube_url = video_url
self.job_log.info(f"Uploaded to YouTube: {video_url}")
else:
self.job_log.warning("YouTube upload did not return a URL")
except Exception as e:
self.job_log.error(f"YouTube upload failed: {e}")
# Don't fail the pipeline - YouTube is optional
async def _upload_to_dropbox(self):
"""Upload files to Dropbox."""
self.job_log.info("Uploading to Dropbox")
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if not dropbox.is_configured:
self.job_log.warning("Dropbox not configured, skipping upload")
return
base_name = f"{self.config.artist} - {self.config.title}"
folder_name = f"{self.result.brand_code or 'TRACK-0000'} - {base_name}"
remote_folder = f"{self.config.dropbox_path}/{folder_name}"
# Upload entire output directory
dropbox.upload_folder(self.config.output_dir, remote_folder)
# Create sharing link
try:
sharing_link = dropbox.create_shared_link(remote_folder)
self.result.dropbox_link = sharing_link
self.job_log.info(f"Dropbox sharing link: {sharing_link}")
except Exception as e:
self.job_log.warning(f"Failed to create Dropbox sharing link: {e}")
self.job_log.info("Dropbox upload complete")
except Exception as e:
self.job_log.error(f"Dropbox upload failed: {e}")
# Don't fail the pipeline - Dropbox is optional
async def _upload_to_gdrive(self):
"""Upload files to Google Drive."""
self.job_log.info("Uploading to Google Drive")
try:
from backend.services.gdrive_service import get_gdrive_service
gdrive = get_gdrive_service()
if not gdrive.is_configured:
self.job_log.warning("Google Drive not configured, skipping upload")
return
base_name = f"{self.config.artist} - {self.config.title}"
brand_code = self.result.brand_code or f"{self.config.brand_prefix or 'TRACK'}-0000"
# Map result files to expected keys
output_files = {
'final_karaoke_lossy_mp4': self.result.final_video_lossy,
'final_karaoke_lossy_720p_mp4': self.result.final_video_720p,
'final_karaoke_cdg_zip': self.result.final_karaoke_cdg_zip,
}
uploaded = gdrive.upload_to_public_share(
root_folder_id=self.config.gdrive_folder_id,
brand_code=brand_code,
base_name=base_name,
output_files=output_files,
)
self.result.gdrive_files = uploaded
self.job_log.info(f"Google Drive upload complete: {len(uploaded)} files")
except Exception as e:
self.job_log.error(f"Google Drive upload failed: {e}")
# Don't fail the pipeline - GDrive is optional
async def _run_notifications(self):
"""Run the notifications stage (Discord)."""
self.job_log.info("Starting notifications stage")
if not self.config.discord_webhook_url:
self.job_log.debug("No Discord webhook configured, skipping notification")
return
if not self.result.youtube_url:
self.job_log.info("No YouTube URL available, skipping Discord notification")
return
try:
discord_service = self._get_discord_service()
discord_service.post_video_notification(self.result.youtube_url)
self.job_log.info("Discord notification sent")
except Exception as e:
self.job_log.error(f"Discord notification failed: {e}")
# Don't fail the pipeline - notifications are optional
def create_orchestrator_config_from_job(
job,
temp_dir: str,
youtube_credentials: Optional[Dict[str, Any]] = None,
cdg_styles: Optional[Dict[str, Any]] = None,
) -> OrchestratorConfig:
"""
Create an OrchestratorConfig from a job object.
This is a helper function to bridge the existing job structure
with the new orchestrator configuration.
Args:
job: Job object from Firestore
temp_dir: Temporary directory with downloaded files
youtube_credentials: Pre-loaded YouTube credentials
cdg_styles: CDG style configuration
Returns:
OrchestratorConfig for the orchestrator
"""
base_name = f"{job.artist} - {job.title}"
# Determine instrumental file path
instrumental_selection = job.state_data.get('instrumental_selection', 'clean')
existing_instrumental = getattr(job, 'existing_instrumental_gcs_path', None)
if existing_instrumental:
ext = Path(existing_instrumental).suffix.lower()
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
else:
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
return OrchestratorConfig(
job_id=job.job_id,
artist=job.artist,
title=job.title,
# Input files
title_video_path=os.path.join(temp_dir, f"{base_name} (Title).mov"),
karaoke_video_path=os.path.join(temp_dir, f"{base_name} (With Vocals).mov"),
instrumental_audio_path=instrumental_path,
end_video_path=os.path.join(temp_dir, f"{base_name} (End).mov"),
lrc_file_path=os.path.join(temp_dir, f"{base_name} (Karaoke).lrc"),
title_jpg_path=os.path.join(temp_dir, f"{base_name} (Title).jpg"),
# Output directory
output_dir=temp_dir,
# Feature flags
enable_cdg=getattr(job, 'enable_cdg', False),
enable_txt=getattr(job, 'enable_txt', False),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
# Service configurations
brand_prefix=getattr(job, 'brand_prefix', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_credentials=youtube_credentials,
youtube_description_template=getattr(job, 'youtube_description_template', None),
cdg_styles=cdg_styles,
# Dropbox/GDrive
dropbox_path=getattr(job, 'dropbox_path', None),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
# Keep existing brand code
keep_brand_code=getattr(job, 'keep_brand_code', None),
# Encoding backend - auto selects GCE if available
encoding_backend="auto",
# Server-side defaults
dry_run=False,
non_interactive=True,
)
+104
View File
@@ -383,6 +383,110 @@ Authorization: Bearer ADMIN_TOKEN
Admins can delete any job. Regular users can only delete their own jobs.
### Audio Search Management (Admin)
#### List Audio Searches
```http
GET /api/admin/audio-searches
GET /api/admin/audio-searches?limit=50&status_filter=awaiting_audio_selection
Authorization: Bearer ADMIN_TOKEN
```
Returns jobs with cached audio search results. Useful for:
- Monitoring search activity
- Identifying stale cached results (YouTube-only when lossless should be available)
- Clearing cache for specific jobs
Response:
```json
{
"jobs": [
{
"job_id": "abc123",
"status": "awaiting_audio_selection",
"user_email": "user@example.com",
"audio_search_artist": "Artist Name",
"audio_search_title": "Song Title",
"created_at": "2026-01-03T12:00:00Z",
"results_count": 5,
"has_lossless": false,
"providers": ["YouTube"],
"results_summary": [...]
}
],
"total": 1
}
```
#### Clear Audio Search Cache
```http
POST /api/admin/audio-searches/{job_id}/clear-cache
Authorization: Bearer ADMIN_TOKEN
```
Clears cached search results and resets job to `pending` status, allowing a new search.
Use when:
- Cached results are stale (e.g., flacfetch was updated with new providers)
- User wants to search again
- Results appear incomplete (YouTube-only when lossless should exist)
Response:
```json
{
"status": "success",
"job_id": "abc123",
"message": "Cleared 5 cached search results. Job reset to pending. Flacfetch cache also cleared.",
"previous_status": "awaiting_audio_selection",
"new_status": "pending",
"results_cleared": 5,
"flacfetch_cache_cleared": true,
"flacfetch_error": null
}
```
Note: This endpoint now also clears the flacfetch GCS cache for the artist/title combination, ensuring the next search hits trackers fresh.
#### Clear All Flacfetch Cache
```http
DELETE /api/admin/cache
Authorization: Bearer ADMIN_TOKEN
```
Clears the entire flacfetch search cache (GCS-backed). Use when flacfetch has been updated and you want all subsequent searches to use fresh tracker results.
Response:
```json
{
"status": "success",
"message": "Cleared 15 cache entries from flacfetch.",
"deleted_count": 15
}
```
#### Get Flacfetch Cache Stats
```http
GET /api/admin/cache/stats
Authorization: Bearer ADMIN_TOKEN
```
Returns statistics about the flacfetch search cache.
Response:
```json
{
"count": 42,
"total_size_bytes": 128000,
"oldest_entry": "2025-12-15T10:30:00Z",
"newest_entry": "2026-01-03T15:45:00Z",
"configured": true
}
```
## Rate Limits
No rate limits currently implemented.
+57 -2
View File
@@ -31,7 +31,8 @@
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
(*) GCE Encoding Worker: C4-standard-8 VM with Intel Granite Rapids for
high-performance FFmpeg encoding. Optional - falls back to Cloud Run.
high-performance FFmpeg encoding. Used for both final video encoding and
preview video generation. Uses LocalEncodingService via GCS wheel deployment.
```
## Processing Pipeline
@@ -57,7 +58,7 @@
▼
4. HUMAN REVIEW (AWAITING_REVIEW)
• React UI for lyrics correction
• Preview video generation
• Preview video generation (optionally via GCE for speed)
• Submit corrections
▼
5. VIDEO RENDERING
@@ -155,6 +156,60 @@ LyricsTranscriber LyricsTranscriber
| Genius | Reference lyrics | Yes |
| YouTube API | Video upload | Optional |
## Firestore Collections
karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses separate Firestore collections:
| Collection | Purpose | Key Fields |
|------------|---------|------------|
| `gen_users` | karaoke-gen user accounts | email, credits, role, is_active |
| `jobs` | Karaoke generation jobs | job_id, user_email, status, state_data |
| `jobs/{job_id}/logs` | Worker log entries (subcollection) | timestamp, level, worker, message, ttl_expiry |
| `sessions` | Magic link auth sessions | user_email, token, expires_at |
| `magic_links` | Passwordless auth tokens | email, token, expires_at, used |
| `beta_feedback` | Beta program feedback | user_email, ratings, comments |
**Note**: The `users` collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
**Worker Logs**: Stored in subcollection (`jobs/{job_id}/logs`) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: `USE_LOG_SUBCOLLECTION` (default: true). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
## Video Worker Orchestrator
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
```text
┌─────────────────────────────────────────────────────────────┐
│ VideoWorkerOrchestrator │
│ │
│ Stage 1: PACKAGING Stage 2: ENCODING │
│ ├─ CDG file generation ├─ Local (FFmpeg) │
│ └─ TXT lyrics export └─ GCE (remote VM) │
│ │
│ Stage 3: ORGANIZATION Stage 4: DISTRIBUTION │
│ ├─ Final path setup ├─ Dropbox upload │
│ └─ File organization ├─ Google Drive upload │
│ └─ YouTube upload │
│ │
│ Stage 5: NOTIFICATIONS │
│ └─ Discord webhooks │
└─────────────────────────────────────────────────────────────┘
```
**Key Design Decision**: The orchestrator coordinates stages rather than embedding logic in monolithic functions. This:
- Ensures all features work with both local and GCE encoding
- Makes each stage independently testable (139 new tests)
- Allows feature flag rollback via `USE_NEW_ORCHESTRATOR` env var
**Service Modules**:
| Module | Purpose |
|--------|---------|
| `youtube_upload_service.py` | YouTube OAuth flow and upload |
| `discord_service.py` | Discord webhook notifications |
| `packaging_service.py` | CDG/TXT package generation |
| `local_encoding_service.py` | FFmpeg encoding (Cloud Run) |
| `encoding_interface.py` | Abstract interface + GCE backend |
| `video_worker_orchestrator.py` | Stage coordination |
## Tech Stack
- **Backend**: FastAPI, Python 3.12, Cloud Run
+25 -5
View File
@@ -110,17 +110,37 @@ Push to `main` triggers GitHub Actions:
3. Deploy to Cloud Run
4. Deploy frontend to Cloudflare Pages
### Manual Backend Deploy
### Infrastructure Changes (Pulumi)
**All infrastructure changes must go through PRs.** The CI pipeline automatically deploys Pulumi changes when PRs merge to main.
```bash
# Preview infrastructure changes
# Preview changes locally (requires GCP auth)
cd infrastructure
pulumi preview
pulumi preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
# Apply changes
pulumi up
# DO NOT run `pulumi up` locally - let CI handle deployment
# CI uses --skip-preview due to limited service account permissions
```
**What's managed by Pulumi:**
- Firestore database and indexes
- GCS buckets
- Cloud Run service configuration
- Cloud Tasks queues
- Secret Manager secrets
- Service accounts and IAM bindings
- Workload Identity Federation
- GCE instances (GitHub runners, encoding worker)
**Workflow:**
1. Make changes in `infrastructure/__main__.py`
2. Run `pulumi preview` locally to verify
3. Create PR and merge
4. CI deploys automatically
See `docs/LESSONS-LEARNED.md` for Pulumi CI gotchas.
### Manual Frontend Deploy
```bash
+11 -6
View File
@@ -14,18 +14,20 @@
## Test Requirements
### 1. New User Registration (MailSlurp)
### 1. New User Registration (testmail.app)
- [ ] Navigate to landing page (`gen.nomadkaraoke.com`)
- [ ] Click "Sign In" or "Get Started" button
- [ ] Create a NEW MailSlurp inbox (fresh email address for each test run)
- [ ] Generate a unique test email address (testmail.app namespace)
- [ ] Enter the test email in the beta enrollment form
- [ ] Submit the form
- [ ] Wait for welcome/magic-link email in MailSlurp inbox
- [ ] Wait for welcome/magic-link email via testmail.app API
- [ ] Extract the magic link from the email
- [ ] Click/navigate to the magic link to authenticate
- [ ] Verify user is logged in and has credits
**Note**: When `use_test_token=true` workflow input is set, steps 1-9 above are skipped and the test uses a pre-authenticated token instead.
### 2. Create Karaoke Job (UI Only)
- [ ] Navigate to the karaoke creation page
@@ -118,8 +120,10 @@
### Environment Variables
```bash
MAILSLURP_API_KEY=xxx # Required - for email testing
# No pre-auth tokens - test handles full signup
TESTMAIL_NAMESPACE=xxx # Required - for email testing (testmail.app)
TESTMAIL_API_KEY=xxx # Required - for email testing (testmail.app)
E2E_TEST_TOKEN=xxx # Optional - skip signup, use existing account
# When use_test_token=true, skips email signup and uses pre-authenticated token
```
### Test Song
@@ -160,7 +164,8 @@ Log detailed progress at each step with timestamps for debugging.
### Secrets Required
- `MAILSLURP_API_KEY` - For email testing
- `TESTMAIL_API_KEY` - For email testing (testmail.app)
- `TESTMAIL_NAMESPACE` - testmail.app namespace for unique email addresses
### Workflow Steps
+665
View File
@@ -4,6 +4,25 @@ Key insights for future AI agents working on this codebase.
## Architecture Decisions
### Separate Collections for Multi-App Projects
**Problem**: karaoke-gen and karaoke-decide shared the same `users` Firestore collection with incompatible schemas. karaoke-gen users have `credits`, `role`, `is_active`; karaoke-decide users have `user_id`, `is_guest`, `quiz_*` fields.
**Symptoms**:
- Admin dashboard showed users from both apps mixed together
- Potential for data corruption if wrong app writes to wrong user
- Hard to run collection-wide queries (indexes, aggregations) for one app
**Solution**: Migrate karaoke-gen to dedicated `gen_users` collection. Created migration script that:
1. Identified karaoke-gen users by schema (has `credits` + `role` + `is_active`, lacks `user_id` + `is_guest`)
2. Copied 42 users to `gen_users` with same document IDs (email addresses)
3. Updated Pulumi indexes from `users` to `gen_users`
4. Left karaoke-decide's `users` collection untouched
**Key insight**: When multiple apps share a GCP project, use distinct collection names from day one. Prefixing collections with app name (e.g., `gen_users`, `decide_users`) avoids schema conflicts and makes ownership clear.
**Migration script**: `scripts/migrate_users_to_gen_users.py` - supports `--dry-run` and `--delete` flags.
### LyricsTranscriber: Library, Not Server
**Problem**: LyricsTranscriber's `ReviewServer` blocks waiting for human input - incompatible with async cloud architecture.
@@ -203,6 +222,52 @@ escaped = filename.replace("'", "\\'")
query = f"name='{escaped}' and '{parent_id}' in parents"
```
### FFmpeg Filter Path Escaping (subprocess without shell)
**Problem**: Song titles with apostrophes (e.g., "I'm With You") cause FFmpeg's ASS subtitle filter to fail because paths aren't escaped correctly.
```python
# BAD - FFmpeg strips the apostrophe, looks for wrong path
ass_filter = f"ass=./Avril Lavigne - I'm With You/cache/temp.ass"
# FFmpeg error: Could not create a libass track when reading file './Avril Lavigne - Im With You/cache/temp.ass'
```
**Wrong approach #1**: Simple backslash escaping like `\'` doesn't work because FFmpeg's filter parser requires more escaping.
**Wrong approach #2**: Shell-style escaping `'\''` (end quote, escaped quote, start quote) only works when passing through a shell. When using Python `subprocess` with a command list, there's no shell - arguments go directly to FFmpeg, so shell escaping patterns don't work.
**Solution**: Use FFmpeg filter escaping (for subprocess without shell):
```python
def _escape_ffmpeg_filter_path(path: str) -> str:
"""Escape a path for FFmpeg filter expressions (for subprocess without shell).
FFmpeg filter escaping rules:
- Backslashes: double them (\ -> \\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\\')
- Spaces: escape with backslash ( -> \ )
"""
# First escape existing backslashes (\ -> \\)
escaped = path.replace("\\", "\\\\")
# Escape single quotes (' -> \\\')
# In Python source: 6 backslashes + apostrophe = 3 backslashes + apostrophe in string
escaped = escaped.replace("'", "\\\\\\'")
# Escape spaces
escaped = escaped.replace(" ", "\\ ")
return escaped
# Result: "I'm With You" becomes "I\\\'m\ With\ You"
# FFmpeg correctly interprets this as: I'm With You
```
**Symptoms**:
- Video generation fails with "Could not create a libass track"
- Error path is subtly different from the path in your code (missing apostrophe)
- Works fine for songs without special characters
**Key insight**: FFmpeg filter escaping is different from shell escaping. When using `subprocess.run(cmd_list)` (no `shell=True`), there's no shell to interpret the arguments. FFmpeg receives the strings directly and applies its own filter expression parsing rules. The `'\''` shell idiom does NOT work in this context.
**Lesson**: When debugging FFmpeg filter path issues, check whether you're using shell or subprocess. For subprocess without shell, escape spaces with `\ ` and apostrophes with `\\\'` (3 backslashes + quote in the actual string).
### Duplicate Code Leads to Inconsistent Behavior
**Problem**: Multiple endpoints computing the same defaults independently. `audio_search.py` had `effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix` but `file_upload.py` was missing it for some endpoints.
@@ -228,6 +293,43 @@ def _get_effective_distribution_settings(...) -> EffectiveDistributionSettings:
**Lesson**: When you see the same logic repeated in multiple places, refactor into a helper function immediately. Otherwise bugs will be fixed in one place but not others.
### Python `or` Operator Can't Override True Defaults with False
**Problem**: Using Python's `or` operator for default logic prevents explicit `false` from overriding a `true` server default.
```python
# BAD - can't explicitly disable if server default is true
effective_value = body.enable_feature or settings.default_enable_feature
# When body.enable_feature=False and settings.default=True:
# False or True = True (ignores explicit False!)
```
**Symptoms**:
- Feature can be enabled but never disabled via API
- Works fine when server default is `false` but breaks when default is `true`
- No errors - just ignores the explicit `false` value
**Solution**: Use `Optional[bool] = None` with explicit `is not None` check:
```python
# Request model
enable_feature: Optional[bool] = None # None = use server default
# Default logic
effective_value = (
body.enable_feature
if body.enable_feature is not None
else settings.default_enable_feature
)
```
This allows three states:
- `None` (not specified) → use server default
- `True` → explicitly enable
- `False` → explicitly disable (overrides server default)
**Lesson**: When a boolean field needs to support "use server default" behavior, use `Optional[bool] = None` instead of `bool = False`. The `or` operator conflates "not specified" with "explicitly false".
### Field Name Mismatches Between Endpoints
**Problem**: Different endpoints setting different field names for the same logical value, where consumers check only one field name.
@@ -569,6 +671,43 @@ GitHub-hosted runners have ~14GB disk, which caused failures during Docker build
See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
### Pulumi CI Deployments Need Skip-Preview for Limited Service Accounts
**Problem**: Running `pulumi up --refresh` in CI requires the service account to have read permissions on ALL managed GCP resources. Pulumi refreshes state by querying each resource, and permission errors cause the deployment to fail before any changes are applied.
**Symptoms**:
```
Error when reading or editing Resource "project" with IAM Member: Role "roles/...":
Error retrieving IAM policy for project "...": googleapi: Error 403: The caller does not have permission
```
**Root cause**: `--refresh` (default behavior) makes read API calls to verify current state matches Pulumi's state. This requires permissions like:
- `resourcemanager.projects.getIamPolicy` (for IAM bindings)
- `secretmanager.secrets.get` (for secrets)
- `cloudtasks.queues.get` (for Cloud Tasks)
- `iam.workloadIdentityPools.get` (for Workload Identity)
- And many more for each resource type managed
**Solution**: Use `--skip-preview` for CI deployments where Pulumi state is authoritative:
```yaml
# In CI workflow
pulumi up --yes --skip-preview --stack org/project/stack
```
**Why this is safe**:
- Infrastructure is only modified through Pulumi (single source of truth)
- Preview/refresh is still useful during local development with broader permissions
- Reduces permissions surface for CI service account (principle of least privilege)
**Alternative** (if refresh is required): Grant the service account `roles/editor` or individual read roles for each resource type. This is less secure but allows drift detection.
**Lesson**: When setting up Pulumi in CI, either:
1. Grant broad read permissions and keep `--refresh` (detects drift)
2. Use minimal permissions and `--skip-preview` (trusts Pulumi state)
Choose based on your threat model. For most projects, trusting Pulumi state is acceptable since manual console changes are rare.
### Docker Disk Space Management on Self-Hosted Runners
**Problem**: Even with 200GB SSDs, self-hosted runners can fill up with Docker images. Each CI build creates new dangling images (~15GB each), and without aggressive cleanup, disk fills in days.
@@ -614,6 +753,95 @@ See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
- Pre-job checks provide fail-fast behavior with clear error messages
- `docker builder prune -af` catches buildkit cache that `system prune` misses
### Fonts in Docker for Video Rendering
**Problem**: Video rendering showed replacement characters (squares with question marks) for special Unicode symbols like `♪` in intro/instrumental screens.
**Root cause**: The `python:3.11-slim` Docker base image has NO fonts installed. FFmpeg/libass falls back to whatever it can find, which often lacks support for musical symbols, emoji, or non-Latin scripts.
**Symptoms**:
- `♪ INTRO (19 seconds) ♪` displayed as `□ INTRO (19 seconds) □`
- Unicode characters render as tofu (replacement boxes)
- Only appears in cloud-generated videos, not local dev (which uses system fonts)
**Solution**: Install comprehensive Noto fonts in the Docker base image:
```dockerfile
RUN apt-get update && apt-get install -y \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
```
**Package breakdown**:
- `fonts-noto-core` (~10MB): Latin, Greek, Cyrillic, musical symbols
- `fonts-noto-cjk` (~150MB): Chinese, Japanese, Korean
- `fonts-noto-extra` (~50MB): Arabic, Hebrew, Thai, Hindi, other scripts
- `fonts-noto-color-emoji` (~10MB): Color emoji (limited libass support)
- `fontconfig`: Font configuration system (required for `fc-cache`)
**Note on color emoji**: FFmpeg's libass subtitle renderer doesn't fully support color emoji fonts - it renders text as vectors and can't display color bitmap glyphs. Color emoji appears as monochrome outlines or may not render at all in ASS subtitles.
**Lesson**: When building Docker images for video/subtitle rendering, install fonts explicitly. The default slim images have nothing, and font fallback behavior differs from local development machines.
### Reuse LLM Model Instances Across Operations
**Problem**: Creating a new LLM model instance for each operation in a loop causes massive overhead. Each instance creation triggers model initialization, connection establishment, and optional "warm-up" calls - adding 2+ seconds per operation.
**Example**: Agentic lyrics correction was creating a new `AgenticCorrector` for each gap (lyric segment needing correction):
```python
# BAD - 31 gaps × 2s initialization = 62s overhead
for gap in gaps_to_correct:
agent = AgenticCorrector.from_model(model_id, ...) # New instance every time!
agent.correct_gap(gap)
```
**Symptoms**:
- Operations that should take ~1 minute take 5+ minutes
- Logs show "Initializing model..." or "Warming up..." repeatedly
- Each loop iteration has a consistent ~2s overhead before actual work
- Total time scales linearly with loop count, not with actual work
**Solution**: Create the model instance ONCE before the loop, then reuse:
```python
# GOOD - 1 initialization, reused for all gaps
agent = AgenticCorrector.from_model(model_id, ...) # Create once
for gap in gaps_to_correct:
agent.correct_gap(gap) # Reuse existing instance
```
**Additional optimization**: If operations are independent, process them in parallel:
```python
# BETTER - parallel processing with shared model
agent = AgenticCorrector.from_model(model_id, ...)
with ThreadPoolExecutor(max_workers=5) as executor:
futures = {executor.submit(agent.correct_gap, gap): gap for gap in gaps}
for future in as_completed(futures):
result = future.result()
```
**Results**:
- Before: 31 gaps → ~5 minutes (2s init + 3s work per gap)
- After: 20 gaps → ~55 seconds (1 init + parallel processing)
- Speedup: ~5-6x
**Why warm-up is often unnecessary**: The warm-up pattern (sending a trivial request to "prime" the model) made sense for cold-start scenarios in older systems. Modern LLM APIs:
- Don't have cold-start penalties per request
- Handle connection pooling internally
- May even rate-limit warm-up calls
If you're already making real requests, warm-up adds latency without benefit.
**Key insight**: LLM client instances are designed for reuse. They maintain connection pools, authentication state, and caching. Creating a new instance per request throws away these benefits.
**Configuration**: Added `AGENTIC_MAX_PARALLEL_GAPS` env var (default: 5) to control parallelism. Higher values increase throughput but may hit rate limits.
## Performance Observations
| Operation | Duration |
@@ -702,6 +930,48 @@ export function getToken() {
**Lesson**: For values that can change (auth tokens, user preferences), read fresh from storage on each access rather than caching in module scope.
### Zustand Module-Level Initialization Doesn't Re-run on Client Navigation
**Problem**: Module-level initialization code in zustand stores only runs once when the module is first imported. Client-side navigation (e.g., `router.push()`) doesn't reload modules, so initialization code doesn't re-run.
**Example bug**: Auth module had initialization that fetched user data if a token existed:
```typescript
// auth.ts - module level (runs once at import)
if (typeof window !== 'undefined') {
const token = getAccessToken()
if (token) {
useAuth.getState().fetchUser() // Only runs on initial page load!
}
}
```
After beta enrollment:
1. User lands on `/` (landing page) - auth.ts imported, no token yet, `fetchUser()` not called
2. User completes enrollment, token stored via `setAccessToken()`
3. `router.push('/app')` - client-side navigation, modules already loaded
4. Auth module initialization doesn't re-run - `user` stays `null`
5. UI shows "Login" button instead of user credits
**Symptoms**:
- Feature works after page reload but not after client-side navigation
- Zustand store has correct persisted data (token) but not computed data (user profile)
- E2E tests that use reload pass; tests that use `router.push` fail
**Solution**: Components that need fresh data should trigger fetches in useEffect:
```typescript
// In /app page component
const { user, fetchUser } = useAuth()
useEffect(() => {
const token = getAccessToken()
if (token && !user) {
fetchUser() // Ensure user data is loaded regardless of how we got here
}
}, [user, fetchUser])
```
**Lesson**: Don't rely on module-level initialization for data that might need to be fetched after client-side navigation. Use useEffect in components to ensure data is loaded when needed.
### GCP Organization Policies Block Service Account Keys
**Problem**: GCP organization policies can block service account key creation even when you have Owner access to the project.
@@ -770,6 +1040,37 @@ result = job.state_data['search_results'][selection_index]
**Lesson**: Design for stateless request handling. Any state that must survive between requests must be persisted externally.
### Firestore Cache Has No Automatic Invalidation
**Problem**: Audio search results are cached in Firestore (`job.state_data.audio_search_results`) with no expiration or invalidation. When a library like flacfetch is updated with new providers or bug fixes, existing cached results remain stale forever.
**Symptoms**:
- Job shows only 5 YouTube results when CLI shows 59 results including lossless sources
- User cannot get better search results without admin intervention
- No "re-search" option in the UI
**Example**: Job `6ddbbef1` was searched when flacfetch had a bug returning only YouTube results. After flacfetch was fixed, the job still showed the old 5-result cache.
**Solution**: Added admin endpoints and UI to manage cached searches:
- `GET /api/admin/audio-searches` - List jobs with cached results
- `POST /api/admin/audio-searches/{job_id}/clear-cache` - Clear both Firestore and flacfetch GCS cache
- `DELETE /api/admin/cache` - Clear entire flacfetch cache
- `GET /api/admin/cache/stats` - View cache statistics
- Admin UI at `/admin/searches` shows cache stats and "Clear All Cache" button
**Key insight**: There are TWO caches involved:
1. **Firestore** (`job.state_data.audio_search_results`) - stores results for user selection UI
2. **Flacfetch GCS** - caches tracker search responses (30-day TTL)
Both must be cleared for a truly fresh search. The clear-cache endpoint now clears both automatically.
**Future consideration**: Add automatic cache invalidation based on:
- Time (TTL)
- flacfetch version change
- User-initiated "re-search" button
**Lesson**: When caching external API results, consider how/when the cache should be invalidated. Permanent caches need admin tooling to manage stale data. When multiple layers cache the same data, clearing one layer isn't enough - you need to clear all of them.
### GCS Downloads Preserve Directory Structure
**Problem**: When downloading files from GCS, the directory structure is preserved. Code that uses non-recursive glob patterns won't find files in subdirectories.
@@ -795,6 +1096,49 @@ input_files = [f for f in input_files if "outputs" not in str(f)]
**Lesson**: When writing code that processes files downloaded from GCS, always use recursive glob patterns and verify the expected directory structure matches what's actually stored.
### GCS Blob Paths vs Full URIs
**Problem**: When uploading files to GCS and returning paths to other services, there's confusion between "blob paths" (just the path within a bucket) and "full GCS URIs" (gs://bucket/path).
**Symptoms**:
- 404 errors when downloading files that were just uploaded
- Paths look like: `bucket/gs://bucket/path/file.mp4` (doubled prefix)
- Storage service prepends bucket name to what's already a full URI
**Example of the bug**:
```python
# GCE worker uploads to gs://bucket/jobs/id/encoded/
# Then returns full URI in response: "gs://bucket/jobs/id/encoded/file.mp4"
# Backend tries: storage.download_file("gs://bucket/jobs/id/encoded/file.mp4")
# Storage service prepends bucket: "bucket/gs://bucket/jobs/id/encoded/file.mp4"
# Result: 404 - No such object
```
**Solution**: Return blob paths (path within bucket), not full GCS URIs:
```python
# output_gcs_path = "gs://bucket/jobs/id/encoded/"
gcs_path = output_gcs_path.replace("gs://", "")
parts = gcs_path.split("/", 1)
bucket = parts[0]
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
# Return blob paths like: "jobs/id/encoded/file.mp4"
blob_path = f"{prefix}/{filename}" if prefix else filename
```
**Related gotcha**: Double slashes in paths. When the prefix already ends with "/" and you concatenate with another "/":
```python
# BAD - creates "encoded//file.mp4"
prefix = "jobs/id/encoded/" # Has trailing slash
blob_name = f"{prefix}/{filename}" # Adds another slash
# GOOD - strip trailing slash first
prefix = parts[1].rstrip("/")
blob_name = f"{prefix}/{filename}" if prefix else filename
```
**Lesson**: When building systems that pass GCS paths between services, establish a clear contract: either always use full URIs (`gs://bucket/path`) or always use blob paths (`path/within/bucket`). Document which format each API expects.
### Library Caches Need GCS Sync in Cloud Run
**Problem**: Libraries like LyricsTranscriber cache API responses to local directories (e.g., `~/lyrics-transcriber-cache`). In Cloud Run, each container is ephemeral - the cache is lost on every restart or scale-up, causing redundant API calls.
@@ -875,6 +1219,327 @@ async def upload(auth_result: AuthResult = Depends(require_auth)):
See `docs/archive/2024-12-31-job-failure-investigation.md` for full details.
### Alternative Code Paths Must Implement All Features
**Problem**: Adding a GCE encoding option created a code path that completely bypassed the existing `KaraokeFinalise` class, which contained YouTube upload, Discord notifications, and CDG/TXT packaging. Jobs using GCE encoding silently missed these features.
```python
# BAD - two divergent code paths
if use_gce_encoding:
await gce_encoding_worker.encode(job_id) # Returns, does nothing else
return # YouTube upload, Discord, CDG never happen!
else:
finalise = KaraokeFinalise(...) # Has all the features
await finalise.run()
```
**Symptoms**:
- Jobs complete successfully but YouTube URL is never set
- No Discord notifications for GCE-encoded jobs
- CDG/TXT files never generated for GCE jobs
- No errors in logs (the code path simply doesn't include those features)
**Solution**: Create an orchestrator that runs all stages regardless of encoding backend:
```python
# GOOD - unified pipeline with swappable encoding
orchestrator = VideoWorkerOrchestrator(config)
result = await orchestrator.run()
# Orchestrator always runs: packaging → encoding → organization → distribution → notifications
# The encoding step uses either local or GCE backend based on config
```
**Key insight**: When adding an alternative implementation (like GCE encoding), audit ALL side effects of the original code path. The new path must replicate them or delegate to shared code.
**Architecture pattern**: The orchestrator pattern separates "what stages run" from "how each stage runs". Stages are always executed; only the implementation varies.
See `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` for full details.
### Firestore Document 1MB Limit with Embedded Arrays
**Problem**: Firestore documents have a 1MB size limit. Storing logs as an embedded array (`worker_logs: [{...}, {...}, ...]`) within job documents caused job failures when logs exceeded the limit during long-running operations like video encoding.
**Example**: Job `501258e1` failed when logs reached 1.26MB (2,547 log entries), representing 98.6% of the maximum document size. The job update failed with:
```
Document cannot be written because its size exceeds the maximum allowed size
```
**Root cause**: Video encoding jobs generate thousands of log entries (FFmpeg output, progress updates). Each log entry is ~500 bytes. 2,000+ entries easily exceed 1MB.
**Solution**: Store logs in a Firestore subcollection instead of embedded array:
```python
# BAD - embedded array hits 1MB limit
doc_ref.update({
"worker_logs": firestore.ArrayUnion([{"message": "..."}])
})
# GOOD - subcollection has no size limit per parent doc
logs_ref = db.collection("jobs").document(job_id).collection("logs")
logs_ref.document(log_id).set({
"timestamp": now,
"level": "INFO",
"worker": "video",
"message": "...",
"ttl_expiry": now + timedelta(days=30)
})
```
**Key benefits of subcollection approach**:
1. Each log entry is its own document (no size limit concerns)
2. Natural association with parent job (path: `jobs/{job_id}/logs/{log_id}`)
3. Can add TTL via Firestore TTL policies for automatic cleanup
4. Efficient queries with composite indexes (worker + timestamp)
5. Concurrent workers can write without race conditions (unlike ArrayUnion which can conflict)
**Migration strategy**: Used feature flag (`USE_LOG_SUBCOLLECTION=true`) for rollback capability. New jobs use subcollection; old jobs with embedded arrays continue to work via fallback read logic.
**Lesson**: Never use embedded arrays for unbounded data. If an array can grow indefinitely based on operation duration or user input, use a subcollection instead. The 1MB limit applies to the entire document, including all fields.
### Splitting Fast Generation from Slow Encoding
**Problem**: Preview video generation took 60+ seconds on Cloud Run, causing poor UX during lyrics review. The bottleneck was FFmpeg encoding (CPU-bound), not the subtitle generation (fast text processing).
**Solution**: Split the operation into two phases with optional offloading:
1. **ASS Generation (Cloud Run)**: Generate ASS subtitle file locally (~2s). This is pure text processing with no CPU-intensive encoding.
2. **Video Encoding (GCE)**: Upload ASS to GCS and offload FFmpeg encoding to the high-performance GCE worker (~15-20s on C4-standard-8).
```python
# Generate ASS only (fast, local)
result = CorrectionOperations.generate_preview_video(
...,
ass_only=True, # Skip video encoding
)
# Upload ASS to GCS
storage.upload_file(result["ass_path"], f"jobs/{job_id}/previews/{preview_hash}.ass")
# Offload encoding to GCE
gce_result = await encoding_service.encode_preview_video(
job_id=preview_hash,
ass_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.ass",
audio_gcs_path=audio_gcs_path,
output_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.mp4",
)
```
**Key insight**: When a pipeline has both fast (text generation) and slow (media encoding) steps, design for separability:
- Add `_only` parameters to skip slow steps
- Return intermediate outputs (paths to generated files)
- Allow external services to handle the slow part
- Keep fallback to local processing for reliability
**Configuration**: Controlled by `USE_GCE_PREVIEW_ENCODING` env var. Falls back to local encoding if GCE fails or is disabled.
**Result**: Preview generation dropped from 60+ seconds to ~15-20 seconds.
### External Service Response Format Mismatches
**Problem**: The GCE encoding worker returned responses in unexpected formats, causing `'list' object has no attribute 'get'` errors that were hard to debug through the async pipeline.
**Issues encountered**:
1. **Status endpoint returning list instead of dict**: The `/status/{job_id}` endpoint sometimes returned a list `[{...}]` instead of a dict `{...}`. Calling `.get()` on the list caused failures.
2. **output_files as list of paths instead of dict**: The worker returned:
```json
"output_files": ["jobs/id/finals/output_4k_lossless.mp4", "jobs/id/finals/output_720p.mp4"]
```
But the backend expected:
```json
"output_files": {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
```
**Symptoms**:
- Job fails late in pipeline (during encoding completion handling)
- Error message is cryptic (`'list' object has no attribute 'get'`)
- Hard to reproduce locally (depends on GCE worker state)
- Multiple similar errors from different root causes
**Solution**: Add defensive type checking for all external service responses:
```python
# Always check response type before accessing
if isinstance(status, list):
logger.warning(f"GCE returned list instead of dict: {status}")
status = status[0] if status and isinstance(status[0], dict) else {}
if not isinstance(status, dict):
status = {}
# Convert list of paths to expected dict format
if isinstance(output_files, list):
output_dict = {}
for path in output_files:
filename = path.split("/")[-1]
if "4k_lossless" in filename:
output_dict["mp4_4k_lossless"] = path
# ... etc
output_files = output_dict
```
**Key insight**: When integrating with external services, never assume the response format. Add explicit type checks and conversions, with logging when unexpected formats are encountered. This is especially important for services you don't control (internal microservices, third-party APIs).
**Debugging tip**: When hitting format errors in async pipelines, check the raw response from the external service directly (curl with API key) before assuming the bug is in your code.
### GCE Encoding Returns GCS Paths, Not Local Files
**Problem**: When using GCE encoding (remote high-performance encoder), the orchestrator received GCS blob paths like `jobs/{id}/finals/output_4k_lossless.mp4` instead of local file paths. Subsequent operations like YouTube upload checked `os.path.isfile()` which returned `False`, causing silent failures.
**Symptoms**:
- YouTube upload logged "No video file available for YouTube upload"
- Other file-based operations silently skipped
- Dropbox/GDrive uploads succeeded (they use different code paths)
- No errors in logs - just warnings that were easy to miss
**Root cause**: The `VideoWorkerOrchestrator._run_encoding()` method stored the paths returned by `GCEEncodingBackend` directly without downloading the files. For local encoding, the paths are already local. For GCE encoding, the paths are GCS blob paths that need to be downloaded first.
**Solution**: Added `_download_gce_encoded_files()` method that:
1. Detects when GCE backend was used (`encoding_backend.name == "gce"`)
2. Downloads each encoded file from GCS to the local temp directory
3. Updates result paths to point to local files
```python
# After encoding completes
if encoding_backend.name == "gce" and self.storage:
await self._download_gce_encoded_files(output)
```
**Key insight**: When an operation can run on multiple backends (local vs remote), ensure the outputs are normalized to a common format before subsequent stages consume them. The orchestrator pattern should abstract away backend differences, not leak them to downstream consumers.
**Testing tip**: Test the full pipeline with each backend type, not just the encoding step in isolation. The bug only manifested when YouTube upload ran after GCE encoding - unit tests of each stage passed.
### Unify Encoding Logic with GCS Wheel Deployment
**Problem**: The GCE encoding worker had simplified, duplicated encoding logic that:
- Didn't concatenate title + karaoke + end screens
- Used generic filenames like `output_4k_lossless.mp4` instead of proper names
- Lacked feature parity with the local CLI's `LocalEncodingService`
This caused cloud-generated videos to be incomplete (no title/end screens) and have unhelpful filenames that didn't match the local CLI output.
**Solution**: Deploy the karaoke-gen wheel to GCS, have the GCE worker install and use `LocalEncodingService`:
1. **CI uploads wheel to GCS**: `deploy-backend` job uploads wheel to `gs://bucket/wheels/`
2. **Dynamic wheel loading**: GCE worker downloads and installs latest wheel at job start:
```python
def ensure_latest_wheel():
"""Download and install latest karaoke-gen wheel from GCS."""
subprocess.run(["gsutil", "cp", "gs://bucket/wheels/karaoke_gen-*.whl", "/tmp/"])
wheel = sorted(glob.glob("/tmp/karaoke_gen-*.whl"))[-1]
subprocess.run([sys.executable, "-m", "pip", "install", "--upgrade", wheel])
async def process_job(job_id, request):
ensure_latest_wheel() # Get latest code at job start
# ... rest of processing
```
3. **Single encoding path**: GCE worker imports and uses `LocalEncodingService`:
```python
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
service = LocalEncodingService(logger=logger)
config = EncodingConfig(
title_video=str(title_video),
karaoke_video=str(karaoke_video),
output_lossless_4k_mp4=f"{base_name} (Final Karaoke Lossless 4k).mp4",
# ... proper naming for all formats
)
result = service.encode_all_formats(config)
```
**Key benefits**:
1. **Single source of truth**: Same encoding logic used by CLI, Cloud Run, and GCE
2. **Hot updates without restart**: New code deployed on next job, not VM restart
3. **In-progress jobs unaffected**: Jobs use the code version they started with
4. **Proper output**: Titles/end screens concatenated, files named correctly
**No fallback by design**: We removed all fallback logic to ensure there's only ONE code path. If the wheel fails to load, the job fails clearly rather than producing inconsistent output.
**Lesson**: When multiple deployment targets (CLI, Cloud Run, GCE) need the same complex logic, package it as a library and deploy via a shared mechanism (GCS wheel). Duplicating logic leads to feature drift and inconsistent behavior.
### Cloud Distribution Must Match Local CLI Output Structure
**Problem**: Cloud-generated karaoke outputs had different file organization than local CLI, causing confusion and missing files in Dropbox uploads:
1. **lyrics/ folder filename differences**:
- Cloud used generic `(Karaoke).json` instead of `(Lyrics Corrections).json`
- Cloud used `(Karaoke).txt` instead of `(Lyrics Corrected).txt`
- Reference lyrics files (`(Lyrics Genius).txt`, etc.) weren't downloaded
2. **Missing files in lyrics/ folder**:
- `(With Vocals).mkv` wasn't copied to lyrics/ subfolder
- `previews/` subfolder with preview ASS files wasn't created
- Reference lyrics from multiple sources weren't included
**Root causes**:
- Lyrics worker uploaded reference files to GCS but didn't track them in `job.files`
- Video worker used generic naming pattern for all lyrics files
- Video worker didn't copy `(With Vocals).mkv` or previews to lyrics folder
**Solution**:
1. **Track all files in job.files** (lyrics_worker.py):
```python
# Upload ALL reference lyrics and track them for distribution
for ref_filename, source_key in reference_files:
if os.path.exists(ref_path):
url = storage.upload_file(ref_path, gcs_path)
# Track in job.files so video_worker can download
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
```
2. **Use proper filenames in distribution** (video_worker.py):
```python
lyrics_file_mappings = [
('lrc', f"{base_name} (Karaoke).lrc"),
('corrections', f"{base_name} (Lyrics Corrections).json"),
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
# ... etc
]
```
3. **Copy (With Vocals).mkv to lyrics folder**:
```python
if videos.get('with_vocals'):
dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
storage.download_file(videos['with_vocals'], dest)
```
4. **Create previews/ subfolder**:
```python
previews_dir = os.path.join(lyrics_dir, "previews")
os.makedirs(previews_dir, exist_ok=True)
preview_files = storage.list_files(f"jobs/{job_id}/previews/")
for blob_name in preview_files:
if blob_name.endswith('.ass'):
storage.download_file(blob_name, dest_path)
```
**Expected output structure** (matches local CLI):
```
NOMAD-XXXX - Artist - Title/
├── stems/
│ ├── Artist - Title (Vocals model_bs_roformer_...).flac
│ └── ...
└── lyrics/
├── Artist - Title (Karaoke).lrc
├── Artist - Title (Karaoke).ass
├── Artist - Title (Lyrics Corrections).json
├── Artist - Title (Lyrics Corrected).txt
├── Artist - Title (Lyrics Uncorrected).txt
├── Artist - Title (Lyrics Genius).txt
├── Artist - Title (With Vocals).mkv
└── previews/
└── preview_abc123 (Karaoke).ass
```
**Lesson**: When building distribution packages from cloud storage, the organization code must replicate the exact structure users expect from local execution. Track ALL intermediate files in `job.files` even if they're "optional" - users may rely on them.
## What We'd Do Differently
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
+20
View File
@@ -36,6 +36,26 @@
## Recent Changes
- **GCE Encoding Unified with LocalEncodingService** (2026-01-06): GCE encoding worker now uses the same `LocalEncodingService` as the local CLI, eliminating duplicated encoding logic. Output files now have proper names (`Artist - Title (Final Karaoke Lossless 4k).mp4` instead of `output_4k_lossless.mp4`) and include title/end screen concatenation. Wheel deployed to GCS; worker installs at job start for hot updates without VM restart. Removed all fallback logic - single code path for consistent output across CLI, Cloud Run, and GCE. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#unify-encoding-logic-with-gcs-wheel-deployment).
- **Cloud Distribution Matches Local CLI** (2026-01-06): Fixed cloud distribution (Dropbox uploads) to match local CLI output structure. The `lyrics/` folder now includes: proper filenames (`(Lyrics Corrections).json` instead of `(Karaoke).json`), all reference lyrics files (`(Lyrics Genius).txt`, `(Lyrics Spotify).txt`, etc.), `(With Vocals).mkv`, and `previews/` subfolder with preview ASS files. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#cloud-distribution-must-match-local-cli-output-structure).
- **Agentic Correction Performance** (2026-01-05): Optimized agentic AI correction from ~5 minutes to ~55 seconds for 20 gaps (~5-6x speedup). Fixed anti-pattern of creating new model instance per gap (caused repeated 2s warm-up overhead). Now creates model once and processes gaps in parallel using ThreadPoolExecutor (default 5 workers). Configure via `AGENTIC_MAX_PARALLEL_GAPS` env var. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#reuse-llm-model-instances-across-operations).
- **Worker Logs Subcollection** (2026-01-04): Moved `worker_logs` from embedded array in job documents to Firestore subcollection (`jobs/{job_id}/logs`). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flag `USE_LOG_SUBCOLLECTION=true` (default). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
- **Video Worker Orchestrator** (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag `USE_NEW_ORCHESTRATOR` (default: true) enables rollback. 139 new tests across 6 new service modules. See [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator) and [archive/2026-01-04-video-worker-orchestrator-refactor.md](archive/2026-01-04-video-worker-orchestrator-refactor.md).
- **Full Unicode Font Support** (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fonts-in-docker-for-video-rendering).
- **Flacfetch Cache API Integration** (2026-01-03): Enhanced audio search cache management to also clear flacfetch's GCS cache. When admins clear a job's cache, both Firestore (job state) and flacfetch (GCS search results) are cleared simultaneously. Added "Clear All Cache" button and cache stats display to admin UI. New endpoints: `DELETE /api/admin/cache`, `GET /api/admin/cache/stats`. See [API.md](API.md#audio-search-management-admin).
- **Audio Search Cache Management** (2026-01-03): Added admin UI at `/admin/searches` to view and manage cached audio search results. Admins can clear stale caches (e.g., when flacfetch is updated with new providers) and allow users to re-search. Fixes issue where jobs showed only YouTube results when lossless sources should be available. See [API.md](API.md#audio-search-management-admin) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-cache-has-no-automatic-invalidation).
- **User Database Separation** (2026-01-03): Migrated karaoke-gen users from shared `users` collection to dedicated `gen_users` collection. This separates karaoke-gen user data from karaoke-decide user data (both apps share the same GCP project/Firestore instance). Migration script in `scripts/migrate_users_to_gen_users.py`. See [ARCHITECTURE.md](ARCHITECTURE.md#firestore-collections) for collection details.
- **Admin Dashboard** (2026-01-03): Added comprehensive admin dashboard at `/admin` with: stats overview (users, jobs, credits), user management (search, sort, add credits, toggle role, enable/disable), job browser (filter by status/user, view details, delete), and beta program monitoring. Query-param based routing for static export compatibility. See [API.md](API.md#admin-endpoints) for backend endpoints.
- **GCE Encoding Worker** (2026-01-03): Added optional high-performance encoding on dedicated GCE VM (C4-standard-8 with Intel Granite Rapids 3.9 GHz). Uses `USE_GCE_ENCODING=true` env var. Falls back to Cloud Run if unavailable. See [ARCHITECTURE.md](ARCHITECTURE.md).
@@ -0,0 +1,101 @@
# Video Worker Orchestrator Refactor - 2026-01-04
## Summary
Major refactor to unify video generation pipeline with a new VideoWorkerOrchestrator. This resolves the GCE vs KaraokeFinalise code path divergence that caused features like YouTube upload, Discord notifications, and CDG/TXT packaging to be missing from the GCE encoding path.
## Problem
Job 5b78ad46 completed successfully via GCE encoding but had no YouTube URL. Investigation revealed:
1. The GCE encoding path was a completely separate code branch
2. When `USE_GCE_ENCODING=true`, the worker triggered GCE encoding and returned immediately
3. This bypassed the entire `KaraokeFinalise` class which contained:
- YouTube upload logic
- Discord webhook notifications
- CDG/TXT package generation
- Dropbox/Google Drive uploads
The root cause wasn't a bug in any individual feature - it was an architectural issue where alternative implementations didn't share common post-processing stages.
## Solution
Created a stage-based orchestrator pattern:
```
VideoWorkerOrchestrator
├── Stage 1: Packaging (CDG, TXT)
├── Stage 2: Encoding (Local or GCE - swappable)
├── Stage 3: Organization (file paths, structure)
├── Stage 4: Distribution (Dropbox, GDrive, YouTube)
└── Stage 5: Notifications (Discord)
```
The orchestrator always runs all stages. Only the encoding implementation varies based on backend selection.
## Changes
### New Service Modules
| File | Lines | Purpose |
|------|-------|---------|
| `youtube_upload_service.py` | ~445 | YouTube OAuth and upload |
| `discord_service.py` | ~172 | Discord webhook notifications |
| `packaging_service.py` | ~287 | CDG/TXT package generation |
| `local_encoding_service.py` | ~590 | FFmpeg encoding on Cloud Run |
| `encoding_interface.py` | ~418 | Abstract interface + GCE backend |
| `video_worker_orchestrator.py` | ~628 | Stage coordination |
### Test Coverage
| Test File | Tests | Coverage |
|-----------|-------|----------|
| `test_youtube_upload_service.py` | 25 | OAuth, upload, error handling |
| `test_discord_service.py` | 18 | Webhook formatting, delivery |
| `test_packaging_service.py` | 22 | CDG/TXT generation |
| `test_local_encoding_service.py` | 24 | FFmpeg encoding variants |
| `test_encoding_interface.py` | 23 | Interface + GCE backend |
| `test_video_worker_orchestrator.py` | 27 | Full pipeline, stages |
| **Total** | **139** | |
### video_worker.py Updates
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: `true`)
- `generate_video()` routes to orchestrator or legacy path
- Legacy code preserved as `generate_video_legacy()` for rollback
- New `generate_video_orchestrated()` builds config and runs orchestrator
## Feature Flag
```bash
# Enable new orchestrator (default)
USE_NEW_ORCHESTRATOR=true
# Rollback to legacy behavior
USE_NEW_ORCHESTRATOR=false
```
## Key Design Decisions
1. **Stage-based orchestration**: Each stage is independent and testable
2. **Strategy pattern for encoding**: `EncodingBackend` interface with local and GCE implementations
3. **Lazy service loading**: Services instantiated only when needed via `_get_*()` methods
4. **Dataclass configs**: `OrchestratorConfig` and `OrchestratorResult` for clear contracts
5. **Feature flag default true**: New code active immediately since tests are comprehensive
## Testing Notes
- All 976 existing tests continue to pass
- 139 new tests added across 6 test files
- E2E verification: Jobs 501258e1 and d21e6ef0 successfully processed through new pipeline
- Both jobs reached `awaiting_instrumental_selection` as expected (requires user input to complete)
## PR
- PR #182: https://github.com/nomadkaraoke/karaoke-gen/pull/182
- Merged: 2026-01-04
- Version: 0.89.0
## Lessons Learned
See [LESSONS-LEARNED.md](../LESSONS-LEARNED.md#alternative-code-paths-must-implement-all-features) for the key architectural lesson about alternative code paths.
+2
View File
@@ -46,6 +46,8 @@ function AdminBreadcrumb() {
breadcrumbs.push({ label: "Jobs", href: "/admin/jobs" })
breadcrumbs.push({ label: decodeURIComponent(segments[2]) })
}
} else if (segments[1] === "searches") {
breadcrumbs.push({ label: "Audio Searches" })
} else if (segments[1] === "beta") {
breadcrumbs.push({ label: "Beta Program" })
}
+473
View File
@@ -0,0 +1,473 @@
"use client"
import { useEffect, useState, useCallback } from "react"
import { adminApi, AudioSearchJobSummary, CacheStatsResponse } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Badge } from "@/components/ui/badge"
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table"
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select"
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@/components/ui/alert-dialog"
import {
Tooltip,
TooltipContent,
TooltipProvider,
TooltipTrigger,
} from "@/components/ui/tooltip"
import {
RefreshCw,
Loader2,
Trash2,
Music,
CheckCircle,
XCircle,
ExternalLink,
Database,
HardDrive,
} from "lucide-react"
import { useToast } from "@/hooks/use-toast"
const statusOptions = [
{ value: "all", label: "All Statuses" },
{ value: "awaiting_audio_selection", label: "Awaiting Selection" },
{ value: "pending", label: "Pending" },
{ value: "complete", label: "Complete" },
{ value: "failed", label: "Failed" },
]
export default function AdminSearchesPage() {
const { toast } = useToast()
const [searches, setSearches] = useState<AudioSearchJobSummary[]>([])
const [loading, setLoading] = useState(true)
const [statusFilter, setStatusFilter] = useState("all")
// Clear cache dialog (single job)
const [clearDialogOpen, setClearDialogOpen] = useState(false)
const [jobToClear, setJobToClear] = useState<AudioSearchJobSummary | null>(null)
const [clearing, setClearing] = useState(false)
// Clear all cache dialog
const [clearAllDialogOpen, setClearAllDialogOpen] = useState(false)
const [clearingAll, setClearingAll] = useState(false)
// Cache stats
const [cacheStats, setCacheStats] = useState<CacheStatsResponse | null>(null)
const [statsLoading, setStatsLoading] = useState(false)
const loadSearches = useCallback(async () => {
try {
setLoading(true)
const data = await adminApi.listAudioSearches({
limit: 100,
status_filter: statusFilter !== "all" ? statusFilter : undefined,
})
setSearches(data.jobs)
} catch (err: any) {
console.error("Failed to load searches:", err)
toast({
title: "Error",
description: err.message || "Failed to load audio searches",
variant: "destructive",
})
} finally {
setLoading(false)
}
}, [statusFilter, toast])
const loadCacheStats = useCallback(async () => {
try {
setStatsLoading(true)
const stats = await adminApi.getCacheStats()
setCacheStats(stats)
} catch (err: any) {
// Silently fail - stats are optional
console.error("Failed to load cache stats:", err)
setCacheStats(null)
} finally {
setStatsLoading(false)
}
}, [])
useEffect(() => {
loadSearches()
loadCacheStats()
}, [loadSearches, loadCacheStats])
const handleClearCache = async () => {
if (!jobToClear) return
try {
setClearing(true)
const result = await adminApi.clearAudioSearchCache(jobToClear.job_id)
toast({
title: "Cache Cleared",
description: result.message,
})
setClearDialogOpen(false)
setJobToClear(null)
loadSearches()
loadCacheStats()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to clear cache",
variant: "destructive",
})
} finally {
setClearing(false)
}
}
const handleClearAllCache = async () => {
try {
setClearingAll(true)
const result = await adminApi.clearAllCache()
toast({
title: "All Cache Cleared",
description: result.message,
})
setClearAllDialogOpen(false)
loadCacheStats()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to clear all cache",
variant: "destructive",
})
} finally {
setClearingAll(false)
}
}
const formatBytes = (bytes: number): string => {
if (bytes === 0) return "0 B"
const k = 1024
const sizes = ["B", "KB", "MB", "GB"]
const i = Math.floor(Math.log(bytes) / Math.log(k))
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + " " + sizes[i]
}
const getStatusVariant = (status: string) => {
if (status === "complete" || status === "prep_complete") return "default"
if (status === "failed") return "destructive"
if (status === "cancelled") return "outline"
if (status.includes("awaiting")) return "secondary"
return "secondary"
}
const formatDate = (dateStr?: string) => {
if (!dateStr) return "—"
return new Date(dateStr).toLocaleString()
}
const getProviderBadgeClass = (provider: string) => {
switch (provider.toLowerCase()) {
case "youtube":
return "bg-red-500/20 text-red-400 border-red-500/30"
case "red":
return "bg-red-600/30 text-red-300 border-red-500/30"
case "ops":
return "bg-blue-500/20 text-blue-400 border-blue-500/30"
default:
return "bg-slate-500/20 text-slate-400 border-slate-500/30"
}
}
return (
<div className="space-y-6">
{/* Header */}
<div className="flex items-center justify-between">
<div>
<h1 className="text-2xl font-bold tracking-tight">Audio Searches</h1>
<p className="text-muted-foreground">
View and manage cached audio search results
</p>
</div>
<div className="flex items-center gap-2">
<Button variant="outline" size="sm" onClick={() => { loadSearches(); loadCacheStats() }} disabled={loading}>
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
Refresh
</Button>
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="outline"
size="sm"
className="text-red-500 hover:text-red-400 hover:bg-red-500/10 border-red-500/30"
onClick={() => setClearAllDialogOpen(true)}
>
<Database className="w-4 h-4 mr-2" />
Clear All Cache
</Button>
</TooltipTrigger>
<TooltipContent>
<p>Clear entire flacfetch search cache</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
</div>
</div>
{/* Cache Stats */}
{cacheStats && (
<div className="flex items-center gap-4 text-sm text-muted-foreground bg-muted/30 rounded-lg px-4 py-2">
<div className="flex items-center gap-2">
<HardDrive className="w-4 h-4" />
<span>Flacfetch Cache:</span>
</div>
<Badge variant="outline" className="font-mono">
{cacheStats.count} entries
</Badge>
<Badge variant="outline" className="font-mono">
{formatBytes(cacheStats.total_size_bytes)}
</Badge>
{cacheStats.newest_entry && (
<span className="text-xs">
Latest: {new Date(cacheStats.newest_entry).toLocaleDateString()}
</span>
)}
</div>
)}
{/* Filters */}
<div className="flex gap-4">
<Select value={statusFilter} onValueChange={setStatusFilter}>
<SelectTrigger className="w-[200px]">
<SelectValue placeholder="Filter by status" />
</SelectTrigger>
<SelectContent>
{statusOptions.map((opt) => (
<SelectItem key={opt.value} value={opt.value}>
{opt.label}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
{/* Table */}
<div className="rounded-md border">
<Table>
<TableHeader>
<TableRow>
<TableHead>Job ID</TableHead>
<TableHead>Search Query</TableHead>
<TableHead>Results</TableHead>
<TableHead>Providers</TableHead>
<TableHead>Status</TableHead>
<TableHead>User</TableHead>
<TableHead>Created</TableHead>
<TableHead className="text-right">Actions</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{loading ? (
<TableRow>
<TableCell colSpan={8} className="text-center py-8">
<Loader2 className="w-6 h-6 animate-spin mx-auto text-muted-foreground" />
</TableCell>
</TableRow>
) : searches.length === 0 ? (
<TableRow>
<TableCell colSpan={8} className="text-center py-8 text-muted-foreground">
<Music className="w-8 h-8 mx-auto mb-2 opacity-50" />
No audio searches found
</TableCell>
</TableRow>
) : (
searches.map((search) => (
<TableRow key={search.job_id}>
<TableCell className="font-mono text-xs">
<a
href={`/admin/jobs?id=${search.job_id}`}
className="text-blue-500 hover:underline"
>
{search.job_id.slice(0, 8)}...
</a>
</TableCell>
<TableCell>
<div className="max-w-[200px]">
<div className="font-medium truncate">
{search.audio_search_artist || "—"}
</div>
<div className="text-sm text-muted-foreground truncate">
{search.audio_search_title || "—"}
</div>
</div>
</TableCell>
<TableCell>
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<div className="flex items-center gap-2">
<span className="font-medium">{search.results_count}</span>
{search.has_lossless ? (
<CheckCircle className="w-4 h-4 text-green-500" />
) : (
<XCircle className="w-4 h-4 text-red-400" />
)}
</div>
</TooltipTrigger>
<TooltipContent>
<p>{search.has_lossless ? "Has lossless sources" : "No lossless sources (YouTube only)"}</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
</TableCell>
<TableCell>
<div className="flex flex-wrap gap-1">
{search.providers.map((provider) => (
<Badge
key={provider}
variant="outline"
className={`text-[10px] ${getProviderBadgeClass(provider)}`}
>
{provider}
</Badge>
))}
</div>
</TableCell>
<TableCell>
<Badge variant={getStatusVariant(search.status)}>
{search.status.replace(/_/g, " ")}
</Badge>
</TableCell>
<TableCell className="text-xs text-muted-foreground">
{search.user_email?.split("@")[0] || "—"}
</TableCell>
<TableCell className="text-xs text-muted-foreground">
{formatDate(search.created_at)}
</TableCell>
<TableCell className="text-right">
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="ghost"
size="icon"
className="h-8 w-8 text-orange-500 hover:text-orange-400 hover:bg-orange-500/10"
onClick={() => {
setJobToClear(search)
setClearDialogOpen(true)
}}
>
<Trash2 className="w-4 h-4" />
</Button>
</TooltipTrigger>
<TooltipContent>
<p>Clear cache & allow re-search</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
</TableCell>
</TableRow>
))
)}
</TableBody>
</Table>
</div>
{/* Summary */}
{!loading && searches.length > 0 && (
<div className="text-sm text-muted-foreground">
Showing {searches.length} jobs with cached search results.
{searches.filter(s => !s.has_lossless).length > 0 && (
<span className="ml-2 text-orange-400">
{searches.filter(s => !s.has_lossless).length} have YouTube-only results (may need cache refresh).
</span>
)}
</div>
)}
{/* Clear Cache Dialog (single job) */}
<AlertDialog open={clearDialogOpen} onOpenChange={setClearDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Clear Search Cache?</AlertDialogTitle>
<AlertDialogDescription>
This will clear the cached search results for job{" "}
<span className="font-mono">{jobToClear?.job_id}</span> and reset it to pending
status, allowing a new search to be performed.
{jobToClear && !jobToClear.has_lossless && (
<span className="block mt-2 text-orange-400">
This job only has YouTube results - clearing the cache may fix this if flacfetch has been updated.
</span>
)}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleClearCache}
disabled={clearing}
className="bg-orange-600 hover:bg-orange-500"
>
{clearing ? (
<Loader2 className="w-4 h-4 animate-spin mr-2" />
) : (
<Trash2 className="w-4 h-4 mr-2" />
)}
Clear Cache
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
{/* Clear All Cache Dialog */}
<AlertDialog open={clearAllDialogOpen} onOpenChange={setClearAllDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Clear Entire Flacfetch Cache?</AlertDialogTitle>
<AlertDialogDescription>
This will delete all cached search results from flacfetch&apos;s GCS cache
{cacheStats && (
<span className="font-medium"> ({cacheStats.count} entries, {formatBytes(cacheStats.total_size_bytes)})</span>
)}.
<span className="block mt-2 text-orange-400">
All subsequent searches will hit the trackers fresh. This does NOT affect
the Firestore job caches - use individual &quot;Clear Cache&quot; buttons for those.
</span>
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleClearAllCache}
disabled={clearingAll}
className="bg-red-600 hover:bg-red-500"
>
{clearingAll ? (
<Loader2 className="w-4 h-4 animate-spin mr-2" />
) : (
<Database className="w-4 h-4 mr-2" />
)}
Clear All Cache
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</div>
)
}
+12
View File
@@ -3,6 +3,7 @@
import { useState, useEffect, useCallback } from "react"
import { useRouter } from "next/navigation"
import { api, Job, getAccessToken } from "@/lib/api"
import { useAuth } from "@/lib/auth"
import { useJobNotifications, useVisibilityRefresh } from "@/hooks/use-notifications"
import { Button } from "@/components/ui/button"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
@@ -28,6 +29,7 @@ export default function AppPage() {
const [isInitialLoad, setIsInitialLoad] = useState(true)
const [isAuthenticated, setIsAuthenticated] = useState<boolean | null>(null)
const { isDarkMode, toggleTheme, mounted } = useTheme()
const { user, fetchUser } = useAuth()
// Memoize loadJobs for use with visibility refresh
const loadJobs = useCallback(async () => {
@@ -67,6 +69,16 @@ export default function AppPage() {
setIsAuthenticated(true)
}, [router])
// Ensure user data is loaded after authentication
// This handles the case where user navigates via client-side navigation
// (e.g., after beta enrollment) and the module-level fetchUser() didn't run
useEffect(() => {
const token = getAccessToken()
if (token && !user) {
fetchUser()
}
}, [user, fetchUser])
// Load jobs on mount (only if authenticated)
useEffect(() => {
if (isAuthenticated !== true) {
@@ -45,7 +45,7 @@ describe('AuthStatus', () => {
// Should show user email and credits
expect(screen.getByText('test@example.com')).toBeInTheDocument()
expect(screen.getByText('5')).toBeInTheDocument()
expect(screen.getByText('5 credits')).toBeInTheDocument()
})
it('opens auth dialog when Login is clicked', () => {
@@ -9,6 +9,7 @@ import {
TestTube2,
ArrowLeft,
Settings,
Search,
} from "lucide-react"
import {
@@ -41,6 +42,11 @@ const navItems = [
href: "/admin/jobs",
icon: Briefcase,
},
{
title: "Audio Searches",
href: "/admin/searches",
icon: Search,
},
{
title: "Beta Program",
href: "/admin/beta",
+24 -2
View File
@@ -12,7 +12,7 @@ import {
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "./ui/dropdown-menu"
import { User, LogOut, Settings } from "lucide-react"
import { User, LogOut, Settings, Shield, Mail } from "lucide-react"
import Link from "next/link"
export function AppHeader() {
@@ -40,7 +40,7 @@ export function AppHeader() {
<Link href="/jobs" className="text-sm font-medium hover:text-primary transition-colors">
My Jobs
</Link>
{user.role === "admin" && (
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
<Link href="/admin" className="text-sm font-medium hover:text-primary transition-colors">
Admin
</Link>
@@ -48,6 +48,14 @@ export function AppHeader() {
</nav>
<div className="flex items-center gap-4">
<a
href="mailto:andrew@nomadkaraoke.com"
className="hidden lg:flex items-center gap-1.5 text-xs text-muted-foreground hover:text-foreground transition-colors"
>
<Mail className="w-3.5 h-3.5" />
<span>Need help?</span>
</a>
<div className="hidden sm:flex items-center gap-2 px-4 py-2 rounded-lg bg-primary/10 border border-primary/20">
<div className="w-2 h-2 rounded-full bg-primary animate-pulse" />
<span className="text-sm font-medium text-primary">{user.credits} credits</span>
@@ -73,10 +81,24 @@ export function AppHeader() {
<span>{user.credits} credits</span>
</div>
</DropdownMenuItem>
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
<DropdownMenuItem asChild>
<Link href="/admin">
<Shield className="w-4 h-4 mr-2" />
Admin Dashboard
</Link>
</DropdownMenuItem>
)}
<DropdownMenuItem>
<Settings className="w-4 h-4 mr-2" />
Settings
</DropdownMenuItem>
<DropdownMenuItem asChild>
<a href="mailto:andrew@nomadkaraoke.com">
<Mail className="w-4 h-4 mr-2" />
Need help? Contact us
</a>
</DropdownMenuItem>
<DropdownMenuSeparator />
<DropdownMenuItem onClick={handleLogout} className="text-destructive focus:text-destructive">
<LogOut className="w-4 h-4 mr-2" />
+14 -2
View File
@@ -1,7 +1,8 @@
"use client"
import { useState, useEffect } from "react"
import { User, LogOut, CreditCard, Coins, KeyRound } from "lucide-react"
import { User, LogOut, CreditCard, Coins, KeyRound, Shield } from "lucide-react"
import Link from "next/link"
import { Button } from "@/components/ui/button"
import {
DropdownMenu,
@@ -69,7 +70,7 @@ export function AuthStatus({ onAuthChange }: AuthStatusProps) {
</span>
<span className="flex items-center gap-1 text-amber-400 font-medium">
<Coins className="w-3 h-3" />
{user.credits}
{user.credits} {user.credits === 1 ? 'credit' : 'credits'}
</span>
</Button>
</DropdownMenuTrigger>
@@ -97,6 +98,17 @@ export function AuthStatus({ onAuthChange }: AuthStatusProps) {
<CreditCard className="w-4 h-4 mr-2" />
<span>Buy More Credits</span>
</DropdownMenuItem>
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
<>
<DropdownMenuSeparator className="bg-slate-700" />
<DropdownMenuItem asChild className="text-slate-300 focus:text-white focus:bg-slate-800">
<Link href="/admin">
<Shield className="w-4 h-4 mr-2" />
<span>Admin Dashboard</span>
</Link>
</DropdownMenuItem>
</>
)}
<DropdownMenuSeparator className="bg-slate-700" />
<DropdownMenuItem
onClick={handleLogout}
+6 -6
View File
@@ -272,8 +272,8 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
Skip lyrics review (non-interactive)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
<span className="text-amber-500 font-medium">Warning:</span> AI lyrics correction isn&apos;t perfect.
Skipping review will likely result in timing errors and incorrect words in your karaoke video.
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn&apos;t perfect.
Skipping review will likely result in incorrect words in your karaoke video.
</p>
</div>
</div>
@@ -377,8 +377,8 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
Skip lyrics review (non-interactive)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
<span className="text-amber-500 font-medium">Warning:</span> AI lyrics correction isn&apos;t perfect.
Skipping review will likely result in timing errors and incorrect words in your karaoke video.
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn&apos;t perfect.
Skipping review will likely result in incorrect words in your karaoke video.
</p>
</div>
</div>
@@ -465,8 +465,8 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
Skip lyrics review (non-interactive)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
<span className="text-amber-500 font-medium">Warning:</span> AI lyrics correction isn&apos;t perfect.
Skipping review will likely result in timing errors and incorrect words in your karaoke video.
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn&apos;t perfect.
Skipping review will likely result in incorrect words in your karaoke video.
</p>
</div>
</div>
@@ -453,8 +453,8 @@ test.describe('Production E2E - Full User Journey', () => {
jobId,
['awaiting_instrumental_selection', 'complete'],
accessToken!,
TIMEOUTS.jobProcessing // Video rendering can take several minutes
).catch(() => ({ status: 'unknown', job: null }));
TIMEOUTS.download // Video rendering can take 10-15 minutes with orchestrator
);
if (statusBeforeInstrumental === 'awaiting_instrumental_selection') {
console.log(' Selecting clean instrumental via API...');
@@ -11,7 +11,7 @@ import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-tes
*
* Test Flow:
* 1. Landing Page - Navigate and verify
* 2. New User Signup - Create MailSlurp inbox, submit beta form, verify email
* 2. New User Signup - Create testmail.app inbox, submit beta form, verify email
* 3. Magic Link Auth - Click magic link to authenticate
* 4. Create Job - Search for test song via UI
* 5. Audio Selection - Select audio source via UI dialog
@@ -24,7 +24,7 @@ import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-tes
* 12. Cleanup - Delete distributed content and job
*
* Environment Variables:
* - MAILSLURP_API_KEY: Required for email testing (can be skipped with E2E_TEST_TOKEN)
* - TESTMAIL_API_KEY + TESTMAIL_NAMESPACE: Required for email testing (can be skipped with E2E_TEST_TOKEN)
* - E2E_TEST_TOKEN: Pre-configured user token to skip signup flow (for iterations)
* - E2E_ADMIN_TOKEN: Required for cleanup endpoint (optional - cleanup skipped if not set)
*/
@@ -138,12 +138,12 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
}) => {
test.setTimeout(TIMEOUTS.fullTest);
// Check if we have a pre-configured token (for iterations without using MailSlurp)
// Check if we have a pre-configured token (for iterations without using testmail.app)
const usePreConfiguredToken = hasPreConfiguredToken();
// Skip if neither pre-configured token nor MailSlurp is available
// Skip if neither pre-configured token nor testmail.app is available
if (!usePreConfiguredToken && !isEmailTestingAvailable()) {
test.skip(true, 'Neither E2E_TEST_TOKEN nor MAILSLURP_API_KEY set - cannot run test');
test.skip(true, 'Neither E2E_TEST_TOKEN nor TESTMAIL_API_KEY/TESTMAIL_NAMESPACE set - cannot run test');
return;
}
@@ -154,6 +154,23 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
// Only create email helper if we're doing the full signup flow
const emailHelper = usePreConfiguredToken ? null : await createEmailHelper();
// Intercept API requests to disable YouTube upload for E2E tests
// This prevents test jobs from uploading to YouTube when server default is true
await page.route('**/api/audio-search/search', async (route) => {
const request = route.request();
if (request.method() === 'POST') {
const postData = request.postDataJSON();
// Explicitly disable YouTube upload for E2E test jobs
postData.enable_youtube_upload = false;
console.log(' [Request Intercept] Disabled YouTube upload for audio search');
await route.continue({
postData: JSON.stringify(postData),
});
} else {
await route.continue();
}
});
try {
// =========================================================================
// STEP 1: Landing Page
@@ -217,7 +234,7 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
console.log('STEP 3 COMPLETE: User authenticated with pre-configured token');
} else {
// ----- FULL PATH: Beta signup with MailSlurp -----
// ----- FULL PATH: Beta signup with testmail.app -----
console.log('\n========================================');
console.log('STEP 2: New User Beta Enrollment');
console.log('========================================');
@@ -171,6 +171,71 @@ test.describe('Authentication - Beta Enrollment', () => {
await page.waitForURL(/\/app/, { timeout: 10000 });
expect(page.url()).toContain('/app');
});
test('beta enrollment shows credits after redirect to app', async ({ page }) => {
// This test verifies that after beta enrollment, the /app page
// correctly fetches and displays user data including credits.
// This catches the bug where client-side navigation didn't trigger fetchUser().
await setupApiFixtures(page, {
mocks: [
{
method: 'POST',
path: '/api/users/beta/enroll',
response: {
status: 200,
body: {
status: 'success',
message: 'Enrolled in beta',
credits_granted: 1,
session_token: 'mock-session-token-456',
},
},
},
{
method: 'GET',
path: '/api/users/me',
response: {
status: 200,
body: {
user: {
email: 'beta-tester@example.com',
credits: 1,
role: 'user',
display_name: null,
total_jobs_created: 0,
total_jobs_completed: 0,
},
has_session: true,
},
},
},
{
method: 'GET',
path: '/api/jobs',
response: { body: [] },
},
],
});
await page.goto('/');
await page.waitForLoadState('networkidle');
// Complete beta enrollment
await page.getByRole('button', { name: /join beta program/i }).click();
await page.locator('#beta-email').fill('beta-tester@example.com');
await page.locator('textarea').fill(
'I want to create a karaoke video for my favorite song!'
);
await page.locator('input[type="checkbox"]').check();
await page.getByRole('button', { name: /get my free credit/i }).click();
// Wait for redirect to /app
await page.waitForURL(/\/app/, { timeout: 10000 });
// Verify credits are displayed in the UI
// The AuthStatus component should show "1 credit" after fetchUser() completes
await expect(page.getByText(/1 credit/i)).toBeVisible({ timeout: 10000 });
});
});
test.describe('Authentication - Token Persistence', () => {
+105
View File
@@ -782,6 +782,60 @@ export interface AdminJobListParams {
limit?: number;
}
// Audio Search Admin Types
export interface AudioSearchResultSummary {
index: number;
provider: string;
artist: string;
title: string;
is_lossless: boolean;
quality?: string;
seeders?: number;
}
export interface AudioSearchJobSummary {
job_id: string;
status: string;
user_email?: string;
audio_search_artist?: string;
audio_search_title?: string;
created_at?: string;
results_count: number;
results_summary: AudioSearchResultSummary[];
has_lossless: boolean;
providers: string[];
}
export interface AudioSearchListResponse {
jobs: AudioSearchJobSummary[];
total: number;
}
export interface ClearSearchCacheResponse {
status: string;
job_id: string;
message: string;
previous_status: string;
new_status: string;
results_cleared: number;
flacfetch_cache_cleared: boolean;
flacfetch_error?: string;
}
export interface ClearAllCacheResponse {
status: string;
message: string;
deleted_count: number;
}
export interface CacheStatsResponse {
count: number;
total_size_bytes: number;
oldest_entry?: string;
newest_entry?: string;
configured: boolean;
}
// Admin API namespace
export const adminApi = {
/**
@@ -933,6 +987,57 @@ export const adminApi = {
);
return handleResponse(response);
},
/**
* List jobs with audio search results
*/
async listAudioSearches(params?: {
limit?: number;
status_filter?: string;
}): Promise<AudioSearchListResponse> {
const searchParams = new URLSearchParams();
if (params?.limit) searchParams.set('limit', String(params.limit));
if (params?.status_filter) searchParams.set('status_filter', params.status_filter);
const url = `${API_BASE_URL}/api/admin/audio-searches${searchParams.toString() ? '?' + searchParams.toString() : ''}`;
const response = await fetch(url, {
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* Clear audio search cache for a job
*/
async clearAudioSearchCache(jobId: string): Promise<ClearSearchCacheResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/audio-searches/${jobId}/clear-cache`,
{ method: 'POST', headers: getAuthHeaders() }
);
return handleResponse(response);
},
/**
* Clear all flacfetch search cache
*/
async clearAllCache(): Promise<ClearAllCacheResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/cache`,
{ method: 'DELETE', headers: getAuthHeaders() }
);
return handleResponse(response);
},
/**
* Get flacfetch cache statistics
*/
async getCacheStats(): Promise<CacheStatsResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/cache/stats`,
{ headers: getAuthHeaders() }
);
return handleResponse(response);
},
};
export { ApiError };
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "karaoke-gen-frontend",
"version": "0.79.0",
"version": "0.80.0",
"private": true,
"scripts": {
"build": "next build",
+429 -100
View File
@@ -144,6 +144,42 @@ firestore_index_gen_users_active_email = firestore.Index(
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# =============================================================================
# Firestore TTL Field Configuration
# TTL (Time-to-Live) policies automatically delete documents after expiration.
# =============================================================================
# Worker logs TTL: logs subcollection documents expire after 30 days
# Collection path for subcollections: "jobs/{job_id}/logs"
# When using subcollection groups, use just the subcollection name "logs"
firestore_field_logs_ttl = firestore.Field(
"firestore-field-logs-ttl",
project=project_id,
database=firestore_db.name,
collection="logs", # Subcollection name (applies to all jobs/{job_id}/logs)
field="ttl_expiry",
ttl_config={}, # Empty block enables TTL based on ttl_expiry field
index_config={}, # Disable indexing on TTL field
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# =============================================================================
# Firestore Composite Indexes for Logs Subcollection
# =============================================================================
# Logs: query by worker, order by timestamp (for filtering logs by worker type)
firestore_index_logs_worker_timestamp = firestore.Index(
"firestore-index-logs-worker-timestamp",
project=project_id,
database=firestore_db.name,
collection="logs", # Subcollection name
fields=[
firestore.IndexFieldArgs(field_path="worker", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="timestamp", order="ASCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Create Cloud Storage Bucket
bucket = storage.Bucket(
"karaoke-storage",
@@ -337,6 +373,22 @@ github_actions_service_usage = gcp.projects.IAMMember(
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Project IAM Admin - required for Pulumi to refresh IAM binding state
github_actions_iam_admin = gcp.projects.IAMMember(
"github-actions-iam-admin",
project=project_id,
role="roles/resourcemanager.projectIamAdmin",
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Secret Manager Admin - required for Pulumi to refresh secret state
github_actions_secretmanager = gcp.projects.IAMMember(
"github-actions-secretmanager",
project=project_id,
role="roles/secretmanager.admin",
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# ==================== Workload Identity Federation ====================
# Allows GitHub Actions to authenticate without service account keys
@@ -811,6 +863,8 @@ pulumi.export("firestore_index_gen_users_active_created", firestore_index_gen_us
pulumi.export("firestore_index_gen_users_active_login", firestore_index_gen_users_active_login.name)
pulumi.export("firestore_index_gen_users_active_credits", firestore_index_gen_users_active_credits.name)
pulumi.export("firestore_index_gen_users_active_email", firestore_index_gen_users_active_email.name)
pulumi.export("firestore_field_logs_ttl", firestore_field_logs_ttl.name)
pulumi.export("firestore_index_logs_worker_timestamp", firestore_index_logs_worker_timestamp.name)
pulumi.export("service_account_email", service_account.email)
pulumi.export("artifact_repo_url", artifact_repo.name.apply(
lambda name: f"us-central1-docker.pkg.dev/{project_id}/karaoke-repo"
@@ -1130,6 +1184,14 @@ encoding_worker_logging_iam = gcp.projects.IAMMember(
member=encoding_worker_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant encoding worker monitoring permissions (for Ops Agent metrics)
encoding_worker_monitoring_iam = gcp.projects.IAMMember(
"encoding-worker-monitoring-access",
project=project_id,
role="roles/monitoring.metricWriter",
member=encoding_worker_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Static external IP for encoding worker (Cloud Run doesn't have VPC access by default)
encoding_worker_ip = gcp.compute.Address(
"encoding-worker-ip",
@@ -1151,6 +1213,13 @@ echo "Starting encoding worker setup at $(date)"
apt-get update
apt-get install -y python3-pip python3-venv docker.io curl git xz-utils
# Install fonts for ASS subtitle rendering (libass uses fontconfig)
# - fonts-noto: Noto Sans (default karaoke font)
# - fonts-noto-cjk: CJK character support
# - fontconfig: Font configuration system
apt-get install -y fonts-noto fonts-noto-cjk fontconfig
fc-cache -fv # Rebuild font cache
# Enable and start Docker
systemctl enable docker
systemctl start docker
@@ -1190,10 +1259,12 @@ fi
# Create the encoding worker service
cat > /opt/encoding-worker/main.py << 'PYTHON_CODE'
import asyncio
import glob as glob_module
import json
import logging
import os
import subprocess
import sys
import tempfile
import uuid
from concurrent.futures import ThreadPoolExecutor
@@ -1235,6 +1306,17 @@ class EncodeRequest(BaseModel):
output_gcs_path: str # gs://bucket/path/to/outputs/
encoding_config: dict # Video formats to generate
class EncodePreviewRequest(BaseModel):
job_id: str
ass_gcs_path: str # gs://bucket/path/to/subtitles.ass
audio_gcs_path: str # gs://bucket/path/to/audio.flac
output_gcs_path: str # gs://bucket/path/to/output.mp4
background_color: str = "black"
background_image_gcs_path: Optional[str] = None
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
class JobStatus(BaseModel):
job_id: str
status: str # pending, running, complete, failed
@@ -1268,12 +1350,12 @@ def upload_to_gcs(local_path: Path, gcs_uri: str):
# Upload a file or folder to GCS
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
prefix = parts[1] if len(parts) > 1 else ""
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
if local_path.is_file():
blob_name = f"{prefix}/{local_path.name}".strip("/")
blob_name = f"{prefix}/{local_path.name}" if prefix else local_path.name
blob = bucket.blob(blob_name)
blob.upload_from_filename(str(local_path))
logger.info(f"Uploaded: {local_path} -> gs://{bucket_name}/{blob_name}")
@@ -1281,128 +1363,312 @@ def upload_to_gcs(local_path: Path, gcs_uri: str):
for file in local_path.rglob("*"):
if file.is_file():
rel_path = file.relative_to(local_path)
blob_name = f"{prefix}/{rel_path}".strip("/")
blob_name = f"{prefix}/{rel_path}" if prefix else str(rel_path)
blob = bucket.blob(blob_name)
blob.upload_from_filename(str(file))
logger.info(f"Uploaded: {file} -> gs://{bucket_name}/{blob_name}")
def run_encoding(job_id: str, work_dir: Path, config: dict):
# Run FFmpeg encoding for all video formats
def download_single_file_from_gcs(gcs_uri: str, local_path: Path):
# Download a single file from GCS
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
blob_name = parts[1] if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
local_path.parent.mkdir(parents=True, exist_ok=True)
blob.download_to_filename(str(local_path))
logger.info(f"Downloaded: {gcs_uri} -> {local_path}")
def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
# Upload a single file to a specific GCS path
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
blob_name = parts[1] if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
blob.upload_from_filename(str(local_path))
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
# Run FFmpeg encoding for preview video (480x270, fast settings)
jobs[job_id]["status"] = "running"
jobs[job_id]["progress"] = 10
try:
# Find input files (search recursively since GCS downloads preserve directory structure)
input_files = list(work_dir.glob("**/*.mkv")) + list(work_dir.glob("**/*.mp4")) + list(work_dir.glob("**/*.mov"))
# Exclude files in the outputs directory to avoid re-encoding previous outputs
input_files = [f for f in input_files if "outputs" not in str(f)]
if not input_files:
raise ValueError(f"No video files found in {work_dir}")
# Download input files
ass_path = work_dir / "subtitles.ass"
audio_path = work_dir / "audio.flac"
# Use first video file as input (With Vocals or Karaoke)
input_video = input_files[0]
logger.info(f"Using input video: {input_video}")
download_single_file_from_gcs(request.ass_gcs_path, ass_path)
jobs[job_id]["progress"] = 20
# Find instrumental audio (search recursively, case-insensitive patterns)
# GCS stores as: instrumental_clean.flac, instrumental_with_backing.flac
instrumental_files = (
list(work_dir.glob("**/*instrumental*.flac")) +
list(work_dir.glob("**/*instrumental*.wav")) +
list(work_dir.glob("**/*Instrumental*.flac")) +
list(work_dir.glob("**/*Instrumental*.wav"))
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
jobs[job_id]["progress"] = 30
# Download background image if provided
bg_image_path = None
if request.background_image_gcs_path:
bg_image_path = work_dir / "background.png"
download_single_file_from_gcs(request.background_image_gcs_path, bg_image_path)
# Download custom font if provided and register with fontconfig
if request.font_gcs_path:
# Use standard fontconfig location that's already in the search path
fonts_dir = Path("/usr/local/share/fonts/custom")
fonts_dir.mkdir(parents=True, exist_ok=True)
font_filename = request.font_gcs_path.split("/")[-1]
font_path = fonts_dir / font_filename
download_single_file_from_gcs(request.font_gcs_path, font_path)
logger.info(f"Downloaded custom font: {font_path}")
# Update fontconfig cache so libass can find the font
subprocess.run(["fc-cache", "-fv"], capture_output=True)
logger.info(f"Updated fontconfig cache with custom font: {font_filename}")
# Build FFmpeg command
output_path = work_dir / "preview.mp4"
# Escape special characters in path for FFmpeg filter syntax
# FFmpeg filter parsing requires escaping: \\ : , [ ] ;
def escape_ffmpeg_filter_path(path: str) -> str:
# Note: Extra escaping needed since this is inside a triple-quoted string in Pulumi
return path.replace("\\\\", "\\\\\\\\").replace(":", "\\\\:").replace(",", "\\\\,").replace("[", "\\\\[").replace("]", "\\\\]").replace(";", "\\\\;")
escaped_ass_path = escape_ffmpeg_filter_path(str(ass_path))
# Base command with frame rate
cmd = ["ffmpeg", "-y", "-r", "24"]
# Video input: background image or solid color
if bg_image_path and bg_image_path.exists():
cmd.extend(["-loop", "1", "-i", str(bg_image_path)])
# Scale and pad background to 480x270
vf = f"scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)/2:(oh-ih)/2,ass={escaped_ass_path}"
else:
# Solid color background
color = request.background_color or "black"
cmd.extend(["-f", "lavfi", "-i", f"color=c={color}:s=480x270:r=24"])
vf = f"ass={escaped_ass_path}"
# Audio input
cmd.extend(["-i", str(audio_path)])
# Video filter and encoding settings
cmd.extend([
"-vf", vf,
"-c:a", "aac", "-b:a", "96k",
"-c:v", "libx264",
"-preset", "superfast",
"-crf", "28",
"-pix_fmt", "yuv420p",
"-movflags", "+faststart",
"-threads", "8",
"-shortest",
str(output_path)
])
jobs[job_id]["progress"] = 40
logger.info(f"Running preview encoding: {' '.join(cmd)}")
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
logger.error(f"FFmpeg failed: {result.stderr}")
raise RuntimeError(f"FFmpeg preview encoding failed: {result.stderr[-500:]}")
jobs[job_id]["progress"] = 80
logger.info(f"Preview encoded: {output_path}")
# Upload output to GCS
upload_single_file_to_gcs(output_path, request.output_gcs_path)
jobs[job_id]["progress"] = 95
jobs[job_id]["output_files"] = [request.output_gcs_path]
return output_path
except Exception as e:
logger.error(f"Preview encoding failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
raise
def ensure_latest_wheel():
"""Download and install latest karaoke-gen wheel from GCS.
Called at the start of each job to enable hot code updates without restart.
In-progress jobs continue with their version, new jobs get latest code.
"""
try:
logger.info("Checking for latest karaoke-gen wheel in GCS...")
# Download latest wheel
result = subprocess.run(
["gsutil", "cp", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl", "/tmp/"],
capture_output=True, text=True, timeout=60
)
instrumental = instrumental_files[0] if instrumental_files else None
# Find the downloaded wheel (get the latest by version sorting)
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
if not wheels:
logger.warning("No wheel found in GCS, using fallback encoding logic")
return False
# Sort to get latest version
wheel_path = sorted(wheels)[-1]
logger.info(f"Installing wheel: {wheel_path}")
# Install (or upgrade) the wheel
install_result = subprocess.run(
[sys.executable, "-m", "pip", "install", "--upgrade", "--quiet", wheel_path],
capture_output=True, text=True, timeout=120
)
if install_result.returncode != 0:
logger.warning(f"Wheel installation failed: {install_result.stderr}")
return False
logger.info(f"Successfully installed wheel: {wheel_path}")
return True
except subprocess.TimeoutExpired:
logger.warning("Wheel download/install timed out, using fallback")
return False
except Exception as e:
logger.warning(f"Failed to ensure latest wheel: {e}")
return False
def find_file(work_dir: Path, *patterns):
"""Find a file matching any of the given glob patterns."""
for pattern in patterns:
matches = list(work_dir.glob(f"**/{pattern}"))
if matches:
return matches[0]
return None
def run_encoding(job_id: str, work_dir: Path, config: dict):
"""Run encoding using LocalEncodingService (single source of truth).
Uses LocalEncodingService from the installed karaoke-gen wheel to ensure
output files match local CLI exactly:
- Proper names like "Artist - Title (Final Karaoke Lossless 4k).mp4"
- Concatenated title + karaoke + end screens
- All formats: lossless 4K MP4, lossy 4K MP4, lossless MKV, 720p MP4
Requires the karaoke-gen wheel to be installed (done by ensure_latest_wheel).
"""
jobs[job_id]["status"] = "running"
jobs[job_id]["progress"] = 10
try:
# Import LocalEncodingService from installed wheel (required, no fallback)
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
logger.info("Using LocalEncodingService from installed wheel")
# Get artist/title from config for proper naming
artist = config.get("artist", "Unknown Artist")
title = config.get("title", "Unknown Title")
base_name = f"{artist} - {title}"
logger.info(f"Encoding for: {base_name}")
# Find input files in work_dir
# Title/end screens are in screens/ subdirectory
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
# Karaoke video - search for With Vocals or main karaoke video
karaoke_video = find_file(
work_dir,
"*With Vocals*.mov", "*With Vocals*.mkv",
"*Vocals*.mov", "*Vocals*.mkv",
"*.mkv", "*.mov"
)
# Exclude title/end/output videos
if karaoke_video:
name_lower = karaoke_video.name.lower()
if "title" in name_lower or "end" in name_lower or "outputs" in str(karaoke_video):
# Search more specifically for karaoke video
karaoke_video = find_file(work_dir, "*Karaoke*.mkv", "*Karaoke*.mov", "*vocals*.mkv")
# Instrumental audio
instrumental = find_file(
work_dir,
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
logger.info(f"Found files:")
logger.info(f" Title video: {title_video}")
logger.info(f" Karaoke video: {karaoke_video}")
logger.info(f" End video: {end_video}")
logger.info(f" Instrumental: {instrumental}")
# Validate required files
if not title_video:
raise ValueError(f"No title video found in {work_dir}. Check screens/ subdirectory.")
if not karaoke_video:
raise ValueError(f"No karaoke video found in {work_dir}")
if not instrumental:
raise ValueError(f"No instrumental audio found in {work_dir}")
output_dir = work_dir / "outputs"
output_dir.mkdir(exist_ok=True)
output_files = []
formats = config.get("formats", ["mp4_4k", "mp4_720p"])
total_formats = len(formats)
jobs[job_id]["progress"] = 20
for i, fmt in enumerate(formats):
progress = 10 + int((i / total_formats) * 80)
jobs[job_id]["progress"] = progress
# Build encoding config with proper file names
encoding_config = EncodingConfig(
title_video=str(title_video),
karaoke_video=str(karaoke_video),
instrumental_audio=str(instrumental),
end_video=str(end_video) if end_video else None,
output_karaoke_mp4=str(output_dir / f"{base_name} (Karaoke).mp4"),
output_with_vocals_mp4=str(output_dir / f"{base_name} (With Vocals).mp4"),
output_lossless_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mp4"),
output_lossy_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 720p).mp4"),
)
if fmt == "mp4_4k_lossless" or fmt == "mp4_4k":
# High quality 4K - CRF 18 (visually lossless)
output_file = output_dir / "output_4k_lossless.mp4"
cmd = [
"ffmpeg", "-y",
"-i", str(input_video),
]
if instrumental:
cmd.extend(["-i", str(instrumental)])
cmd.extend([
"-map", "0:v",
"-map", "1:a" if instrumental else "0:a",
"-c:v", "libx264",
"-preset", "medium",
"-crf", "18",
"-c:a", "aac",
"-b:a", "256k",
"-threads", "24",
str(output_file)
])
elif fmt == "mp4_4k_lossy":
# Smaller 4K file - CRF 23 (good quality, smaller size)
output_file = output_dir / "output_4k_lossy.mp4"
cmd = [
"ffmpeg", "-y",
"-i", str(input_video),
]
if instrumental:
cmd.extend(["-i", str(instrumental)])
cmd.extend([
"-map", "0:v",
"-map", "1:a" if instrumental else "0:a",
"-c:v", "libx264",
"-preset", "medium",
"-crf", "23",
"-c:a", "aac",
"-b:a", "192k",
"-threads", "24",
str(output_file)
])
elif fmt == "mp4_720p":
output_file = output_dir / "output_720p.mp4"
cmd = [
"ffmpeg", "-y",
"-i", str(input_video),
]
if instrumental:
cmd.extend(["-i", str(instrumental)])
cmd.extend([
"-map", "0:v",
"-map", "1:a" if instrumental else "0:a",
"-vf", "scale=-1:720",
"-c:v", "libx264",
"-preset", "medium",
"-crf", "20",
"-c:a", "aac",
"-b:a", "192k",
"-threads", "24",
str(output_file)
])
else:
logger.warning(f"Unknown format: {fmt}")
continue
# Create service and run encoding
service = LocalEncodingService(logger=logger)
logger.info(f"Running: {' '.join(cmd)}")
result = subprocess.run(cmd, capture_output=True, text=True)
jobs[job_id]["progress"] = 30
logger.info("Starting LocalEncodingService.encode_all_formats()")
if result.returncode != 0:
logger.error(f"FFmpeg failed: {result.stderr}")
raise RuntimeError(f"FFmpeg encoding failed: {result.stderr[-500:]}")
result = service.encode_all_formats(encoding_config)
output_files.append(str(output_file))
logger.info(f"Encoded: {output_file}")
if not result.success:
raise RuntimeError(f"Encoding failed: {result.error}")
jobs[job_id]["output_files"] = output_files
jobs[job_id]["progress"] = 90
# Collect output files
output_files = [str(f) for f in output_dir.glob("*") if f.is_file()]
jobs[job_id]["output_files"] = output_files
logger.info(f"Encoding complete. Output files: {output_files}")
return output_dir
except ImportError as e:
# No fallback - wheel must be installed
error_msg = (
f"LocalEncodingService not available: {e}. "
"The karaoke-gen wheel must be installed. "
"Check that ensure_latest_wheel() succeeded and wheel exists in GCS."
)
logger.error(error_msg)
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = error_msg
raise RuntimeError(error_msg) from e
except Exception as e:
logger.error(f"Encoding failed: {e}")
jobs[job_id]["status"] = "failed"
@@ -1413,6 +1679,10 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
async def process_job(job_id: str, request: EncodeRequest):
# Process an encoding job asynchronously
try:
# Download and install latest wheel at job start (allows hot updates without restart)
# This means in-progress jobs continue with their version, new jobs get latest code
ensure_latest_wheel()
with tempfile.TemporaryDirectory() as temp_dir:
work_dir = Path(temp_dir) / "work"
work_dir.mkdir()
@@ -1439,7 +1709,7 @@ async def process_job(job_id: str, request: EncodeRequest):
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
# We need paths like "jobs/id/encoded/output_4k.mp4"
# We need paths like "jobs/id/encoded/Artist - Title (Final Karaoke Lossless 4k).mp4"
gcs_path = request.output_gcs_path.replace("gs://", "")
parts = gcs_path.split("/", 1)
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
@@ -1462,6 +1732,65 @@ async def process_job(job_id: str, request: EncodeRequest):
jobs[job_id]["error"] = str(e)
async def process_preview_job(job_id: str, request: EncodePreviewRequest):
# Process a preview encoding job asynchronously
try:
with tempfile.TemporaryDirectory() as temp_dir:
work_dir = Path(temp_dir) / "work"
work_dir.mkdir()
# Run preview encoding in thread pool (CPU-bound)
# Note: run_preview_encoding handles download/upload internally
loop = asyncio.get_event_loop()
await loop.run_in_executor(
executor,
run_preview_encoding,
job_id,
work_dir,
request
)
jobs[job_id]["status"] = "complete"
jobs[job_id]["progress"] = 100
logger.info(f"Preview job {job_id} complete")
except Exception as e:
logger.error(f"Preview job {job_id} failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
@app.post("/encode-preview")
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit a preview encoding job
job_id = request.job_id
# If job already exists, return cached result or current status
if job_id in jobs:
existing_job = jobs[job_id]
if existing_job["status"] == "complete":
# Return cached result - preview already encoded
return {"status": "cached", "job_id": job_id, "output_path": existing_job.get("output_path")}
elif existing_job["status"] == "failed":
# Previous attempt failed, allow retry by replacing the job
pass
else:
# Job is still in progress
return {"status": "in_progress", "job_id": job_id}
jobs[job_id] = {
"job_id": job_id,
"status": "pending",
"progress": 0,
"error": None,
"output_files": None,
}
background_tasks.add_task(process_preview_job, job_id, request)
return {"status": "accepted", "job_id": job_id}
@app.post("/encode")
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit an encoding job
+3 -1
View File
@@ -70,7 +70,9 @@ DEFAULT_KARAOKE_STYLE = {
"background_color": "#000000",
"background_image": None,
# Font settings
"font": "Arial",
# Using "Noto Sans" for full Unicode support (CJK, Arabic, symbols, etc.)
# Requires fonts-noto-* packages installed in Docker image
"font": "Noto Sans",
"font_path": "", # Must be string, not None (for ASS generator)
"ass_name": "Default",
# Colors in "R, G, B, A" format (required by ASS)
+9 -4
View File
@@ -156,15 +156,20 @@ class VideoBackgroundProcessor:
"""
Escape a file path for use in ffmpeg filter expressions.
FFmpeg filter syntax uses single quotes to protect special characters
like spaces, colons, and semicolons. Single quotes within the path
are escaped using the '\\'' pattern (end quote, literal quote, start quote).
Args:
path: File path to escape
Returns:
str: Escaped path
str: Escaped path (wrapped in single quotes)
"""
# Escape backslashes and colons for ffmpeg filter syntax
escaped = path.replace("\\", "\\\\").replace(":", "\\:")
return escaped
# Escape single quotes: ' becomes '\'' (end quote, \', start quote)
escaped = path.replace("'", "'\\''")
# Wrap entire path in single quotes
return f"'{escaped}'"
def build_video_filter(self, ass_subtitles_path, darkness_percent, fonts_dir=None):
"""
@@ -35,10 +35,14 @@ class ProviderConfig:
circuit_breaker_failure_threshold: int = 3
circuit_breaker_open_seconds: int = 60
# Initialization timeouts - fail fast instead of hanging forever
# These are separate from request_timeout to catch connection establishment issues
initialization_timeout_seconds: float = 30.0 # Model creation + warm-up
warmup_timeout_seconds: float = 15.0 # Just the warm-up call
# Initialization timeout - fail fast instead of hanging forever
# This is separate from request_timeout to catch connection establishment issues
initialization_timeout_seconds: float = 30.0 # Model creation
# Parallel processing settings
# Process multiple gaps concurrently to reduce total correction time
# Set to 1 to disable parallelism, higher values increase throughput but may hit rate limits
max_parallel_gaps: int = 5
@staticmethod
def from_env(cache_dir: Optional[str] = None) -> "ProviderConfig":
@@ -70,7 +74,7 @@ class ProviderConfig:
circuit_breaker_failure_threshold=int(os.getenv("AGENTIC_CIRCUIT_THRESHOLD", "3")),
circuit_breaker_open_seconds=int(os.getenv("AGENTIC_CIRCUIT_OPEN_SECONDS", "60")),
initialization_timeout_seconds=float(os.getenv("AGENTIC_INIT_TIMEOUT_SECONDS", "30.0")),
warmup_timeout_seconds=float(os.getenv("AGENTIC_WARMUP_TIMEOUT_SECONDS", "15.0")),
max_parallel_gaps=int(os.getenv("AGENTIC_MAX_PARALLEL_GAPS", "5")),
)
def validate_environment(self, logger: Optional[object] = None) -> None:
@@ -97,7 +97,6 @@ class LangChainBridge(BaseAIProvider):
# Lazy-initialized chat model
self._chat_model: Optional[Any] = None
self._warmed_up: bool = False
def name(self) -> str:
"""Return provider name for logging."""
@@ -164,13 +163,7 @@ class LangChainBridge(BaseAIProvider):
) from None
init_elapsed = time.time() - init_start
logger.info(f"🤖 Model created in {init_elapsed:.2f}s, starting warm-up...")
# Warm up the model to establish connection before real work
self._warm_up_model()
total_elapsed = time.time() - init_start
logger.info(f"🤖 Model initialization complete in {total_elapsed:.2f}s")
logger.info(f"🤖 Model initialized in {init_elapsed:.2f}s")
except InitializationTimeoutError as e:
self._circuit_breaker.record_failure(self._model)
@@ -271,47 +264,4 @@ class LangChainBridge(BaseAIProvider):
return content
def _warm_up_model(self) -> None:
"""Send a lightweight request to warm up the model connection.
This helps establish the REST connection and potentially warm up any
server-side resources before processing real correction requests.
The warm-up uses a timeout to fail fast if the service is unresponsive.
"""
if self._warmed_up:
return
timeout = self._config.warmup_timeout_seconds
# Use print with flush=True for visibility when output is redirected
print(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...", flush=True)
logger.info(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...")
warmup_start = time.time()
try:
from langchain_core.messages import HumanMessage
# Minimal prompt that requires almost no processing
warm_up_prompt = 'Respond with exactly: {"status":"ready"}'
# Use ThreadPoolExecutor for timeout on warm-up call
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(
self._chat_model.invoke,
[HumanMessage(content=warm_up_prompt)]
)
try:
future.result(timeout=timeout)
except FuturesTimeoutError:
raise TimeoutError(f"Warm-up timed out after {timeout}s") from None
elapsed = time.time() - warmup_start
self._warmed_up = True
print(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s", flush=True)
logger.info(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s")
except Exception as e:
elapsed = time.time() - warmup_start
# Don't fail the actual request if warm-up fails
# Just log and continue - the real request might still work
print(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)", flush=True)
logger.warning(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)")
@@ -268,12 +268,26 @@ class LyricsCorrector:
_adapt = None
_ModelRouter = None
# Pre-initialized agentic corrector (created once, reused for all gaps)
_agentic_agent = None
if use_agentic_env:
try:
from lyrics_transcriber.correction.agentic.agent import AgenticCorrector as _AgenticCorrector
from lyrics_transcriber.correction.agentic.adapter import adapt_proposals_to_word_corrections as _adapt
from lyrics_transcriber.correction.agentic.router import ModelRouter as _ModelRouter
self.logger.info("🤖 Agentic modules imported successfully - running in AGENTIC-ONLY mode")
# Create agent ONCE and reuse for all gaps (avoids repeated model initialization)
_router = _ModelRouter()
model_id = _router.choose_model("gap", uncertainty=0.5) # Use default uncertainty
self.logger.info(f"🤖 Creating single AgenticCorrector with model: {model_id}")
_agentic_agent = _AgenticCorrector.from_model(
model=model_id,
session_id=session_id,
cache_dir=str(self._cache_dir)
)
self.logger.info("🤖 AgenticCorrector initialized and ready for all gaps")
except Exception as e:
self.logger.error(f"🤖 Failed to import agentic modules but USE_AGENTIC_AI=1: {e}")
raise RuntimeError(f"Agentic AI correction is enabled but required modules could not be imported: {e}") from e
@@ -443,16 +457,186 @@ class LyricsCorrector:
sys.exit(0)
# === END TEMPORARY CODE ===
# AGENTIC-ONLY MODE: Process all gaps in parallel for better performance
if use_agentic_env:
from concurrent.futures import ThreadPoolExecutor, as_completed
from lyrics_transcriber.correction.agentic.providers.config import ProviderConfig
# Get parallel processing config
_config = ProviderConfig.from_env()
max_workers = _config.max_parallel_gaps
self.logger.info(f"🤖 Processing {len(gap_sequences)} gaps in parallel (max_workers={max_workers})")
# Pre-compute shared data structures once (not per-gap)
all_transcribed_words = []
for seg in segments:
all_transcribed_words.extend(seg.words)
word_position = {w.id: idx for idx, w in enumerate(all_transcribed_words)}
# Build reference contexts once (same for all gaps)
reference_contexts = {}
for source, lyrics_data in self.reference_lyrics.items():
if lyrics_data and lyrics_data.segments:
ref_words = []
for seg in lyrics_data.segments:
ref_words.extend([w.text for w in seg.words])
reference_contexts[source] = " ".join(ref_words)
# Get artist and title once
artist = metadata.get("artist") if metadata else None
title = metadata.get("title") if metadata else None
# Prepare all gap inputs upfront
gap_inputs = []
for i, gap in enumerate(gap_sequences, 1):
# Prepare gap words data
gap_words_data = []
for word_id in gap.transcribed_word_ids:
if word_id in word_map:
word = word_map[word_id]
gap_words_data.append({
"id": word_id,
"text": word.text,
"start_time": getattr(word, 'start_time', 0),
"end_time": getattr(word, 'end_time', 0)
})
# Compute context words
gap_positions = [word_position[wid] for wid in gap.transcribed_word_ids if wid in word_position]
preceding_words = ""
following_words = ""
if gap_positions:
first_gap_pos = min(gap_positions)
last_gap_pos = max(gap_positions)
# Get 10 words before
start_pos = max(0, first_gap_pos - 10)
preceding_list = [all_transcribed_words[idx].text for idx in range(start_pos, first_gap_pos) if idx < len(all_transcribed_words)]
preceding_words = " ".join(preceding_list)
# Get 10 words after
end_pos = min(len(all_transcribed_words), last_gap_pos + 11)
following_list = [all_transcribed_words[idx].text for idx in range(last_gap_pos + 1, end_pos) if idx < len(all_transcribed_words)]
following_words = " ".join(following_list)
gap_inputs.append({
'index': i,
'gap': gap,
'gap_id': f"gap_{i}",
'gap_words': gap_words_data,
'preceding_words': preceding_words,
'following_words': following_words,
'reference_contexts': reference_contexts,
'artist': artist,
'title': title
})
# Function to process a single gap (runs in thread pool)
def process_single_gap(gap_input):
"""Process a single gap and return proposals. Thread-safe."""
idx = gap_input['index']
try:
proposals = _agentic_agent.propose_for_gap(
gap_id=gap_input['gap_id'],
gap_words=gap_input['gap_words'],
preceding_words=gap_input['preceding_words'],
following_words=gap_input['following_words'],
reference_contexts=gap_input['reference_contexts'],
artist=gap_input['artist'],
title=gap_input['title']
)
return {'index': idx, 'gap': gap_input['gap'], 'proposals': proposals, 'error': None}
except Exception as e:
return {'index': idx, 'gap': gap_input['gap'], 'proposals': None, 'error': str(e)}
# Process gaps in parallel
results = [None] * len(gap_inputs)
completed_count = 0
errors = []
with ThreadPoolExecutor(max_workers=max_workers) as executor:
# Submit all tasks
future_to_input = {executor.submit(process_single_gap, g): g for g in gap_inputs}
# Collect results as they complete
for future in as_completed(future_to_input):
# Check deadline
if deadline and time.time() > deadline:
self.logger.warning(
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {completed_count}/{len(gap_sequences)} gaps. "
"Cancelling remaining gaps - human review will correct any issues."
)
# Cancel remaining futures (use list() to avoid mutating dict during iteration)
for f in list(future_to_input.keys()):
f.cancel()
break
result = future.result()
idx = result['index'] - 1 # Convert 1-based to 0-based
results[idx] = result
completed_count += 1
if result['error']:
errors.append(f"Gap {result['index']}: {result['error']}")
self.logger.error(f"🤖 Gap {result['index']} failed: {result['error']}")
else:
proposal_count = len(result['proposals']) if result['proposals'] else 0
self.logger.info(f"🤖 Gap {result['index']}/{len(gap_sequences)} completed ({proposal_count} proposals)")
self.logger.info(f"🤖 Parallel processing complete: {completed_count}/{len(gap_sequences)} gaps processed")
# If any errors occurred, fail fast
if errors:
raise RuntimeError(f"Agentic AI correction failed for {len(errors)} gaps: {'; '.join(errors)}")
# Apply corrections sequentially (must be in order due to segment modifications)
for result in results:
if result is None:
continue # Skipped due to deadline
i = result['index']
gap = result['gap']
_proposals = result['proposals']
_agentic_corrections = _adapt(_proposals, word_map, linear_position_map) if _proposals else []
if _agentic_corrections:
self.logger.info(f"🤖 Applying {len(_agentic_corrections)} agentic corrections for gap {i}")
affected_word_ids = [w.id for w in self._get_affected_words(gap, segments)]
affected_segment_ids = [s.id for s in self._get_affected_segments(gap, segments)]
updated_segments = self._apply_corrections_to_segments(self._get_affected_segments(gap, segments), _agentic_corrections)
for correction in _agentic_corrections:
if correction.word_id and correction.corrected_word_id:
word_id_map[correction.word_id] = correction.corrected_word_id
for old_seg, new_seg in zip(self._get_affected_segments(gap, segments), updated_segments):
segment_id_map[old_seg.id] = new_seg.id
step = CorrectionStep(
handler_name="AgenticCorrector",
affected_word_ids=affected_word_ids,
affected_segment_ids=affected_segment_ids,
corrections=_agentic_corrections,
segments_before=self._get_affected_segments(gap, segments),
segments_after=updated_segments,
created_word_ids=[w.id for w in self._get_new_words(updated_segments, affected_word_ids)],
deleted_word_ids=[id for id in affected_word_ids if not self._word_exists(id, updated_segments)],
)
correction_steps.append(step)
all_corrections.extend(_agentic_corrections)
# Log corrections made
for correction in _agentic_corrections:
self.logger.info(
f"Made correction: '{correction.original_word}' -> '{correction.corrected_word}' "
f"(confidence: {correction.confidence:.2f}, reason: {correction.reason})"
)
else:
self.logger.debug(f"🤖 No agentic corrections needed for gap {i}")
# RULE-BASED MODE: Process gaps sequentially
for i, gap in enumerate(gap_sequences, 1):
# Check deadline before processing each gap (agentic mode only)
# This allows us to abort early and return uncorrected results for human review
if deadline and use_agentic_env and time.time() > deadline:
self.logger.warning(
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {i-1}/{len(gap_sequences)} gaps. "
"Skipping remaining gaps - human review will correct any issues."
)
# Break out of loop - continue with whatever corrections we have (likely none)
break
# Skip if we already processed in agentic mode
if use_agentic_env:
continue
self.logger.info(f"Processing gap {i}/{len(gap_sequences)} at position {gap.transcription_position}")
@@ -460,128 +644,6 @@ class LyricsCorrector:
gap_words = [word_map[word_id] for word_id in gap.transcribed_word_ids]
self.logger.debug(f"Gap text: '{' '.join(w.text for w in gap_words)}'")
# AGENTIC-ONLY MODE: Use agentic correction exclusively
if use_agentic_env:
self.logger.info(f"🤖 Attempting agentic correction for gap {i}/{len(gap_sequences)}")
try:
# Prepare gap data for classification-first workflow
gap_words_data = []
for word_id in gap.transcribed_word_ids:
if word_id in word_map:
word = word_map[word_id]
gap_words_data.append({
"id": word_id,
"text": word.text,
"start_time": getattr(word, 'start_time', 0),
"end_time": getattr(word, 'end_time', 0)
})
# Get context words
all_transcribed_words = []
for seg in segments:
all_transcribed_words.extend(seg.words)
word_position = {w.id: idx for idx, w in enumerate(all_transcribed_words)}
gap_positions = [word_position[wid] for wid in gap.transcribed_word_ids if wid in word_position]
preceding_words = ""
following_words = ""
if gap_positions:
first_gap_pos = min(gap_positions)
last_gap_pos = max(gap_positions)
# Get 10 words before
start_pos = max(0, first_gap_pos - 10)
preceding_list = [all_transcribed_words[idx].text for idx in range(start_pos, first_gap_pos) if idx < len(all_transcribed_words)]
preceding_words = " ".join(preceding_list)
# Get 10 words after
end_pos = min(len(all_transcribed_words), last_gap_pos + 11)
following_list = [all_transcribed_words[idx].text for idx in range(last_gap_pos + 1, end_pos) if idx < len(all_transcribed_words)]
following_words = " ".join(following_list)
# Get reference contexts from all sources
reference_contexts = {}
for source, lyrics_data in self.reference_lyrics.items():
if lyrics_data and lyrics_data.segments:
ref_words = []
for seg in lyrics_data.segments:
ref_words.extend([w.text for w in seg.words])
# For now, use full text (handlers will extract relevant portions)
reference_contexts[source] = " ".join(ref_words)
# Get artist and title from metadata
artist = metadata.get("artist") if metadata else None
title = metadata.get("title") if metadata else None
# Choose model via router
_router = _ModelRouter()
uncertainty = 0.3 if len(gap_words_data) <= 2 else 0.7
model_id = _router.choose_model("gap", uncertainty)
self.logger.debug(f"🤖 Router selected model: {model_id}")
# Create agent and use new classification-first workflow
self.logger.debug(f"🤖 Creating AgenticCorrector with model: {model_id}")
_agent = _AgenticCorrector.from_model(
model=model_id,
session_id=session_id,
cache_dir=str(self._cache_dir)
)
# Use new propose_for_gap method
self.logger.debug(f"🤖 Calling agent.propose_for_gap() for gap {i}")
_proposals = _agent.propose_for_gap(
gap_id=f"gap_{i}",
gap_words=gap_words_data,
preceding_words=preceding_words,
following_words=following_words,
reference_contexts=reference_contexts,
artist=artist,
title=title
)
self.logger.debug(f"🤖 Agent returned {len(_proposals) if _proposals else 0} proposals")
_agentic_corrections = _adapt(_proposals, word_map, linear_position_map) if _proposals else []
self.logger.debug(f"🤖 Adapter returned {len(_agentic_corrections)} corrections")
if _agentic_corrections:
self.logger.info(f"🤖 Applying {len(_agentic_corrections)} agentic corrections for gap {i}")
affected_word_ids = [w.id for w in self._get_affected_words(gap, segments)]
affected_segment_ids = [s.id for s in self._get_affected_segments(gap, segments)]
updated_segments = self._apply_corrections_to_segments(self._get_affected_segments(gap, segments), _agentic_corrections)
for correction in _agentic_corrections:
if correction.word_id and correction.corrected_word_id:
word_id_map[correction.word_id] = correction.corrected_word_id
for old_seg, new_seg in zip(self._get_affected_segments(gap, segments), updated_segments):
segment_id_map[old_seg.id] = new_seg.id
step = CorrectionStep(
handler_name="AgenticCorrector",
affected_word_ids=affected_word_ids,
affected_segment_ids=affected_segment_ids,
corrections=_agentic_corrections,
segments_before=self._get_affected_segments(gap, segments),
segments_after=updated_segments,
created_word_ids=[w.id for w in self._get_new_words(updated_segments, affected_word_ids)],
deleted_word_ids=[id for id in affected_word_ids if not self._word_exists(id, updated_segments)],
)
correction_steps.append(step)
all_corrections.extend(_agentic_corrections)
# Log corrections made
for correction in _agentic_corrections:
self.logger.info(
f"Made correction: '{correction.original_word}' -> '{correction.corrected_word}' "
f"(confidence: {correction.confidence:.2f}, reason: {correction.reason})"
)
else:
self.logger.info(f"🤖 No agentic corrections needed for gap {i}")
except Exception as e:
# In agentic-only mode, fail fast instead of falling back
self.logger.error(f"🤖 Agentic correction failed for gap {i}: {e}", exc_info=True)
raise RuntimeError(f"Agentic AI correction failed for gap {i}: {e}") from e
# Skip rule-based handlers completely in agentic mode
continue
# RULE-BASED MODE: Try each handler in order
for handler in self.handlers:
handler_name = handler.__class__.__name__
@@ -274,11 +274,12 @@ class CorrectionOperations:
audio_filepath: str,
artist: Optional[str] = None,
title: Optional[str] = None,
logger: Optional[logging.Logger] = None
logger: Optional[logging.Logger] = None,
ass_only: bool = False,
) -> Dict[str, Any]:
"""
Generate a preview video with current corrections.
Args:
correction_result: Current correction result
updated_data: Updated correction data for preview
@@ -287,10 +288,12 @@ class CorrectionOperations:
artist: Optional artist name
title: Optional title
logger: Optional logger instance
ass_only: If True, generate only ASS subtitles without video encoding.
Useful when video encoding is offloaded to external service.
Returns:
Dict with status, preview_hash, and video_path
Dict with status, preview_hash, and video_path (or ass_path if ass_only)
Raises:
ValueError: If preview video generation fails
"""
@@ -338,15 +341,27 @@ class CorrectionOperations:
audio_filepath=audio_filepath,
artist=artist,
title=title,
ass_only=ass_only,
)
# When ass_only, we only need the ASS file (video encoding done externally)
if ass_only:
if not preview_outputs.ass:
raise ValueError("Preview ASS generation failed")
logger.info(f"Generated preview ASS: {preview_outputs.ass}")
return {
"status": "success",
"preview_hash": preview_hash,
"ass_path": preview_outputs.ass,
}
if not preview_outputs.video:
raise ValueError("Preview video generation failed")
logger.info(f"Generated preview video: {preview_outputs.video}")
return {
"status": "success",
"preview_hash": preview_hash,
"video_path": preview_outputs.video
"video_path": preview_outputs.video,
}
@@ -2,7 +2,7 @@
"name": "lyrics-transcriber-frontend",
"private": true,
"homepage": "https://nomadkaraoke.github.io/lyrics-transcriber-frontend",
"version": "0.83.0",
"version": "0.84.0",
"type": "module",
"scripts": {
"dev": "vite",
@@ -1 +1 @@
{"root":["./src/app.tsx","./src/api.ts","./src/main.tsx","./src/theme.ts","./src/types.ts","./src/validation.ts","./src/vite-env.d.ts","./src/components/aifeedbackmodal.tsx","./src/components/addlyricsmodal.tsx","./src/components/agenticcorrectionmetrics.tsx","./src/components/audioplayer.tsx","./src/components/correctedwordwithactions.tsx","./src/components/correctionannotationmodal.tsx","./src/components/correctiondetailcard.tsx","./src/components/correctionmetrics.tsx","./src/components/durationtimelineview.tsx","./src/components/editactionbar.tsx","./src/components/editmodal.tsx","./src/components/edittimelinesection.tsx","./src/components/editwordlist.tsx","./src/components/fileupload.tsx","./src/components/findreplacemodal.tsx","./src/components/header.tsx","./src/components/lyricsanalyzer.tsx","./src/components/metricsdashboard.tsx","./src/components/modeselectionmodal.tsx","./src/components/modeselector.tsx","./src/components/modelselector.tsx","./src/components/previewvideosection.tsx","./src/components/referenceview.tsx","./src/components/replacealllyricsmodal.tsx","./src/components/reviewchangesmodal.tsx","./src/components/segmentdetailsmodal.tsx","./src/components/timelineeditor.tsx","./src/components/timingoffsetmodal.tsx","./src/components/transcriptionview.tsx","./src/components/worddivider.tsx","./src/components/lyricssynchronizer/synccontrols.tsx","./src/components/lyricssynchronizer/timelinecanvas.tsx","./src/components/lyricssynchronizer/upcomingwordsbar.tsx","./src/components/lyricssynchronizer/index.tsx","./src/components/shared/constants.ts","./src/components/shared/styles.ts","./src/components/shared/types.ts","./src/components/shared/components/highlightedtext.tsx","./src/components/shared/components/sourceselector.tsx","./src/components/shared/components/word.tsx","./src/components/shared/hooks/usewordclick.ts","./src/components/shared/utils/keyboardhandlers.ts","./src/components/shared/utils/localstorage.ts","./src/components/shared/utils/referencelinecalculator.ts","./src/components/shared/utils/segmentoperations.ts","./src/components/shared/utils/timingutils.ts","./src/components/shared/utils/wordutils.ts","./src/hooks/usemanualsync.ts","./src/types/global.d.ts"],"version":"5.6.3"}
{"root":["./src/app.tsx","./src/api.ts","./src/main.tsx","./src/theme.ts","./src/types.ts","./src/validation.ts","./src/vite-env.d.ts","./src/components/aifeedbackmodal.tsx","./src/components/addlyricsmodal.tsx","./src/components/agenticcorrectionmetrics.tsx","./src/components/appheader.tsx","./src/components/audioplayer.tsx","./src/components/correctedwordwithactions.tsx","./src/components/correctionannotationmodal.tsx","./src/components/correctiondetailcard.tsx","./src/components/correctionmetrics.tsx","./src/components/durationtimelineview.tsx","./src/components/editactionbar.tsx","./src/components/editmodal.tsx","./src/components/edittimelinesection.tsx","./src/components/editwordlist.tsx","./src/components/fileupload.tsx","./src/components/findreplacemodal.tsx","./src/components/header.tsx","./src/components/lyricsanalyzer.tsx","./src/components/metricsdashboard.tsx","./src/components/modeselectionmodal.tsx","./src/components/modeselector.tsx","./src/components/modelselector.tsx","./src/components/previewvideosection.tsx","./src/components/referenceview.tsx","./src/components/replacealllyricsmodal.tsx","./src/components/reviewchangesmodal.tsx","./src/components/segmentdetailsmodal.tsx","./src/components/timelineeditor.tsx","./src/components/timingoffsetmodal.tsx","./src/components/transcriptionview.tsx","./src/components/worddivider.tsx","./src/components/lyricssynchronizer/synccontrols.tsx","./src/components/lyricssynchronizer/timelinecanvas.tsx","./src/components/lyricssynchronizer/upcomingwordsbar.tsx","./src/components/lyricssynchronizer/index.tsx","./src/components/shared/constants.ts","./src/components/shared/styles.ts","./src/components/shared/types.ts","./src/components/shared/components/highlightedtext.tsx","./src/components/shared/components/sourceselector.tsx","./src/components/shared/components/word.tsx","./src/components/shared/hooks/usewordclick.ts","./src/components/shared/utils/keyboardhandlers.ts","./src/components/shared/utils/localstorage.ts","./src/components/shared/utils/referencelinecalculator.ts","./src/components/shared/utils/segmentoperations.ts","./src/components/shared/utils/timingutils.ts","./src/components/shared/utils/wordutils.ts","./src/hooks/usemanualsync.ts","./src/types/global.d.ts"],"version":"5.6.3"}
@@ -3,6 +3,11 @@ import react from '@vitejs/plugin-react';
// https://vite.dev/config/
export default defineConfig({
plugins: [react()],
server: {
port: 5173,
// Enable CORS for local development
cors: true,
},
build: {
minify: false,
sourcemap: true,
@@ -1 +0,0 @@
{"version":3,"file":"index-BECn1o8Q.js","sources":["../../node_modules/react/cjs/react.production.min.js","../../node_modules/react/index.js","../../node_modules/react/cjs/react-jsx-runtime.production.min.js","../../node_modules/react/jsx-runtime.js","../../node_modules/scheduler/cjs/scheduler.production.min.js","../../node_modules/scheduler/index.js","../../node_modules/react-dom/cjs/react-dom.production.min.js","../../node_modules/react-dom/index.js","../../node_modules/react-dom/client.js","../../node_modules/@mui/utils/esm/formatMuiErrorMessage/formatMuiErrorMessage.js","../../node_modules/@mui/material/styles/identifier.js","../../node_modules/@babel/runtime/helpers/esm/extends.js","../../node_modules/@emotion/sheet/dist/emotion-sheet.esm.js","../../node_modules/stylis/src/Enum.js","../../node_modules/stylis/src/Utility.js","../../node_modules/stylis/src/Tokenizer.js","../../node_modules/stylis/src/Parser.js","../../node_modules/stylis/src/Serializer.js","../../node_modules/stylis/src/Middleware.js","../../node_modules/@emotion/memoize/dist/emotion-memoize.esm.js","../../node_modules/@emotion/cache/dist/emotion-cache.browser.esm.js","../../node_modules/react-is/cjs/react-is.production.min.js","../../node_modules/react-is/index.js","../../node_modules/hoist-non-react-statics/dist/hoist-non-react-statics.cjs.js","../../node_modules/@emotion/utils/dist/emotion-utils.browser.esm.js","../../node_modules/@emotion/hash/dist/emotion-hash.esm.js","../../node_modules/@emotion/unitless/dist/emotion-unitless.esm.js","../../node_modules/@emotion/serialize/dist/emotion-serialize.esm.js","../../node_modules/@emotion/use-insertion-effect-with-fallbacks/dist/emotion-use-insertion-effect-with-fallbacks.browser.esm.js","../../node_modules/@emotion/react/dist/emotion-element-f0de968e.browser.esm.js","../../node_modules/@emotion/react/dist/emotion-react.browser.esm.js","../../node_modules/@emotion/is-prop-valid/dist/emotion-is-prop-valid.esm.js","../../node_modules/@emotion/styled/base/dist/emotion-styled-base.browser.esm.js","../../node_modules/@emotion/styled/dist/emotion-styled.browser.esm.js","../../node_modules/@mui/styled-engine/GlobalStyles/GlobalStyles.js","../../node_modules/@mui/styled-engine/index.js","../../node_modules/@mui/utils/node_modules/react-is/cjs/react-is.production.js","../../node_modules/@mui/utils/node_modules/react-is/index.js","../../node_modules/@mui/utils/esm/deepmerge/deepmerge.js","../../node_modules/@mui/system/esm/createBreakpoints/createBreakpoints.js","../../node_modules/@mui/system/esm/cssContainerQueries/cssContainerQueries.js","../../node_modules/@mui/system/esm/createTheme/shape.js","../../node_modules/@mui/system/esm/merge/merge.js","../../node_modules/@mui/system/esm/breakpoints/breakpoints.js","../../node_modules/@mui/utils/esm/capitalize/capitalize.js","../../node_modules/@mui/system/esm/style/style.js","../../node_modules/@mui/system/esm/memoize/memoize.js","../../node_modules/@mui/system/esm/spacing/spacing.js","../../node_modules/@mui/system/esm/createTheme/createSpacing.js","../../node_modules/@mui/system/esm/compose/compose.js","../../node_modules/@mui/system/esm/borders/borders.js","../../node_modules/@mui/system/esm/cssGrid/cssGrid.js","../../node_modules/@mui/system/esm/palette/palette.js","../../node_modules/@mui/system/esm/sizing/sizing.js","../../node_modules/@mui/system/esm/styleFunctionSx/defaultSxConfig.js","../../node_modules/@mui/system/esm/styleFunctionSx/styleFunctionSx.js","../../node_modules/@mui/system/esm/createTheme/applyStyles.js","../../node_modules/@mui/system/esm/createTheme/createTheme.js","../../node_modules/@mui/system/esm/useThemeWithoutDefault/useThemeWithoutDefault.js","../../node_modules/@mui/system/esm/useTheme/useTheme.js","../../node_modules/@mui/system/esm/GlobalStyles/GlobalStyles.js","../../node_modules/@mui/system/esm/styleFunctionSx/extendSxProp.js","../../node_modules/@mui/utils/esm/ClassNameGenerator/ClassNameGenerator.js","../../node_modules/clsx/dist/clsx.mjs","../../node_modules/@mui/system/esm/createBox/createBox.js","../../node_modules/@mui/utils/esm/generateUtilityClass/generateUtilityClass.js","../../node_modules/@mui/utils/esm/generateUtilityClasses/generateUtilityClasses.js","../../node_modules/@mui/system/esm/preprocessStyles.js","../../node_modules/@mui/system/esm/createStyled/createStyled.js","../../node_modules/@mui/system/esm/styled/styled.js","../../node_modules/@mui/utils/esm/resolveProps/resolveProps.js","../../node_modules/@mui/system/esm/useThemeProps/getThemeProps.js","../../node_modules/@mui/system/esm/useThemeProps/useThemeProps.js","../../node_modules/@mui/utils/esm/useEnhancedEffect/useEnhancedEffect.js","../../node_modules/@mui/system/esm/useMediaQuery/useMediaQuery.js","../../node_modules/@mui/utils/esm/clamp/clamp.js","../../node_modules/@mui/system/esm/colorManipulator/colorManipulator.js","../../node_modules/@mui/utils/esm/createChainedFunction/createChainedFunction.js","../../node_modules/@mui/utils/esm/debLine truncated
@@ -0,0 +1 @@
{"version":3,"file":"index-V91xgc4E.js","sources":["../../node_modules/react/cjs/react.production.min.js","../../node_modules/react/index.js","../../node_modules/react/cjs/react-jsx-runtime.production.min.js","../../node_modules/react/jsx-runtime.js","../../node_modules/scheduler/cjs/scheduler.production.min.js","../../node_modules/scheduler/index.js","../../node_modules/react-dom/cjs/react-dom.production.min.js","../../node_modules/react-dom/index.js","../../node_modules/react-dom/client.js","../../node_modules/@mui/utils/node_modules/react-is/cjs/react-is.production.js","../../node_modules/@mui/utils/node_modules/react-is/index.js","../../node_modules/@mui/utils/esm/deepmerge/deepmerge.js","../../node_modules/@mui/utils/esm/formatMuiErrorMessage/formatMuiErrorMessage.js","../../node_modules/@mui/utils/esm/capitalize/capitalize.js","../../node_modules/@mui/utils/esm/createChainedFunction/createChainedFunction.js","../../node_modules/@mui/utils/esm/debounce/debounce.js","../../node_modules/@mui/utils/esm/isMuiElement/isMuiElement.js","../../node_modules/@mui/utils/esm/ownerDocument/ownerDocument.js","../../node_modules/@mui/utils/esm/ownerWindow/ownerWindow.js","../../node_modules/@mui/utils/esm/setRef/setRef.js","../../node_modules/@mui/utils/esm/useEnhancedEffect/useEnhancedEffect.js","../../node_modules/@mui/utils/esm/useId/useId.js","../../node_modules/@mui/utils/esm/useControlled/useControlled.js","../../node_modules/@mui/utils/esm/useEventCallback/useEventCallback.js","../../node_modules/@mui/utils/esm/useForkRef/useForkRef.js","../../node_modules/@mui/utils/esm/useLazyRef/useLazyRef.js","../../node_modules/@mui/utils/esm/useOnMount/useOnMount.js","../../node_modules/@mui/utils/esm/useTimeout/useTimeout.js","../../node_modules/@mui/utils/esm/isFocusVisible/isFocusVisible.js","../../node_modules/@mui/utils/esm/getScrollbarSize/getScrollbarSize.js","../../node_modules/@mui/utils/esm/getValidReactChildren/getValidReactChildren.js","../../node_modules/@mui/utils/esm/visuallyHidden/visuallyHidden.js","../../node_modules/@mui/utils/esm/resolveProps/resolveProps.js","../../node_modules/@mui/utils/esm/composeClasses/composeClasses.js","../../node_modules/@mui/utils/esm/ClassNameGenerator/ClassNameGenerator.js","../../node_modules/@mui/utils/esm/generateUtilityClass/generateUtilityClass.js","../../node_modules/@mui/utils/esm/generateUtilityClasses/generateUtilityClasses.js","../../node_modules/@mui/utils/esm/clamp/clamp.js","../../node_modules/@mui/utils/esm/isHostComponent/isHostComponent.js","../../node_modules/@mui/utils/esm/appendOwnerState/appendOwnerState.js","../../node_modules/clsx/dist/clsx.mjs","../../node_modules/@mui/utils/esm/extractEventHandlers/extractEventHandlers.js","../../node_modules/@mui/utils/esm/omitEventHandlers/omitEventHandlers.js","../../node_modules/@mui/utils/esm/mergeSlotProps/mergeSlotProps.js","../../node_modules/@mui/utils/esm/resolveComponentProps/resolveComponentProps.js","../../node_modules/@mui/utils/esm/useSlotProps/useSlotProps.js","../../node_modules/@mui/utils/esm/getReactElementRef/getReactElementRef.js","../../node_modules/@mui/system/esm/merge/merge.js","../../node_modules/@mui/system/esm/cssContainerQueries/cssContainerQueries.js","../../node_modules/@mui/system/esm/breakpoints/breakpoints.js","../../node_modules/@mui/system/esm/style/style.js","../../node_modules/@mui/system/esm/memoize/memoize.js","../../node_modules/@mui/system/esm/spacing/spacing.js","../../node_modules/@mui/system/esm/compose/compose.js","../../node_modules/@mui/system/esm/borders/borders.js","../../node_modules/@mui/system/esm/cssGrid/cssGrid.js","../../node_modules/@mui/system/esm/palette/palette.js","../../node_modules/@mui/system/esm/sizing/sizing.js","../../node_modules/@mui/system/esm/styleFunctionSx/defaultSxConfig.js","../../node_modules/@mui/system/esm/styleFunctionSx/styleFunctionSx.js","../../node_modules/@mui/system/esm/styleFunctionSx/extendSxProp.js","../../node_modules/@babel/runtime/helpers/esm/extends.js","../../node_modules/@emotion/sheet/dist/emotion-sheet.esm.js","../../node_modules/stylis/src/Enum.js","../../node_modules/stylis/src/Utility.js","../../node_modules/stylis/src/Tokenizer.js","../../node_modules/stylis/src/Parser.js","../../node_modules/stylis/src/Serializer.js","../../node_modules/stylis/src/Middleware.js","../../node_modules/@emotion/memoize/dist/emotion-memoize.esm.js","../../node_modules/@emotion/cache/dist/emotion-cache.browser.esm.js","../../node_modules/react-is/cjs/react-is.production.min.js","../../node_modules/react-is/index.js","../../node_modules/hoist-non-react-statics/dist/hoist-non-react-statics.cjs.js","../../node_modules/@emotion/utils/dist/emotion-utils.browser.esm.js","../../node_modules/@emotion/hash/dist/emotion-hash.esm.js","../../node_modules/@emotion/unitless/dist/emotion-unitless.esm.js","../../node_modules/@emotion/serialize/dist/emotion-serialize.esm.js","../../node_modules/@emotion/use-insertion-effect-with-fallbacks/dist/emotion-useLine truncated
@@ -2,6 +2,10 @@
<html lang="en">
<head>
<meta charset="UTF-8" />
<!-- Load Inter font to match karaoke-gen styling -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&display=swap" rel="stylesheet">
<link rel="icon" type="image/x-icon" href="/favicon.ico" />
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
@@ -9,8 +13,8 @@
<link rel="icon" type="image/png" sizes="192x192" href="/android-chrome-192x192.png" />
<link rel="icon" type="image/png" sizes="512x512" href="/android-chrome-512x512.png" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Nomad Karaoke: Lyrics Review</title>
<script type="module" crossorigin src="/assets/index-BECn1o8Q.js"></script>
<title>Nomad Karaoke: Lyrics Transcription Review</title>
<script type="module" crossorigin src="/assets/index-V91xgc4E.js"></script>
</head>
<body>
<div id="root"></div>
@@ -0,0 +1,5 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" version="1.1" viewBox="45 60 255 135">
<g data-v-70b83f88="" fill="#ff7acc" class="basesvg" transform="translate(48.368003845214844,62.62239074707031)"><g fill-rule="" class="tp-name" transform="translate(0,0)"><g transform="scale(1.4000000000000004)"><g stroke="#ff7acc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" paint-order="stroke" data-gra-attr="stroke" fill-opacity="0"><path d="M1.4-33.44L1.4 0C31.48 0 3.96 0 34.04 0L34.04-32.64 17.81-32.64 17.81-20.61 1.4-33.44ZM52.21-33.76C42.6-33.76 34.81-25.97 34.81-16.37 34.81-6.76 42.6 1.03 52.21 1.03 61.81 1.03 69.6-6.76 69.6-16.37 69.6-25.97 61.81-33.76 52.21-33.76ZM70.7 0L103.34 0 103.25-33.9 86.93-21.54 70.61-33.9 70.7 0ZM102.34 0C138.2 0 107.52 0 143.56 0L122.95-34.93 102.34 0ZM158.84 0.05C167.84 0.05 175.16-7.27 175.16-16.27 175.16-25.27 167.84-32.64 158.84-32.64L142.52-32.64C142.52 0.75 142.52-33.34 142.52 0.05L158.84 0.05Z" transform="translate(-1.399999976158142, 34.93000030517578)"></path></g><g transform="translate(0,38.959999084472656)"><g stroke="#ff7acc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" paint-order="stroke" data-gra-attr="stroke" fill="#ff7acc" fill-opacity="0" transform="scale(1.02719)"><path d="M24.43 0L17.67 0 7.98-11.87 7.98 0 2.63 0 2.63-26.64 7.98-26.64 7.98-14.69 17.67-26.64 24.12-26.64 13.13-13.44 24.43 0ZM45.03 0L43.27-5.08 32.66-5.08 30.91 0 25.3 0 34.88-26.68 41.1-26.68 50.68 0 45.03 0ZM34.11-9.35L41.82-9.35 37.97-20.5 34.11-9.35ZM73.19 0L67.01 0 61.13-10.38 58.61-10.38 58.61 0 53.27 0 53.27-26.64 63.27-26.64Q66.36-26.64 68.53-25.55 70.71-24.47 71.8-22.61 72.89-20.76 72.89-18.47L72.89-18.47Q72.89-15.84 71.36-13.72 69.83-11.6 66.82-10.8L66.82-10.8 73.19 0ZM58.61-22.21L58.61-14.39 63.08-14.39Q65.25-14.39 66.32-15.44 67.39-16.49 67.39-18.36L67.39-18.36Q67.39-20.19 66.32-21.2 65.25-22.21 63.08-22.21L63.08-22.21 58.61-22.21ZM94.82 0L93.07-5.08 82.46-5.08 80.7 0 75.09 0 84.67-26.68 90.89-26.68 100.47 0 94.82 0ZM83.91-9.35L91.62-9.35 87.76-20.5 83.91-9.35ZM115.43 0.27Q111.69 0.27 108.56-1.49 105.43-3.24 103.6-6.35 101.76-9.47 101.76-13.4L101.76-13.4Q101.76-17.29 103.6-20.4 105.43-23.51 108.56-25.27 111.69-27.02 115.43-27.02L115.43-27.02Q119.21-27.02 122.32-25.27 125.43-23.51 127.24-20.4 129.05-17.29 129.05-13.4L129.05-13.4Q129.05-9.47 127.24-6.35 125.43-3.24 122.3-1.49 119.17 0.27 115.43 0.27L115.43 0.27ZM115.43-4.5Q117.83-4.5 119.66-5.59 121.5-6.68 122.53-8.7 123.56-10.73 123.56-13.4L123.56-13.4Q123.56-16.07 122.53-18.07 121.5-20.08 119.66-21.15 117.83-22.21 115.43-22.21L115.43-22.21Q113.02-22.21 111.17-21.15 109.32-20.08 108.29-18.07 107.26-16.07 107.26-13.4L107.26-13.4Q107.26-10.73 108.29-8.7 109.32-6.68 111.17-5.59 113.02-4.5 115.43-4.5L115.43-4.5ZM153.82 0L147.06 0 137.37-11.87 137.37 0 132.02 0 132.02-26.64 137.37-26.64 137.37-14.69 147.06-26.64 153.51-26.64 142.52-13.44 153.82 0ZM171.79-22.33L161.67-22.33 161.67-15.65 170.64-15.65 170.64-11.41 161.67-11.41 161.67-4.35 171.79-4.35 171.79 0 156.33 0 156.33-26.68 171.79-26.68 171.79-22.33Z" transform="translate(-2.630000114440918, 27.020000457763672)"></path></g></g></g></g> <g data-gra="path-slogan" fill-rule="" class="tp-slogan" fill="#ffdf6b" transform="translate(5,111.788818359375)"><rect x="0" height="1" y="5.9832000732421875" width="14.278396606445312"></rect> <rect height="1" y="5.9832000732421875" width="14.278396606445312" x="218.985595703125"></rect> <g transform="translate(17.278396606445312,0)"><g transform="scale(1.2800000000000002)"><path d="M10.21-8.38L12.02-8.38L9.68 0L7.70 0L6.13-5.96L4.49 0L2.52 0.01L0.26-8.38L2.06-8.38L3.54-1.87L5.24-8.38L7.12-8.38L8.72-1.91L10.21-8.38ZM18.38-8.38L20.06-8.38L20.06 0L18.38 0L18.38-3.56L14.80-3.56L14.80 0L13.12 0L13.12-8.38L14.80-8.38L14.80-4.93L18.38-4.93L18.38-8.38ZM26.58-7.02L23.40-7.02L23.40-4.92L26.22-4.92L26.22-3.59L23.40-3.59L23.40-1.37L26.58-1.37L26.58 0L21.72 0L21.72-8.39L26.58-8.39L26.58-7.02ZM34.37 0L32.42 0L30.58-3.26L29.78-3.26L29.78 0L28.10 0L28.10-8.38L31.25-8.38Q32.22-8.38 32.90-8.03Q33.59-7.69 33.93-7.11Q34.27-6.53 34.27-5.81L34.27-5.81Q34.27-4.98 33.79-4.31Q33.31-3.65 32.36-3.40L32.36-3.40L34.37 0ZM29.78-6.98L29.78-4.52L31.19-4.52Q31.87-4.52 32.21-4.85Q32.54-5.18 32.54-5.77L32.54-5.77Q32.54-6.35 32.21-6.67Q31.87-6.98 31.19-6.98L31.19-6.98L29.78-6.98ZM40.66-7.02L37.48-7.02L37.48-4.92L40.30-4.92L40.30-3.59L37.48-3.59L37.48-1.37L40.66-1.37L40.66 0L35.80 0L35.80-8.39L40.66-8.39L40.66-7.02ZM47.92-8.38L49.70-8.38L46.63 0L44.59 0L41.52-8.38L43.32-8.38L45.62-1.72L47.92-8.38ZM55.57-7.02L52.39-7.02L52.39-4.92L55.21-4.92L55.21-3.59L52.39-3.59L52.39-1.37L55.57-1.37L55.57 0L50.71 0L50.71-8.39L55.57-8.39L55.57-7.02ZM63.36 0L61.42 0L59.57-3.26L58.78-3.26L58.78 0L57.10 0L57.10-8.38L60.24-8.38Q61.21-8.38 61.90-8.03Q62.58-7.69 62.92-7.11Q63.26-6.53 63.26-5.81L63.26-5.81Q63.26-4.98 62.78-4.31Q62.30-3.65 61.36-3.40L61.36-3.40L63.36 0ZM58.78-6.98L58.78-4.52L60.18-4.52Q60.86-4.52 61.20-4.85Q61.54-5.18 61.54-5.77L61.54-5.77Q61.54-6.35 61.20-6.67Q60.86-6.98 60.18-6.98L60.18-6.98L58.78-6.98ZM72.43-8.38L74.30-8.38L71.47-2.92L71.47 0L69.Line truncated
</svg>

After

Width:  |  Height:  |  Size: 9.0 KiB

@@ -148,8 +148,21 @@ class OutputGenerator:
audio_filepath: str,
artist: Optional[str] = None,
title: Optional[str] = None,
ass_only: bool = False,
) -> OutputPaths:
"""Generate all requested output formats."""
"""Generate all requested output formats.
Args:
transcription_corrected: Corrected transcription data
lyrics_results: Lyrics data from various providers
output_prefix: Prefix for output filenames
audio_filepath: Path to audio file
artist: Optional artist name
title: Optional title
ass_only: If True (only in preview_mode), generate only ASS subtitles
without video encoding. Useful when video encoding is offloaded
to an external service.
"""
outputs = OutputPaths()
try:
@@ -165,8 +178,9 @@ class OutputGenerator:
# Generate ASS subtitles for preview
outputs.ass = self.subtitle.generate_ass(transcription_corrected.resized_segments, output_prefix, audio_filepath)
# Generate preview video
outputs.video = self.video.generate_preview_video(outputs.ass, audio_filepath, output_prefix)
# Generate preview video (unless ass_only mode for GCE encoding)
if not ass_only:
outputs.video = self.video.generate_preview_video(outputs.ass, audio_filepath, output_prefix)
return outputs
@@ -352,19 +352,44 @@ class VideoGenerator:
self.logger.error(f"Failed to resize background image: {e.output}")
raise
def _escape_ffmpeg_filter_path(self, path: str) -> str:
"""Escape a path for FFmpeg filter expressions (for subprocess without shell).
When using subprocess with a command list (no shell), FFmpeg receives the
filter string directly. FFmpeg's filter parser requires escaping:
- Backslashes: double them (\ -> \\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\')
- Spaces: escape with backslash ( -> \ )
Note: This is different from shell escaping. The '\\'\\''' pattern used for
shell escaping does NOT work when subprocess passes args directly to FFmpeg.
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
"""
# First escape existing backslashes (\ -> \\)
escaped = path.replace("\\", "\\\\")
# Escape single quotes (' -> \\')
# In the actual string we need 3 backslashes before the quote
escaped = escaped.replace("'", "\\\\\\'")
# Escape spaces
escaped = escaped.replace(" ", "\\ ")
return escaped
def _build_ass_filter(self, ass_path: str) -> str:
"""Build ASS filter with font directory support."""
ass_filter = f"ass={ass_path}"
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
ass_filter = f"ass={escaped_ass_path}"
# Get font path from styles configuration
karaoke_styles = self.styles.get("karaoke", {})
font_path = karaoke_styles.get("font_path")
if font_path and os.path.isfile(font_path):
font_dir = os.path.dirname(font_path)
ass_filter += f":fontsdir={font_dir}"
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
ass_filter += f":fontsdir={escaped_font_dir}"
self.logger.info(f"Returning ASS filter with fonts dir: {ass_filter}")
return ass_filter
def _build_ffmpeg_command(self, ass_path: str, audio_path: str, output_path: str) -> List[str]:
@@ -210,6 +210,7 @@ def test_build_ass_filter_no_font_path(video_generator):
"""Test _build_ass_filter with no font path configured."""
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping needed
assert result == "ass=test.ass"
@@ -220,13 +221,16 @@ def test_build_ass_filter_with_valid_font_path(video_generator, tmp_path):
font_dir.mkdir()
font_file = font_dir / "test.ttf"
font_file.touch()
# Update styles to include font path
video_generator.styles["karaoke"]["font_path"] = str(font_file)
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
expected = f"ass=test.ass:fontsdir={str(font_dir)}"
# Paths with no special chars don't need escaping
# But font_dir path from tmp_path may have spaces on some systems
escaped_font_dir = str(font_dir).replace("\\", "\\\\").replace("'", "\\\\'").replace(" ", "\\ ")
expected = f"ass=test.ass:fontsdir={escaped_font_dir}"
assert result == expected
@@ -234,9 +238,10 @@ def test_build_ass_filter_with_invalid_font_path(video_generator):
"""Test _build_ass_filter with invalid font path (file doesn't exist)."""
# Set non-existent font path
video_generator.styles["karaoke"]["font_path"] = "/nonexistent/font.ttf"
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -244,9 +249,10 @@ def test_build_ass_filter_with_empty_font_path(video_generator):
"""Test _build_ass_filter with empty font path."""
# Set empty font path
video_generator.styles["karaoke"]["font_path"] = ""
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -254,9 +260,10 @@ def test_build_ass_filter_with_none_font_path(video_generator):
"""Test _build_ass_filter with None font path."""
# Set None font path
video_generator.styles["karaoke"]["font_path"] = None
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -269,9 +276,10 @@ def test_build_ass_filter_no_karaoke_section(tmp_path):
video_resolution=(1920, 1080),
styles=styles,
)
ass_path = "test.ass"
result = generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -279,19 +287,73 @@ def test_build_ass_filter_logging(video_generator, tmp_path, caplog):
"""Test _build_ass_filter logging behavior."""
# Set log level to INFO to capture the logs
caplog.set_level(logging.INFO)
# Create a test font file
font_dir = tmp_path / "fonts"
font_dir.mkdir()
font_file = font_dir / "test.ttf"
font_file.touch()
# Update styles to include font path
video_generator.styles["karaoke"]["font_path"] = str(font_file)
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# Check that info log was generated
assert "Returning ASS filter with fonts dir:" in caplog.text
assert str(font_dir) in caplog.text
def test_escape_ffmpeg_filter_path(video_generator):
"""Test _escape_ffmpeg_filter_path uses FFmpeg filter escaping (not shell escaping).
FFmpeg filter escaping rules (for subprocess without shell):
- Backslashes: double them (\ -> \\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\')
- Spaces: escape with backslash ( -> \ )
"""
# Test simple path without special chars (no escaping needed)
assert video_generator._escape_ffmpeg_filter_path("/path/to/file.ass") == "/path/to/file.ass"
# Test apostrophe escaping (fixes paths like "I'm With You")
# Apostrophe becomes \\\' (three backslashes + quote) in the actual string
result = video_generator._escape_ffmpeg_filter_path("./Avril Lavigne - I'm With You/file.ass")
assert result == "./Avril\\ Lavigne\\ -\\ I\\\\\\'m\\ With\\ You/file.ass"
# Test path with spaces (escaped with backslash)
assert video_generator._escape_ffmpeg_filter_path("/path/with spaces/file.ass") == "/path/with\\ spaces/file.ass"
# Test path with colons (no escaping needed - colon is only special in filter options)
assert video_generator._escape_ffmpeg_filter_path("C:/path/file.ass") == "C:/path/file.ass"
# Test multiple apostrophes
result = video_generator._escape_ffmpeg_filter_path("It's Bob's file.ass")
assert result == "It\\\\\\'s\\ Bob\\\\\\'s\\ file.ass"
def test_build_ass_filter_escapes_path_with_apostrophe(video_generator):
"""Test _build_ass_filter properly escapes paths containing apostrophes."""
ass_path = "./Avril Lavigne - I'm With You/cache/temp.ass"
result = video_generator._build_ass_filter(ass_path)
# Spaces escaped with backslash, apostrophe escaped with \\\'
assert result == "ass=./Avril\\ Lavigne\\ -\\ I\\\\\\'m\\ With\\ You/cache/temp.ass"
def test_build_ass_filter_escapes_font_dir_with_special_chars(video_generator, tmp_path):
"""Test _build_ass_filter properly escapes font directory with special characters."""
# Create a font directory with special characters
font_dir = tmp_path / "It's a Font Dir"
font_dir.mkdir()
font_file = font_dir / "test.ttf"
font_file.touch()
# Update styles to include font path
video_generator.styles["karaoke"]["font_path"] = str(font_file)
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# Should have fontsdir with escaped path
assert "fontsdir=" in result
# Apostrophe should be escaped using \\\' pattern (3 backslashes + quote)
assert "\\\\\\'s" in result
Generated
+467 -454
View File
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.87.1"
version = "0.92.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
+41
View File
@@ -0,0 +1,41 @@
#!/bin/bash
# Admin API helper script
# Usage: ./scripts/admin_api.sh <endpoint> [method] [data]
#
# Examples:
# ./scripts/admin_api.sh /api/jobs # List all jobs (admin sees all)
# ./scripts/admin_api.sh /api/jobs?limit=10 # List 10 jobs
# ./scripts/admin_api.sh /api/jobs/abc123 # Get specific job
# ./scripts/admin_api.sh /api/jobs/abc123/retry POST # Retry a job
# ./scripts/admin_api.sh /api/jobs/abc123/select-instrumental POST '{"selection":"clean"}'
set -e
API_URL="${KARAOKE_API_URL:-https://api.nomadkaraoke.com}"
# Get admin token from Secret Manager if not set
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke 2>/dev/null)
fi
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
echo "Error: Could not get admin token. Run 'gcloud auth login' first." >&2
exit 1
fi
ENDPOINT="${1:-/api/admin/jobs}"
METHOD="${2:-GET}"
DATA="$3"
if [ "$METHOD" = "GET" ]; then
curl -s "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
elif [ -n "$DATA" ]; then
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d "$DATA" | jq .
else
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
fi
+34 -24
View File
@@ -121,16 +121,25 @@ class TestVideoBackgroundProcessor:
processor.get_audio_duration("/path/to/audio.wav")
def test_escape_filter_path(self, processor):
"""Test path escaping for ffmpeg filters."""
# Test backslash escaping
assert processor.escape_filter_path("C:\\Users\\test\\file.ass") == "C\\:\\\\Users\\\\test\\\\file.ass"
# Test colon escaping
assert processor.escape_filter_path("/path/to/file.ass") == "/path/to/file.ass"
assert processor.escape_filter_path("C:/path/file.ass") == "C\\:/path/file.ass"
# Test combined escaping
assert processor.escape_filter_path("C:\\path:with:colons\\file.ass") == "C\\:\\\\path\\:with\\:colons\\\\file.ass"
"""Test path escaping for ffmpeg filters using single-quote wrapping."""
# Test simple path (wrapped in single quotes)
assert processor.escape_filter_path("/path/to/file.ass") == "'/path/to/file.ass'"
# Test path with spaces (protected by single quotes)
assert processor.escape_filter_path("/path/with spaces/file.ass") == "'/path/with spaces/file.ass'"
# Test apostrophe escaping (fixes paths like "I'm With You")
# Single quote becomes '\'' (end quote, escaped quote, start quote)
assert processor.escape_filter_path("./Avril Lavigne - I'm With You/file.ass") == "'./Avril Lavigne - I'\\''m With You/file.ass'"
# Test path with colons (protected by single quotes)
assert processor.escape_filter_path("C:/path/file.ass") == "'C:/path/file.ass'"
# Test Windows-style path (backslashes preserved, wrapped in quotes)
assert processor.escape_filter_path("C:\\Users\\test\\file.ass") == "'C:\\Users\\test\\file.ass'"
# Test multiple apostrophes
assert processor.escape_filter_path("It's Bob's file.ass") == "'It'\\''s Bob'\\''s file.ass'"
def test_build_video_filter_no_darkness(self, processor):
"""Test video filter building without darkness overlay."""
@@ -139,13 +148,13 @@ class TestVideoBackgroundProcessor:
darkness_percent=0,
fonts_dir=None
)
# Should contain scale and crop
assert "scale=w=3840:h=2160:force_original_aspect_ratio=increase,crop=3840:2160" in vf_filter
# Should contain ASS filter
assert "ass=/path/to/subs.ass" in vf_filter
# Should contain ASS filter with single-quoted path
assert "ass='/path/to/subs.ass'" in vf_filter
# Should NOT contain darkening
assert "drawbox" not in vf_filter
@@ -156,10 +165,10 @@ class TestVideoBackgroundProcessor:
darkness_percent=50,
fonts_dir=None
)
# Should contain darkening with correct alpha
assert "drawbox=x=0:y=0:w=iw:h=ih:color=black@0.50:t=fill" in vf_filter
# Should contain ASS filter after darkening
assert vf_filter.index("drawbox") < vf_filter.index("ass=")
@@ -171,20 +180,21 @@ class TestVideoBackgroundProcessor:
darkness_percent=0,
fonts_dir="/path/to/fonts"
)
# Should contain fonts directory in ASS filter
assert "fontsdir=/path/to/fonts" in vf_filter
# Should contain fonts directory in ASS filter (single-quoted)
assert "fontsdir='/path/to/fonts'" in vf_filter
def test_build_video_filter_path_escaping(self, processor):
"""Test video filter building with special characters in paths."""
# Test with apostrophe in path
vf_filter = processor.build_video_filter(
ass_subtitles_path="C:\\Users\\test\\subs.ass",
ass_subtitles_path="./Avril Lavigne - I'm With You/subs.ass",
darkness_percent=0,
fonts_dir=None
)
# Paths should be escaped
assert "C\\:\\\\Users\\\\test\\\\subs.ass" in vf_filter
# Paths should be single-quoted with apostrophes escaped as '\''
assert "'./Avril Lavigne - I'\\''m With You/subs.ass'" in vf_filter
@patch('subprocess.run')
def test_execute_command_with_fallback_gpu_success(self, mock_run, processor_with_nvenc, mock_logger):