Compare commits

...
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.5 fddb06cf04 chore: Bump version to 0.91.0
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 00:52:54 -05:00
Andrew Beveridge 792f609d46 Merge remote-tracking branch 'origin/main' into feat/gce-preview-encoding 2026-01-05 00:52:40 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 f2639754a9 fix: Download GCE-encoded files locally for YouTube upload (#201)
## Summary
- Fix YouTube upload failing when using GCE encoding backend
- GCE encoding returns GCS blob paths, but YouTube upload expected local
files
- Added `_download_gce_encoded_files()` to download encoded videos from
GCS before distribution

## Root Cause
Job `62a95e01` completed but didn't upload to YouTube. Investigation
showed:
1. GCE encoding stores files in GCS and returns blob paths like
`jobs/{id}/finals/output_4k_lossless.mp4`
2. The orchestrator stored these paths directly without downloading
3. YouTube upload checks `os.path.isfile()` which returns `False` for
GCS paths
4. Result: "No video file available for YouTube upload" warning

## Changes
| File | Change |
|------|--------|
| `backend/workers/video_worker_orchestrator.py` | Added
`_download_gce_encoded_files()` method called after GCE encoding |
| `backend/tests/test_video_worker_orchestrator.py` | Added 2 tests for
GCE download behavior |
| `docs/LESSONS-LEARNED.md` | Documented the issue and fix |

## Test plan
- [x] All 30 orchestrator tests pass
- [x] New test `test_run_encoding_gce_downloads_files` verifies GCE
paths are downloaded
- [x] New test `test_run_encoding_local_does_not_download` verifies
local encoding skips download
- [ ] Manual test: Create job via web UI and verify YouTube upload works

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 05:51:52 +00:00
Andrew BeveridgeandClaude Opus 4.5 04fb2c7c83 feat: Offload preview video encoding to GCE for faster generation
Move the slow FFmpeg encoding step of preview video generation to the
high-performance GCE encoding worker (C4-standard-8, 3.9 GHz) to reduce
preview generation time from 60+ seconds to ~15-20 seconds.

Key changes:
- Add /encode-preview endpoint to GCE encoding worker
- Add encode_preview_video() method to EncodingService
- Add ass_only parameter to generate ASS without video encoding
- Modify review route to use GCE preview encoding with fallback
- Add USE_GCE_PREVIEW_ENCODING config setting

The approach splits ASS subtitle generation (fast, ~2s on Cloud Run) from
video encoding (slow, offloaded to GCE). Falls back to local encoding if
GCE is unavailable or disabled.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 00:39:10 -05:00
Andrew Beveridge ☄️ 94182277fa Update pyproject.toml (#199) 2026-01-05 04:14:37 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 e16da0fa5f fix: Allow explicit override of YouTube upload server default (#198)
## Summary
- Changed `enable_youtube_upload` from `bool=False` to
`Optional[bool]=None` in request models
- Allows explicit `false` to override server default when
`DEFAULT_ENABLE_YOUTUBE_UPLOAD=true`
- E2E tests now intercept requests to disable YouTube upload for test
jobs

## Changes
- **Request models**: Changed to `Optional[bool] = None` to support
three states (None/True/False)
- **API routes**: All job creation endpoints now compute effective value
using `if is not None else` pattern
- **E2E test**: Added Playwright request interception to set
`enable_youtube_upload=false` for test jobs
- **Tests**: Updated to expect `None` instead of `False` for request
model defaults
- **Docs**: Added lesson learned about Python `or` operator gotcha with
boolean defaults

## Testing
- [x] Backend tests pass (`make test`)
- [x] No TypeScript errors in frontend changes

## Review
- [x] CodeRabbit CLI review completed locally
- [x] No issues found

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 23:04:06 -05:00
Andrew Beveridge ☄️andClaude b8e15b2af5 fix: Ensure user data is fetched after client-side navigation to /app (#197)
## Summary

- Fix E2E happy path test failure at Step 3 (Verify Credits)
- After beta enrollment, client-side navigation to `/app` didn't fetch
user data
- Credits weren't displayed because zustand auth store had `user: null`

## Changes

- **frontend/app/app/page.tsx**: Added `useEffect` that calls
`fetchUser()` when there's a token but no user data loaded. This ensures
user data is fetched regardless of how the user navigated to the page
(page reload vs client-side navigation).

- **frontend/e2e/regression/authentication.spec.ts**: Added new test
`'beta enrollment shows credits after redirect to app'` that
specifically verifies credits are displayed after the beta enrollment
flow.

- **docs/LESSONS-LEARNED.md**: Documented the root cause (zustand
module-level initialization only runs once, not on client-side
navigation) for future reference.

## Root Cause

The auth module's initialization code (`fetchUser()` call) only runs
once when the module is first imported. After beta enrollment,
`router.push('/app')` performs client-side navigation which doesn't
reload modules. The user data was never fetched, leaving `user: null` in
the store.

## Testing

- [x] New regression test passes locally (9/9 auth tests pass)
- [x] Build succeeds
- [x] CodeRabbit CLI review completed locally

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 21:33:01 -05:00
Andrew Beveridge ☄️ 2ee3cabc92 Added admin API script (#196) 2026-01-05 01:33:17 +00:00
Andrew Beveridge ☄️andClaude ae85ec0a09 feat: Move worker logs to Firestore subcollection to avoid 1MB limit (#195)
## Summary

Move `worker_logs` from embedded array in job documents to a Firestore
subcollection (`jobs/{job_id}/logs`) to avoid the 1MB document size
limit. This fixes job failures when logs exceed the limit during
long-running operations like video encoding.

**Problem:** Job `501258e1` failed when logs reached 1.26MB (2,547 log
entries), representing 98.6% of the maximum document size.

## Changes

### New Files
- `backend/models/worker_log.py` - WorkerLogEntry dataclass with factory
methods, serialization (to_dict, to_legacy_dict, from_dict)
- `backend/tests/test_worker_log_subcollection.py` - 24 unit tests
- `backend/tests/emulator/test_worker_logs_subcollection.py` - 11
integration tests using Firestore emulator

### Modified Files
- `backend/services/firestore_service.py` - Added subcollection methods
(append, get, count, delete)
- `backend/services/job_manager.py` - Use subcollection with fallback to
legacy embedded arrays
- `backend/api/routes/jobs.py` - Updated logs endpoint to use new
methods
- `backend/config.py` - Added `USE_LOG_SUBCOLLECTION` feature flag
(default: true)
- `infrastructure/__main__.py` - Added TTL policy (30 days) and
composite index for logs subcollection
- `pyproject.toml` - Version bump 0.89.6 → 0.90.0

### Documentation
- `docs/README.md` - Added to Recent Changes
- `docs/ARCHITECTURE.md` - Added logs subcollection to Firestore
Collections table
- `docs/LESSONS-LEARNED.md` - Added "Firestore Document 1MB Limit with
Embedded Arrays" section

## Testing

- [x] 24 unit tests pass
(`backend/tests/test_worker_log_subcollection.py`)
- [x] 11 emulator integration tests pass
(`backend/tests/emulator/test_worker_logs_subcollection.py`)
- [x] All 68 existing emulator tests pass

## Review

- [x] CodeRabbit CLI review completed locally
- [x] Review feedback addressed (timestamp formatting fix)

## Rollback Plan

Set `USE_LOG_SUBCOLLECTION=false` environment variable and redeploy. New
logs will go back to embedded array; subcollection logs will remain but
won't be read.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 20:14:25 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 a2282357da docs: Add lessons learned for GCE encoding response format handling (#194)
## Summary
Document lessons learned from debugging GCE encoding response format
issues.

## Changes
- Add "External Service Response Format Mismatches" section to
LESSONS-LEARNED.md
- Document defensive type checking patterns for API integration
- Add recent changes entry for GCE encoding fixes (PRs #186-193)
- Reference the worker logs rearchitecture plan

## Context
This session fixed multiple response format bugs in the GCE encoding
integration:
- Status endpoint returning list instead of dict
- output_files as list of paths instead of dict with format keys
- Created plan for worker logs to use Firestore subcollection (avoid 1MB
limit)

These lessons are valuable for future debugging of external service
integrations.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 00:29:23 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 44c44da95e fix: Handle GCE worker output_files as list of paths (#193)
## Summary
Fix the actual root cause of `'list' object has no attribute 'get'`
error in GCE encoding.

## Problem
The GCE encoding worker returns `output_files` as a **list of paths**:
```json
"output_files": [
    "jobs/501258e1/finals/output_4k_lossless.mp4",
    "jobs/501258e1/finals/output_4k_lossy.mp4",
    "jobs/501258e1/finals/output_720p.mp4"
]
```

But the code expected a **dict** with format keys:
```json
{"mp4_4k_lossless": "path/...", "mp4_4k_lossy": "path/...", ...}
```

This caused the error when calling `.get()` on the list.

## Solution
Added conversion logic to map the list of paths to the expected dict
format by matching filename patterns (e.g., `output_4k_lossless.mp4` ->
`mp4_4k_lossless`).

## Testing
- [x] Verified GCE worker response format
- [ ] Deploy and test with job 501258e1

## Notes
PR #191 fixed a similar issue in `wait_for_completion` where the entire
`status` response could be a list. This PR fixes a different issue where
`output_files` within the response is a list.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 23:23:21 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 16dc451bdf docs: Add API auth guide and worker logs rearchitecture plan (#192)
## Summary
- Add API authentication documentation to CLAUDE.md for future agents
- Add comprehensive plan for rearchitecting worker logs storage

## Changes

### CLAUDE.md
- New "API Authentication (for Agents)" section covering:
  - Internal worker endpoints with X-Admin-Token
  - Firestore direct access
  - GCE encoding worker SSH commands
  - Common pitfalls to avoid

### .claude/plans/worker-logs-rearchitecture.plan.md
- Plan to move worker_logs from embedded array to Firestore
subcollection
- Fixes 1MB document size limit issue (job 501258e1 hit 1.26 MB)
- Includes TTL policy for automatic cleanup (30 days)

## Testing
- [x] Documentation review
- No code changes

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 22:37:17 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 278690b373 fix: Handle list response in encoding service wait_for_completion (#191)
## Summary
- Fix `'list' object has no attribute 'get'` error in GCE encoding
wait_for_completion
- The GCE encoding worker sometimes returns a list instead of a dict
from the /status endpoint
- PR #186 added handling in encode(), but same issue occurs in
wait_for_completion()

## Changes
- Added defensive handling to convert list responses to dicts in
wait_for_completion()

## Testing
- [x] Code review
- [ ] Deploy and test with job 501258e1

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 22:35:55 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 0802e79261 fix: Pass job_id in encoding options for GCE backend (#190)
## Summary
- Pass `job_id` in the encoding options for GCE backend
- Fixes "Encoding job unknown already exists" error on retry

## Root Cause
The `EncodingInput.options` dict was missing `job_id`, causing it to
fall back to "unknown".

## Testing
- [x] Unit tests pass
- [ ] Test with job `501258e1` after deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 20:40:44 +00:00
Andrew Beveridge ☄️ 504d1159ec chore: force backend deployment (#189)
Trigger deployment for admin endpoint

@coderabbitai ignore
2026-01-04 20:12:12 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 7acc2c9f17 feat: Add admin endpoint to reset stale worker state (#188)
## Summary
- Add `/api/admin/jobs/{id}/reset-worker-state` endpoint
- Allows admins to reset stale worker progress when jobs get stuck

## Context
When a job fails, the worker progress (e.g., `video_progress.stage =
'running'`)
can get stuck, preventing retries. This endpoint resets all worker
progress to 'pending'.

## Testing
- [ ] Test with job `501258e1` after deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 20:00:21 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 a3f5c68660 fix: Reset worker progress state on job retry (#187)
## Summary
- Fix job retry getting stuck because worker progress state wasn't reset
- When retrying, idempotency check saw `video_progress.stage =
'running'` from previous failed run
- Now properly resets progress state to 'pending' for video, render, and
screens workers

## Changes
- `backend/api/routes/jobs.py` - Add `update_state_data` calls to reset
progress

## Testing
- [x] Encoding tests pass locally
- [ ] Test with job `501258e1` after deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 19:24:02 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 27495fb4fe fix: Handle list response from GCE encoding worker (#186)
## Summary
- Fix "'list' object has no attribute 'get'" error when GCE worker
returns a list instead of dict
- Add defensive handling in `GCEEncodingBackend.encode()` to extract
first dict from list responses
- Add local tests to catch encoding integration issues earlier

## Changes
- `backend/services/encoding_interface.py` - Handle list/non-dict
responses from GCE worker
- `backend/tests/test_encoding_interface.py` - Add tests for list
response handling and backend options
- `backend/tests/test_video_worker_orchestrator.py` - Add test for GCS
paths structure
- `backend/tests/conftest.py` - Move env var setup to top before imports

## Testing
- [x] All 57 tests pass locally (`make test`)
- [x] `test_encode_handles_list_result` validates fix
- [ ] Deploy and test with existing job at
`awaiting_instrumental_selection`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 18:48:22 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 b32014a9a0 fix: Pass GCS paths to encoding backend for GCE encoding (#185)
## Summary
- Fixed `GCE encoding requires input_gcs_path and output_gcs_path in
options` error
- Jobs were failing at video generation because orchestrator wasn't
passing required GCS paths

## Changes
- Added GCS path construction in
`VideoWorkerOrchestrator._run_encoding()`
- Paths are now passed in `EncodingInput.options` dict

## Testing
- [x] All 27 orchestrator tests pass
- [x] Ready for quick deploy and test with existing job at
`awaiting_instrumental_selection`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 17:34:35 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 bb19a6e1fc fix: Add dry_run parameter to GCEEncodingBackend (#184)
## Summary
- Fixed `GCEEncodingBackend.__init__() got an unexpected keyword
argument 'dry_run'` error
- Jobs were failing during video generation when using GCE encoding

## Changes
- Added `dry_run` parameter to `GCEEncodingBackend.__init__()` for
interface consistency with `LocalEncodingBackend`
- Parameter is stored but not used since GCE is a remote service that
doesn't support dry run mode

## Testing
- [x] All 23 encoding interface tests pass
- [x] No breaking changes to existing functionality

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 11:54:27 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 1b0521cb83 docs: Add video worker orchestrator documentation (#183)
## Summary
- Add entry to README.md Recent Changes section for orchestrator
refactor
- Add Video Worker Orchestrator section to ARCHITECTURE.md with diagram
- Add "Alternative Code Paths Must Implement All Features" lesson to
LESSONS-LEARNED.md
- Create archive entry documenting the refactor

## Changes
- `docs/README.md` - Added recent change entry
- `docs/ARCHITECTURE.md` - Added orchestrator section with stage diagram
- `docs/LESSONS-LEARNED.md` - Added lesson about code path divergence
- `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` - Full
documentation

## Context
Documentation for PR #182 which unified the video worker pipeline.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 11:07:16 -05:00
Andrew Beveridge ☄️ c1588992ce refactor: Unify video worker pipeline with new orchestrator (#182)
Resolves GCE vs KaraokeFinalise code path divergence that caused features like YouTube upload, Discord notifications, and CDG/TXT packaging to be missing from the GCE encoding path.

- New services: youtube_upload_service, discord_service, packaging_service, local_encoding_service, encoding_interface
- New VideoWorkerOrchestrator coordinates all stages (packaging, encoding, distribution, notifications)
- Works with any encoding backend (GCE or local)
- 139 new tests, 976 total tests passing
- Feature flag USE_NEW_ORCHESTRATOR (default: true) with rollback capability
2026-01-04 00:25:23 -05:00
Andrew Beveridge ☄️andClaude b3045c5487 fix: Add Noto fonts for full Unicode support in video rendering (#181)
## Summary

- Fixes rendering of musical symbols (♪) in INTRO/INSTRUMENTAL/OUTRO
screens
- Adds comprehensive international font support for CJK and other
scripts
- Changes default karaoke font from Arial to Noto Sans

## Changes

- **backend/Dockerfile.base**: Install Noto font packages (core, CJK,
extra, color-emoji) with fontconfig
- **karaoke_gen/style_loader.py**: Change default font from "Arial" to
"Noto Sans"
- **backend/tests/test_routes_review.py**: Update test assertions to
match new default font
- **docs/LESSONS-LEARNED.md**: Add "Fonts in Docker for Video Rendering"
section
- **docs/README.md**: Add "Full Unicode Font Support" to Recent Changes

## Testing

- [x] Tests pass locally (`make test`)
- [x] Manual review of changes completed

## Notes

- Adds ~200-300MB to Docker base image for comprehensive font coverage
- Color emoji has limited support in FFmpeg/libass (renders as
monochrome)
- Requires base image rebuild to take effect

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 23:11:30 -05:00
Andrew Beveridge ☄️andClaude 8a7cbb4a8b feat: Integrate flacfetch cache management API (#180)
## Summary

- Integrate karaoke-gen with flacfetch's new cache management API
(v0.15.4)
- When admins clear a job's cache, both Firestore AND flacfetch GCS
cache are now cleared
- Add bulk cache management (clear all, view stats) to admin UI

## Changes

### Backend
- **FlacfetchClient** (`backend/services/flacfetch_client.py`): Add
cache management methods
  - `clear_search_cache(artist, title)` - clears specific cache entry
  - `clear_all_cache()` - clears entire flacfetch cache  
  - `get_cache_stats()` - returns cache statistics

- **Admin API** (`backend/api/routes/admin.py`): 
- Update `POST /api/admin/audio-searches/{job_id}/clear-cache` to also
call flacfetch cache API
  - Add `DELETE /api/admin/cache` - clear entire flacfetch cache
  - Add `GET /api/admin/cache/stats` - get cache statistics

### Frontend
- **API Client** (`frontend/lib/api.ts`): Add `clearAllCache()` and
`getCacheStats()` methods
- **Admin Searches Page** (`frontend/app/admin/searches/page.tsx`):
  - Add "Clear All Cache" button with confirmation dialog
  - Display cache stats (entry count, size, newest entry)

### Documentation
- Update API.md with new endpoints
- Update README.md with recent changes
- Update LESSONS-LEARNED.md with two-layer caching insight

## Architecture

```
User clicks "Clear Cache" on job
    ↓
karaoke-gen clears Firestore state_data (existing)
    ↓
karaoke-gen calls flacfetch DELETE /cache/search (NEW)
    ↓
flacfetch deletes GCS cache entry
    ↓
Next search hits trackers fresh ✓
```

## Testing

- [x] Backend tests pass (`make test` - 57 tests)
- [x] Frontend tests pass (`npm run test:unit` - 128 tests)
- [x] Frontend builds successfully
- [x] CodeRabbit CLI review completed locally

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 22:48:29 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 3ffed23880 feat: Add customer support link to app header (#179)
## Summary
- Add "Need help?" mailto link visible in header on large screens
- Add support contact option in user dropdown menu for mobile/tablet
users
- Links to andrew@nomadkaraoke.com

## Test plan
- [ ] Verify link appears in header on desktop (lg breakpoint and above)
- [ ] Verify link appears in user dropdown menu
- [ ] Verify clicking opens email client with correct address

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 21:01:47 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 ac39cc5ff7 fix: Add admin link to AuthStatus component dropdown (#178)
## Summary
- The `/app` page uses the `AuthStatus` component for its header
dropdown menu, not the `app-header.tsx` component
- This adds the admin link for users with admin role or
@nomadkaraoke.com email to the correct component

## Test plan
- [ ] Log in as admin user at https://gen.nomadkaraoke.com/app
- [ ] Click user dropdown in header
- [ ] Verify "Admin Dashboard" link appears between "Buy More Credits"
and "Sign Out"

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 20:27:43 -05:00
Andrew Beveridge ☄️andClaude d47634016d feat: Add admin link to header for admin users (#177)
## Summary

- Add "Admin" link to header navigation for users with admin access
- Add "Admin Dashboard" option to user dropdown menu
- Uses same access logic as admin layout (role=admin OR
@nomadkaraoke.com email)

## Changes

- `frontend/components/app-header.tsx` - Added admin link to nav and
dropdown menu with Shield icon

## Testing

- [x] TypeScript compiles
- [ ] Manual testing needed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 20:20:04 -05:00
Andrew Beveridge ☄️andClaude 68188e9a28 feat: Add admin UI for audio search cache management (#176)
## Summary

- Add admin UI at `/admin/searches` to view and manage cached audio
search results
- Allow admins to clear stale caches when flacfetch is updated with new
providers
- Fixes issue where jobs showed only YouTube results when lossless
sources should be available

## Changes

- **Backend**: New admin endpoints `GET /api/admin/audio-searches` and
`POST /api/admin/audio-searches/{job_id}/clear-cache`
- **Frontend**: New admin page with search results table, status
indicators, and cache clear action
- **Safety**: Status validation prevents clearing cache on
active/complete jobs
- **Docs**: Updated API.md, LESSONS-LEARNED.md, and README.md

## Testing

- [x] Backend module imports successfully
- [x] Frontend TypeScript compiles without errors
- [x] CodeRabbit CLI review completed and issues addressed

## Review

- [x] CodeRabbit CLI review completed locally (3 cycles)
- [x] Fixed: `has_lossless` now computed from ALL results, not just
first 10
- [x] Fixed: Added status validation before cache clear
- [x] Noted (not fixed): Scalability optimization for large-scale -
acceptable for admin tool

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 20:08:43 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 cbf3b815dd docs: Add gen_users migration documentation (#174)
## Summary
Adds documentation for the gen_users migration completed in PR #173.

- **README.md**: Add entry in Recent Changes about user database
separation
- **ARCHITECTURE.md**: Add Firestore Collections section documenting
`gen_users` and other collections
- **LESSONS-LEARNED.md**: Add lesson about separate collections for
multi-app projects

## Test plan
- [x] Documentation-only change, no testing required

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 18:09:57 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 6aa8f150df docs: Add GCS path lessons and update E2E spec for testmail.app (#175)
## Summary

- Add "GCS Blob Paths vs Full URIs" section to LESSONS-LEARNED.md
documenting the path format confusion bug and double-slash gotcha that
was fixed in PR #168
- Update E2E-HAPPY-PATH-TEST-SPEC.md secrets section to correctly
reflect testmail.app migration (was referencing MAILSLURP)

## Test plan

- [ ] Documentation review - verify changes are accurate and helpful

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 18:08:09 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 925dd3c262 fix: Complete mailslurp to testmail.app migration (#168)
## Summary
- Complete the migration from MailSlurp to testmail.app for E2E testing
- Update all workflow files to use `TESTMAIL_API_KEY` and
`TESTMAIL_NAMESPACE` secrets
- Update documentation and test comments

## Changes
- `.github/workflows/e2e-happy-path.yml` - Use testmail.app secrets
- `.github/workflows/ci.yml` - Use testmail.app secrets  
- `docs/E2E-HAPPY-PATH-TEST-SPEC.md` - Update documentation
- `frontend/e2e/production/happy-path-real-user.spec.ts` - Update
comments

## Test plan
- [x] Verify no remaining mailslurp references in active files (archives
excluded)
- [ ] E2E Happy Path test should run (not skip) with testmail.app
secrets

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 17:32:53 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 ab05ec8bee feat: Migrate user storage to gen_users collection (#173)
## Summary
Separates karaoke-gen user data from karaoke-decide by using a dedicated
`gen_users` collection instead of the shared `users` collection.

**Problem:** Both karaoke-gen and karaoke-decide were using the same
`users` collection in Firestore with incompatible schemas:
- karaoke-gen: email as doc ID, has `credits`, `role`, `is_active`
- karaoke-decide: hash as doc ID, has `user_id`, `is_guest`, `quiz_*`
fields

**Solution:** karaoke-gen now uses its own `gen_users` collection.

## Changes
- Updated `USERS_COLLECTION` constant from `"users"` to `"gen_users"`
- Updated all hardcoded `db.collection("users")` references to use the
constant
- Updated Pulumi Firestore indexes for `gen_users` collection
- Added migration script (already executed)

## Migration Status
✅ 42 karaoke-gen users migrated to `gen_users` collection
✅ 36 karaoke-decide users remain in `users` collection (untouched)
✅ Firestore indexes created for `gen_users`

## Test plan
- [ ] Admin dashboard loads user list from `gen_users`
- [ ] User login/signup works
- [ ] Credit operations work

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 21:43:47 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 69dd904f60 fix: Handle string dates in admin user detail endpoint (#172)
## Summary
Some jobs have created_at stored as string instead of datetime, causing
500 errors in admin user detail.

## Changes
Check for isoformat method before calling it.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 21:21:20 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 06bba48199 fix: Admin dashboard improvements (#171)
## Summary
- Fix user detail page navigation (404 when clicking on user in admin
panel)
- Add admin token support to `/api/users/me` endpoint (allows
token-based login)
- Add Firestore composite indexes for admin user queries (fixes 500
errors)

## Changes
1. **Frontend**: Fixed navigation from `/admin/users/{email}` to
`/admin/users/detail?email={email}`
2. **Backend**: Updated `/me` endpoint to accept admin tokens in
addition to session tokens
3. **Infrastructure**: Added 4 Firestore indexes for `users` collection
(already deployed via Pulumi)

## Test plan
- [ ] Navigate to `/admin/users` and click on a user - should load
detail page
- [ ] Use admin token in "use access token instead" login - should
authenticate as admin

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 20:39:33 +00:00
Andrew Beveridge ☄️andClaude cc7e5cd1c4 fix: allow @nomadkaraoke.com emails to access admin dashboard (#170)
Updates AdminLayout to grant admin access to users with
@nomadkaraoke.com email addresses, matching the admin validation logic
in E2E tests.

Previously, only users with role="admin" could access /admin, causing
users with @nomadkaraoke.com emails to be redirected to /app.

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 14:38:14 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 9793c0598b test: add Playwright E2E tests for admin dashboard (#169)
## Summary
- Add production E2E tests for the admin dashboard
- Tests validate token is admin before running (gracefully skip if not)
- Cover dashboard, users, jobs, and beta pages
- Test admin API endpoints

## Test Coverage
| Test | Description |
|------|-------------|
| admin dashboard loads | Verifies stats cards appear |
| sidebar navigation | Tests nav between admin pages |
| users page loads | Checks user table and search |
| jobs page loads | Checks job table and filters |
| beta page loads | Checks beta stats display |
| API stats endpoint | Tests `/api/admin/stats/overview` |
| API users endpoint | Tests `/api/users/admin/users` |

## Usage
```bash
# Requires admin token (@nomadkaraoke.com email)
KARAOKE_ADMIN_TOKEN=xxx npx playwright test --config=playwright.production.config.ts e2e/production/admin-dashboard.spec.ts
```

## Test plan
- [x] Tests skip gracefully without admin token
- [x] Clear error message shown explaining requirements
- [ ] Tests pass with valid admin token (manual verification needed)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 13:05:06 -05:00
Andrew Beveridge ☄️andClaude c29c2a6c73 feat: add comprehensive admin dashboard for platform management (#167)
## Summary

Add a full admin dashboard at `/admin` for platform management with:
- **Dashboard**: Stats overview showing users, jobs by status, credits
issued
- **User Management**: Search, sort, pagination, add credits, toggle
role, enable/disable
- **Job Browser**: Filter by status/user, view details with timeline and
logs, delete jobs
- **Beta Program**: Stats and feedback list

## Changes

### Backend
- New `/api/admin/stats/overview` endpoint for dashboard metrics
- Enhanced user list with search, pagination, and sorting
- New user detail endpoint with credit history and recent jobs

### Frontend
- Admin layout with responsive sidebar navigation
- Dashboard page with stats cards and job status breakdown
- User management pages (list + detail) with all admin actions
- Job browser pages (list + detail) with filters and log viewer
- Beta program page with stats and feedback
- Query-param based routing (`/admin/users/detail?email=`) for static
export compatibility

### Documentation
- Added Admin Endpoints section to API.md
- Updated README with admin dashboard in What Works and Current State

## Testing
- [x] Backend tests pass (`make test` - 57 passed)
- [x] Frontend tests pass (`npm run test:all` - 64 passed)
- [x] Frontend build passes (`npm run build`)
- [x] Manual testing completed

## Review
- [x] CodeRabbit CLI review completed locally (2 cycles)
- [x] All important feedback addressed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 12:39:42 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 593aac9e09 fix: Use recursive glob for GCE encoding worker file discovery (#166)
## Summary
- Fix GCE encoding worker failing with "No video files found" error
- Files downloaded from GCS preserve directory structure (e.g.,
`videos/with_vocals.mkv`, `stems/instrumental_clean.flac`)
- Original code used non-recursive glob patterns that only searched root
directory

## Changes
- Use recursive glob patterns (`**/*.mkv`) to find files in
subdirectories
- Add filter to exclude outputs directory to avoid re-encoding previous
outputs
- Add case-insensitive patterns for instrumental files (GCS uses
lowercase)

## Test plan
- [x] Fix deployed to production via Pulumi (VM recreated with updated
startup script)
- [x] Encoding worker health check verified:
`{"status":"ok","active_jobs":0,"queue_length":0}`
- [ ] Manual test: Submit a job through the frontend to verify GCE
encoding works end-to-end

## Notes
This fix was hotfixed to production via Pulumi to unblock users. The GCE
encoding worker VM was replaced with the corrected startup script. This
PR captures the fix in git.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 12:26:08 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 db2c86c3a0 fix: add karaoke-decide to WIF binding for GitHub Actions deploys (#165)
## Summary
- Add `karaoke-decide` repo to the Workload Identity Federation binding
- Fixes GitHub Actions deploy failures for karaoke-decide

## Problem
The `IAMBinding` resource is authoritative (replaces all members), and
`karaoke-decide` was missing from the list. This caused GitHub Actions
deploys to fail with:
```
Permission 'iam.serviceAccounts.getAccessToken' denied on resource
```

## Solution
Add karaoke-decide to the WIF binding alongside karaoke-gen and
flacfetch.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 11:07:03 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 1f8a1826e8 Add GCE encoding worker for high-performance video encoding (#164)
## Summary
- Adds dedicated GCE encoding worker for 2-3x faster video encoding
- Offloads FFmpeg libx264 encoding from Cloud Run to high-performance C4
instance
- Enabled via `USE_GCE_ENCODING=true` environment variable

## Changes
- **Infrastructure**: C4-standard-8 GCE instance with Intel Granite
Rapids 3.9 GHz CPU
- **Backend**: New `encoding_service.py` client for dispatching jobs to
GCE worker
- **video_worker.py**: Conditional GCE encoding path when enabled
- **Health check**: Added encoding worker status to
`/api/health/detailed`
- **CI/CD**: Added encoding worker URL and API key secrets to Cloud Run
deployment

## Architecture
```
Cloud Run (video_worker) --> GCE Encoding Worker --> GCS
                                   |
                            C4-standard-8
                         Intel Granite Rapids
                            3.9 GHz CPU
```

## Testing
- [x] Backend tests pass
- [x] Encoding worker health check verified
- [ ] End-to-end encoding test in prod

## Notes
- Currently using c4-standard-8 (8 vCPUs) due to quota limits
- Can upgrade to c4-standard-96 after requesting quota increase
- GCE encoding path skips CDG/TXT generation (not commonly used)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 10:54:51 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 fbaf8f960b fix(ci): Add automatic disk space cleanup for self-hosted runners (#163)
## Summary

- Fixed CI failures caused by self-hosted runners (11-20) running out of
disk space
- Changed Docker cleanup from daily/7-day-old to hourly/threshold-based
(70%)
- Added pre-job disk checks to all 9 self-hosted CI jobs
- Deployed fix immediately to all 20 runners (emergency cleanup
reclaimed ~57GB each)

## Problem

Runners 11-20 were at 98-100% disk usage causing jobs to fail with `No
space left on device`. The existing cleanup cron only removed images
older than 7 days, but CI builds create new dangling images (~15GB each)
faster than cleanup could remove them.

## Changes

| File | Change |
|------|--------|
| `.github/workflows/ci.yml` | Added disk check step to all 9
self-hosted jobs |
| `infrastructure/__main__.py` | Changed cleanup to
hourly/threshold-based, removed age filter |
| `docs/LESSONS-LEARNED.md` | Added section on Docker disk space
management |
| `docs/README.md` | Added recent change entry |

## Test plan

- [x] Verified all 20 runners now healthy (14-37% usage)
- [x] Deployed new cleanup script to all runners immediately
- [x] CI workflow YAML validates correctly
- [ ] CI jobs will run disk check on next execution

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 06:14:05 +00:00
Andrew Beveridge ☄️andClaude bf7faed6e4 feat: improve download filenames, add With Vocals button, add URL copy buttons (#161)
## Summary

Improves the download experience for completed karaoke jobs:
- Downloads now use proper filenames (e.g., "Artist - Title (Final
Karaoke Lossy 4k).mp4")
- Added "With Vocals" download button for easy access to vocals version
- Added copy-to-clipboard buttons for YouTube and Dropbox URLs

## Changes

### Backend (`backend/api/routes/jobs.py`)
- Added `DOWNLOAD_FILENAME_SUFFIXES` mapping for proper download
filenames
- Downloads now generate filenames like "Artist - Title (Final Karaoke
Lossy 4k).mp4" instead of "lossy_4k_mp4.mp4"
- Added filename sanitization to remove unsafe characters (`/ \ : * ? "
< > |`)
- Filename parts limited to 100 chars to prevent overly long names

### Frontend (`frontend/components/job/OutputLinks.tsx`)
- Added "With Vocals" download button (purple) for `videos/with_vocals`
- Added copy-to-clipboard buttons next to YouTube and Dropbox links with
"Copied!" feedback
- Fixed 720p download key mismatch (`lossy_720p` → `lossy_720p_mp4`)
- Added `aria-label` attributes for accessibility
- Added `type="button"` to prevent form submission issues
- Added clipboard API availability check and proper timeout cleanup on
unmount

## Testing

- [x] Backend tests pass (`make test` - 57 passed)
- [x] Frontend tests pass (`npm run test:all` - 64 passed)
- [ ] Manual testing of downloads and copy buttons

## Review

- [x] CodeRabbit CLI review completed locally
- [x] All review feedback addressed (6 issues fixed)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 06:11:01 +00:00
Andrew Beveridge ☄️andClaude fd92040dc7 feat: Make non-interactive mode job-specific instead of global toggle (#162)
## Summary

- Changes non-interactive/auto mode from a global toggle to a per-job
option selected during job creation
- Adds clear warning to users that skipping lyrics review will likely
result in mistakes
- Uses existing backing vocals analyzer for smart instrumental selection
in non-interactive mode

## Changes

### Backend
- Added `non_interactive: bool = False` field to Job and JobCreate
models
- Accept `non_interactive` in all job creation endpoints (upload, URL,
audio search)

### Frontend
- Added checkbox with warning in JobSubmission form for all 3 tabs
(Search, Upload, URL)
- Updated AutoProcessor to check `job.non_interactive` instead of global
flag
- AutoProcessor now uses backing vocals analyzer recommendation for
instrumental selection
- Added "Auto" badge on JobCard for non-interactive jobs
- Removed global auto mode toggle button and banner from header
- Deleted unused `auto-mode.ts` store and its tests
- Updated JobCard tests for new per-job behavior

## Testing
- [x] Backend tests pass (57 passed)
- [x] Frontend unit tests pass (128 passed)
- [x] Version bumped to 0.86.7

## Review
- [ ] CodeRabbit CLI review was rate limited - skipping for this PR

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 01:09:01 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 30f6fc6c05 perf: Improve cloud karaoke generation performance (#160)
## Summary

Infrastructure changes to address 3x performance gap between local and
cloud karaoke generation:

- **Keep Cloud Run warm** with `min-instances: 4` (eliminates 77s cold
starts)
- **Upgrade FFmpeg** from Debian 5.1.8 to static 7.0.2 (~20-40% faster
encoding)
- **Add diagnostics** to `/api/health/detailed` endpoint (FFmpeg
version, CPU info)
- **Add performance scripts** for benchmarking and comparison

## Changes

- `.github/workflows/ci.yml`: Changed `--min-instances` from 0 to 4
- `backend/Dockerfile.base`: Replace `apt-get install ffmpeg` with John
Van Sickle's static build
- `backend/api/routes/health.py`: Add `get_system_info()` and
`get_ffmpeg_info()` to detailed health endpoint
- `docs/DEVELOPMENT.md`: Add new scripts to Key Scripts table
- `docs/LESSONS-LEARNED.md`: Document Cloud Run cold start and FFmpeg
optimization findings
- `scripts/benchmark_ffmpeg.py`: Isolated FFmpeg encoding benchmark
- `scripts/compare_local_vs_remote.py`: Compare local vs cloud
karaoke-gen performance

## Expected Performance Improvement

| Metric | Before | After |
|--------|--------|-------|
| Cold start | 77s | 0s (warm instances) |
| Video encoding | 174s | ~120-140s (FFmpeg 7.x) |

## Testing

- [ ] Tests pass locally
- [x] FFmpeg static build verified in Docker container
- [ ] Manual testing after deploy

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 00:00:44 -05:00
Andrew Beveridge ☄️andClaude dfb2ebc184 feat(frontend): migrate from MailSlurp to testmail.app for E2E testing (#159)
## Summary

- Replace MailSlurp email testing service with testmail.app for E2E
magic link authentication tests
- Simpler API with namespace-based email addresses
(`{namespace}.{tag}@inbox.testmail.app`)
- No new dependencies - uses native `fetch` instead of mailslurp-client

## Changes

- **`frontend/e2e/helpers/email-testing.ts`** - Complete rewrite using
testmail.app JSON API
  - Uses `livequery=true` to wait for emails (no polling needed)
  - Generates unique tags per test run to avoid collisions
  - Same `EmailHelper` interface maintained for minimal test changes
- **`frontend/e2e/production/full-user-journey.spec.ts`** - Update env
var references and messages
- **`frontend/playwright.record.config.ts`** - Update env var comment
- **`frontend/package.json`** - Remove `mailslurp-client` dependency,
bump version to 0.79.0
- **`frontend/.env.local.example`** - New file documenting required env
vars for local testing
- **`docs/DEVELOPMENT.md`** - Add testmail.app env vars to frontend
section

## Environment Variables

| Old | New |
|-----|-----|
| `MAILSLURP_API_KEY` | `TESTMAIL_API_KEY` |
| (none) | `TESTMAIL_NAMESPACE` |

**Note:** GitHub Actions org secrets already configured with new env
vars.

## Testing

- [x] Backend tests pass (`make test` - 57 tests)
- [x] Frontend unit tests pass (139 tests)
- [x] Frontend E2E tests pass (64 tests)
- [x] TypeScript compilation clean for changed files

## Review

- [x] Local CodeRabbit CLI review completed
- [x] Review feedback addressed (removed exposed credentials from plan
file)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-02 22:24:19 -05:00
Andrew Beveridge ☄️andClaude ffa73c3194 Add comprehensive E2E happy path test with real user interactions (#156)
## Summary

- Add E2E test that validates complete karaoke generation flow using
ONLY browser UI interactions (no API shortcuts)
- Add GitHub Actions workflow for daily/on-demand E2E testing
- Add cleanup-distribution API endpoint for test cleanup
- Add delete methods to Dropbox and Google Drive services

## Test Coverage

| Step | Action | Method |
|------|--------|--------|
| 1 | Landing page | Browser |
| 2 | New user signup | MailSlurp + beta form |
| 3 | Magic link auth | Email verification |
| 4 | Create job | Search UI |
| 5 | Audio selection | Dialog UI |
| 6 | Lyrics review | **Review UI (not API)** |
| 7 | Instrumental selection | **Selection UI (not API)** |
| 8 | Download verification | Click links |
| 9 | Distribution cleanup | YouTube/Dropbox/GDrive |

## Test Plan

- [x] Add `e2e-happy-path` label to trigger the test
- [ ] Iterate until test passes

Slack thread:
https://nomadkaraoke.slack.com/archives/C0A77BFQL9W/p1767281650999499

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* CI: label-triggered E2E "happy-path" job and a scheduled/manual
happy-path workflow with run summaries, notifications, artifacts, and
extended timeout.

* **Tests**
* New full UI "Happy Path" E2E and multiple focused production E2E tests
(lyrics review, instrumental selection, existing-job stages) producing
logs, reports, and video artifacts.

* **Backend**
* Admin endpoint to clean up distributed content across external
services with resilient deletion handling.

* **Documentation**
* Added E2E happy-path spec, lessons-learned, and updated workflow
guidance.

* **Chores**
  * Version bumped to 0.86.6

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-02 21:48:19 -05:00
Andrew Beveridge ☄️andClaude 2edd5ee7c1 Add GCS-backed cache for LyricsTranscriber to persist API responses (#158)
## Summary

- Add GCS-backed cache persistence for LyricsTranscriber to avoid
redundant API calls
- AudioShake transcription and lyrics provider responses now cached
across Cloud Run instances
- Significantly reduces API costs and speeds up repeated processing of
same songs

## Changes

- **New:** `backend/services/lyrics_cache_service.py` - Service to sync
cache files with GCS
  - Computes MD5 hashes matching LyricsTranscriber's algorithm
  - Downloads relevant cache files from GCS before processing
  - Uploads new cache files to GCS after processing
  - Best-effort with graceful error handling

- **Modified:** `backend/workers/lyrics_worker.py`
  - Sets `LYRICS_TRANSCRIBER_CACHE_DIR` environment variable
  - Syncs cache from GCS before transcription
  - Syncs cache to GCS after transcription
  - Error handling ensures cache failures don't fail jobs

- **New:** `tests/unit/test_lyrics_cache_service.py` - 16 unit tests
- **Updated:** `docs/README.md` and `docs/LESSONS-LEARNED.md`

## Cache Structure

| Type | Key | Files |
|------|-----|-------|
| Transcription | MD5(audio_file) | `audioshake_{hash}_raw.json`,
`audioshake_{hash}_converted.json` |
| Lyrics | MD5(artist_title) | `genius_{hash}_raw.json`,
`lrclib_{hash}_converted.json`, etc. |

Providers cached: `audioshake`, `whisper`, `genius`, `spotify`,
`lrclib`, `musixmatch`

## Testing

- [x] All 16 new unit tests pass
- [x] All 57 existing tests pass (`make test`)
- [x] CodeRabbit CLI review completed locally

## Review

- [x] Local CodeRabbit review completed
- [x] Review feedback addressed (added error handling for cache
operations)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-02 17:08:11 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 8206d3fa12 docs: Document worker timeout fixes (PRs #153-155) (#157)
## Summary

Documents the worker timeout fixes from PRs #153, #154, and #155 that
resolved job failures:

- **Archive doc**: `docs/archive/2026-01-01-worker-timeout-fixes.md` -
Full investigation and fix details
- **LESSONS-LEARNED**: Added Cloud Tasks `dispatch_deadline` insight
(separate from Cloud Run timeout)
- **README**: Updated recent changes, removed outdated known issue

## Key Insights Documented

1. **Cloud Tasks dispatch_deadline** (default 10 min) is SEPARATE from
Cloud Run service timeout
2. For tasks >30 minutes, Cloud Tasks HTTP targets won't work - use
Cloud Run Jobs
3. Table comparing all three timeout types and their limits

## Test plan

- [x] Docs render correctly in markdown preview
- [x] Links to archive doc are valid

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
  * Removed known issue regarding long audio file timeouts
* Added documentation on worker timeout improvements for audio, lyrics,
and video processing
  * Documented new timeout handling patterns for long-running tasks

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-01 15:42:24 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 aa5ae10126 Enable Cloud Run Jobs for video encoding to prevent timeout (#155)
## Summary
- Enable `USE_CLOUD_RUN_JOBS_FOR_VIDEO=true` in Cloud Run deployment
- Video encoding now uses Cloud Run Jobs (1-hour timeout) instead of
Cloud Tasks (30-min limit)
- This fixes the final video encoding timeout issue observed during E2E
testing

## Problem
The video encoding step (`KaraokeFinalise.process()`) was timing out
after 30 minutes:
- Cloud Run service has 30-minute request timeout
- Cloud Tasks dispatch_deadline was set to 30 minutes
- But video encoding takes 30-40+ minutes for average songs

## Solution
Enable the existing `USE_CLOUD_RUN_JOBS_FOR_VIDEO` feature flag which:
1. Triggers video encoding as a Cloud Run Job instead of Cloud Tasks
2. Cloud Run Jobs have 1-hour timeout (configured in infrastructure)
3. Better suited for batch processing workloads

This completes the timeout fixes for all workers:
- **Audio**: dispatch_deadline=1800s (PR #154) ✅
- **Lyrics**: 20-minute inner timeout (PR #153) ✅
- **Video**: Cloud Run Jobs with 1-hour timeout (this PR)

## Test plan
- [x] Verified audio/lyrics workers complete successfully with previous
fixes
- [ ] Deploy and verify video encoding completes for stuck job d12fcc4d
(or new job)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-31 22:28:13 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 9617996bc2 Add dispatch_deadline to Cloud Tasks to prevent audio worker timeout (#154)
## Summary
- Add explicit `dispatch_deadline` configuration to Cloud Tasks for each
worker type
- Audio worker gets 30 minutes (max) to accommodate Modal API separation
times
- Lyrics worker gets 25 minutes for AudioShake + agentic correction
- This fixes the issue where audio workers were being killed after ~10
minutes (Cloud Tasks default) before Modal could complete

## Problem
The audio worker was being terminated before Modal API could complete
audio separation:
1. Job created, audio task enqueued to Cloud Tasks
2. Cloud Tasks dispatches to Cloud Run, starts 10-minute default
deadline timer
3. Audio worker submits to Modal API and waits
4. After 10 minutes, Cloud Tasks considers the dispatch failed
5. Cloud Run container gets killed, Modal task returns 404

## Solution
Set explicit `dispatch_deadline` for each worker type based on their
expected execution time:
```python
WORKER_DISPATCH_DEADLINES = {
    "audio": 1800,       # 30 min - Modal separation can take 15-20 min
    "lyrics": 1500,      # 25 min - TRANSCRIPTION_TIMEOUT_SECONDS is 1200s (20 min)
    "screens": 600,      # 10 min - Screen generation is fast
    "render-video": 1800,  # 30 min - Video encoding can be slow
    "video": 1800,       # 30 min - Video encoding can be slow
}
```

## Test plan
- [x] All 57 backend tests pass
- [ ] Deploy and run E2E test to verify audio separation completes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Improvements**
* Enhanced task scheduling with optimized processing deadlines for
different job types, improving system efficiency and reliability.

* **Chores**
  * Updated project version to 0.86.4.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-31 21:17:40 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 b755ed54dc Increase lyrics transcription timeout to 20 minutes (#153)
## Summary
- Increase outer transcription timeout from 600s to 1200s (20 min) to
prevent timeout during agentic AI correction
- Update E2E test to use API calls for human review steps instead of UI
navigation
- Extend E2E test timeouts for longer processing

## Problem
Jobs were failing with "Lyrics transcription timed out after 600s"
because the outer timeout didn't account for:
- Cloud Run cold start / worker initialization (1-5 min)
- AudioShake transcription (1-2 min)
- spaCy model loading for correction (2-3 min on cold start)
- Agentic AI correction (1-3 min)

## Test plan
- [ ] Deploy to production
- [ ] Run `npm run test:e2e:prod` in frontend directory
- [ ] Verify job completes through to distribution

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* End-to-end tests updated to rely more on API-driven steps and to
verify longer processing flows and output artifacts.

* **Improvements**
* Increased processing timeouts to accommodate extended transcription,
audio separation, and post-processing steps.

* **Chores**
  * Package version bumped to 0.86.3
* Test and CI/playwright timeouts adjusted for longer-running
operations.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-31 20:43:18 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 77de1587e2 Fix audio search cache not persisting across Cloud Run instances (#152)
## Summary
- Fixes audio download failing when Cloud Run scales horizontally
- Fixes agentic mode timeout failing entire job instead of proceeding
with partial results
- Version bump: 0.86.0 -> 0.86.2

## Problem 1: Audio Search Cache
When a user searches for audio on one Cloud Run instance (A) and then
selects a result that is handled by a different instance (B), the
download would fail with "No cached result for index X" because the
in-memory cache doesn't persist across instances.

**Solution**: When local cache is empty but we have a
`remote_search_id`, use the remote flacfetch service which maintains its
own cache by search_id.

## Problem 2: Agentic Timeout Fails Job
The outer timeout for agentic mode was only 240s (agentic_timeout + 60),
but this didn't account for AudioShake transcription time which can take
2-5+ minutes. If AudioShake took longer than expected, the outer timeout
would trigger and fail the job instead of proceeding with partial
results.

**Solution**: Changed outer timeout to match non-agentic mode (10
minutes) since the inner deadline check in corrector.py properly handles
the 3-minute correction limit by returning partial results gracefully.

## Changes
- `audio_search_service.py`: Check for local cache first, fall back to
remote download when cache is empty
- `_download_remote`: Handle missing local cache gracefully (metadata
comes from remote service)
- `lyrics_worker.py`: Use TRANSCRIPTION_TIMEOUT_SECONDS (600s) for outer
timeout in agentic mode
- Test update: Explicitly disable remote client when testing local cache
validation

## Test plan
- [x] All audio search tests pass (49 tests)
- [ ] CI passes
- [ ] Production E2E test passes after deploy

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Prefer remote audio downloads when local cache is missing; handle
missing metadata gracefully during remote downloads
* Align outer transcription timeout behavior and clarify timeout
handling

* **Tests**
* Updated test to exercise the local-cache download path for
invalid-index scenarios

* **Chores**
  * Version bumped to 0.86.2

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-31 17:36:31 -05:00
Andrew Beveridge ☄️ aea59f298e Update pyproject.toml (#151) 2025-12-31 21:36:36 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 c68c221ba3 feat(agentic): Add 3-minute timeout for lyrics correction (#149)
## Summary

- Adds configurable timeout (default 3 minutes) for agentic AI lyrics
correction
- When timeout occurs, skips remaining gaps and returns uncorrected
transcription
- Job proceeds to human review where user can manually correct lyrics
- Prevents stuck jobs in production (songs with 74+ gaps could take 30+
minutes)

## Implementation

Two-layer timeout approach:
1. **Inner deadline check** in `corrector.py` gap loop - cooperative
exit before each gap
2. **Outer `asyncio.wait_for()`** in `lyrics_worker.py` - safety net
catches hung LLM calls

## Files Changed

- `backend/config.py` - Add `agentic_correction_timeout_seconds` config
- `backend/workers/lyrics_worker.py` - Add deadline calculation and
outer timeout
- `karaoke_gen/lyrics_processor.py` - Pass deadline through to
controller
- `lyrics_transcriber_temp/lyrics_transcriber/core/controller.py` -
Accept deadline parameter
- `lyrics_transcriber_temp/lyrics_transcriber/correction/corrector.py` -
Add deadline check in gap loop
- `pyproject.toml` - Bump version to 0.85.0

## Test plan

- [x] Unit tests pass (812 passed, 29 skipped)
- [ ] Deploy to production and monitor agentic correction jobs
- [ ] Verify timeout triggers after ~3 minutes on long-running
corrections
- [ ] Verify job proceeds to `awaiting_review` with uncorrected lyrics

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

**New Features**
- Agentic AI lyrics correction now includes a configurable timeout
(default 3 minutes). If correction exceeds the timeout, jobs proceed
with raw transcription available for human review.

**Documentation**
- Added implementation notes on timeout behavior and the two-layer
timeout pattern architecture.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-31 16:13:46 -05:00
Andrew Beveridge ☄️ ed2305d446 Fix 4 critical job failure issues (#150)
Fixes 4 distinct issues causing job failures:

1. **user_email not set on jobs (CRITICAL)** - Jobs weren't associated with users
2. **YouTube URL download race condition (HIGH)** - Lyrics worker started before audio downloaded  
3. **Audio search cache not persisting (HIGH)** - Cloud Run scaling broke download
4. **Jobs stuck in downloading state (MEDIUM)** - Added 10-minute transcription timeout

Changes:
- Updated all job creation endpoints to extract `user_email` from `AuthResult`
- Changed URL jobs to trigger audio worker only, which then triggers lyrics worker
- Added direct YouTube download using URL from `state_data` (cache-independent)
- Added 10-minute timeout around AudioShake transcription
- Added 25 regression tests covering all 4 issues
- Added lessons learned to documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2025-12-31 14:15:51 -05:00
Andrew Beveridge ☄️ 7a39631c63 Merge pull request #148 from nomadkaraoke/bump-ver
Bump version from 0.83.0 to 0.84.0
2025-12-31 00:51:28 -05:00
Andrew Beveridge ☄️ 208d80ff3c Bump version from 0.83.0 to 0.84.0 2025-12-31 00:51:00 -05:00
Andrew Beveridge ☄️ 7bee09bffb Merge pull request #147 from nomadkaraoke/feature/work-20251230-225419
fix(agentic): Use Vertex AI backend with service account auth on Cloud Run
2025-12-31 00:23:10 -05:00
Andrew BeveridgeandClaude Opus 4.5 9c8d2835f0 docs: Add archive doc for Vertex AI auth fix
- Create docs/archive/2025-12-31-vertex-ai-auth-fix.md
- Update README.md with recent changes entry
- Correct status: Gemini via Vertex AI (not Google AI Studio)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-31 00:06:57 -05:00
Andrew BeveridgeandClaude Opus 4.5 679498328d fix(agentic): Use Vertex AI backend with service account auth on Cloud Run
PR #145 migrated from ChatVertexAI to ChatGoogleGenerativeAI but only
configured API key authentication. This broke Cloud Run deployments
which rely on service account credentials (ADC).

ChatGoogleGenerativeAI supports BOTH auth methods:
- Vertex AI backend (service account/ADC) - when `project` param is set
- Google AI Studio (API key) - when only `google_api_key` is set

This fix:
- Passes `project` parameter when GOOGLE_CLOUD_PROJECT is set
- Falls back to API key auth if no project is set
- Works automatically on Cloud Run using the attached service account

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-31 00:03:49 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 25be810442 feat(agentic): Migrate from ChatVertexAI to ChatGoogleGenerativeAI (#145)
## Summary

- Migrate from deprecated `langchain-google-vertexai` (gRPC) to
`langchain-google-genai` (REST) to fix silent hang issues during model
initialization
- Add initialization and warm-up timeouts using ThreadPoolExecutor for
cross-platform support
- Upgrade key logs from DEBUG to INFO with timing information for better
visibility

## Problem

Job `b1b2648c` hung indefinitely at "LangFuse prompt service
initialized" with no visible errors. Root cause: `ChatVertexAI` uses
gRPC which can hang during connection establishment with no timeout.

## Changes

| File | Changes |
|------|---------|
| `pyproject.toml` | Replace `langchain-google-vertexai` with
`langchain-google-genai` |
| `lyrics_transcriber_temp/pyproject.toml` | Same dependency change |
| `.../providers/config.py` | Add `initialization_timeout_seconds` (30s)
and `warmup_timeout_seconds` (15s) |
| `.../providers/model_factory.py` | Migrate to
`ChatGoogleGenerativeAI`, add timing logs |
| `.../providers/langchain_bridge.py` | Add ThreadPoolExecutor timeout
wrapper, upgrade logs to INFO |
| `.../agent.py` | Add timing logs for `classify_gap()` |

## Test plan

- [x] All 812 tests pass
- [x] E2E emulator tests pass
- [ ] Test with actual GOOGLE_API_KEY in staging

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Configurable initialization and warmup timeouts.
* Support for Google AI Studio (new Google generative API) as a model
backend.

* **Improvements**
* Enhanced logging and timing visibility across model initialization,
warm‑up, invocation, and gap classification.
* Warm‑up timeouts treated as non‑fatal; initialization timeouts
surfaced clearly and reported.

* **Bug Fixes**
* Better handling/reporting for initialization timeouts and missing
Google API key.

* **Chores**
* Dependency key updated to new Google generative client; package
versions bumped.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 22:50:16 -05:00
Andrew Beveridge ☄️ 03b37af506 Merge pull request #143 from nomadkaraoke/feature/session-2025-12-30-1824
fix(agentic): Improve Vertex AI resilience with timeout and warm-up
2025-12-30 20:51:01 -05:00
Andrew BeveridgeandClaude Opus 4.5 94e29f89b4 fix(agentic): Flush print output for warm-up visibility
Add flush=True to warm-up print statements so they appear immediately
when output is redirected to a file, rather than being buffered.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 20:35:05 -05:00
Andrew Beveridge 9345c06eac Merge remote-tracking branch 'origin/main' into feature/session-2025-12-30-1824 2025-12-30 20:18:44 -05:00
Andrew BeveridgeandClaude Opus 4.5 46a4540093 fix(agentic): Strip markdown code fences from LLM responses
- Add markdown fence stripping to response parser (```json ... ```)
- Add print statements to warm-up for debugging visibility

The model sometimes wraps JSON in markdown code blocks which caused
parsing failures and classification errors.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 20:10:24 -05:00
Andrew BeveridgeandClaude Opus 4.5 5b44f4cd73 docs: Add LangChain Vertex AI migration plan
Document the planned migration from deprecated ChatVertexAI to
ChatGoogleGenerativeAI. Key points:
- ChatVertexAI deprecated in langchain-google-vertexai 3.2.0
- New ChatGoogleGenerativeAI supports Vertex AI via vertexai=True
- REST instead of gRPC may resolve connection issues

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 20:05:28 -05:00
Andrew Beveridge ☄️ cc244d5d41 Merge pull request #144 from nomadkaraoke/feature/session-2025-12-30-1913
fix: Handle both duration field names in instrumental review waveform
2025-12-30 19:59:42 -05:00
Andrew BeveridgeandClaude Opus 4.5 40e730c4c9 fix: Handle both duration field names in instrumental review waveform
The local waveform-data API returns `duration` but the frontend was only
checking for `duration_seconds`. This caused the global duration variable
to stay at 0, which made the click handler early-return and prevented
all waveform seeking.

Now handles both field names for compatibility with cloud (duration_seconds)
and local (duration) APIs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 19:50:43 -05:00
Andrew BeveridgeandClaude Opus 4.5 b2c0fb7a58 fix(agentic): Improve Vertex AI resilience with timeout and warm-up
- Increase default timeout from 30s to 120s to handle connection
  establishment delays (observed 499 errors at ~60s)
- Add warm-up mechanism that sends lightweight request after model
  creation to establish gRPC connection before real work begins
- Increase retry backoff base from 0.2s to 2.0s - immediate retries
  after timeouts are unlikely to succeed

Addresses issues observed in LangFuse traces where first 3 requests
to Gemini 3 Flash timed out (~61s each) while subsequent requests
completed normally (9-17s).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 18:55:09 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 b49e49b428 feat: Add /worktree-cleanup slash command (#142)
## Summary

- Add `/worktree-cleanup` Claude slash command for git worktree
maintenance
- Automates the process of identifying merged worktrees for cleanup
- Suggests renaming generic branch names to descriptive ones

## What the command does

1. Lists all worktrees and checks PR status for each branch
2. Categorizes into: merged (cleanup), open (keep), no PR (investigate)
3. Flags generic names (e.g., `work-YYYYMMDD-*`) and suggests renames
based on PR content
4. Asks user before executing cleanup/renames
5. Handles uncommitted changes safely

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 23:19:07 +00:00
128 changed files with 23388 additions and 1724 deletions

No files matched your search

-118
View File
@@ -1,118 +0,0 @@
---
description: Review and address CodeRabbit comments on current PR
allowed-tools: Read, Glob, Grep, Edit, Write, Bash, mcp__github__pull_request_read, mcp__github__get_me, mcp__github__list_pull_requests, mcp__github__add_issue_comment
---
# CodeRabbit Comment Review
You are reviewing the current PR to address all CodeRabbit automated review comments.
## Your Task
### 1. Identify the Current PR
First, determine the current branch and find the associated PR:
```bash
git branch --show-current
```
Then use GitHub tools to find the PR for this branch in the `nomadkaraoke/karaoke-gen` repository.
### 2. Fetch CodeRabbit Comments
Get all reviews, review comments, and regular comments on the PR:
**PR Reviews (MOST IMPORTANT - Contains Nitpicks and Issues)**
- Use `pull_request_read` with `method: "get_reviews"` to get all PR reviews
- Look for reviews from the `coderabbitai[bot]` user
- The review `body` field contains the detailed analysis with:
- "Nitpick comments" section with specific code suggestions
- "Potential issues" flagged in the diff
- Code improvement suggestions with file paths and line numbers
- **This is the primary source of CodeRabbit feedback** - parse this carefully
**Review Comments (Inline Code Comments)**
- Use `pull_request_read` with `method: "get_review_comments"` to get inline code comments
- These are comments attached to specific lines in the diff
- Also check for `coderabbitai[bot]` comments here
**PR Comments (Summary)**
- Use `pull_request_read` with `method: "get_comments"` to get regular PR comments
- CodeRabbit posts a walkthrough/summary comment here
- Less actionable than the review body, but may contain additional context
### 3. Analyze Each Comment
For each CodeRabbit comment:
1. Read the comment carefully to understand the concern
2. Look at the referenced code location
3. Decide if a code change is warranted:
- **Make changes** for: valid bugs, security issues, clear improvements, style violations
- **Skip changes** for: subjective preferences, false positives, already addressed, not applicable
4. If skipping, note the reason
### 4. Make Code Changes
For each comment requiring action:
1. Navigate to the file and line mentioned
2. Understand the surrounding context
3. Make the minimal change needed to address the concern
4. Verify the change doesn't break anything obvious
### 5. Commit and Push
After making all changes:
```bash
# Stage all changes
git add -A
# Commit with descriptive message
git commit -m "fix: address CodeRabbit review comments
- [list each fix briefly]
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
# Push to the PR branch
git push
```
### 6. Resolve Threads
After pushing:
- Reply to each CodeRabbit review thread indicating what was done
- If a comment was addressed, reply with what was changed
- If a comment was intentionally skipped, explain why
- This helps mark threads as resolved so the PR can be merged
## Output Format
Provide a summary report:
```
## CodeRabbit Review Summary
### Comments Addressed
- [ ] File:line - Brief description of issue → What was changed
### Comments Skipped (with reason)
- [ ] File:line - Issue → Reason for skipping
### Actions Taken
- Committed: [commit hash]
- Pushed to: [branch name]
- Threads replied to: [count]
### Status
Ready for merge: Yes/No (if No, explain why)
```
## Guidelines
- Focus on code quality improvements, not just making CodeRabbit happy
- Don't blindly accept every suggestion - use judgment
- Group related fixes into logical commits if needed
- Keep changes minimal and focused on the specific issues raised
- If a suggestion would require significant refactoring, note it for future work instead
-74
View File
@@ -1,74 +0,0 @@
---
description: Periodic documentation maintenance and organization check
allowed-tools: Read, Glob, Grep, Edit, Write, Bash
---
# Documentation Maintenance Check
You are performing periodic maintenance on the project documentation.
## Checks to Perform
### 1. Structure Compliance
Verify docs/ follows the correct structure:
```
docs/
├── README.md # Current status + navigation
├── ARCHITECTURE.md # System design
├── DEVELOPMENT.md # Dev setup, testing, deployment
├── API.md # Backend API reference
├── LESSONS-LEARNED.md # Accumulated wisdom
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
```
Report any files that don't belong or are misplaced.
### 2. Freshness Check
- Read `docs/README.md` - is the status section current?
- Check recent git commits - do they suggest docs should be updated?
- Look for stale information that contradicts current code
### 3. Archive Organization
- Verify all archive files have YYYY-MM-DD prefix
- Check for any docs that should be archived (completed work, old plans)
- Remove truly obsolete content (empty files, duplicate info)
### 4. Cross-Reference Check
- Verify CLAUDE.md points to current doc locations
- Check that docs reference each other correctly
- Ensure no broken internal links
### 5. Content Quality
- Flag any docs over 500 lines (may need splitting)
- Identify duplicate information across docs
- Note sections that seem outdated
## Actions to Take
1. **Fix structural issues** - Move/rename misplaced files
2. **Update stale content** - Refresh outdated information
3. **Archive completed work** - Move old docs to archive/
4. **Report findings** - Summarize what was found and fixed
## Output Format
Provide a maintenance report:
```
## Documentation Maintenance Report - YYYY-MM-DD
### Issues Found
- [list of issues]
### Actions Taken
- [list of fixes made]
### Recommendations
- [suggestions for future attention]
### Status
- [ ] Structure: OK/Fixed
- [ ] Freshness: OK/Updated
- [ ] Archive: OK/Organized
- [ ] Cross-refs: OK/Fixed
- [ ] Quality: OK/Flagged
```
-44
View File
@@ -1,44 +0,0 @@
---
description: Review current session for documentation updates before merging PR
allowed-tools: Read, Glob, Grep, Edit, Write
---
# Pre-Merge Documentation Review
You are reviewing this chat session to identify documentation updates needed before merging the current PR.
## Your Task
1. **Analyze the current session** - What work was done? What was learned? What changed?
2. **Check if docs need updates:**
- Does `docs/README.md` status section need updating?
- Should anything be added to `docs/LESSONS-LEARNED.md`?
- Does `docs/ARCHITECTURE.md` need changes (if architecture changed)?
- Does `docs/DEVELOPMENT.md` need changes (if dev workflow changed)?
- Does `docs/API.md` need changes (if API changed)?
3. **For significant completed work:**
- Create `docs/archive/YYYY-MM-DD-topic.md` summarizing the work
- Use today's date and a descriptive topic name
4. **Update docs/README.md** with:
- Current status if it changed
- Any new known issues discovered
- Link to new archive doc if created
## Guidelines
- **IMPORTANT**: Make all documentation changes on the current feature branch (git worktree), NOT on main. Check `git branch` or the worktree path to ensure you're editing files in the correct location.
- Keep updates concise and factual
- Focus on information useful for future AI agents
- Don't duplicate information that's in the code
- Archive docs should capture "what was done and why" not implementation details
## Output
After reviewing, either:
- Make the necessary documentation updates
- Report "No documentation updates needed" with brief explanation why
Remember: Good documentation helps the next agent continue where you left off.
@@ -0,0 +1,106 @@
# Plan: Migrate from MailSlurp to testmail.app
**Created:** 2026-01-02
**Branch:** feat/sess-20260102-2109-migrate-mailslurp-testmail
**Status:** Complete
## Overview
Replace MailSlurp email testing service with testmail.app for E2E testing of magic link authentication. testmail.app uses a simpler API with namespace-based email addresses that require no inbox creation/deletion - emails just arrive at `{namespace}.{tag}@inbox.testmail.app`.
## Requirements
- [x] Remove all code referencing MailSlurp
- [x] Remove `mailslurp-client` npm dependency
- [x] Implement testmail.app API integration
- [x] Update environment variable names (`TESTMAIL_API_KEY`, `TESTMAIL_NAMESPACE`)
- [x] Maintain same `EmailHelper` interface for minimal test changes
- [x] Update archive docs to note migration (optional - historical accuracy)
## Technical Approach
### API Differences
| Feature | MailSlurp | testmail.app |
|---------|-----------|--------------|
| Email format | Random `xxx@mailslurp.com` | `{namespace}.{tag}@inbox.testmail.app` |
| Inbox creation | Required (`createInbox()`) | Not needed - use any tag |
| Inbox deletion | Required (`deleteInbox()`) | Not needed |
| Get emails | `waitForLatestEmail(inboxId)` | GET `/api/json?apikey=X&namespace=Y&tag=Z&livequery=true` |
| Auth | API key in constructor | API key as query param |
| Email body | `email.body` | `email.html` or `email.text` |
### Implementation Strategy
1. **Keep same `EmailHelper` interface** - `createInbox()`, `waitForEmail()`, `extractMagicLink()`, `deleteInbox()`
2. **Use testmail.app's live query feature** - `livequery=true` waits for email to arrive (up to timeout)
3. **Generate unique tags** - Use timestamp + random suffix for each test run to avoid collisions
4. **Inbox becomes a simple object** - `{ id: tag, emailAddress: '{namespace}.{tag}@inbox.testmail.app' }`
5. **Delete becomes no-op** - testmail.app emails auto-expire, no cleanup needed
### Environment Variables
| Old | New |
|-----|-----|
| `MAILSLURP_API_KEY` | `TESTMAIL_API_KEY` |
| (none) | `TESTMAIL_NAMESPACE` |
Values: Set via environment variables (see `frontend/.env.local.example`)
## Implementation Steps
1. [x] **Update `email-testing.ts`** - Replace MailSlurp implementation with testmail.app
- Remove `mailslurp-client` import
- Add testmail.app API fetch implementation
- Use native `fetch` (no new dependencies)
- Generate unique tag for each "inbox"
- Implement `livequery=true` for waiting
- Adapt email body extraction (`html`/`text` instead of `body`)
2. [x] **Update `full-user-journey.spec.ts`** - Change references
- Update doc comments (env var name)
- Update error messages
- Update console logs
3. [x] **Update `playwright.record.config.ts`** - Change env var comment
4. [x] **Update `package.json`** - Remove `mailslurp-client` dependency
5. [x] **Run `npm install`** - Update package-lock.json
6. [x] **Create `.env.local.example`** - Document required env vars for local testing
7. [x] **Update archive docs** - Skipped intentionally; historical docs preserved for accuracy
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `frontend/e2e/helpers/email-testing.ts` | Modify | Replace MailSlurp with testmail.app |
| `frontend/e2e/production/full-user-journey.spec.ts` | Modify | Update env var refs and messages |
| `frontend/playwright.record.config.ts` | Modify | Update env var comment |
| `frontend/package.json` | Modify | Remove `mailslurp-client` |
| `frontend/package-lock.json` | Modify | Auto-updated by npm install |
| `frontend/.env.local.example` | Create | Document env vars for local testing |
## Testing Strategy
- **Manual testing**: Run production E2E test with testmail.app credentials
```bash
cd frontend
TESTMAIL_API_KEY=your-api-key \
TESTMAIL_NAMESPACE=your-namespace \
npm run test:e2e:prod:headed
```
- **Verify**: Email received, magic link extracted, auth flow completes
- **CI testing**: Env vars already configured in GitHub Actions org secrets
## Open Questions
None - requirements are clear.
## Rollback Plan
1. Revert the commit
2. Run `npm install` to restore mailslurp-client
3. Re-add MAILSLURP_API_KEY to secrets if removed
@@ -0,0 +1,401 @@
# Plan: Video Worker Stage Refactor
**Created:** 2026-01-03
**Branch:** feat/sess-20260103-2250-investigate-youtube-upload
**Status:** COMPLETE - Ready for PR
**Last Updated:** 2026-01-04
## Overview
Refactor `video_worker.py` and `KaraokeFinalise` into separate, composable stages following SOLID principles. This eliminates the GCE vs KaraokeFinalise code path divergence that causes features like YouTube upload to be missing from the GCE encoding path.
## Three Execution Modes
Understanding the three ways this system runs is critical to the refactor:
### 1. Local CLI (`karaoke-gen`)
- Runs entirely on user's machine
- KaraokeFinalise.process() orchestrates everything
- Writes directly to local Dropbox sync folder (e.g., `~/Dropbox/Karaoke/Tracks-Organized/`)
- **Produces full output structure** with stems/, lyrics/, visualizations/, etc.
- **This is the source of truth for output format**
### 2. Remote CLI (`karaoke-gen-remote`)
- User's CLI talks to cloud backend (api.nomadkaraoke.com)
- Cloud does ALL processing (audio, lyrics, video encoding)
- CLI just monitors job status and downloads final outputs
- With Dropbox configured: cloud uploads full structure to user's Dropbox
- Without Dropbox: user downloads individual files
### 3. Web Frontend (`gen.nomadkaraoke.com`)
- Same cloud backend as remote CLI
- User sees:
- YouTube video link
- Dropbox folder link
- Direct download: final karaoke video (720p)
- Direct download: 4K video
- Direct download: "with vocals" video
- Direct download: CDG zip (if enabled)
## Problem Statement
Currently, `video_worker.py` has two completely divergent code paths:
```python
if use_gce_encoding:
# GCE PATH: Only encoding, missing YouTube/Discord/CDG/TXT
result = await _encode_via_gce(...)
# result DOES NOT include youtube_url!
else:
# STANDARD PATH: KaraokeFinalise does everything
finalise = KaraokeFinalise(...)
result = finalise.process()
# result includes youtube_url from upload
```
### What GCE Encoding Broke
| Feature | KaraokeFinalise Path | GCE Path |
|---------|---------------------|----------|
| Video encoding | ✅ 4 formats | ✅ 4 formats |
| YouTube upload | ✅ | ❌ **BROKEN** |
| Discord notification | ✅ | ❌ **BROKEN** |
| CDG/TXT packaging | ✅ | ❌ **BROKEN** |
| Brand code generation | ✅ | ✅ (via Dropbox service) |
| Dropbox upload | ✅ (rclone) | ✅ (native SDK) |
| Google Drive upload | ❌ | ✅ (native SDK) |
### Root Cause
`KaraokeFinalise.process()` is a monolith that handles:
1. CDG/TXT packaging
2. Video encoding (the CPU-intensive part)
3. YouTube upload
4. Discord notifications
5. Folder organization
The GCE encoding was added as a bypass around the entire class, rather than replacing just the encoding step.
## Requirements
### Functional Requirements
- [ ] YouTube upload works regardless of encoding backend (GCE or local)
- [ ] Discord notifications work regardless of encoding backend
- [ ] CDG/TXT packaging works regardless of encoding backend
- [ ] Brand code generation works consistently
- [ ] Dropbox/Google Drive upload works (already works in both paths)
- [ ] All existing CLI functionality continues to work
- [ ] Output structure matches local CLI exactly
### Output Structure Parity
Cloud backend must produce identical output to local CLI:
```
NOMAD-XXXX - Artist - Title/
├── Artist - Title (Title).mov
├── Artist - Title (Title).jpg
├── Artist - Title (End).mov
├── Artist - Title (End).jpg
├── Artist - Title (Final Karaoke Lossless 4k).mp4
├── Artist - Title (Final Karaoke Lossless 4k).mkv
├── Artist - Title (Final Karaoke Lossy 4k).mp4
├── Artist - Title (Final Karaoke Lossy 720p).mp4
├── Artist - Title (Final Karaoke CDG).zip
├── Artist - Title (Final Karaoke TXT).zip
├── Artist - Title (Instrumental {model}).flac
├── Artist - Title (Instrumental +BV {model}).flac
├── Artist - Title (Karaoke).mp4
├── Artist - Title (Karaoke).cdg
├── Artist - Title (Karaoke).mp3
├── Artist - Title (Karaoke).lrc
├── Artist - Title (With Vocals).mp4
├── stems/
│ ├── Artist - Title (Instrumental {model}).flac
│ ├── Artist - Title (Vocals {model}).flac
│ ├── Artist - Title (Lead Vocals {model}).flac
│ ├── Artist - Title (Backing Vocals {model}).flac
│ └── ... (other stems)
├── lyrics/
│ ├── Artist - Title (Karaoke).lrc
│ ├── Artist - Title (Karaoke).ass
│ ├── Artist - Title (Karaoke).srt
│ └── ... (corrections JSONs)
├── previews/
├── visualizations/
└── styles_assets/
```
### Non-Functional Requirements
- [ ] Single Responsibility: Each stage has one job
- [ ] Open/Closed: Easy to add new encoding backends without modifying stages
- [ ] Testable: Each stage can be unit tested in isolation
- [ ] No duplicate code between paths
- [ ] Backwards compatible with existing KaraokeFinalise CLI usage
## Technical Approach
### Architecture: Stage-Based Pipeline
```
video_worker.py (VideoWorkerOrchestrator)
│
├── 1. EncodingStage (interface)
│ ├── LocalEncodingService (wraps KaraokeFinalise encoding methods)
│ └── GCEEncodingService (existing encoding_service.py)
│
├── 2. PackagingStage
│ ├── CDGPackager (extracts from KaraokeFinalise)
│ └── TXTPackager (extracts from KaraokeFinalise)
│
├── 3. DistributionStage
│ ├── YouTubeUploadService (extracts from KaraokeFinalise)
│ ├── DropboxService (existing)
│ ├── GoogleDriveService (existing)
│ └── DiscordService (extracts from KaraokeFinalise)
│
└── 4. OrganizationStage
└── BrandCodeService (extracts from KaraokeFinalise + DropboxService)
```
### Key Design Decisions
1. **Extract, Don't Rewrite**: Pull methods out of KaraokeFinalise into services, don't rewrite from scratch
2. **Keep KaraokeFinalise for CLI**: Local CLI still uses KaraokeFinalise.process() - we extract services that it can also use
3. **Interface-Based Encoding**: EncodingStage interface allows swapping GCE/local without changing orchestration
4. **Existing Service Pattern**: Follow singleton + factory pattern already used in codebase
5. **Local CLI stays sync**: Local karaoke-gen CLI remains synchronous as it is today
### Data Flow
```
JobConfig (from Firestore)
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ VideoWorkerOrchestrator │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 1. Setup: Download files, prepare directories ││
│ │ - Download title.mov, end.mov, with_vocals.mkv ││
│ │ - Download instrumental audio ││
│ │ - Prepare temp directory ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 2. Packaging: CDG/TXT (if enabled) - runs BEFORE encoding ││
│ │ Reason: CDG depends on LRC file which is available ││
│ │ Input: LRC file, instrumental audio ││
│ │ Output: CDG.zip, TXT.zip, .cdg, .mp3 ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 3. Encoding: GCE or Local (via interface) ││
│ │ Input: title.mov, karaoke.mov, end.mov ││
│ │ Output: 4K lossless, 4K lossy, 720p, MKV ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 4. Organization: Brand code, folder structure ││
│ │ - Generate brand code (NOMAD-XXXX) ││
│ │ - Prepare full output directory structure ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 5. Distribution: Upload to destinations ││
│ │ - YouTube upload (if enabled) ││
│ │ - Dropbox upload (if configured) ││
│ │ - Google Drive upload (if configured) ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ 6. Notification: Discord, Email ││
│ │ Input: YouTube URL, brand code, Dropbox link ││
│ └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘
│
▼
Result (youtube_url, brand_code, dropbox_link, file_paths, etc.)
```
## Implementation Steps
### Phase 1: Extract Services from KaraokeFinalise (Non-Breaking) ✅ COMPLETE
These services can be used by both the cloud orchestrator AND KaraokeFinalise.
1. [x] **Create `backend/services/youtube_upload_service.py`** ✅
- Extract YouTube upload logic from KaraokeFinalise
- Methods: `upload_video()`, `check_duplicate()`, `delete_video()`, `truncate_title()`
- Supports pre-loaded credentials (server-side) or client secrets file (CLI)
- 23 unit tests with mocked YouTube API
2. [x] **Create `backend/services/discord_service.py`** ✅
- Extract Discord notification from KaraokeFinalise
- Methods: `post_message()`, `post_video_notification()`, `is_enabled()`
- Simple HTTP POST to webhook with validation
- 20 unit tests with mocked requests
3. [x] **Create `backend/services/packaging_service.py`** ✅
- Extract CDG/TXT generation from KaraokeFinalise
- Methods: `create_cdg_package()`, `create_txt_package()`
- Uses lyrics_converter and CDGGenerator (existing dependencies)
- 18 unit tests with temp files
4. [x] **Create `backend/services/local_encoding_service.py`** ✅
- Extract encoding methods from KaraokeFinalise
- Methods: `encode_all_formats()`, individual encoding methods
- Hardware acceleration detection with software fallback
- 28 unit tests with mocked subprocess
### Phase 2: Create Encoding Interface ✅ COMPLETE
5. [x] **Create `backend/services/encoding_interface.py`** ✅
- Define ABC `EncodingBackend` with methods: `encode()`, `is_available()`, `get_status()`
- `EncodingInput` and `EncodingOutput` dataclasses for config/results
- Factory function `get_encoding_backend()` with "local", "gce", "auto" options
6. [x] **`LocalEncodingBackend`** ✅
- Implements `EncodingBackend` interface
- Wraps `LocalEncodingService` with `asyncio.to_thread()`
- 17 unit tests
7. [x] **`GCEEncodingBackend`** ✅
- Implements `EncodingBackend` interface
- Wraps existing `EncodingService` (encoding_service.py)
- Requires GCS paths in options for cloud execution
- 6 additional unit tests
### Phase 3: Create Orchestrator ✅ COMPLETE
8. [x] **Create `backend/workers/video_worker_orchestrator.py`** ✅
- New class that coordinates all stages
- Replaces the branching logic in `generate_video()`
- Methods:
- `__init__(config, job_manager, storage, job_logger)`
- `async run() -> OrchestratorResult`
- `_run_packaging()` - CDG/TXT
- `_run_encoding()` - GCE or local via EncodingBackend interface
- `_run_organization()` - Brand code, folder structure
- `_run_distribution()` - YouTube, Dropbox, GDrive
- `_run_notifications()` - Discord
- 27 unit tests (test_video_worker_orchestrator.py)
9. [x] **Update `backend/workers/video_worker.py`** ✅
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: true)
- `generate_video()` routes to orchestrator or legacy path
- `generate_video_orchestrated()` - new unified pipeline
- `generate_video_legacy()` - original code for rollback
- Legacy GCE and KaraokeFinalise paths preserved for safety
### Phase 4: Update KaraokeFinalise to Use Services (OPTIONAL)
10. [ ] **Update `karaoke_gen/karaoke_finalise/karaoke_finalise.py`**
- Import and use the new services
- Replace inline YouTube/Discord/CDG code with service calls
- Keep process() working for CLI backwards compatibility
- Services become the single source of truth
- **NOTE**: This is optional for MVP - the orchestrator already uses the extracted services directly
### Phase 5: Testing & Validation ✅ COMPLETE
11. [x] **Add unit tests for each service** ✅
- test_youtube_upload_service.py (23 tests)
- test_discord_service.py (20 tests)
- test_packaging_service.py (18 tests)
- test_local_encoding_service.py (28 tests)
- test_encoding_interface.py (23 tests)
- test_video_worker_orchestrator.py (27 tests)
- **Total: 139 new tests**
12. [x] **Full test suite passes** ✅
- 976 tests passed (86 skipped - emulator/integration tests)
- No regressions from changes
13. [ ] **Manual E2E testing** (After deployment)
- [ ] GCE encoding job with YouTube upload enabled
- [ ] Local encoding job with YouTube upload enabled
- [ ] CLI karaoke-gen with all features
- [ ] Job with CDG/TXT enabled
- [ ] Verify output structure matches local CLI
## Files Created/Modified
| File | Action | Description | Status |
|------|--------|-------------|--------|
| `backend/services/youtube_upload_service.py` | Create | YouTube upload logic (~446 lines) | ✅ |
| `backend/services/discord_service.py` | Create | Discord notification service (~167 lines) | ✅ |
| `backend/services/packaging_service.py` | Create | CDG/TXT package generation (~266 lines) | ✅ |
| `backend/services/local_encoding_service.py` | Create | Local FFmpeg encoding service (~512 lines) | ✅ |
| `backend/services/encoding_interface.py` | Create | Abstract interface + backends (~418 lines) | ✅ |
| `backend/workers/video_worker_orchestrator.py` | Create | Stage orchestration logic (~550 lines) | ✅ |
| `backend/workers/video_worker.py` | Modify | Added orchestrator path + feature flag | ✅ |
| `backend/tests/test_youtube_upload_service.py` | Create | 23 unit tests | ✅ |
| `backend/tests/test_discord_service.py` | Create | 20 unit tests | ✅ |
| `backend/tests/test_packaging_service.py` | Create | 18 unit tests | ✅ |
| `backend/tests/test_local_encoding_service.py` | Create | 28 unit tests | ✅ |
| `backend/tests/test_encoding_interface.py` | Create | 23 unit tests | ✅ |
| `backend/tests/test_video_worker_orchestrator.py` | Create | 27 unit tests | ✅ |
| `karaoke_gen/karaoke_finalise/karaoke_finalise.py` | Modify | Use extracted services (optional) | Deferred |
## Testing Strategy
### Unit Tests
- **youtube_upload_service**: Mock YouTube API, test upload/delete/duplicate check
- **discord_service**: Mock requests, test webhook POST
- **packaging_service**: Use temp files, test CDG/TXT generation
- **local_encoding_service**: Mock subprocess, test FFmpeg command construction
### Integration Tests
- **video_worker_orchestrator**: Mock all services, test stage sequencing
- **video_worker**: Test with mocked orchestrator
### E2E Verification
- [ ] GCE encoding job with YouTube upload enabled → YouTube link appears
- [ ] Local encoding job with YouTube upload enabled → Same behavior
- [ ] CLI `karaoke-gen` with all features → No regression
- [ ] Job with CDG/TXT enabled → Packages generated
- [ ] Dropbox output structure → Matches local CLI exactly
## Open Questions
- [x] Should we keep KaraokeFinalise.process() or deprecate it?
- **Decision**: Keep it for CLI backwards compatibility. It will use the new services internally.
- [x] Should local_encoding_service run FFmpeg sync or async?
- **Decision**: Sync implementation, async wrapper via `asyncio.to_thread()` for interface compatibility.
- [x] Order of packaging vs encoding?
- **Decision**: CDG/TXT runs BEFORE encoding (depends on LRC file, not videos). KaraokeFinalise does it this way too.
## Rollback Plan
1. The refactor is additive - new services don't replace old code until orchestrator is wired up
2. Feature flag: `USE_NEW_ORCHESTRATOR=true` to enable new code path
3. If issues found in production:
- Set `USE_NEW_ORCHESTRATOR=false` to revert to old behavior
- Old `generate_video()` code preserved behind feature flag
4. After 1 week stable, remove feature flag and old code
## Success Criteria
1. Job `5b78ad46` scenario (GCE + YouTube) would upload to YouTube
2. All 803+ existing tests pass
3. E2E happy path test passes
4. No regression in local CLI functionality
5. Output structure matches local CLI exactly
6. Code coverage maintained or improved
## Estimated Scope
- **Phase 1** (Extract Services): 4 new service files + tests (~400-600 lines each)
- **Phase 2** (Interface): 1 new file, 2 modifications (~100 lines)
- **Phase 3** (Orchestrator): 2 files (~300-400 lines)
- **Phase 4** (KaraokeFinalise): 1 modification (~100 lines of changes)
- **Phase 5** (Testing): Integration tests + manual verification
This is a medium-sized refactor that can be done incrementally. Each phase produces working, tested code.
@@ -0,0 +1,222 @@
# Plan: Worker Logs Rearchitecture
**Created:** 2026-01-04
**Branch:** `feat/sess-20260104-1931-logs-research`
**Status:** Implementation Complete - Pending PR Merge & Deployment
## Overview
Move `worker_logs` from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).
## Problem Statement
- Firestore documents have a 1 MB size limit
- `worker_logs` is stored as an array field within job documents
- Long-running jobs (especially video encoding) can generate thousands of log entries
- When the limit is exceeded, job updates fail with: `Document cannot be written because its size exceeds the maximum allowed size`
## Requirements
- [x] Logs must remain associated with their job
- [x] Logs should be queryable by job_id, worker type, timestamp
- [x] Old logs should auto-cleanup (TTL: 30 days)
- [x] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
- [x] Frontend must continue to display logs without changes
- [x] Migration path for existing jobs (optional - can just start fresh)
## Technical Approach
### Option A: Subcollection (Recommended)
Store logs in `jobs/{job_id}/logs/{log_id}` subcollection.
**Pros:**
- Natural association with job (delete job = delete logs with cascading)
- Each log entry is its own document (no size limit concerns)
- Can add TTL via Firestore TTL policies
- Efficient queries with composite indexes
**Cons:**
- More Firestore read operations for paginated log fetching
- Slightly more complex write logic
### Option B: Separate Collection
Store logs in `job_logs` collection with `job_id` field.
**Pros:**
- Can query across all jobs
- Simpler TTL management
**Cons:**
- Orphaned logs if job deleted (need cleanup logic)
- Extra index needed for job_id queries
### Decision: Option A (Subcollection)
## Implementation Steps
### Phase 1: Backend Changes
1. [x] **Create log entry model** - `backend/models/worker_log.py`
- `WorkerLogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
- Factory method `create()` for easy instantiation
- `to_dict()` for Firestore storage
- `to_legacy_dict()` for API compatibility
- `from_dict()` for Firestore retrieval
2. [x] **Update FirestoreService** - `backend/services/firestore_service.py`
- Added `append_log_to_subcollection(job_id, log_entry)` method
- Added `get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset)` method
- Added `get_logs_count_from_subcollection(job_id)` method
- Added `delete_logs_subcollection(job_id, batch_size)` method
- Old `append_worker_log` marked as deprecated
3. [x] **Update JobManager** - `backend/services/job_manager.py`
- Modified `append_worker_log()` to use subcollection when `USE_LOG_SUBCOLLECTION=true`
- Modified `get_worker_logs()` to read from subcollection with fallback to embedded array
- Added `get_worker_logs_count()` for total count
- Updated `delete_job()` to clean up logs subcollection
4. [x] **Update jobs API** - `backend/api/routes/jobs.py`
- `GET /api/jobs/{id}/logs` - Updated to use new methods
- Response format stays the same (backward compatible)
- Fixed `next_index` calculation
5. [x] **Add feature flag** - `backend/config.py`
- Added `USE_LOG_SUBCOLLECTION` environment variable (default: true)
6. [x] **Add TTL configuration** - `infrastructure/__main__.py`
- Added Firestore TTL policy for logs subcollection (30 days)
- Added composite index for worker + timestamp queries
### Phase 2: Testing
7. [x] **Unit tests** - `backend/tests/test_worker_log_subcollection.py`
- 24 tests covering:
- WorkerLogEntry model creation and conversion
- FirestoreService subcollection methods
- JobManager logging with feature flag
- Edge cases (unicode, long messages, empty messages)
8. [x] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
- 11 tests using real Firestore emulator:
- Single and sequential writes
- TTL field verification
- Concurrent writes (no race conditions)
- Large volume (500 logs)
- Worker filtering
- Timestamp ordering
- Subcollection deletion
- Concurrent workers interleaved
- Job deletion cleanup
- Comparison with embedded array
### Phase 3: Deployment & Migration
9. [ ] **Deploy with feature flag**
- Add `USE_LOG_SUBCOLLECTION=true` env var (default is true)
- Deploy to production
10. [ ] **Monitor**
- Watch for errors in Cloud Logging
- Verify logs appear in frontend
11. [x] **Backward compatibility**
- Old jobs with embedded arrays still work (fallback in get_worker_logs)
- No migration needed - new jobs use subcollection, old jobs use array
### Phase 4: Cleanup (Future)
12. [ ] **Remove old code path** (after confidence period)
- Remove `worker_logs` field from job model
- Remove old `append_worker_log` method
- Remove feature flag
## Files Created/Modified
| File | Action | Description |
|------|--------|-------------|
| `backend/models/worker_log.py` | Created | WorkerLogEntry dataclass |
| `backend/services/firestore_service.py` | Modified | Added subcollection methods |
| `backend/services/job_manager.py` | Modified | Switch to subcollection with fallback |
| `backend/api/routes/jobs.py` | Modified | Updated logs endpoint |
| `backend/config.py` | Modified | Added USE_LOG_SUBCOLLECTION setting |
| `backend/tests/test_worker_log_subcollection.py` | Created | Unit tests (24 tests) |
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Created | Integration tests (11 tests) |
| `infrastructure/__main__.py` | Modified | Added TTL policy and index |
## Log Entry Schema
```python
@dataclass
class WorkerLogEntry:
timestamp: datetime
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
worker: str # "audio", "lyrics", "video", "render", "screens", "distribution"
message: str
id: str # Auto-generated UUID
job_id: str
ttl_expiry: datetime # timestamp + 30 days
metadata: Optional[Dict[str, Any]] = None
```
## Firestore Structure
```
jobs/
{job_id}/
... (job fields, worker_logs still present for old jobs)
logs/ # subcollection for new jobs
{log_id}/
id: string
job_id: string
timestamp: datetime
level: string
worker: string
message: string
metadata: map (optional)
ttl_expiry: datetime
```
## API Response (Unchanged)
```json
{
"logs": [
{
"timestamp": "2026-01-04T12:00:00Z",
"level": "INFO",
"worker": "video",
"message": "Starting video generation"
}
],
"next_index": 42,
"total_logs": 150
}
```
## Testing Results
All tests pass:
- 24 unit tests in `test_worker_log_subcollection.py`
- 11 emulator integration tests in `test_worker_logs_subcollection.py`
- 68 total tests in emulator suite pass
## Rollback Plan
1. Set `USE_LOG_SUBCOLLECTION=false` env var
2. Redeploy backend
3. New logs will go back to embedded array
4. Subcollection logs will remain but won't be read
## Answers to Open Questions
- [x] **Should we paginate logs in the API?** - Yes, added `offset` parameter support
- [x] **What's the optimal TTL period?** - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
- [x] **Should we index by timestamp for efficient range queries?** - Yes, added composite index (worker + timestamp)
## References
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
- [Pulumi Firestore Field](https://www.pulumi.com/registry/packages/gcp/api-docs/firestore/field/)
+215 -3
View File
@@ -4,6 +4,7 @@ on:
push:
pull_request:
branches: [main]
types: [opened, synchronize, reopened, labeled]
workflow_dispatch:
permissions:
@@ -143,6 +144,23 @@ jobs:
needs.changes.outputs.frontend == 'true' &&
contains(github.event.pull_request.labels.*.name, 'e2e')
steps:
- name: Check disk space
run: |
echo "Checking disk space..."
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Current disk usage: ${USAGE}%"
if [ "$USAGE" -gt 70 ]; then
echo "Disk usage high, running cleanup..."
docker system prune -af || true
docker builder prune -af || true
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "After cleanup: ${USAGE}%"
fi
if [ "$USAGE" -gt 85 ]; then
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
exit 1
fi
- uses: actions/checkout@v4
- name: Setup Node.js with caching
@@ -172,6 +190,64 @@ jobs:
path: frontend/playwright-report/
retention-days: 7
# Happy Path Production Test - runs the comprehensive real-user test against prod
frontend-e2e-happy-path:
name: "Frontend - E2E Happy Path (Production)"
runs-on: ubuntu-latest
needs: changes
# Only runs when PR has 'e2e-happy-path' label
if: >-
github.ref != 'refs/heads/main' &&
contains(github.event.pull_request.labels.*.name, 'e2e-happy-path')
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps chromium
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Run Happy Path Production Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ secrets.E2E_TEST_TOKEN }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
run: |
echo "Starting E2E Happy Path Test against Production"
echo "Time: $(date)"
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=chromium \
--reporter=list \
--timeout=3600000
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results
path: |
frontend/test-results/
frontend/playwright-report/
retention-days: 30
frontend-build:
name: "Frontend - Build"
runs-on: ubuntu-latest
@@ -207,6 +283,23 @@ jobs:
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- name: Check disk space
run: |
echo "Checking disk space..."
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Current disk usage: ${USAGE}%"
if [ "$USAGE" -gt 70 ]; then
echo "Disk usage high, running cleanup..."
docker system prune -af || true
docker builder prune -af || true
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "After cleanup: ${USAGE}%"
fi
if [ "$USAGE" -gt 85 ]; then
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
exit 1
fi
- uses: actions/checkout@v4
with:
submodules: true
@@ -265,6 +358,23 @@ jobs:
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- name: Check disk space
run: |
echo "Checking disk space..."
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Current disk usage: ${USAGE}%"
if [ "$USAGE" -gt 70 ]; then
echo "Disk usage high, running cleanup..."
docker system prune -af || true
docker builder prune -af || true
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "After cleanup: ${USAGE}%"
fi
if [ "$USAGE" -gt 85 ]; then
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
exit 1
fi
- uses: actions/checkout@v4
with:
submodules: true
@@ -318,6 +428,23 @@ jobs:
(needs.package-integration-tests.result == 'success' || needs.package-integration-tests.result == 'skipped') &&
(needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- name: Check disk space
run: |
echo "Checking disk space..."
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Current disk usage: ${USAGE}%"
if [ "$USAGE" -gt 70 ]; then
echo "Disk usage high, running cleanup..."
docker system prune -af || true
docker builder prune -af || true
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "After cleanup: ${USAGE}%"
fi
if [ "$USAGE" -gt 85 ]; then
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
exit 1
fi
- uses: actions/checkout@v4
with:
submodules: true
@@ -351,6 +478,23 @@ jobs:
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- name: Check disk space
run: |
echo "Checking disk space..."
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Current disk usage: ${USAGE}%"
if [ "$USAGE" -gt 70 ]; then
echo "Disk usage high, running cleanup..."
docker system prune -af || true
docker builder prune -af || true
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "After cleanup: ${USAGE}%"
fi
if [ "$USAGE" -gt 85 ]; then
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
exit 1
fi
- name: Checkout code
uses: actions/checkout@v4
with:
@@ -388,6 +532,23 @@ jobs:
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- name: Check disk space
run: |
echo "Checking disk space..."
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Current disk usage: ${USAGE}%"
if [ "$USAGE" -gt 70 ]; then
echo "Disk usage high, running cleanup..."
docker system prune -af || true
docker builder prune -af || true
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "After cleanup: ${USAGE}%"
fi
if [ "$USAGE" -gt 85 ]; then
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
exit 1
fi
- name: Checkout code
uses: actions/checkout@v4
with:
@@ -435,6 +596,23 @@ jobs:
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true')
steps:
- name: Check disk space
run: |
echo "Checking disk space..."
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Current disk usage: ${USAGE}%"
if [ "$USAGE" -gt 70 ]; then
echo "Disk usage high, running cleanup..."
docker system prune -af || true
docker builder prune -af || true
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "After cleanup: ${USAGE}%"
fi
if [ "$USAGE" -gt 85 ]; then
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
exit 1
fi
- name: Checkout code
uses: actions/checkout@v4
with:
@@ -510,6 +688,23 @@ jobs:
runs-on: [self-hosted, linux, gcp]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
steps:
- name: Check disk space
run: |
echo "Checking disk space..."
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Current disk usage: ${USAGE}%"
if [ "$USAGE" -gt 70 ]; then
echo "Disk usage high, running cleanup..."
docker system prune -af || true
docker builder prune -af || true
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "After cleanup: ${USAGE}%"
fi
if [ "$USAGE" -gt 85 ]; then
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
exit 1
fi
- uses: actions/checkout@v4
with:
submodules: true
@@ -720,6 +915,23 @@ jobs:
BASE_IMAGE_NAME: karaoke-backend-base
APP_IMAGE_NAME: karaoke-backend
steps:
- name: Check disk space
run: |
echo "Checking disk space..."
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Current disk usage: ${USAGE}%"
if [ "$USAGE" -gt 70 ]; then
echo "Disk usage high, running cleanup..."
docker system prune -af || true
docker builder prune -af || true
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "After cleanup: ${USAGE}%"
fi
if [ "$USAGE" -gt 85 ]; then
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
exit 1
fi
- name: Checkout code
uses: actions/checkout@v4
with:
@@ -872,9 +1084,9 @@ jobs:
--timeout 1800 \
--concurrency 1 \
--max-instances 50 \
--min-instances 0 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest"
--min-instances 4 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest"
- name: Verify deployment and version
run: |
+173
View File
@@ -0,0 +1,173 @@
name: E2E Happy Path Test
# Comprehensive end-to-end test that validates the full karaoke generation journey
# using ONLY browser interactions (no API shortcuts)
#
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
on:
# Daily run at 6 AM UTC
schedule:
- cron: '0 6 * * *'
# Manual trigger with optional parameters
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
use_test_token:
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
required: false
default: 'false'
type: boolean
permissions:
contents: read
jobs:
e2e-happy-path:
name: "Full Happy Path (Real User)"
runs-on: ubuntu-latest
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: |
BROWSER=${{ inputs.browser || 'chromium' }}
npx playwright install --with-deps $BROWSER
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Run E2E Happy Path Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "Starting E2E Happy Path Test"
echo "Browser: ${{ inputs.browser || 'chromium' }}"
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
echo "Time: $(date)"
echo ""
# Run the real user happy path test
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
echo "result=success" >> $GITHUB_OUTPUT
echo "Test PASSED"
else
echo "result=failure" >> $GITHUB_OUTPUT
echo "Test FAILED - see artifacts for details"
fi
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output.log
retention-days: 30
- name: Upload video recordings
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [ -f frontend/test-output.log ]; then
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
fi
- name: Send email notification
if: always() && github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.sendgrid.net
server_port: 587
username: apikey
password: ${{ secrets.SENDGRID_API_KEY }}
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Happy Path Test Results
===========================
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
Time: ${{ github.event.repository.updated_at }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
This is an automated notification from the daily E2E happy path test.
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
+55 -9
View File
@@ -29,11 +29,8 @@
## Essential Rules
### Git Workflow
- **Never commit directly to `main`** - always use feature branches
- **Use git worktrees** for all changes:
```bash
git worktree add -b feature/xyz ../karaoke-gen-feat-xyz main
```
- **Never commit directly to `main`** - use `/new-worktree <description>` to start
- **Follow global workflow** - see `~/.claude/CLAUDE.md` for command sequence
- Create PR with summary, changes, testing info
- Merge only after CI passes
@@ -55,6 +52,52 @@ cd frontend && npm run test:all 2>&1 | tail -n 200
- `gcloud` CLI for reading/debugging only
- Stop and notify user on auth issues
## API Authentication (for Agents)
When you need to call backend APIs programmatically:
### Internal Worker Endpoints
Use the `X-Admin-Token` header with the token from Secret Manager:
```bash
# Trigger video worker
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
-H "Content-Type: application/json" \
-d '{"job_id": "YOUR_JOB_ID"}'
# Other internal endpoints follow the same pattern
```
### Firestore Direct Access
```bash
# Set the correct project
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
# Use Python with google-cloud-firestore
python3 << 'EOF'
import os
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
from google.cloud import firestore
db = firestore.Client(project='nomadkaraoke')
# Query/update documents...
EOF
```
### GCE Encoding Worker
```bash
# SSH to restart service (clears in-memory job queue)
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
# Check health
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
```
### What Doesn't Work
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
## Documentation Maintenance
### Structure
@@ -90,9 +133,12 @@ Location: `/Users/andrew/Projects/flacfetch`
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
## Pre-Commit Checklist
## Pre-PR Checklist
- [ ] Tests pass (`make test` / `npm run test:all`)
Follow global workflow (`~/.claude/CLAUDE.md`):
- [ ] Tests pass: `/test` (or `make test` / `npm run test:all`)
- [ ] Version bumped (if code changed)
- [ ] `/docs-review` run (check for doc updates)
- [ ] PR created with clear description
- [ ] Docs checked: `/docs-review`
- [ ] Code reviewed: `/review` (CodeRabbit CLI)
- [ ] PR created: `/pr` (adds @coderabbitai ignore)
+30 -1
View File
@@ -10,14 +10,43 @@ FROM python:3.11-slim
# Note: Transmission daemon removed - torrent downloads now handled by
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
RUN apt-get update && apt-get install -y \
ffmpeg \
libsndfile1 \
libsox-dev \
sox \
build-essential \
curl \
xz-utils \
&& rm -rf /var/lib/apt/lists/*
# Install comprehensive font support for international lyrics and symbols
# Noto fonts provide full Unicode coverage including:
# - Latin, Greek, Cyrillic (core)
# - CJK: Chinese, Japanese, Korean (~150MB)
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
# - Color emoji (limited support in libass, but available)
# - Musical symbols (♪ ♫ etc.)
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
RUN apt-get update && apt-get install -y \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
# Install optimized static FFmpeg build (7.0.2) instead of Debian package (5.1.8)
# John Van Sickle's builds are compiled with better optimizations for x86_64
# This provides ~20-40% performance improvement for encoding tasks
# Source: https://johnvansickle.com/ffmpeg/
RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz && \
tar -xf /tmp/ffmpeg.tar.xz -C /tmp && \
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/ && \
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/ && \
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe && \
rm -rf /tmp/ffmpeg* && \
ffmpeg -version
# Set working directory
WORKDIR /app
+582
View File
@@ -0,0 +1,582 @@
"""
Admin API routes for dashboard statistics and management.
Handles:
- Dashboard overview statistics
- System-wide metrics
- Admin-only operations
- Audio search cache management
"""
import logging
from datetime import datetime, timedelta
from typing import Tuple, List, Optional, Any, Dict
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from backend.api.dependencies import require_admin
from backend.services.auth_service import UserType
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
from backend.services.job_manager import JobManager
from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchServiceError
from backend.models.job import JobStatus
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/admin", tags=["admin"])
# =============================================================================
# Response Models
# =============================================================================
class JobsByStatusResponse(BaseModel):
"""Breakdown of jobs by status."""
pending: int = 0
processing: int = 0
awaiting_review: int = 0
awaiting_instrumental: int = 0
complete: int = 0
failed: int = 0
cancelled: int = 0
class AdminStatsOverview(BaseModel):
"""Overview statistics for admin dashboard."""
total_users: int
active_users_7d: int
active_users_30d: int
total_jobs: int
jobs_last_7d: int
jobs_last_30d: int
jobs_by_status: JobsByStatusResponse
total_credits_issued_30d: int
total_beta_testers: int
# =============================================================================
# Admin Stats Endpoints
# =============================================================================
@router.get("/stats/overview", response_model=AdminStatsOverview)
async def get_admin_stats_overview(
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
"""
Get overview statistics for admin dashboard.
Includes:
- User counts (total, active in 7d, active in 30d)
- Job counts (total, by status, recent)
- Credit statistics
- Beta program stats
"""
from google.cloud.firestore_v1 import FieldFilter
from google.cloud.firestore_v1 import aggregation
db = user_service.db
now = datetime.utcnow()
seven_days_ago = now - timedelta(days=7)
thirty_days_ago = now - timedelta(days=30)
# Helper function to get count using aggregation
def get_count(query) -> int:
try:
agg_query = aggregation.AggregationQuery(query)
agg_query.count(alias="count")
results = agg_query.get()
return results[0][0].value if results else 0
except Exception as e:
logger.warning(f"Aggregation query failed: {e}")
return 0
# User statistics
users_collection = db.collection(USERS_COLLECTION)
total_users = get_count(users_collection)
active_users_7d = get_count(
users_collection.where(filter=FieldFilter("last_login_at", ">=", seven_days_ago))
)
active_users_30d = get_count(
users_collection.where(filter=FieldFilter("last_login_at", ">=", thirty_days_ago))
)
total_beta_testers = get_count(
users_collection.where(filter=FieldFilter("is_beta_tester", "==", True))
)
# Job statistics
jobs_collection = db.collection("jobs")
total_jobs = get_count(jobs_collection)
jobs_last_7d = get_count(
jobs_collection.where(filter=FieldFilter("created_at", ">=", seven_days_ago))
)
jobs_last_30d = get_count(
jobs_collection.where(filter=FieldFilter("created_at", ">=", thirty_days_ago))
)
# Jobs by status - map multiple statuses to simplified categories
processing_statuses = [
"downloading", "downloading_audio", "searching_audio", "awaiting_audio_selection",
"separating_stage1", "separating_stage2", "transcribing", "correcting",
"generating_screens", "applying_padding", "rendering_video",
"instrumental_selected", "generating_video", "encoding", "packaging",
"uploading", "notifying"
]
jobs_by_status = JobsByStatusResponse(
pending=get_count(
jobs_collection.where(filter=FieldFilter("status", "==", "pending"))
),
processing=sum(
get_count(jobs_collection.where(filter=FieldFilter("status", "==", status)))
for status in processing_statuses
),
awaiting_review=get_count(
jobs_collection.where(filter=FieldFilter("status", "==", "awaiting_review"))
) + get_count(
jobs_collection.where(filter=FieldFilter("status", "==", "in_review"))
),
awaiting_instrumental=get_count(
jobs_collection.where(filter=FieldFilter("status", "==", "awaiting_instrumental_selection"))
),
complete=get_count(
jobs_collection.where(filter=FieldFilter("status", "==", "complete"))
) + get_count(
jobs_collection.where(filter=FieldFilter("status", "==", "prep_complete"))
),
failed=get_count(
jobs_collection.where(filter=FieldFilter("status", "==", "failed"))
),
cancelled=get_count(
jobs_collection.where(filter=FieldFilter("status", "==", "cancelled"))
),
)
# Credit statistics - sum credits added in last 30 days
# This is more expensive, so we'll just estimate from users
total_credits_issued_30d = 0
try:
# Get all users and sum recent credit transactions
users_docs = users_collection.limit(500).stream()
for user_doc in users_docs:
user_data = user_doc.to_dict()
transactions = user_data.get("credit_transactions", [])
for txn in transactions:
txn_date = txn.get("created_at")
if txn_date:
# Handle both datetime and string formats
if isinstance(txn_date, str):
try:
txn_date = datetime.fromisoformat(txn_date.replace("Z", "+00:00"))
except Exception:
continue
if isinstance(txn_date, datetime):
txn_date = txn_date.replace(tzinfo=None)
else:
continue
if txn_date >= thirty_days_ago:
amount = txn.get("amount", 0)
if amount > 0: # Only count additions, not deductions
total_credits_issued_30d += amount
except Exception as e:
logger.warning(f"Error calculating credits: {e}")
return AdminStatsOverview(
total_users=total_users,
active_users_7d=active_users_7d,
active_users_30d=active_users_30d,
total_jobs=total_jobs,
jobs_last_7d=jobs_last_7d,
jobs_last_30d=jobs_last_30d,
jobs_by_status=jobs_by_status,
total_credits_issued_30d=total_credits_issued_30d,
total_beta_testers=total_beta_testers,
)
# =============================================================================
# Audio Search Management Models
# =============================================================================
class AudioSearchResultSummary(BaseModel):
"""Summary of a single audio search result."""
index: int
provider: str
artist: str
title: str
is_lossless: bool
quality: Optional[str] = None
seeders: Optional[int] = None
class AudioSearchJobSummary(BaseModel):
"""Summary of a job with audio search results."""
job_id: str
status: str
user_email: Optional[str] = None
audio_search_artist: Optional[str] = None
audio_search_title: Optional[str] = None
created_at: Optional[datetime] = None
results_count: int
results_summary: List[AudioSearchResultSummary]
has_lossless: bool
providers: List[str]
class AudioSearchListResponse(BaseModel):
"""Response for listing audio search jobs."""
jobs: List[AudioSearchJobSummary]
total: int
class ClearSearchCacheResponse(BaseModel):
"""Response for clearing search cache."""
status: str
job_id: str
message: str
previous_status: str
new_status: str
results_cleared: int
flacfetch_cache_cleared: bool = False
flacfetch_error: Optional[str] = None
class ClearAllCacheResponse(BaseModel):
"""Response for clearing all flacfetch cache."""
status: str
message: str
deleted_count: int
class CacheStatsResponse(BaseModel):
"""Response for cache statistics."""
count: int
total_size_bytes: int
oldest_entry: Optional[str] = None
newest_entry: Optional[str] = None
configured: bool
# =============================================================================
# Audio Search Management Endpoints
# =============================================================================
@router.get("/audio-searches", response_model=AudioSearchListResponse)
async def list_audio_searches(
limit: int = 50,
status_filter: Optional[str] = None,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
"""
List jobs with audio search results.
Returns jobs that have cached audio search results, useful for:
- Monitoring search activity
- Identifying stale cached results
- Clearing cache for specific jobs
Args:
limit: Maximum number of jobs to return (default 50)
status_filter: Optional filter by job status (e.g., 'awaiting_audio_selection')
"""
from google.cloud.firestore_v1 import FieldFilter
db = user_service.db
jobs_collection = db.collection("jobs")
# Query jobs - we'll filter for those with audio_search_results in Python
# since Firestore can't query for existence of nested fields efficiently
query = jobs_collection.order_by("created_at", direction="DESCENDING").limit(500)
if status_filter:
query = jobs_collection.where(
filter=FieldFilter("status", "==", status_filter)
).order_by("created_at", direction="DESCENDING").limit(500)
jobs_with_searches = []
for doc in query.stream():
data = doc.to_dict()
state_data = data.get("state_data", {})
audio_results = state_data.get("audio_search_results", [])
if not audio_results:
continue
# Compute has_lossless and providers from ALL results (not just first 10)
has_lossless = any(r.get("is_lossless", False) for r in audio_results)
providers = {r.get("provider", "Unknown") for r in audio_results}
# Build summary from first 10 results only
results_summary = []
for r in audio_results[:10]:
results_summary.append(AudioSearchResultSummary(
index=r.get("index", 0),
provider=r.get("provider", "Unknown"),
artist=r.get("artist", ""),
title=r.get("title", ""),
is_lossless=r.get("is_lossless", False),
quality=r.get("quality"),
seeders=r.get("seeders"),
))
jobs_with_searches.append(AudioSearchJobSummary(
job_id=doc.id,
status=data.get("status", "unknown"),
user_email=data.get("user_email"),
audio_search_artist=data.get("audio_search_artist"),
audio_search_title=data.get("audio_search_title"),
created_at=data.get("created_at"),
results_count=len(audio_results),
results_summary=results_summary,
has_lossless=has_lossless,
providers=sorted(providers),
))
if len(jobs_with_searches) >= limit:
break
return AudioSearchListResponse(
jobs=jobs_with_searches,
total=len(jobs_with_searches),
)
@router.post("/audio-searches/{job_id}/clear-cache", response_model=ClearSearchCacheResponse)
async def clear_audio_search_cache(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
"""
Clear the cached audio search results for a job.
This will:
1. Remove the cached search results from job.state_data
2. Reset the job status to 'pending' so a new search can be performed
3. Clear the flacfetch GCS cache for this artist/title (if available)
Use this when:
- Cached results are stale (e.g., flacfetch was updated)
- User wants to search again with different terms
- Results appear incomplete or incorrect
"""
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Get current state
state_data = job.state_data or {}
audio_results = state_data.get("audio_search_results", [])
results_count = len(audio_results)
previous_status = job.status
if not audio_results:
raise HTTPException(
status_code=400,
detail=f"Job {job_id} has no cached audio search results"
)
# Validate job status - only allow cache clear for appropriate states
# Don't allow clearing cache for jobs that are actively processing or complete
forbidden_statuses = {
"downloading", "downloading_audio", "searching_audio",
"separating_stage1", "separating_stage2", "transcribing", "correcting",
"generating_screens", "applying_padding", "rendering_video",
"generating_video", "encoding", "packaging", "uploading",
"complete", "prep_complete",
}
if previous_status in forbidden_statuses:
raise HTTPException(
status_code=400,
detail=f"Cannot clear cache for job in '{previous_status}' state. "
f"Only jobs in pending, awaiting_audio_selection, failed, or cancelled states can have cache cleared."
)
# Clear the cache by removing the keys
db = user_service.db
job_ref = db.collection("jobs").document(job_id)
from google.cloud.firestore_v1 import DELETE_FIELD
# Update job: clear cache and reset status
job_ref.update({
"state_data.audio_search_results": DELETE_FIELD,
"state_data.audio_search_count": DELETE_FIELD,
"state_data.remote_search_id": DELETE_FIELD,
"status": "pending",
"progress": 0,
"message": "Audio search cache cleared by admin. Ready for new search.",
"updated_at": datetime.utcnow(),
})
# Also clear flacfetch's GCS cache if we have artist/title
flacfetch_cache_cleared = False
flacfetch_error = None
artist = job.audio_search_artist
title = job.audio_search_title
if artist and title:
flacfetch_client = get_flacfetch_client()
if flacfetch_client:
try:
flacfetch_cache_cleared = await flacfetch_client.clear_search_cache(artist, title)
logger.info(
f"Cleared flacfetch cache for '{artist}' - '{title}': "
f"{'deleted' if flacfetch_cache_cleared else 'no entry found'}"
)
except FlacfetchServiceError as e:
flacfetch_error = str(e)
logger.warning(f"Failed to clear flacfetch cache: {e}")
else:
flacfetch_error = "flacfetch client not configured"
logger.debug("Skipping flacfetch cache clear - client not configured")
else:
flacfetch_error = "missing artist or title"
logger.debug(f"Skipping flacfetch cache clear - missing artist ({artist}) or title ({title})")
logger.info(
f"Admin {auth_data[0]} cleared audio search cache for job {job_id}. "
f"Cleared {results_count} results. Status changed from {previous_status} to pending. "
f"Flacfetch cache cleared: {flacfetch_cache_cleared}"
)
message = f"Cleared {results_count} cached search results. Job reset to pending."
if flacfetch_cache_cleared:
message += " Flacfetch cache also cleared."
elif flacfetch_error:
message += f" Note: flacfetch cache not cleared ({flacfetch_error})."
return ClearSearchCacheResponse(
status="success",
job_id=job_id,
message=message,
previous_status=previous_status,
new_status="pending",
results_cleared=results_count,
flacfetch_cache_cleared=flacfetch_cache_cleared,
flacfetch_error=flacfetch_error,
)
@router.post("/jobs/{job_id}/reset-worker-state")
async def reset_worker_state(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Reset stale worker progress state for a job.
This clears the video_progress, render_progress, and screens_progress
from state_data, allowing workers to be re-triggered.
Use this when a job is stuck because worker progress shows 'running'
from a previous failed attempt.
"""
from backend.services.job_manager import JobManager
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Reset worker progress states
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
logger.info(f"Admin {auth_data[0]} reset worker state for job {job_id}")
return {
"status": "success",
"job_id": job_id,
"message": "Worker progress states reset to pending"
}
@router.delete("/cache", response_model=ClearAllCacheResponse)
async def clear_all_flacfetch_cache(
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Clear the entire flacfetch search cache.
This will delete all cached search results from flacfetch's GCS cache.
Use with caution - this will cause all subsequent searches to hit
the trackers fresh.
Note: This does NOT clear Firestore job.state_data caches, only the
flacfetch-side GCS cache.
"""
flacfetch_client = get_flacfetch_client()
if not flacfetch_client:
raise HTTPException(
status_code=503,
detail="flacfetch client not configured"
)
try:
deleted_count = await flacfetch_client.clear_all_cache()
logger.info(
f"Admin {auth_data[0]} cleared all flacfetch cache. "
f"Deleted {deleted_count} entries."
)
return ClearAllCacheResponse(
status="success",
message=f"Cleared {deleted_count} cache entries from flacfetch.",
deleted_count=deleted_count,
)
except FlacfetchServiceError as e:
logger.error(f"Failed to clear all flacfetch cache: {e}")
raise HTTPException(
status_code=502,
detail=f"Failed to clear flacfetch cache: {e}"
)
@router.get("/cache/stats", response_model=CacheStatsResponse)
async def get_flacfetch_cache_stats(
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Get statistics about the flacfetch search cache.
Returns information about:
- Number of cached entries
- Total size in bytes
- Oldest and newest cache entries
- Whether cache is configured
"""
flacfetch_client = get_flacfetch_client()
if not flacfetch_client:
raise HTTPException(
status_code=503,
detail="flacfetch client not configured"
)
try:
stats = await flacfetch_client.get_cache_stats()
return CacheStatsResponse(
count=stats.get("count", 0),
total_size_bytes=stats.get("total_size_bytes", 0),
oldest_entry=stats.get("oldest_entry"),
newest_entry=stats.get("newest_entry"),
configured=stats.get("configured", True),
)
except FlacfetchServiceError as e:
logger.error(f"Failed to get flacfetch cache stats: {e}")
raise HTTPException(
status_code=502,
detail=f"Failed to get cache stats: {e}"
)
# Trigger deployment
+40 -8
View File
@@ -35,7 +35,7 @@ from backend.services.audio_search_service import (
from backend.config import get_settings
from backend.version import VERSION
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType
from backend.services.auth_service import UserType, AuthResult
from pathlib import Path
logger = logging.getLogger(__name__)
@@ -95,7 +95,7 @@ class AudioSearchRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
@@ -117,6 +117,9 @@ class AudioSearchRequest(BaseModel):
# Style params JSON and related assets (background images, fonts)
style_files: Optional[List[StyleFileRequest]] = Field(None, description="Style files to upload (style_params JSON, backgrounds, fonts)")
# Non-interactive mode
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
class AudioSearchResultResponse(BaseModel):
"""A single audio search result.
@@ -372,8 +375,31 @@ async def _download_and_start_processing(
target_file=target_file,
download_url=download_url,
)
elif source_name == 'YouTube' and download_url:
# For YouTube without remote, download directly using URL from state_data
# This avoids relying on in-memory cache which doesn't persist across instances
logger.info(f"Downloading YouTube audio directly from URL: {download_url}")
from backend.workers.audio_worker import download_from_url
local_path = await download_from_url(
download_url,
temp_dir,
selected.get('artist'),
selected.get('title')
)
if not local_path or not os.path.exists(local_path):
raise DownloadError(f"Failed to download from YouTube: {download_url}")
# Create a result-like object
class DownloadResult:
def __init__(self, filepath):
self.filepath = filepath
result = DownloadResult(local_path)
else:
# Fallback to search-based download
# Fallback to search-based download (may fail if cache is empty)
logger.warning(f"Falling back to cache-based download for {source_name} - this may fail on multi-instance deployments")
result = audio_search_service.download(
result_index=selection_index,
output_dir=temp_dir,
@@ -436,7 +462,7 @@ async def search_audio(
request: Request,
background_tasks: BackgroundTasks,
body: AudioSearchRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
auth_result: AuthResult = Depends(require_auth)
):
"""
Search for audio by artist and title, creating a new job.
@@ -459,7 +485,8 @@ async def search_audio(
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
# Apply defaults for YouTube/Dropbox distribution (for web service)
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
# Use explicit value if provided, otherwise fall back to server default
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
effective_youtube_description = body.youtube_description or settings.default_youtube_description
@@ -500,6 +527,9 @@ async def search_audio(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Use authenticated user's email
effective_user_email = auth_result.user_email
# Create job
job_create = JobCreate(
artist=body.artist,
@@ -515,6 +545,7 @@ async def search_audio(
discord_webhook_url=effective_discord_webhook_url,
dropbox_path=effective_dropbox_path,
gdrive_folder_id=effective_gdrive_folder_id,
user_email=effective_user_email,
lyrics_artist=body.lyrics_artist,
lyrics_title=body.lyrics_title,
subtitle_offset_ms=body.subtitle_offset_ms,
@@ -525,10 +556,11 @@ async def search_audio(
audio_search_title=body.title,
auto_download=body.auto_download,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
)
job = job_manager.create_job(job_create)
job_id = job.job_id
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
# Update job with audio search fields
@@ -766,7 +798,7 @@ async def search_audio(
@router.get("/audio-search/{job_id}/results")
async def get_audio_search_results(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
auth_result: AuthResult = Depends(require_auth)
):
"""
Get audio search results for a job.
@@ -801,7 +833,7 @@ async def select_audio_source(
job_id: str,
background_tasks: BackgroundTasks,
body: AudioSelectRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
auth_result: AuthResult = Depends(require_auth)
):
"""
Select an audio source and start job processing.
+86 -27
View File
@@ -33,7 +33,7 @@ from backend.config import get_settings
from backend.version import VERSION
from backend.services.metrics import metrics
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType
from backend.services.auth_service import UserType, AuthResult
logger = logging.getLogger(__name__)
router = APIRouter(tags=["jobs"])
@@ -69,7 +69,7 @@ class CreateJobFromUrlRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
@@ -92,6 +92,9 @@ class CreateJobFromUrlRequest(BaseModel):
backing_vocals_models: Optional[List[str]] = Field(None, description="Models for backing vocals separation")
other_stems_models: Optional[List[str]] = Field(None, description="Models for other stems")
# Non-interactive mode
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
class CreateJobFromUrlResponse(BaseModel):
"""Response from creating a job from URL."""
@@ -120,7 +123,7 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
@@ -150,6 +153,9 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
prep_only: bool = Field(False, description="Stop after review phase, don't run finalisation")
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code instead of generating new one")
# Non-interactive mode
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
class SignedUploadUrl(BaseModel):
"""Signed URL for uploading a file."""
@@ -247,6 +253,16 @@ async def _trigger_workers_parallel(job_id: str) -> None:
)
async def _trigger_audio_worker_only(job_id: str) -> None:
"""
Trigger only the audio worker.
Used for URL jobs where audio needs to be downloaded first.
The audio worker will trigger the lyrics worker after download completes.
"""
await worker_service.trigger_audio_worker(job_id)
def _resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool],
@@ -368,7 +384,7 @@ def _prepare_theme_for_job(
async def upload_and_create_job(
request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_auth),
auth_result: AuthResult = Depends(require_auth),
# Required fields
file: UploadFile = File(..., description="Audio file to process"),
artist: str = Form(..., description="Artist name"),
@@ -390,7 +406,7 @@ async def upload_and_create_job(
enable_txt: Optional[bool] = Form(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise"),
# Finalisation options
brand_prefix: Optional[str] = Form(None, description="Brand code prefix (e.g., NOMAD)"),
enable_youtube_upload: bool = Form(False, description="Upload to YouTube"),
enable_youtube_upload: Optional[bool] = Form(None, description="Upload to YouTube. None = use server default"),
youtube_description: Optional[str] = Form(None, description="YouTube video description text"),
discord_webhook_url: Optional[str] = Form(None, description="Discord webhook URL for notifications"),
webhook_url: Optional[str] = Form(None, description="Generic webhook URL"),
@@ -409,6 +425,8 @@ async def upload_and_create_job(
clean_instrumental_model: Optional[str] = Form(None, description="Model for clean instrumental separation (e.g., model_bs_roformer_ep_317_sdr_12.9755.ckpt)"),
backing_vocals_models: Optional[str] = Form(None, description="Comma-separated list of models for backing vocals separation"),
other_stems_models: Optional[str] = Form(None, description="Comma-separated list of models for other stems (bass, drums, guitar, etc.)"),
# Non-interactive mode
non_interactive: bool = Form(False, description="Skip interactive steps (lyrics review, instrumental selection)"),
):
"""
Upload an audio file and create a karaoke generation job with full style configuration.
@@ -492,12 +510,19 @@ async def upload_and_create_job(
if dist.brand_prefix and not brand_prefix:
logger.info(f"Using default brand_prefix: {dist.brand_prefix}")
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload
if effective_enable_youtube_upload and enable_youtube_upload is None:
logger.info("Using default enable_youtube_upload: True (from env)")
# Validate credentials for requested distribution services (including defaults)
# This prevents accepting jobs that will fail later due to missing credentials
invalid_services = []
credential_manager = get_credential_manager()
if enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -563,6 +588,9 @@ async def upload_and_create_job(
if other_stems_models:
parsed_other_stems_models = [m.strip() for m in other_stems_models.split(',') if m.strip()]
# Prefer authenticated user's email over form parameter
effective_user_email = auth_result.user_email or user_email
# Create job first to get job_id
job_create = JobCreate(
artist=artist,
@@ -573,11 +601,11 @@ async def upload_and_create_job(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=youtube_description,
discord_webhook_url=dist.discord_webhook_url,
webhook_url=webhook_url,
user_email=user_email,
user_email=effective_user_email,
# Native API distribution (preferred for remote CLI)
dropbox_path=dist.dropbox_path,
gdrive_folder_id=dist.gdrive_folder_id,
@@ -593,6 +621,8 @@ async def upload_and_create_job(
other_stems_models=parsed_other_stems_models,
# Request metadata for tracking and filtering
request_metadata=request_metadata,
# Non-interactive mode
non_interactive=non_interactive,
)
job = job_manager.create_job(job_create)
job_id = job.job_id
@@ -792,11 +822,12 @@ async def upload_and_create_job(
"using_default": gdrive_folder_id is None,
}
if enable_youtube_upload:
if effective_enable_youtube_upload:
youtube_result = credential_manager.check_youtube_credentials()
distribution_services["youtube"] = {
"enabled": True,
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
"using_default": enable_youtube_upload is None,
}
if dist.discord_webhook_url:
@@ -885,7 +916,7 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code from folder name")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
@@ -973,7 +1004,7 @@ def _get_gcs_path_for_file(job_id: str, file_type: str, filename: str) -> str:
async def create_job_with_upload_urls(
request: Request,
body: CreateJobWithUploadUrlsRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
auth_result: AuthResult = Depends(require_auth)
):
"""
Create a karaoke generation job and return signed URLs for direct file upload to GCS.
@@ -1025,11 +1056,16 @@ async def create_job_with_upload_urls(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1069,6 +1105,9 @@ async def create_job_with_upload_urls(
# Check if style_params is being uploaded (overrides theme)
has_style_params_upload = any(f.file_type == 'style_params' for f in body.files)
# Prefer authenticated user's email over request body
effective_user_email = auth_result.user_email or body.user_email
# Create job
job_create = JobCreate(
artist=body.artist,
@@ -1079,12 +1118,12 @@ async def create_job_with_upload_urls(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
youtube_description_template=body.youtube_description, # video_worker reads this field
discord_webhook_url=dist.discord_webhook_url,
webhook_url=body.webhook_url,
user_email=body.user_email,
user_email=effective_user_email,
dropbox_path=dist.dropbox_path,
gdrive_folder_id=dist.gdrive_folder_id,
organised_dir_rclone_root=body.organised_dir_rclone_root,
@@ -1095,6 +1134,7 @@ async def create_job_with_upload_urls(
backing_vocals_models=body.backing_vocals_models,
other_stems_models=body.other_stems_models,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
)
job = job_manager.create_job(job_create)
job_id = job.job_id
@@ -1160,7 +1200,7 @@ async def mark_uploads_complete(
job_id: str,
background_tasks: BackgroundTasks,
body: UploadsCompleteRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
auth_result: AuthResult = Depends(require_auth)
):
"""
Mark file uploads as complete and start job processing.
@@ -1406,7 +1446,7 @@ async def create_job_from_url(
request: Request,
background_tasks: BackgroundTasks,
body: CreateJobFromUrlRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
auth_result: AuthResult = Depends(require_auth)
):
"""
Create a karaoke generation job from a YouTube or other online video URL.
@@ -1439,11 +1479,16 @@ async def create_job_from_url(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1481,6 +1526,9 @@ async def create_job_from_url(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Prefer authenticated user's email over request body
effective_user_email = auth_result.user_email or body.user_email
# Create job with URL
job_create = JobCreate(
url=body.url,
@@ -1492,11 +1540,11 @@ async def create_job_from_url(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
discord_webhook_url=dist.discord_webhook_url,
webhook_url=body.webhook_url,
user_email=body.user_email,
user_email=effective_user_email,
dropbox_path=dist.dropbox_path,
gdrive_folder_id=dist.gdrive_folder_id,
organised_dir_rclone_root=body.organised_dir_rclone_root,
@@ -1507,6 +1555,7 @@ async def create_job_from_url(
backing_vocals_models=body.backing_vocals_models,
other_stems_models=body.other_stems_models,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
)
job = job_manager.create_job(job_create)
job_id = job.job_id
@@ -1542,10 +1591,11 @@ async def create_job_from_url(
progress=5,
message="Starting audio download from URL"
)
# Trigger workers in parallel
# The audio worker will handle downloading from URL
background_tasks.add_task(_trigger_workers_parallel, job_id)
# For URL jobs, trigger ONLY audio worker first
# The audio worker will trigger lyrics worker after download completes
# This prevents race condition where lyrics worker times out waiting for audio
background_tasks.add_task(_trigger_audio_worker_only, job_id)
return CreateJobFromUrlResponse(
status="success",
@@ -1608,7 +1658,7 @@ def _get_gcs_path_for_finalise_file(job_id: str, file_type: str, filename: str)
async def create_finalise_only_job(
request: Request,
body: CreateFinaliseOnlyJobRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
auth_result: AuthResult = Depends(require_auth)
):
"""
Create a finalise-only job for continuing from a local prep phase.
@@ -1680,11 +1730,16 @@ async def create_finalise_only_job(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate distribution credentials if services are requested
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1721,6 +1776,9 @@ async def create_finalise_only_job(
# Check if style_params is being uploaded (overrides theme)
has_style_params_upload = any(f.file_type == 'style_params' for f in body.files)
# Use authenticated user's email
effective_user_email = auth_result.user_email
# Create job with finalise_only=True
job_create = JobCreate(
artist=body.artist,
@@ -1731,11 +1789,12 @@ async def create_finalise_only_job(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
discord_webhook_url=dist.discord_webhook_url,
dropbox_path=dist.dropbox_path,
gdrive_folder_id=dist.gdrive_folder_id,
user_email=effective_user_email,
finalise_only=True,
keep_brand_code=body.keep_brand_code,
request_metadata=request_metadata,
@@ -1804,7 +1863,7 @@ async def mark_finalise_uploads_complete(
job_id: str,
background_tasks: BackgroundTasks,
body: UploadsCompleteRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
auth_result: AuthResult = Depends(require_auth)
):
"""
Mark finalise-only file uploads as complete and start video generation.
+144
View File
@@ -3,6 +3,8 @@ Health check routes.
"""
import os
import logging
import subprocess
import platform
from fastapi import APIRouter
from typing import Dict, Any, Optional
@@ -10,11 +12,114 @@ from backend.version import VERSION
from backend.services.flacfetch_client import get_flacfetch_client
from backend.services.email_service import get_email_service
from backend.services.stripe_service import get_stripe_service
from backend.services.encoding_service import get_encoding_service
router = APIRouter()
logger = logging.getLogger(__name__)
def get_system_info() -> Dict[str, Any]:
"""Get system information for performance analysis."""
info = {
"platform": platform.system(),
"platform_release": platform.release(),
"python_version": platform.python_version(),
}
# Get CPU info
try:
cpu_count = os.cpu_count() or 0
info["cpu_count"] = cpu_count
# Try to get more detailed CPU info on Linux
if platform.system() == "Linux":
try:
with open("/proc/cpuinfo", "r") as f:
for line in f:
if "model name" in line:
info["cpu_model"] = line.split(":")[1].strip()
break
except Exception:
pass
except Exception as e:
info["cpu_error"] = str(e)
# Get memory info
try:
if platform.system() == "Linux":
with open("/proc/meminfo", "r") as f:
for line in f:
if line.startswith("MemTotal:"):
mem_kb = int(line.split()[1])
info["memory_gb"] = round(mem_kb / (1024 * 1024), 1)
break
except Exception as e:
info["memory_error"] = str(e)
return info
def get_ffmpeg_info() -> Dict[str, Any]:
"""Get FFmpeg version and configuration for debugging encoding performance."""
try:
# Get FFmpeg version
result = subprocess.run(
["ffmpeg", "-version"],
capture_output=True,
text=True,
timeout=10
)
if result.returncode == 0:
lines = result.stdout.split("\n")
version_line = lines[0] if lines else "unknown"
# Check for hardware acceleration support
hw_accel = {}
config_result = subprocess.run(
["ffmpeg", "-hwaccels"],
capture_output=True,
text=True,
timeout=10
)
if config_result.returncode == 0:
hw_lines = config_result.stdout.strip().split("\n")
hw_accel["available"] = [h.strip() for h in hw_lines[1:] if h.strip()]
# Check encoder availability
encoders = {}
encoder_result = subprocess.run(
["ffmpeg", "-encoders"],
capture_output=True,
text=True,
timeout=10
)
if encoder_result.returncode == 0:
encoder_output = encoder_result.stdout
# Check for specific encoders we care about
encoders["libx264"] = "libx264" in encoder_output
encoders["h264_nvenc"] = "h264_nvenc" in encoder_output
encoders["h264_vaapi"] = "h264_vaapi" in encoder_output
encoders["hevc_nvenc"] = "hevc_nvenc" in encoder_output
return {
"available": True,
"version": version_line,
"hw_acceleration": hw_accel,
"encoders": encoders,
}
else:
return {
"available": False,
"error": result.stderr[:500] if result.stderr else "Unknown error",
}
except subprocess.TimeoutExpired:
return {"available": False, "error": "Timeout getting FFmpeg info"}
except FileNotFoundError:
return {"available": False, "error": "FFmpeg not found"}
except Exception as e:
return {"available": False, "error": str(e)}
def check_transmission_status() -> Dict[str, Any]:
"""Check if Transmission daemon is available and responsive."""
try:
@@ -101,6 +206,37 @@ async def check_flacfetch_service_status() -> Dict[str, Any]:
}
async def check_encoding_worker_status() -> Dict[str, Any]:
"""Check if GCE encoding worker is available and healthy."""
encoding_service = get_encoding_service()
if not encoding_service.is_configured:
return {
"configured": False,
"enabled": encoding_service.settings.use_gce_encoding,
"message": "GCE encoding worker not configured (ENCODING_WORKER_URL or API key not set)",
}
try:
health = await encoding_service.health_check()
return {
"configured": True,
"enabled": encoding_service.is_enabled,
"available": health.get("status") == "ok",
"status": health.get("status"),
"active_jobs": health.get("active_jobs", 0),
"queue_length": health.get("queue_length", 0),
"ffmpeg_version": health.get("ffmpeg_version"),
}
except Exception as e:
return {
"configured": True,
"enabled": encoding_service.is_enabled,
"available": False,
"error": str(e),
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
@@ -110,6 +246,7 @@ async def detailed_health_check() -> Dict[str, Any]:
"""
transmission_status = check_transmission_status()
flacfetch_status = await check_flacfetch_service_status()
encoding_status = await check_encoding_worker_status()
# Check email service
email_service = get_email_service()
@@ -141,13 +278,20 @@ async def detailed_health_check() -> Dict[str, Any]:
if not stripe_status["configured"]:
logger.warning("Stripe service not configured - payments will not work")
# Get system and FFmpeg info for performance analysis
system_info = get_system_info()
ffmpeg_info = get_ffmpeg_info()
return {
"status": "healthy",
"service": "karaoke-gen-backend",
"version": VERSION,
"system": system_info,
"ffmpeg": ffmpeg_info,
"dependencies": {
"transmission_local": transmission_status,
"flacfetch_remote": flacfetch_status,
"encoding_worker": encoding_status,
},
"services": {
"email": email_status,
+186 -14
View File
@@ -87,6 +87,11 @@ async def create_job(
user_email = request.user_email
logger.info(f"Admin {auth_result.user_email} creating job on behalf of {user_email}")
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = request.enable_youtube_upload if request.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Create job with all preferences
job_create = JobCreate(
url=str(request.url),
@@ -94,7 +99,7 @@ async def create_job(
title=request.title,
enable_cdg=request.enable_cdg,
enable_txt=request.enable_txt,
enable_youtube_upload=request.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=request.youtube_description,
webhook_url=request.webhook_url,
user_email=user_email
@@ -1082,6 +1087,18 @@ async def get_download_urls(
}
# Map file keys to human-readable suffixes (matching Dropbox naming from karaoke_finalise.py)
DOWNLOAD_FILENAME_SUFFIXES = {
"lossless_4k_mp4": " (Final Karaoke Lossless 4k).mp4",
"lossless_4k_mkv": " (Final Karaoke Lossless 4k).mkv",
"lossy_4k_mp4": " (Final Karaoke Lossy 4k).mp4",
"lossy_720p_mp4": " (Final Karaoke Lossy 720p).mp4",
"cdg_zip": " (Final Karaoke CDG).zip",
"txt_zip": " (Final Karaoke TXT).zip",
"with_vocals": " (With Vocals).mkv",
}
@router.get("/{job_id}/download/{category}/{file_key}")
async def download_file(
job_id: str,
@@ -1146,8 +1163,28 @@ async def download_file(
}
content_type = content_types.get(ext, 'application/octet-stream')
# Get filename for Content-Disposition
filename = gcs_path.split('/')[-1]
# Build proper filename: "Artist - Title (Final Karaoke Lossy 4k).mp4"
# Sanitize artist/title to remove filesystem-unsafe characters
def sanitize_filename_part(s: str) -> str:
if not s:
return ""
# Remove/replace characters unsafe for filesystems: / \ : * ? " < > |
unsafe_chars = r'[/\\:*?"<>|]'
import re
sanitized = re.sub(unsafe_chars, '', s)
# Collapse multiple spaces and trim
sanitized = ' '.join(sanitized.split())
# Limit length to prevent overly long filenames
return sanitized[:100].strip()
artist_clean = sanitize_filename_part(job.artist) if job.artist else None
title_clean = sanitize_filename_part(job.title) if job.title else None
base_name = f"{artist_clean} - {title_clean}" if artist_clean and title_clean else None
if base_name and file_key in DOWNLOAD_FILENAME_SUFFIXES:
filename = f"{base_name}{DOWNLOAD_FILENAME_SUFFIXES[file_key]}"
else:
filename = gcs_path.split('/')[-1] # Fallback to original
try:
# Download to temp file and stream
@@ -1259,12 +1296,13 @@ async def retry_job(
logger.info(f"Job {job_id}: Has rendered video and instrumental selection, retrying video generation")
# Clear error state
# Clear error state and reset worker progress for idempotency
job_manager.update_job(job_id, {
'error_message': None,
'error_details': None,
})
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
# Reset to INSTRUMENTAL_SELECTED and trigger video worker
if not job_manager.transition_to_state(
job_id=job_id,
@@ -1292,13 +1330,14 @@ async def retry_job(
file_urls.get('screens', {}).get('title')):
logger.info(f"Job {job_id}: Has corrections and screens, retrying from render stage")
# Clear error state
# Clear error state and reset worker progress for idempotency
job_manager.update_job(job_id, {
'error_message': None,
'error_details': None,
})
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
# Reset to REVIEW_COMPLETE and trigger render worker
if not job_manager.transition_to_state(
job_id=job_id,
@@ -1326,13 +1365,14 @@ async def retry_job(
file_urls.get('lyrics', {}).get('corrections')):
logger.info(f"Job {job_id}: Has stems and corrections, retrying from screens stage")
# Clear error state
# Clear error state and reset worker progress for idempotency
job_manager.update_job(job_id, {
'error_message': None,
'error_details': None,
})
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
# Reset to a state before screens and trigger screens worker
if not job_manager.transition_to_state(
job_id=job_id,
@@ -1417,10 +1457,14 @@ async def get_worker_logs(
This endpoint returns worker logs stored in Firestore.
Use `since_index` for efficient polling (returns only new logs).
Logs are stored in a subcollection (jobs/{job_id}/logs) to avoid
the 1MB document size limit. Older jobs may have logs in an embedded
array (worker_logs field) - this endpoint handles both transparently.
Args:
job_id: Job ID
since_index: Return only logs after this index (for pagination/polling)
worker: Filter by worker name (audio, lyrics, screens, video, render)
worker: Filter by worker name (audio, lyrics, screens, video, render, distribution)
Returns:
{
@@ -1438,11 +1482,139 @@ async def get_worker_logs(
raise HTTPException(status_code=403, detail="You don't have permission to access logs for this job")
logs = job_manager.get_worker_logs(job_id, since_index=since_index, worker=worker)
total = len(job.worker_logs) if job.worker_logs else 0
total = job_manager.get_worker_logs_count(job_id)
return {
"logs": logs,
"next_index": total,
"next_index": since_index + len(logs),
"total_logs": total
}
@router.post("/{job_id}/cleanup-distribution")
async def cleanup_distribution(
job_id: str,
delete_job: bool = True,
auth_result: AuthResult = Depends(require_admin)
) -> dict:
"""
Clean up all distributed content for a job (YouTube, Dropbox, Google Drive).
This admin-only endpoint is designed for E2E test cleanup. It:
1. Deletes YouTube video (if uploaded)
2. Deletes Dropbox folder (if uploaded)
3. Deletes Google Drive files (if uploaded)
4. Optionally deletes the job itself
Args:
job_id: Job ID to clean up
delete_job: If True, also delete the job after cleaning up distribution
Returns:
Cleanup results for each service
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
state_data = job.state_data or {}
results = {
"job_id": job_id,
"youtube": {"status": "skipped", "reason": "no youtube_url in state_data"},
"dropbox": {"status": "skipped", "reason": "no brand_code or dropbox_path"},
"gdrive": {"status": "skipped", "reason": "no gdrive_files in state_data"},
"job_deleted": False
}
# Clean up YouTube
youtube_url = state_data.get('youtube_url')
if youtube_url:
try:
# Extract video ID from URL (format: https://youtu.be/VIDEO_ID or https://www.youtube.com/watch?v=VIDEO_ID)
import re
video_id_match = re.search(r'(?:youtu\.be/|youtube\.com/watch\?v=)([^&\s]+)', youtube_url)
if video_id_match:
video_id = video_id_match.group(1)
# Import and use karaoke_finalise for YouTube deletion
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
from backend.services.youtube_service import get_youtube_service
youtube_service = get_youtube_service()
if youtube_service.is_configured:
# Create minimal KaraokeFinalise instance for deletion
finalise = KaraokeFinalise(
dry_run=False,
non_interactive=True,
enable_youtube=True,
user_youtube_credentials=youtube_service.get_credentials_dict()
)
success = finalise.delete_youtube_video(video_id)
results["youtube"] = {
"status": "success" if success else "failed",
"video_id": video_id
}
else:
results["youtube"] = {"status": "failed", "reason": "YouTube credentials not configured"}
else:
results["youtube"] = {"status": "failed", "reason": f"Could not extract video ID from {youtube_url}"}
except Exception as e:
logger.error(f"Error cleaning up YouTube for job {job_id}: {e}", exc_info=True)
results["youtube"] = {"status": "error", "error": str(e)}
# Clean up Dropbox
brand_code = state_data.get('brand_code')
dropbox_path = getattr(job, 'dropbox_path', None)
if brand_code and dropbox_path:
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if dropbox.is_configured:
base_name = f"{job.artist} - {job.title}"
folder_name = f"{brand_code} - {base_name}"
full_path = f"{dropbox_path}/{folder_name}"
success = dropbox.delete_folder(full_path)
results["dropbox"] = {
"status": "success" if success else "failed",
"path": full_path
}
else:
results["dropbox"] = {"status": "failed", "reason": "Dropbox credentials not configured"}
except Exception as e:
logger.error(f"Error cleaning up Dropbox for job {job_id}: {e}", exc_info=True)
results["dropbox"] = {"status": "error", "error": str(e)}
# Clean up Google Drive
gdrive_files = state_data.get('gdrive_files')
if gdrive_files:
try:
from backend.services.gdrive_service import get_gdrive_service
gdrive = get_gdrive_service()
if gdrive.is_configured:
# gdrive_files is a dict like {"mp4": "file_id", "mp4_720p": "file_id", "cdg": "file_id"}
file_ids = list(gdrive_files.values()) if isinstance(gdrive_files, dict) else []
delete_results = gdrive.delete_files(file_ids)
all_success = all(delete_results.values())
results["gdrive"] = {
"status": "success" if all_success else "partial",
"files": delete_results
}
else:
results["gdrive"] = {"status": "failed", "reason": "Google Drive credentials not configured"}
except Exception as e:
logger.error(f"Error cleaning up Google Drive for job {job_id}: {e}", exc_info=True)
results["gdrive"] = {"status": "error", "error": str(e)}
# Delete the job if requested
if delete_job:
try:
job_manager.delete_job(job_id, delete_files=True)
results["job_deleted"] = True
logger.info(f"Deleted job {job_id} after distribution cleanup")
except Exception as e:
logger.error(f"Error deleting job {job_id}: {e}", exc_info=True)
results["job_deleted"] = False
results["job_delete_error"] = str(e)
return results
+103 -40
View File
@@ -27,8 +27,10 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_logging import job_log_context, JobLogger
from backend.services.tracing import create_span, add_span_attribute, add_span_event
from backend.services.encoding_service import get_encoding_service
from backend.api.dependencies import require_auth, require_review_auth
from backend.services.auth_service import UserType
from backend.config import get_settings
# LyricsTranscriber imports for preview generation
from lyrics_transcriber.types import CorrectionResult
@@ -385,28 +387,35 @@ async def generate_preview_video(
):
"""
Generate a preview video with the current corrections.
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
video with the user's current corrections applied.
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
the high-performance GCE worker for faster generation (15-20s vs 60+s).
"""
job_manager = JobManager()
storage = StorageService()
settings = get_settings()
encoding_service = get_encoding_service()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Job must be in review state to generate preview
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
logger.info(f"Job {job_id}: Generating preview video")
# Check if GCE preview encoding is enabled
use_gce_preview = encoding_service.is_preview_enabled
logger.info(f"Job {job_id}: Generating preview video (GCE preview: {use_gce_preview})")
# Use tracing and job_log_context for full observability
with create_span("generate-preview-video", {"job_id": job_id}) as span:
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
with job_log_context(job_id, worker="preview"):
try:
# Create temp directory for this preview operation
@@ -416,19 +425,19 @@ async def generate_preview_video(
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
corrections_path = os.path.join(temp_dir, "corrections.json")
storage.download_file(corrections_gcs, corrections_path)
with open(corrections_path, 'r', encoding='utf-8') as f:
original_data = json.load(f)
# 2. Download input audio
audio_path = os.path.join(temp_dir, "audio.flac")
storage.download_file(job.input_media_gcs_path, audio_path)
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
# 3. Load original as CorrectionResult
correction_result = CorrectionResult.from_dict(original_data)
add_span_event("corrections_loaded")
# 4. Get or create styles file for preview using unified style loader
with create_span("load-styles") as styles_span:
styles_path, _ = load_styles_from_gcs(
@@ -439,54 +448,108 @@ async def generate_preview_video(
logger=logger,
)
styles_span.set_attribute("styles_path", styles_path)
# 5. Set up output config for preview
output_dir = os.path.join(temp_dir, "output")
cache_dir = os.path.join(temp_dir, "cache")
os.makedirs(output_dir, exist_ok=True)
os.makedirs(cache_dir, exist_ok=True)
output_config = OutputConfig(
output_styles_json=styles_path,
output_dir=output_dir,
cache_dir=cache_dir,
video_resolution="360p",
)
# 6. Generate preview video using CorrectionOperations (heavy operation)
with create_span("render-preview-video") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger
)
add_span_event("render_complete")
# 7. Upload preview video to GCS for serving
preview_hash = result["preview_hash"]
video_path = result["video_path"]
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# 6. Generate preview (ASS-only if using GCE, or full video if local)
preview_gcs_path = None
if use_gce_preview:
# GCE path: Generate ASS only, then offload encoding to GCE
try:
with create_span("generate-ass-subtitles") as ass_span:
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger,
ass_only=True, # Only generate ASS, skip video encoding
)
preview_hash = result["preview_hash"]
ass_path = result["ass_path"]
ass_span.set_attribute("ass_path", ass_path)
add_span_event("ass_generated")
# Upload ASS to GCS
with create_span("upload-ass-to-gcs") as upload_ass_span:
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
storage.upload_file(ass_path, ass_gcs_path)
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
# Call GCE encoding service
with create_span("gce-preview-encoding") as gce_span:
bucket_name = settings.gcs_bucket_name
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
gce_result = await encoding_service.encode_preview_video(
job_id=f"preview_{job_id}_{preview_hash}",
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
background_color="black",
)
gce_span.set_attribute("gce_status", gce_result.get("status"))
add_span_event("gce_encoding_complete")
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
except Exception as gce_error:
# Fall back to local encoding if GCE fails
logger.warning(
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
)
span.set_attribute("gce_fallback", True)
use_gce_preview = False # Fall through to local encoding below
if not use_gce_preview:
# Local path: Generate full preview video locally
with create_span("render-preview-video-local") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger,
ass_only=False, # Generate full video locally
)
preview_hash = result["preview_hash"]
video_path = result["video_path"]
add_span_event("render_complete")
# Upload preview video to GCS
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# Store the GCS path for serving
if job_id not in _preview_videos:
_preview_videos[job_id] = {}
_preview_videos[job_id][preview_hash] = preview_gcs_path
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
span.set_attribute("preview_hash", preview_hash)
span.set_attribute("success", True)
return {"status": "success", "preview_hash": preview_hash}
except Exception as e:
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
span.set_attribute("error", str(e))
+199 -20
View File
@@ -29,7 +29,7 @@ from backend.models.user import (
BetaFeedbackResponse,
BetaTesterFeedback,
)
from backend.services.user_service import get_user_service, UserService
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
from backend.services.email_service import get_email_service, EmailService
from backend.services.stripe_service import get_stripe_service, StripeService, CREDIT_PACKAGES
from backend.api.dependencies import require_admin
@@ -228,15 +228,34 @@ async def get_current_user(
"""
Get the current user's profile.
Requires a valid session token.
Requires a valid session token or admin token.
"""
from backend.services.auth_service import get_auth_service
if not authorization:
raise HTTPException(status_code=401, detail="Not authenticated")
# Extract token
token = authorization.replace("Bearer ", "") if authorization.startswith("Bearer ") else authorization
# Validate session
# First try auth service (handles admin tokens and auth_tokens)
auth_service = get_auth_service()
auth_result = auth_service.validate_token_full(token)
if auth_result.is_valid and auth_result.user_email:
# Get or create user record for the authenticated email
user = user_service.get_or_create_user(auth_result.user_email)
user_public = UserPublic(
email=user.email,
role=user.role if not auth_result.is_admin else UserRole.ADMIN,
credits=user.credits,
display_name=user.display_name,
total_jobs_created=user.total_jobs_created,
total_jobs_completed=user.total_jobs_completed,
)
return UserProfileResponse(user=user_public, has_session=True)
# Fall back to session validation (for magic link sessions)
valid, user, message = user_service.validate_session(token)
if not valid or not user:
@@ -607,31 +626,191 @@ async def get_feedback_form_data(
# Admin Endpoints
# =============================================================================
@router.get("/admin/users", response_model=UserListResponse)
class UserListResponsePaginated(BaseModel):
"""Paginated response for user list."""
users: list[UserPublic]
total: int
offset: int
limit: int
has_more: bool
class UserDetailResponse(BaseModel):
"""Detailed user information for admin view."""
email: str
role: UserRole
credits: int
display_name: Optional[str] = None
is_active: bool = True
email_verified: bool = False
created_at: Optional[str] = None
updated_at: Optional[str] = None
last_login_at: Optional[str] = None
total_jobs_created: int = 0
total_jobs_completed: int = 0
is_beta_tester: bool = False
beta_tester_status: Optional[str] = None
credit_transactions: list[dict] = []
recent_jobs: list[dict] = []
active_sessions_count: int = 0
@router.get("/admin/users", response_model=UserListResponsePaginated)
async def list_users(
limit: int = 100,
limit: int = 50,
offset: int = 0,
search: Optional[str] = None,
sort_by: str = "created_at",
sort_order: str = "desc",
include_inactive: bool = False,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
"""
List all users (admin only).
List all users with search, pagination, and sorting (admin only).
Args:
limit: Maximum users to return (default 50, max 100)
offset: Number of users to skip for pagination
search: Search by email (case-insensitive prefix match)
sort_by: Field to sort by (created_at, last_login_at, credits, email)
sort_order: Sort direction (asc, desc)
include_inactive: Include disabled users
"""
users = user_service.list_users(limit=limit, include_inactive=include_inactive)
from google.cloud import firestore
from google.cloud.firestore_v1 import FieldFilter
users_public = [
UserPublic(
email=u.email,
role=u.role,
credits=u.credits,
display_name=u.display_name,
total_jobs_created=u.total_jobs_created,
total_jobs_completed=u.total_jobs_completed,
)
for u in users
]
# Validate and cap limit
limit = min(limit, 100)
return UserListResponse(users=users_public, total=len(users_public))
db = user_service.db
query = db.collection(USERS_COLLECTION)
# Filter inactive users
if not include_inactive:
query = query.where(filter=FieldFilter('is_active', '==', True))
# Search by email prefix (case-insensitive via range query)
if search:
search_lower = search.lower()
# Use range query for prefix matching
query = query.where(filter=FieldFilter('email', '>=', search_lower))
query = query.where(filter=FieldFilter('email', '<', search_lower + '\uffff'))
# Sorting
direction = firestore.Query.DESCENDING if sort_order == "desc" else firestore.Query.ASCENDING
if sort_by in ["created_at", "last_login_at", "credits", "email"]:
query = query.order_by(sort_by, direction=direction)
else:
query = query.order_by("created_at", direction=direction)
# Get total count (without pagination) for has_more calculation
# Note: This is expensive for large datasets, consider caching
all_docs = list(query.stream())
total_count = len(all_docs)
# Apply pagination manually (Firestore doesn't support offset well)
paginated_docs = all_docs[offset:offset + limit]
users_public = []
for doc in paginated_docs:
data = doc.to_dict()
users_public.append(UserPublic(
email=data.get("email", ""),
role=data.get("role", UserRole.USER),
credits=data.get("credits", 0),
display_name=data.get("display_name"),
total_jobs_created=data.get("total_jobs_created", 0),
total_jobs_completed=data.get("total_jobs_completed", 0),
))
return UserListResponsePaginated(
users=users_public,
total=total_count,
offset=offset,
limit=limit,
has_more=(offset + limit) < total_count,
)
@router.get("/admin/users/{email}/detail", response_model=UserDetailResponse)
async def get_user_detail(
email: str,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
"""
Get detailed user information including credit history and recent jobs (admin only).
"""
from google.cloud import firestore
from google.cloud.firestore_v1 import FieldFilter
from urllib.parse import unquote
# URL decode the email (handles @ and other special chars)
email = unquote(email).lower()
user = user_service.get_user(email)
if not user:
raise HTTPException(status_code=404, detail="User not found")
db = user_service.db
# Get recent jobs for this user
jobs_query = db.collection("jobs").where(
filter=FieldFilter("user_email", "==", email)
).order_by("created_at", direction=firestore.Query.DESCENDING).limit(10)
recent_jobs = []
for job_doc in jobs_query.stream():
job_data = job_doc.to_dict()
created_at = job_data.get("created_at")
# Handle both datetime objects and ISO strings
if created_at:
created_at_str = created_at.isoformat() if hasattr(created_at, 'isoformat') else str(created_at)
else:
created_at_str = None
recent_jobs.append({
"job_id": job_data.get("job_id"),
"status": job_data.get("status"),
"artist": job_data.get("artist"),
"title": job_data.get("title"),
"created_at": created_at_str,
})
# Count active sessions
sessions_query = db.collection("sessions").where(
filter=FieldFilter("user_email", "==", email)
).where(
filter=FieldFilter("is_active", "==", True)
)
active_sessions_count = sum(1 for _ in sessions_query.stream())
# Format credit transactions
credit_transactions = []
for txn in user.credit_transactions[-20:]: # Last 20 transactions
if hasattr(txn, 'model_dump'):
credit_transactions.append(txn.model_dump(mode='json'))
elif isinstance(txn, dict):
credit_transactions.append(txn)
return UserDetailResponse(
email=user.email,
role=user.role,
credits=user.credits,
display_name=user.display_name,
is_active=user.is_active,
email_verified=user.email_verified,
created_at=user.created_at.isoformat() if user.created_at else None,
updated_at=user.updated_at.isoformat() if user.updated_at else None,
last_login_at=user.last_login_at.isoformat() if user.last_login_at else None,
total_jobs_created=user.total_jobs_created,
total_jobs_completed=user.total_jobs_completed,
is_beta_tester=user.is_beta_tester,
beta_tester_status=user.beta_tester_status,
credit_transactions=credit_transactions,
recent_jobs=recent_jobs,
active_sessions_count=active_sessions_count,
)
@router.post("/admin/credits", response_model=AddCreditsResponse)
@@ -773,7 +952,7 @@ async def get_beta_stats(
from google.cloud.firestore_v1 import aggregation
# Count beta testers by status using efficient aggregation queries
users_collection = user_service.db.collection("users")
users_collection = user_service.db.collection(USERS_COLLECTION)
# Helper function to get count using aggregation
def get_count(query) -> int:
+24 -1
View File
@@ -45,7 +45,10 @@ class Settings(BaseSettings):
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
# Timeout for agentic correction in seconds. If correction takes longer, abort and
# use uncorrected transcription - human review will fix any issues.
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
# Cloud Tasks (for scalable worker coordination)
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
# When disabled (default), workers are triggered via direct HTTP (for development)
@@ -57,9 +60,29 @@ class Settings(BaseSettings):
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
# GCE Encoding Worker (for high-performance video encoding)
# When enabled, video encoding is offloaded to a dedicated C4 GCE instance
# with faster CPU (Intel Granite Rapids 3.9 GHz) instead of Cloud Run.
# This provides 2-3x faster encoding times for CPU-bound FFmpeg libx264 encoding.
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
# GCE Preview Encoding (for faster preview video generation)
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
# Worker logs storage mode
# When enabled, worker logs are stored in a Firestore subcollection (jobs/{job_id}/logs)
# instead of an embedded array. This avoids the 1MB document size limit.
# Default is true for new deployments.
use_log_subcollection: bool = os.getenv("USE_LOG_SUBCOLLECTION", "true").lower() in ("true", "1", "yes")
# Flacfetch remote service (for torrent downloads)
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
+2 -1
View File
@@ -7,7 +7,7 @@ from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.middleware.audit_logging import AuditLoggingMiddleware
@@ -114,6 +114,7 @@ app.include_router(auth.router, prefix="/api") # OAuth credential management
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
app.include_router(themes.router, prefix="/api") # Theme selection for styles
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
@app.get("/")
+2
View File
@@ -204,6 +204,7 @@ class Job(BaseModel):
youtube_description: Optional[str] = None # YouTube video description
webhook_url: Optional[str] = None # Webhook for notifications
user_email: Optional[str] = None # Email for notifications
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
@@ -419,6 +420,7 @@ class JobCreate(BaseModel):
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
+2 -2
View File
@@ -14,7 +14,7 @@ class URLSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
enable_youtube_upload: Optional[bool] = None # None = use server default
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
@@ -28,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
enable_youtube_upload: Optional[bool] = None # None = use server default
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
+164
View File
@@ -0,0 +1,164 @@
"""
Worker log entry model for subcollection storage.
This module defines the LogEntry model for storing worker logs in a Firestore
subcollection instead of an embedded array. This avoids the 1MB document size
limit that caused job 501258e1 to fail when logs reached 1.26 MB.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone, timedelta
from typing import Optional, Dict, Any
import uuid
# Default TTL for log entries (30 days)
DEFAULT_LOG_TTL_DAYS = 30
@dataclass
class WorkerLogEntry:
"""
Worker log entry for Firestore subcollection storage.
Stored at: jobs/{job_id}/logs/{log_id}
This model is separate from the legacy LogEntry (in job.py) which is
stored as an embedded array in the job document. This new model supports
the subcollection approach with TTL and richer metadata.
"""
# Core fields (required)
timestamp: datetime
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
worker: str # "audio", "lyrics", "screens", "video", "render", "distribution"
message: str
# Identifiers
id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = "" # Set when writing to Firestore
# TTL for automatic cleanup
ttl_expiry: datetime = field(
default_factory=lambda: datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
)
# Optional metadata for debugging
metadata: Optional[Dict[str, Any]] = None
@classmethod
def create(
cls,
job_id: str,
worker: str,
level: str,
message: str,
metadata: Optional[Dict[str, Any]] = None,
ttl_days: int = DEFAULT_LOG_TTL_DAYS
) -> "WorkerLogEntry":
"""
Factory method to create a new log entry.
Args:
job_id: Job ID this log belongs to
worker: Worker name (audio, lyrics, screens, video, render, distribution)
level: Log level (DEBUG, INFO, WARNING, ERROR)
message: Log message (truncated to 1000 chars)
metadata: Optional additional metadata
ttl_days: Days until log expires (default 30)
Returns:
New WorkerLogEntry instance
"""
now = datetime.now(timezone.utc)
return cls(
id=str(uuid.uuid4()),
job_id=job_id,
timestamp=now,
level=level.upper(),
worker=worker,
message=message[:1000], # Truncate long messages
metadata=metadata,
ttl_expiry=now + timedelta(days=ttl_days)
)
def to_dict(self) -> Dict[str, Any]:
"""
Convert to dictionary for Firestore storage.
Returns:
Dictionary representation for Firestore
"""
result = {
"id": self.id,
"job_id": self.job_id,
"timestamp": self.timestamp,
"level": self.level,
"worker": self.worker,
"message": self.message,
"ttl_expiry": self.ttl_expiry
}
if self.metadata:
result["metadata"] = self.metadata
return result
def to_legacy_dict(self) -> Dict[str, Any]:
"""
Convert to legacy format for backward compatibility with API responses.
The existing API returns logs in this format:
{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}
Returns:
Dictionary in legacy format
"""
# Format timestamp: use "Z" suffix for UTC, isoformat() for others
if self.timestamp.tzinfo is None:
# Naive datetime - use as-is
timestamp_str = self.timestamp.isoformat()
elif self.timestamp.utcoffset() == timedelta(0):
# UTC datetime - replace +00:00 with Z for cleaner format
timestamp_str = self.timestamp.replace(tzinfo=None).isoformat() + "Z"
else:
# Non-UTC timezone-aware - use full isoformat with offset
timestamp_str = self.timestamp.isoformat()
return {
"timestamp": timestamp_str,
"level": self.level,
"worker": self.worker,
"message": self.message
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "WorkerLogEntry":
"""
Create from Firestore document data.
Args:
data: Dictionary from Firestore document
Returns:
WorkerLogEntry instance
"""
# Handle timestamp conversion (Firestore returns datetime objects)
timestamp = data.get("timestamp")
if isinstance(timestamp, str):
# Parse ISO format string
timestamp = datetime.fromisoformat(timestamp.rstrip("Z")).replace(tzinfo=timezone.utc)
ttl_expiry = data.get("ttl_expiry")
if isinstance(ttl_expiry, str):
ttl_expiry = datetime.fromisoformat(ttl_expiry.rstrip("Z")).replace(tzinfo=timezone.utc)
elif ttl_expiry is None:
# Default TTL if not set
ttl_expiry = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
return cls(
id=data.get("id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
timestamp=timestamp or datetime.now(timezone.utc),
level=data.get("level", "INFO"),
worker=data.get("worker", "unknown"),
message=data.get("message", ""),
metadata=data.get("metadata"),
ttl_expiry=ttl_expiry
)
+37 -16
View File
@@ -276,20 +276,35 @@ class AudioSearchService:
Raises:
DownloadError: If download fails or no cached result for index
"""
if result_index < 0 or result_index >= len(self._cached_results):
raise DownloadError(
f"No cached result for index {result_index}. "
f"Available indices: 0-{len(self._cached_results) - 1}. "
"Run search() first."
)
# Use provided search_id or fall back to service-level cache
effective_search_id = remote_search_id or self._remote_search_id
# Check if we have local cached results
has_local_cache = result_index >= 0 and result_index < len(self._cached_results)
# If no local cache but we have remote search_id, use remote download
# This handles multi-instance deployments where the cache doesn't persist
if not has_local_cache:
if effective_search_id and self._remote_client:
logger.info(f"No local cache, using remote download with search_id={effective_search_id}, index={result_index}")
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
else:
# Provide clear error message based on whether cache is empty or index out of bounds
if len(self._cached_results) == 0:
raise DownloadError(
f"No cached result for index {result_index}. "
"No cached results available. Run search() first."
)
raise DownloadError(
f"No cached result for index {result_index}. "
f"Available indices: 0-{len(self._cached_results) - 1}. "
"Run search() first."
)
result = self._cached_results[result_index]
logger.info(f"Downloading: {result.artist} - {result.title} from {result.provider}")
# Use provided search_id or fall back to service-level cache
effective_search_id = remote_search_id or self._remote_search_id
# Check if this was a remote search (torrent sources need remote download)
if effective_search_id and self._remote_client:
# Check if this is a torrent source that needs remote handling
@@ -481,8 +496,12 @@ class AudioSearchService:
if not effective_search_id:
raise DownloadError("No remote search ID - run search() first")
result = self._cached_results[result_index]
# Try to get local result for metadata, but it's optional
# The remote service maintains its own cache by search_id
result = None
if result_index >= 0 and result_index < len(self._cached_results):
result = self._cached_results[result_index]
try:
# Enable nested event loops (needed when called from FastAPI async context)
try:
@@ -529,13 +548,15 @@ class AudioSearchService:
raise DownloadError("Remote download completed but no file path returned")
logger.info(f"Remote download complete: {filepath}")
# Use local result metadata if available, otherwise use empty strings
# (the actual download is handled by the remote service using search_id)
return AudioDownloadResult(
filepath=filepath,
artist=result.artist,
title=result.title,
provider=result.provider,
quality=result.quality,
artist=result.artist if result else "",
title=result.title if result else "",
provider=result.provider if result else "remote",
quality=result.quality if result else "",
)
except FlacfetchServiceError as e:
+172
View File
@@ -0,0 +1,172 @@
"""
Discord Notification Service.
Provides Discord webhook notification functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- Posting messages to Discord webhooks
- Video upload notifications
- Validation of webhook URLs
"""
import logging
import re
from typing import Optional
import requests
logger = logging.getLogger(__name__)
class DiscordNotificationService:
"""
Service for sending Discord webhook notifications.
Supports posting messages to Discord channels via webhooks,
commonly used to notify when new karaoke videos are uploaded.
"""
# Discord webhook URL pattern
WEBHOOK_URL_PATTERN = re.compile(
r"^https://discord\.com/api/webhooks/\d+/[a-zA-Z0-9_-]+$"
)
def __init__(
self,
webhook_url: Optional[str] = None,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the Discord notification service.
Args:
webhook_url: Discord webhook URL for posting notifications
dry_run: If True, log actions without performing them
logger: Optional logger instance
"""
self.webhook_url = webhook_url
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
# Validate webhook URL if provided
if webhook_url:
self._validate_webhook_url(webhook_url)
def _validate_webhook_url(self, url: str) -> None:
"""
Validate that the webhook URL is a valid Discord webhook URL.
Args:
url: URL to validate
Raises:
ValueError: If URL is not a valid Discord webhook URL
"""
url = url.strip()
if not url.startswith("https://discord.com/api/webhooks/"):
raise ValueError(
f"Invalid Discord webhook URL: {url}. "
"URL must start with 'https://discord.com/api/webhooks/'"
)
def post_message(
self,
message: str,
webhook_url: Optional[str] = None,
) -> bool:
"""
Post a message to a Discord channel via webhook.
Args:
message: The message content to post
webhook_url: Optional webhook URL (uses instance URL if not provided)
Returns:
True if message was posted successfully, False otherwise
Raises:
ValueError: If no webhook URL is available
requests.HTTPError: If the webhook request fails
"""
url = webhook_url or self.webhook_url
if not url:
raise ValueError("No Discord webhook URL provided")
url = url.strip()
self._validate_webhook_url(url)
if self.dry_run:
self.logger.info(
f"DRY RUN: Would post Discord message: '{message}' to {url}"
)
return True
self.logger.info(f"Posting message to Discord webhook...")
data = {"content": message}
response = requests.post(url, json=data, timeout=30)
response.raise_for_status()
self.logger.info("Message posted to Discord successfully")
return True
def post_video_notification(
self,
youtube_url: str,
webhook_url: Optional[str] = None,
) -> bool:
"""
Post a notification about a new video upload.
Args:
youtube_url: The YouTube URL of the uploaded video
webhook_url: Optional webhook URL (uses instance URL if not provided)
Returns:
True if notification was posted successfully, False otherwise
"""
if not youtube_url:
self.logger.info("Skipping Discord notification - no YouTube URL available")
return False
message = f"New upload: {youtube_url}"
return self.post_message(message, webhook_url)
def is_enabled(self) -> bool:
"""
Check if Discord notifications are enabled.
Returns:
True if a valid webhook URL is configured
"""
return bool(self.webhook_url)
# Singleton instance and factory function (following existing service pattern)
_discord_notification_service: Optional[DiscordNotificationService] = None
def get_discord_notification_service(
webhook_url: Optional[str] = None,
**kwargs
) -> DiscordNotificationService:
"""
Get a Discord notification service instance.
Args:
webhook_url: Discord webhook URL for notifications
**kwargs: Additional arguments passed to DiscordNotificationService
Returns:
DiscordNotificationService instance
"""
global _discord_notification_service
# Create new instance if webhook URL changed or not yet created
if _discord_notification_service is None or webhook_url:
_discord_notification_service = DiscordNotificationService(
webhook_url=webhook_url,
**kwargs
)
return _discord_notification_service
+32
View File
@@ -263,6 +263,38 @@ class DropboxService:
return links.links[0].url
raise
def delete_folder(self, path: str) -> bool:
"""
Delete a folder and all its contents from Dropbox.
Args:
path: Dropbox path to delete (e.g., "/Karaoke/Tracks-Organized/NOMAD-1234 - Artist - Title")
Returns:
True if deleted successfully, False otherwise
"""
from dropbox.exceptions import ApiError
# Ensure path starts with /
if not path.startswith("/"):
path = f"/{path}"
logger.info(f"Deleting Dropbox folder: {path}")
try:
self.client.files_delete_v2(path)
logger.info(f"Successfully deleted: {path}")
return True
except ApiError as e:
if e.error.is_path_lookup() and e.error.get_path_lookup().is_not_found():
logger.warning(f"Folder not found (already deleted?): {path}")
return True # Consider it success if already gone
logger.error(f"Failed to delete Dropbox folder: {e}")
return False
except Exception as e:
logger.error(f"Error deleting Dropbox folder: {e}")
return False
def get_dropbox_service() -> DropboxService:
"""Factory function to get a DropboxService instance."""
+451
View File
@@ -0,0 +1,451 @@
"""
Encoding Interface.
Defines the abstract interface for video encoding backends, allowing
the video worker orchestrator to use either GCE or local encoding
interchangeably.
This follows the Strategy pattern - different encoding implementations
can be swapped without changing the orchestration logic.
"""
import logging
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Any
logger = logging.getLogger(__name__)
@dataclass
class EncodingInput:
"""
Input configuration for video encoding.
Contains all the paths and metadata needed to encode a karaoke video.
"""
# Required input files
title_video_path: str # Title card video (MOV)
karaoke_video_path: str # Main karaoke video with vocals (MOV/MKV)
instrumental_audio_path: str # Instrumental audio track (FLAC)
# Optional input files
end_video_path: Optional[str] = None # End credits video (MOV)
# Metadata for output naming
artist: str = ""
title: str = ""
brand_code: Optional[str] = None
# Output directory
output_dir: str = ""
# Additional options
options: Dict[str, Any] = field(default_factory=dict)
@dataclass
class EncodingOutput:
"""
Output from video encoding.
Contains paths to all generated video files and status information.
"""
success: bool
error_message: Optional[str] = None
# Output file paths (relative to output_dir or absolute)
karaoke_mp4_path: Optional[str] = None # Karaoke video with instrumental audio
with_vocals_mp4_path: Optional[str] = None # With vocals MP4 version
lossless_4k_mp4_path: Optional[str] = None # Final lossless 4K MP4
lossy_4k_mp4_path: Optional[str] = None # Final lossy 4K MP4
lossless_mkv_path: Optional[str] = None # MKV with FLAC (for YouTube)
lossy_720p_mp4_path: Optional[str] = None # 720p web version
# All output files as a dict for convenience
output_files: Dict[str, str] = field(default_factory=dict)
# Encoding metadata
encoding_time_seconds: Optional[float] = None
encoding_backend: Optional[str] = None # "gce" or "local"
class EncodingBackend(ABC):
"""
Abstract base class for video encoding backends.
Implementations:
- GCEEncodingBackend: Cloud-based encoding using Google Compute Engine
- LocalEncodingBackend: Local FFmpeg-based encoding
"""
@property
@abstractmethod
def name(self) -> str:
"""Return the name of this encoding backend."""
pass
@abstractmethod
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video files according to the input configuration.
This method should:
1. Create karaoke video with instrumental audio
2. Convert with-vocals video to MP4 if needed
3. Encode lossless 4K MP4 (concatenating title + karaoke + end)
4. Encode lossy 4K MP4 with AAC audio
5. Create MKV with FLAC audio for YouTube
6. Encode 720p version for web
Args:
input_config: EncodingInput with all required paths and options
Returns:
EncodingOutput with paths to generated files and status
"""
pass
@abstractmethod
async def is_available(self) -> bool:
"""
Check if this encoding backend is available and configured.
Returns:
True if the backend can be used, False otherwise
"""
pass
@abstractmethod
async def get_status(self) -> Dict[str, Any]:
"""
Get the current status of the encoding backend.
Returns:
Dict with status information (availability, queue length, etc.)
"""
pass
class LocalEncodingBackend(EncodingBackend):
"""
Local FFmpeg-based encoding backend.
Wraps the LocalEncodingService to implement the EncodingBackend interface.
Uses asyncio.to_thread() to run synchronous FFmpeg operations asynchronously.
"""
def __init__(
self,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the local encoding backend.
Args:
dry_run: If True, log operations without executing
logger: Optional logger instance
"""
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
self._service = None
@property
def name(self) -> str:
return "local"
def _get_service(self):
"""Lazy-load the local encoding service."""
if self._service is None:
from backend.services.local_encoding_service import LocalEncodingService
self._service = LocalEncodingService(
dry_run=self.dry_run,
logger=self.logger
)
return self._service
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video using local FFmpeg.
Args:
input_config: Encoding input configuration
Returns:
EncodingOutput with results
"""
import asyncio
import time
from backend.services.local_encoding_service import EncodingConfig
start_time = time.time()
# Build output file paths
base_name = f"{input_config.artist} - {input_config.title}"
output_dir = input_config.output_dir or "."
import os
config = EncodingConfig(
title_video=input_config.title_video_path,
karaoke_video=input_config.karaoke_video_path,
instrumental_audio=input_config.instrumental_audio_path,
end_video=input_config.end_video_path,
output_karaoke_mp4=os.path.join(output_dir, f"{base_name} (Karaoke).mp4"),
output_with_vocals_mp4=os.path.join(output_dir, f"{base_name} (With Vocals).mp4"),
output_lossless_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mp4"),
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
)
# Run encoding in thread pool to avoid blocking
service = self._get_service()
result = await asyncio.to_thread(service.encode_all_formats, config)
encoding_time = time.time() - start_time
if result.success:
return EncodingOutput(
success=True,
karaoke_mp4_path=config.output_karaoke_mp4,
with_vocals_mp4_path=config.output_with_vocals_mp4,
lossless_4k_mp4_path=config.output_lossless_4k_mp4,
lossy_4k_mp4_path=config.output_lossy_4k_mp4,
lossless_mkv_path=config.output_lossless_mkv,
lossy_720p_mp4_path=config.output_720p_mp4,
output_files=result.output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
else:
return EncodingOutput(
success=False,
error_message=result.error,
output_files=result.output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
async def is_available(self) -> bool:
"""Check if local encoding is available (FFmpeg installed)."""
import subprocess
import asyncio
try:
result = await asyncio.to_thread(
subprocess.run,
["ffmpeg", "-version"],
capture_output=True,
timeout=10
)
return result.returncode == 0
except (FileNotFoundError, subprocess.TimeoutExpired):
return False
async def get_status(self) -> Dict[str, Any]:
"""Get local encoding status."""
available = await self.is_available()
service = self._get_service()
return {
"backend": self.name,
"available": available,
"hwaccel_available": service.hwaccel_available if available else False,
"video_encoder": service.video_encoder if available else None,
}
class GCEEncodingBackend(EncodingBackend):
"""
GCE-based encoding backend.
Wraps the existing EncodingService to implement the EncodingBackend interface.
Submits jobs to a remote GCE worker for high-performance encoding.
"""
def __init__(
self,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the GCE encoding backend.
Args:
dry_run: Ignored for GCE backend (remote service doesn't support dry run)
logger: Optional logger instance
"""
self.dry_run = dry_run # Stored but not used (GCE is remote)
self.logger = logger or logging.getLogger(__name__)
self._service = None
@property
def name(self) -> str:
return "gce"
def _get_service(self):
"""Lazy-load the GCE encoding service."""
if self._service is None:
from backend.services.encoding_service import get_encoding_service
self._service = get_encoding_service()
return self._service
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
"""
Encode video using GCE worker.
Note: GCE encoding requires files to be in GCS, so this method
expects input_config.options to contain 'input_gcs_path' and
'output_gcs_path'.
Args:
input_config: Encoding input configuration
Returns:
EncodingOutput with results
"""
import time
start_time = time.time()
service = self._get_service()
# GCE requires GCS paths
input_gcs_path = input_config.options.get("input_gcs_path")
output_gcs_path = input_config.options.get("output_gcs_path")
job_id = input_config.options.get("job_id", input_config.brand_code or "unknown")
if not input_gcs_path or not output_gcs_path:
return EncodingOutput(
success=False,
error_message="GCE encoding requires input_gcs_path and output_gcs_path in options",
encoding_backend=self.name
)
try:
# Build encoding config
encoding_config = {
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
"artist": input_config.artist,
"title": input_config.title,
}
# Submit and wait for completion
result = await service.encode_videos(
job_id=job_id,
input_gcs_path=input_gcs_path,
output_gcs_path=output_gcs_path,
encoding_config=encoding_config,
)
encoding_time = time.time() - start_time
# Extract output file paths from result
# Handle case where GCE worker returns a list or unexpected format
if isinstance(result, list):
# If result is a list, try to find the output_files in the first dict
self.logger.warning(f"GCE returned list instead of dict: {result}")
result = result[0] if result and isinstance(result[0], dict) else {}
if not isinstance(result, dict):
self.logger.error(f"Unexpected GCE result type: {type(result)}")
result = {}
raw_output_files = result.get("output_files", {})
# Convert output_files from list of paths to dict if needed
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
if isinstance(raw_output_files, list):
output_files = {}
for path in raw_output_files:
if not isinstance(path, str):
continue
filename = path.split("/")[-1] if "/" in path else path
# Map filename patterns to output format keys
if "4k_lossless" in filename:
output_files["mp4_4k_lossless"] = path
elif "4k_lossy" in filename:
output_files["mp4_4k_lossy"] = path
elif "720p" in filename:
output_files["mp4_720p"] = path
elif filename.endswith(".mkv"):
output_files["mkv_4k"] = path
self.logger.info(f"Converted output_files list to dict: {output_files}")
else:
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
return EncodingOutput(
success=True,
lossless_4k_mp4_path=output_files.get("mp4_4k_lossless"),
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
lossless_mkv_path=output_files.get("mkv_4k"),
lossy_720p_mp4_path=output_files.get("mp4_720p"),
output_files=output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
except Exception as e:
encoding_time = time.time() - start_time
self.logger.error(f"GCE encoding failed: {e}")
return EncodingOutput(
success=False,
error_message=str(e),
encoding_time_seconds=encoding_time,
encoding_backend=self.name
)
async def is_available(self) -> bool:
"""Check if GCE encoding is available and configured."""
service = self._get_service()
return service.is_enabled
async def get_status(self) -> Dict[str, Any]:
"""Get GCE encoding status."""
service = self._get_service()
status = {
"backend": self.name,
"available": service.is_enabled,
"configured": service.is_configured,
}
if service.is_configured:
try:
health = await service.health_check()
status["health"] = health
except Exception as e:
status["health_error"] = str(e)
return status
# Factory function to get an encoding backend
def get_encoding_backend(
backend_type: str = "auto",
**kwargs
) -> EncodingBackend:
"""
Get an encoding backend instance.
Args:
backend_type: Type of backend - "local", "gce", or "auto"
**kwargs: Additional arguments for the backend
Returns:
EncodingBackend instance
Raises:
ValueError: If backend_type is unknown
"""
if backend_type == "local":
return LocalEncodingBackend(**kwargs)
elif backend_type == "gce":
return GCEEncodingBackend(**kwargs)
elif backend_type == "auto":
# Check if GCE is available, otherwise use local
gce_backend = GCEEncodingBackend(**kwargs)
if gce_backend._get_service().is_enabled:
return gce_backend
return LocalEncodingBackend(**kwargs)
else:
raise ValueError(f"Unknown encoding backend type: {backend_type}")
+388
View File
@@ -0,0 +1,388 @@
"""
GCE Encoding Worker Service.
This service dispatches video encoding jobs to a dedicated high-performance
GCE instance (C4-standard with Intel Granite Rapids CPU) for faster encoding.
The GCE worker provides:
- 3.9 GHz all-core frequency (vs 3.7 GHz on Cloud Run)
- Dedicated vCPUs (no contention)
- 2-3x faster FFmpeg libx264 encoding
Usage:
encoding_service = get_encoding_service()
if encoding_service.is_configured:
result = await encoding_service.encode_videos(job_id, input_gcs_path, config)
"""
import asyncio
import logging
from typing import Optional, Dict, Any
import aiohttp
from backend.config import get_settings
logger = logging.getLogger(__name__)
class EncodingService:
"""Service for dispatching encoding jobs to GCE worker."""
def __init__(self):
self.settings = get_settings()
self._url = None
self._api_key = None
self._initialized = False
def _load_credentials(self):
"""Load encoding worker URL and API key from config/secrets."""
if self._initialized:
return
# Try environment variables first, then Secret Manager
self._url = self.settings.encoding_worker_url
self._api_key = self.settings.encoding_worker_api_key
# Fall back to Secret Manager
if not self._url:
self._url = self.settings.get_secret("encoding-worker-url")
if not self._api_key:
self._api_key = self.settings.get_secret("encoding-worker-api-key")
self._initialized = True
@property
def is_configured(self) -> bool:
"""Check if encoding service is configured with URL and API key."""
self._load_credentials()
return bool(self._url and self._api_key)
@property
def is_enabled(self) -> bool:
"""Check if GCE encoding is enabled and configured."""
return self.settings.use_gce_encoding and self.is_configured
@property
def is_preview_enabled(self) -> bool:
"""Check if GCE preview encoding is enabled and configured."""
return self.settings.use_gce_preview_encoding and self.is_configured
async def submit_encoding_job(
self,
job_id: str,
input_gcs_path: str,
output_gcs_path: str,
encoding_config: Dict[str, Any],
) -> Dict[str, Any]:
"""
Submit an encoding job to the GCE worker.
Args:
job_id: Unique job identifier
input_gcs_path: GCS path to input files (gs://bucket/path/)
output_gcs_path: GCS path for output files (gs://bucket/path/)
encoding_config: Configuration for encoding (formats, quality, etc.)
Returns:
Response from the encoding worker
Raises:
Exception: If submission fails
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._url}/encode"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
"input_gcs_path": input_gcs_path,
"output_gcs_path": output_gcs_path,
"encoding_config": encoding_config,
}
logger.info(f"[job:{job_id}] Submitting encoding job to GCE worker: {url}")
async with aiohttp.ClientSession() as session:
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
if resp.status == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp.status == 409:
raise RuntimeError(f"Encoding job {job_id} already exists")
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"Failed to submit encoding job: {resp.status} - {text}")
return await resp.json()
async def get_job_status(self, job_id: str) -> Dict[str, Any]:
"""
Get the status of an encoding job.
Args:
job_id: Job identifier
Returns:
Job status including: status, progress, error, output_files
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._url}/status/{job_id}"
headers = {"X-API-Key": self._api_key}
async with aiohttp.ClientSession() as session:
async with session.get(url, headers=headers, timeout=30) as resp:
if resp.status == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp.status == 404:
raise RuntimeError(f"Encoding job {job_id} not found")
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"Failed to get job status: {resp.status} - {text}")
return await resp.json()
async def wait_for_completion(
self,
job_id: str,
poll_interval: float = 10.0,
timeout: float = 3600.0,
progress_callback=None,
) -> Dict[str, Any]:
"""
Poll for encoding job completion.
Args:
job_id: Job identifier
poll_interval: Seconds between status checks
timeout: Maximum time to wait (default 1 hour)
progress_callback: Optional callback(progress: int) for progress updates
Returns:
Final job status with output files
Raises:
TimeoutError: If job doesn't complete within timeout
RuntimeError: If job fails
"""
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
start_time = asyncio.get_event_loop().time()
last_progress = 0
while True:
elapsed = asyncio.get_event_loop().time() - start_time
if elapsed > timeout:
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
status = await self.get_job_status(job_id)
# Handle case where GCE worker returns a list instead of dict
if isinstance(status, list):
logger.warning(f"[job:{job_id}] GCE returned list instead of dict: {status}")
status = status[0] if status and isinstance(status[0], dict) else {}
if not isinstance(status, dict):
logger.error(f"[job:{job_id}] Unexpected status type: {type(status)}")
status = {}
job_status = status.get("status", "unknown")
progress = status.get("progress", 0)
# Report progress
if progress != last_progress:
logger.info(f"[job:{job_id}] Encoding progress: {progress}%")
last_progress = progress
if progress_callback:
try:
progress_callback(progress)
except Exception as e:
logger.warning(f"Progress callback failed: {e}")
if job_status == "complete":
logger.info(f"[job:{job_id}] GCE encoding complete in {elapsed:.1f}s")
return status
if job_status == "failed":
error = status.get("error", "Unknown error")
raise RuntimeError(f"Encoding job {job_id} failed: {error}")
await asyncio.sleep(poll_interval)
async def encode_videos(
self,
job_id: str,
input_gcs_path: str,
output_gcs_path: str,
encoding_config: Optional[Dict[str, Any]] = None,
progress_callback=None,
) -> Dict[str, Any]:
"""
Submit encoding job and wait for completion.
This is a convenience method that combines submit + wait.
Args:
job_id: Unique job identifier
input_gcs_path: GCS path to input files
output_gcs_path: GCS path for output files
encoding_config: Optional encoding configuration
progress_callback: Optional callback for progress updates
Returns:
Final job status with output files
"""
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
# Submit the job
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
# Wait for completion
return await self.wait_for_completion(
job_id, progress_callback=progress_callback
)
async def submit_preview_encoding_job(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
) -> Dict[str, Any]:
"""
Submit a preview video encoding job to the GCE worker.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
Returns:
Response from the encoding worker
Raises:
Exception: If submission fails
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._url}/encode-preview"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
"ass_gcs_path": ass_gcs_path,
"audio_gcs_path": audio_gcs_path,
"output_gcs_path": output_gcs_path,
"background_color": background_color,
}
if background_image_gcs_path:
payload["background_image_gcs_path"] = background_image_gcs_path
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
async with aiohttp.ClientSession() as session:
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
if resp.status == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp.status == 409:
raise RuntimeError(f"Preview encoding job {job_id} already exists")
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"Failed to submit preview encoding job: {resp.status} - {text}")
return await resp.json()
async def encode_preview_video(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
timeout: float = 90.0,
poll_interval: float = 2.0,
) -> Dict[str, Any]:
"""
Submit preview encoding job and wait for completion.
This is a convenience method that combines submit + wait with shorter
timeouts suitable for preview videos.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
timeout: Maximum time to wait (default 90s for preview)
poll_interval: Seconds between status checks (default 2s)
Returns:
Final job status with output files
"""
# Submit the job
await self.submit_preview_encoding_job(
job_id=job_id,
ass_gcs_path=ass_gcs_path,
audio_gcs_path=audio_gcs_path,
output_gcs_path=output_gcs_path,
background_color=background_color,
background_image_gcs_path=background_image_gcs_path,
)
# Wait for completion with shorter timeout
return await self.wait_for_completion(
job_id=job_id,
poll_interval=poll_interval,
timeout=timeout,
)
async def health_check(self) -> Dict[str, Any]:
"""
Check the health of the encoding worker.
Returns:
Health status including active jobs and FFmpeg version
"""
self._load_credentials()
if not self.is_configured:
return {"status": "not_configured"}
url = f"{self._url}/health"
headers = {"X-API-Key": self._api_key}
try:
async with aiohttp.ClientSession() as session:
async with session.get(url, headers=headers, timeout=10) as resp:
if resp.status == 200:
return await resp.json()
return {"status": "error", "code": resp.status}
except Exception as e:
return {"status": "error", "error": str(e)}
# Singleton instance
_encoding_service: Optional[EncodingService] = None
def get_encoding_service() -> EncodingService:
"""Get the singleton encoding service instance."""
global _encoding_service
if _encoding_service is None:
_encoding_service = EncodingService()
return _encoding_service
+165 -2
View File
@@ -9,6 +9,7 @@ from google.cloud.firestore_v1 import FieldFilter
from backend.config import settings
from backend.models.job import Job, JobStatus, TimelineEvent
from backend.models.worker_log import WorkerLogEntry
logger = logging.getLogger(__name__)
@@ -245,10 +246,13 @@ class FirestoreService:
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
"""
Atomically append a log entry to worker_logs using ArrayUnion.
This avoids the race condition of read-modify-write when multiple
workers are logging concurrently.
DEPRECATED: Use append_log_to_subcollection() instead to avoid
the 1MB document size limit.
Args:
job_id: Job ID
log_entry: Log entry dict with timestamp, level, worker, message
@@ -263,6 +267,165 @@ class FirestoreService:
except Exception as e:
# Log but don't raise - logging shouldn't break workers
logger.debug(f"Error appending worker log for job {job_id}: {e}")
# ============================================
# Worker Log Subcollection Methods
# ============================================
# These methods store logs in a subcollection (jobs/{job_id}/logs)
# instead of an embedded array to avoid the 1MB document size limit.
def append_log_to_subcollection(self, job_id: str, log_entry: WorkerLogEntry) -> None:
"""
Append a log entry to the logs subcollection.
Stores logs at: jobs/{job_id}/logs/{log_id}
This approach avoids the 1MB document size limit by storing each
log entry as a separate document in a subcollection.
Args:
job_id: Job ID
log_entry: WorkerLogEntry instance
"""
try:
# Ensure job_id is set on the log entry
log_entry.job_id = job_id
# Get subcollection reference
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
# Add document with auto-generated ID or use log_entry.id
doc_ref = logs_ref.document(log_entry.id)
doc_ref.set(log_entry.to_dict())
# Don't log every append - too spammy
except Exception as e:
# Log but don't raise - logging shouldn't break workers
logger.debug(f"Error appending log to subcollection for job {job_id}: {e}")
def get_logs_from_subcollection(
self,
job_id: str,
limit: int = 500,
since_timestamp: Optional[datetime] = None,
worker: Optional[str] = None,
offset: int = 0
) -> List[WorkerLogEntry]:
"""
Get log entries from the logs subcollection.
Args:
job_id: Job ID
limit: Maximum number of logs to return (default 500)
since_timestamp: Return only logs after this timestamp
worker: Filter by worker name (optional)
offset: Number of logs to skip (for pagination)
Returns:
List of WorkerLogEntry instances, ordered by timestamp ascending
"""
try:
# Get subcollection reference
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
# Build query
query = logs_ref.order_by("timestamp", direction=firestore.Query.ASCENDING)
if since_timestamp:
query = query.where(filter=FieldFilter("timestamp", ">", since_timestamp))
if worker:
query = query.where(filter=FieldFilter("worker", "==", worker))
# Apply offset and limit
if offset > 0:
query = query.offset(offset)
query = query.limit(limit)
# Execute query
docs = query.stream()
logs = [WorkerLogEntry.from_dict(doc.to_dict()) for doc in docs]
return logs
except Exception as e:
logger.error(f"Error getting logs from subcollection for job {job_id}: {e}")
return []
def get_logs_count_from_subcollection(self, job_id: str) -> int:
"""
Get the total count of log entries in the subcollection.
Args:
job_id: Job ID
Returns:
Total count of log entries
"""
try:
# Get subcollection reference
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
# Use aggregation query for efficient counting
count_query = logs_ref.count()
result = count_query.get()
# Result is a list of AggregationResult, we want the first one's count
if result and len(result) > 0:
return result[0][0].value
return 0
except Exception as e:
logger.error(f"Error counting logs for job {job_id}: {e}")
return 0
def delete_logs_subcollection(self, job_id: str, batch_size: int = 500) -> int:
"""
Delete all log entries in the logs subcollection.
This is used when deleting a job to clean up its logs.
Args:
job_id: Job ID
batch_size: Number of documents to delete per batch
Returns:
Number of logs deleted
"""
try:
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
deleted_count = 0
while True:
# Get a batch of documents
docs = logs_ref.limit(batch_size).stream()
deleted_in_batch = 0
# Delete in a batch
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_in_batch += 1
if deleted_in_batch == 0:
break
batch.commit()
deleted_count += deleted_in_batch
# If we deleted less than batch_size, we're done
if deleted_in_batch < batch_size:
break
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} logs for job {job_id}")
return deleted_count
except Exception as e:
logger.error(f"Error deleting logs subcollection for job {job_id}: {e}")
return 0
# ============================================
# Token Management Methods
+103 -3
View File
@@ -408,11 +408,11 @@ class FlacfetchClient:
) -> Dict[str, Any]:
"""
Trigger disk cleanup.
Args:
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
target_free_gb: Target free space to achieve
Returns:
Cleanup response with removed count and freed space
"""
@@ -429,12 +429,112 @@ class FlacfetchClient:
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
# =========================================================================
# Cache Management
# =========================================================================
async def clear_search_cache(self, artist: str, title: str) -> bool:
"""
Clear a specific cached search result by artist and title.
Args:
artist: Artist name
title: Track title
Returns:
True if a cache entry was deleted, False if no entry existed
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.request(
"DELETE",
f"{self.base_url}/cache/search",
headers=self._headers(),
json={"artist": artist, "title": title},
timeout=30,
)
resp.raise_for_status()
data = resp.json()
deleted = data.get("deleted", False)
logger.info(
f"Cleared flacfetch cache for '{artist}' - '{title}': "
f"{'deleted' if deleted else 'no entry found'}"
)
return deleted
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
)
async def clear_all_cache(self) -> int:
"""
Clear all cached search results.
Returns:
Number of cache entries deleted
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.delete(
f"{self.base_url}/cache",
headers=self._headers(),
timeout=60,
)
resp.raise_for_status()
data = resp.json()
deleted_count = data.get("deleted_count", 0)
logger.info(f"Cleared all flacfetch cache: {deleted_count} entries deleted")
return deleted_count
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
)
async def get_cache_stats(self) -> Dict[str, Any]:
"""
Get statistics about the cache.
Returns:
Dict with count, total_size_bytes, oldest_entry, newest_entry, configured
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/cache/stats",
headers=self._headers(),
timeout=30,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
)
# Singleton client instance
_client: Optional[FlacfetchClient] = None
+39
View File
@@ -300,6 +300,45 @@ class GoogleDriveService:
logger.info(f"Public share upload complete: {len(uploaded_files)} files uploaded")
return uploaded_files
def delete_file(self, file_id: str) -> bool:
"""
Delete a file from Google Drive.
Args:
file_id: Google Drive file ID to delete
Returns:
True if deleted successfully, False otherwise
"""
logger.info(f"Deleting Google Drive file: {file_id}")
try:
self.service.files().delete(fileId=file_id).execute()
logger.info(f"Successfully deleted file: {file_id}")
return True
except Exception as e:
# Check if it's a 404 (already deleted)
if hasattr(e, 'resp') and e.resp.status == 404:
logger.warning(f"File not found (already deleted?): {file_id}")
return True
logger.error(f"Failed to delete Google Drive file: {e}")
return False
def delete_files(self, file_ids: list[str]) -> dict[str, bool]:
"""
Delete multiple files from Google Drive.
Args:
file_ids: List of Google Drive file IDs to delete
Returns:
Dictionary mapping file_id to success status
"""
results = {}
for file_id in file_ids:
results[file_id] = self.delete_file(file_id)
return results
# Singleton instance
_gdrive_service: Optional[GoogleDriveService] = None
+90 -28
View File
@@ -13,7 +13,9 @@ import uuid
from datetime import datetime
from typing import Optional, Dict, Any, List
from backend.config import settings
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
from backend.models.worker_log import WorkerLogEntry
from backend.services.firestore_service import FirestoreService
from backend.services.storage_service import StorageService
@@ -224,7 +226,7 @@ class JobManager:
return urls
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
"""Delete a job and optionally its files."""
"""Delete a job, its files, and its logs subcollection."""
if delete_files:
job = self.get_job(job_id)
if job and job.output_files:
@@ -233,7 +235,15 @@ class JobManager:
self.storage.delete_file(gcs_path)
except Exception as e:
logger.error(f"Error deleting file {gcs_path}: {e}")
# Delete logs subcollection first (must be done before deleting parent doc)
try:
deleted_logs = self.firestore.delete_logs_subcollection(job_id)
if deleted_logs > 0:
logger.info(f"Deleted {deleted_logs} log entries for job {job_id}")
except Exception as e:
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
self.firestore.delete_job(job_id)
logger.info(f"Deleted job {job_id}")
@@ -579,33 +589,45 @@ class JobManager:
worker: str,
level: str,
message: str,
max_logs: int = 500
max_logs: int = 500,
metadata: Optional[Dict[str, Any]] = None
) -> None:
"""
Append a log entry to the job's worker_logs atomically.
Uses Firestore ArrayUnion for atomic append to avoid race conditions
when multiple workers (audio + lyrics) are logging concurrently.
Note: max_logs is not enforced per-append (would require transactions).
Logs may grow beyond max_logs; trim periodically if needed.
Append a log entry to the job's logs.
By default (USE_LOG_SUBCOLLECTION=true), logs are stored in a Firestore
subcollection (jobs/{job_id}/logs) to avoid the 1MB document size limit.
For backward compatibility (USE_LOG_SUBCOLLECTION=false), logs are stored
in the embedded worker_logs array using atomic ArrayUnion.
Args:
job_id: Job ID
worker: Worker name (audio, lyrics, screens, video, render)
worker: Worker name (audio, lyrics, screens, video, render, distribution)
level: Log level (DEBUG, INFO, WARNING, ERROR)
message: Log message
max_logs: Not used (kept for API compatibility)
metadata: Optional additional metadata dict
"""
log_entry = {
'timestamp': datetime.utcnow().isoformat() + 'Z',
'level': level,
'worker': worker,
'message': message[:1000] # Truncate long messages
}
# Use atomic ArrayUnion to avoid race conditions
self.firestore.append_worker_log(job_id, log_entry)
if settings.use_log_subcollection:
# New subcollection approach - avoids 1MB limit
log_entry = WorkerLogEntry.create(
job_id=job_id,
worker=worker,
level=level,
message=message,
metadata=metadata
)
self.firestore.append_log_to_subcollection(job_id, log_entry)
else:
# Legacy embedded array approach
log_entry = {
'timestamp': datetime.utcnow().isoformat() + 'Z',
'level': level,
'worker': worker,
'message': message[:1000] # Truncate long messages
}
self.firestore.append_worker_log(job_id, log_entry)
def get_worker_logs(
self,
@@ -615,28 +637,68 @@ class JobManager:
) -> List[Dict[str, Any]]:
"""
Get worker logs for a job, optionally filtered by worker and index.
By default (USE_LOG_SUBCOLLECTION=true), logs are read from the
subcollection. Falls back to embedded array for older jobs.
Args:
job_id: Job ID
since_index: Return only logs after this index (for pagination)
worker: Filter by worker name (optional)
Returns:
List of log entries as dicts
List of log entries as dicts (in legacy format for API compatibility)
"""
if settings.use_log_subcollection:
# Try subcollection first
subcollection_logs = self.firestore.get_logs_from_subcollection(
job_id=job_id,
offset=since_index,
worker=worker,
limit=500
)
if subcollection_logs:
# Convert to legacy format for API compatibility
return [log.to_legacy_dict() for log in subcollection_logs]
# Fall through to check embedded array for older jobs
# Embedded array approach (legacy jobs or fallback)
job = self.get_job(job_id)
if not job or not job.worker_logs:
return []
logs = [log.dict() if hasattr(log, 'dict') else log for log in job.worker_logs]
# Filter by index
if since_index > 0:
logs = logs[since_index:]
# Filter by worker
if worker:
logs = [log for log in logs if log.get('worker') == worker]
return logs
def get_worker_logs_count(self, job_id: str) -> int:
"""
Get the total count of worker logs for a job.
Args:
job_id: Job ID
Returns:
Total count of logs
"""
if settings.use_log_subcollection:
# Try subcollection first
count = self.firestore.get_logs_count_from_subcollection(job_id)
if count > 0:
return count
# Fall through to check embedded array
# Embedded array (legacy jobs or fallback)
job = self.get_job(job_id)
if not job or not job.worker_logs:
return 0
return len(job.worker_logs)
+590
View File
@@ -0,0 +1,590 @@
"""
Local Encoding Service.
Provides local FFmpeg-based video encoding functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker as a fallback) and local CLI.
This service handles:
- Concatenating title, karaoke, and end videos
- Encoding to multiple output formats (4K lossless, 4K lossy, 720p, MKV)
- Hardware acceleration detection and fallback to software encoding
- Remuxing video with instrumental audio
"""
import logging
import os
import shlex
import subprocess
import sys
from dataclasses import dataclass
from typing import Optional, List, Dict, Any, Tuple
logger = logging.getLogger(__name__)
@dataclass
class EncodingConfig:
"""Configuration for video encoding."""
title_video: str # Path to title video
karaoke_video: str # Path to karaoke video (with vocals)
instrumental_audio: str # Path to instrumental audio
end_video: Optional[str] = None # Optional path to end credits video
# Output paths
output_karaoke_mp4: Optional[str] = None
output_with_vocals_mp4: Optional[str] = None
output_lossless_4k_mp4: Optional[str] = None
output_lossy_4k_mp4: Optional[str] = None
output_lossless_mkv: Optional[str] = None
output_720p_mp4: Optional[str] = None
@dataclass
class EncodingResult:
"""Result of video encoding operation."""
success: bool
output_files: Dict[str, str]
error: Optional[str] = None
class LocalEncodingService:
"""
Service for local FFmpeg-based video encoding.
Supports hardware acceleration (NVENC) with automatic fallback
to software encoding (libx264) when unavailable.
"""
# MP4 flags for better compatibility
MP4_FLAGS = "-movflags +faststart"
def __init__(
self,
dry_run: bool = False,
log_level: int = logging.INFO,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the local encoding service.
Args:
dry_run: If True, log commands without executing them
log_level: Logging level (affects FFmpeg verbosity)
logger: Optional logger instance
"""
self.dry_run = dry_run
self.log_level = log_level
self.logger = logger or logging.getLogger(__name__)
# Hardware acceleration settings (detected on first use)
self._hwaccel_available: Optional[bool] = None
self._video_encoder: Optional[str] = None
self._scale_filter: Optional[str] = None
self._hwaccel_decode_flags: Optional[str] = None
self._aac_codec: Optional[str] = None
# Build FFmpeg base command
self._ffmpeg_base_command = self._build_ffmpeg_base_command()
def _build_ffmpeg_base_command(self) -> str:
"""Build the FFmpeg base command with appropriate flags."""
# Use bundled FFmpeg for frozen builds
ffmpeg_path = os.path.join(sys._MEIPASS, "ffmpeg.exe") if getattr(sys, "frozen", False) else "ffmpeg"
base_cmd = f"{ffmpeg_path} -hide_banner -nostats -y"
if self.log_level == logging.DEBUG:
base_cmd += " -loglevel verbose"
else:
base_cmd += " -loglevel fatal"
return base_cmd
def _detect_hardware_acceleration(self) -> Tuple[bool, str, str, str, str]:
"""
Detect available hardware acceleration.
Returns:
Tuple of (hwaccel_available, video_encoder, scale_filter,
hwaccel_decode_flags, aac_codec)
"""
self.logger.info("Detecting hardware acceleration capabilities...")
# Try NVENC (NVIDIA)
try:
test_cmd = f"{self._ffmpeg_base_command} -hide_banner -loglevel error " \
f"-f lavfi -i testsrc=duration=1:size=320x240:rate=1 " \
f"-c:v h264_nvenc -f null -"
subprocess.run(
test_cmd, shell=True, check=True,
capture_output=True, timeout=30
)
self.logger.info("NVIDIA NVENC hardware acceleration available")
return (
True,
"h264_nvenc",
"scale_cuda",
"-hwaccel cuda -hwaccel_output_format cuda",
"aac"
)
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
pass
# No hardware acceleration available
self.logger.info("No hardware acceleration available, using software encoding")
return (False, "libx264", "scale", "", "aac")
@property
def hwaccel_available(self) -> bool:
"""Check if hardware acceleration is available."""
if self._hwaccel_available is None:
self._detect_and_set_hwaccel()
return self._hwaccel_available
@property
def video_encoder(self) -> str:
"""Get the video encoder to use."""
if self._video_encoder is None:
self._detect_and_set_hwaccel()
return self._video_encoder
@property
def scale_filter(self) -> str:
"""Get the scale filter to use."""
if self._scale_filter is None:
self._detect_and_set_hwaccel()
return self._scale_filter
@property
def hwaccel_decode_flags(self) -> str:
"""Get hardware acceleration decode flags."""
if self._hwaccel_decode_flags is None:
self._detect_and_set_hwaccel()
return self._hwaccel_decode_flags
@property
def aac_codec(self) -> str:
"""Get the AAC codec to use."""
if self._aac_codec is None:
self._detect_and_set_hwaccel()
return self._aac_codec
def _detect_and_set_hwaccel(self) -> None:
"""Detect and set hardware acceleration settings."""
(
self._hwaccel_available,
self._video_encoder,
self._scale_filter,
self._hwaccel_decode_flags,
self._aac_codec
) = self._detect_hardware_acceleration()
def _get_nvenc_quality_settings(self, preset: str = "medium") -> str:
"""Get NVENC quality settings for different presets."""
if not self.hwaccel_available:
return ""
settings = {
"lossless": "-preset p7 -tune hq -rc vbr -cq 0 -qmin 0 -qmax 0",
"high": "-preset p7 -tune hq -rc vbr -cq 19 -b:v 0",
"medium": "-preset p4 -tune hq -rc vbr -cq 23 -b:v 0",
"fast": "-preset p2 -tune ll -rc vbr -cq 28 -b:v 0",
}
return settings.get(preset, settings["medium"])
def _execute_command(
self,
command: str,
description: str,
timeout: int = 3600, # 1 hour default
) -> bool:
"""
Execute an FFmpeg command.
Args:
command: The FFmpeg command to execute
description: Human-readable description of the operation
timeout: Command timeout in seconds
Returns:
True if successful, False otherwise
"""
self.logger.info(f"Executing: {description}")
self.logger.debug(f"Command: {command}")
if self.dry_run:
self.logger.info(f"DRY RUN: Would execute: {command}")
return True
try:
result = subprocess.run(
command,
shell=True,
check=True,
capture_output=True,
text=True,
timeout=timeout
)
self.logger.info(f"Completed: {description}")
return True
except subprocess.CalledProcessError as e:
self.logger.error(f"Failed: {description}")
self.logger.error(f"Error: {e.stderr}")
return False
except subprocess.TimeoutExpired:
self.logger.error(f"Timeout: {description}")
return False
def _execute_command_with_fallback(
self,
gpu_command: str,
cpu_command: str,
description: str,
) -> bool:
"""
Execute a command with GPU, falling back to CPU if it fails.
Args:
gpu_command: Hardware-accelerated command
cpu_command: Software fallback command
description: Human-readable description
Returns:
True if successful, False otherwise
"""
if self.hwaccel_available:
self.logger.info(f"Trying hardware-accelerated encoding for: {description}")
if self._execute_command(gpu_command, description):
return True
self.logger.warning(f"Hardware encoding failed, falling back to software")
return self._execute_command(cpu_command, description)
def remux_with_instrumental(
self,
input_video: str,
instrumental_audio: str,
output_file: str,
) -> bool:
"""
Remux video with instrumental audio track.
Args:
input_video: Path to input video file
instrumental_audio: Path to instrumental audio file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_video}" '
f'-i "{instrumental_audio}" -map 0:v -map 1:a -c copy '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command(command, "Remuxing with instrumental audio")
def convert_mov_to_mp4(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Convert MOV to MP4 format.
Args:
input_file: Path to input MOV file
output_file: Path for output MP4 file
Returns:
True if successful, False otherwise
"""
gpu_command = (
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
f'-c:v {self.video_encoder} -c:a copy {self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v libx264 -c:a copy {self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Converting MOV to MP4"
)
def encode_lossless_mp4(
self,
title_video: str,
karaoke_video: str,
output_file: str,
end_video: Optional[str] = None,
) -> bool:
"""
Create lossless 4K MP4 by concatenating title, karaoke, and optionally end videos.
Args:
title_video: Path to title video
karaoke_video: Path to karaoke video
output_file: Path for output file
end_video: Optional path to end credits video
Returns:
True if successful, False otherwise
"""
# Quote file paths
title_quoted = shlex.quote(os.path.abspath(title_video))
karaoke_quoted = shlex.quote(os.path.abspath(karaoke_video))
# Build filter and inputs for concatenation
if end_video and os.path.isfile(end_video):
end_quoted = shlex.quote(os.path.abspath(end_video))
extra_input = f"-i {end_quoted}"
concat_filter = (
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0]'
'concat=n=3:v=1:a=1[outv][outa]"'
)
else:
extra_input = ""
concat_filter = (
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0]'
'concat=n=2:v=1:a=1[outv][outa]"'
)
gpu_command = (
f"{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i {title_quoted} "
f"{self.hwaccel_decode_flags} -i {karaoke_quoted} {extra_input} "
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v {self.video_encoder} '
f'{self._get_nvenc_quality_settings("lossless")} -c:a pcm_s16le '
f'{self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f"{self._ffmpeg_base_command} -i {title_quoted} -i {karaoke_quoted} {extra_input} "
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v libx264 -c:a pcm_s16le '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Encoding lossless 4K MP4"
)
def encode_lossy_mp4(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create lossy 4K MP4 with AAC audio.
Args:
input_file: Path to input file (typically lossless MP4)
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v copy -c:a {self.aac_codec} -ar 48000 -b:a 320k '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command(command, "Encoding lossy 4K MP4 with AAC")
def encode_lossless_mkv(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create MKV with FLAC audio (for YouTube upload).
Args:
input_file: Path to input file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v copy -c:a flac "{output_file}"'
)
return self._execute_command(command, "Creating MKV with FLAC for YouTube")
def encode_720p(
self,
input_file: str,
output_file: str,
) -> bool:
"""
Create 720p MP4 with AAC audio.
Args:
input_file: Path to input file
output_file: Path for output file
Returns:
True if successful, False otherwise
"""
gpu_command = (
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
f'-c:v {self.video_encoder} -vf "{self.scale_filter}=1280:720" '
f'{self._get_nvenc_quality_settings("medium")} -b:v 2000k '
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
f'{self.MP4_FLAGS} "{output_file}"'
)
cpu_command = (
f'{self._ffmpeg_base_command} -i "{input_file}" '
f'-c:v libx264 -vf "scale=1280:720" -b:v 2000k -preset medium -tune animation '
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
f'{self.MP4_FLAGS} "{output_file}"'
)
return self._execute_command_with_fallback(
gpu_command, cpu_command, "Encoding 720p MP4"
)
def encode_all_formats(
self,
config: EncodingConfig,
) -> EncodingResult:
"""
Encode video to all output formats.
This performs the full encoding pipeline:
1. Remux with instrumental audio
2. Convert to MP4 if needed
3. Encode lossless 4K MP4 (concatenated)
4. Encode lossy 4K MP4
5. Encode lossless MKV (for YouTube)
6. Encode 720p MP4
Args:
config: Encoding configuration with input/output paths
Returns:
EncodingResult with success status and output file paths
"""
output_files = {}
try:
# Step 1: Remux with instrumental audio
if config.output_karaoke_mp4:
self.logger.info("[Step 1/6] Remuxing video with instrumental audio...")
if not self.remux_with_instrumental(
config.karaoke_video,
config.instrumental_audio,
config.output_karaoke_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to remux with instrumental audio"
)
output_files["karaoke_mp4"] = config.output_karaoke_mp4
# Step 2: Convert to MP4 if needed
if config.output_with_vocals_mp4:
if not config.karaoke_video.endswith(".mp4"):
self.logger.info("[Step 2/6] Converting karaoke video to MP4...")
if not self.convert_mov_to_mp4(
config.karaoke_video,
config.output_with_vocals_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to convert to MP4"
)
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
else:
self.logger.info("[Step 2/6] Skipped - video already MP4")
# Step 3: Encode lossless 4K MP4
if config.output_lossless_4k_mp4:
self.logger.info("[Step 3/6] Encoding lossless 4K MP4...")
karaoke_for_concat = config.output_karaoke_mp4 or config.karaoke_video
if not self.encode_lossless_mp4(
config.title_video,
karaoke_for_concat,
config.output_lossless_4k_mp4,
config.end_video
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode lossless 4K MP4"
)
output_files["lossless_4k_mp4"] = config.output_lossless_4k_mp4
# Step 4: Encode lossy 4K MP4
if config.output_lossy_4k_mp4 and config.output_lossless_4k_mp4:
self.logger.info("[Step 4/6] Encoding lossy 4K MP4...")
if not self.encode_lossy_mp4(
config.output_lossless_4k_mp4,
config.output_lossy_4k_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode lossy 4K MP4"
)
output_files["lossy_4k_mp4"] = config.output_lossy_4k_mp4
# Step 5: Create MKV with FLAC audio
if config.output_lossless_mkv and config.output_lossless_4k_mp4:
self.logger.info("[Step 5/6] Creating MKV with FLAC audio...")
if not self.encode_lossless_mkv(
config.output_lossless_4k_mp4,
config.output_lossless_mkv
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to create MKV"
)
output_files["lossless_mkv"] = config.output_lossless_mkv
# Step 6: Encode 720p version
if config.output_720p_mp4 and config.output_lossless_4k_mp4:
self.logger.info("[Step 6/6] Encoding 720p MP4...")
if not self.encode_720p(
config.output_lossless_4k_mp4,
config.output_720p_mp4
):
return EncodingResult(
success=False,
output_files=output_files,
error="Failed to encode 720p"
)
output_files["720p_mp4"] = config.output_720p_mp4
self.logger.info("All encoding steps completed successfully")
return EncodingResult(success=True, output_files=output_files)
except Exception as e:
self.logger.error(f"Encoding failed with exception: {e}")
return EncodingResult(
success=False,
output_files=output_files,
error=str(e)
)
# Singleton instance and factory function (following existing service pattern)
_local_encoding_service: Optional[LocalEncodingService] = None
def get_local_encoding_service(**kwargs) -> LocalEncodingService:
"""
Get a local encoding service instance.
Args:
**kwargs: Arguments passed to LocalEncodingService
Returns:
LocalEncodingService instance
"""
global _local_encoding_service
if _local_encoding_service is None:
_local_encoding_service = LocalEncodingService(**kwargs)
return _local_encoding_service
+216
View File
@@ -0,0 +1,216 @@
"""
LyricsTranscriber cache synchronization with GCS.
This service persists LyricsTranscriber's cache files to GCS so that
cloud workers (Cloud Run instances) can share cache across containers.
Cache files are stored flat in GCS under lyrics-transcriber-cache/ prefix:
- Transcription: {provider}_{audio_hash}_raw.json, {provider}_{audio_hash}_converted.json
- Lyrics: {provider}_{artist_title_hash}_raw.json, {provider}_{artist_title_hash}_converted.json
Hash computation matches LyricsTranscriber's implementation exactly:
- Audio hash: MD5 of audio file bytes
- Lyrics hash: MD5 of "{artist.lower()}_{title.lower()}"
"""
import hashlib
import logging
import os
from typing import Dict, List, Optional
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
# Providers that use audio file hash as cache key
TRANSCRIPTION_PROVIDERS = ["audioshake", "whisper", "localwhisper"]
# Providers that use artist+title hash as cache key
LYRICS_PROVIDERS = ["genius", "spotify", "lrclib", "musixmatch"]
# Cache file suffixes
CACHE_SUFFIXES = ["raw", "converted"]
class LyricsCacheService:
"""Service to sync LyricsTranscriber cache with GCS."""
GCS_CACHE_PREFIX = "lyrics-transcriber-cache/"
def __init__(self, storage: Optional[StorageService] = None):
"""Initialize the cache service.
Args:
storage: StorageService instance. If None, creates a new one.
"""
self.storage = storage or StorageService()
def compute_audio_hash(self, audio_path: str) -> str:
"""Compute MD5 hash of audio file bytes.
This matches LyricsTranscriber's _get_file_hash() method exactly.
Args:
audio_path: Path to the audio file.
Returns:
MD5 hex digest of the file contents.
"""
md5_hash = hashlib.md5()
with open(audio_path, "rb") as f:
for chunk in iter(lambda: f.read(4096), b""):
md5_hash.update(chunk)
return md5_hash.hexdigest()
def compute_lyrics_hash(self, artist: str, title: str) -> str:
"""Compute MD5 hash of artist and title.
This matches LyricsTranscriber's _get_artist_title_hash() method exactly.
Args:
artist: Artist name.
title: Track title.
Returns:
MD5 hex digest of "{artist.lower()}_{title.lower()}".
"""
combined = f"{artist.lower()}_{title.lower()}"
return hashlib.md5(combined.encode()).hexdigest()
def _get_cache_filenames(
self, providers: List[str], hash_value: str
) -> List[str]:
"""Generate list of possible cache filenames for given providers and hash.
Args:
providers: List of provider names (e.g., ["audioshake", "whisper"]).
hash_value: The hash to use in filenames.
Returns:
List of filenames like ["audioshake_abc123_raw.json", ...].
"""
filenames = []
for provider in providers:
for suffix in CACHE_SUFFIXES:
filenames.append(f"{provider}_{hash_value}_{suffix}.json")
return filenames
def sync_cache_from_gcs(
self,
local_cache_dir: str,
audio_hash: str,
lyrics_hash: str,
) -> Dict[str, int]:
"""Download relevant cache files from GCS to local directory.
Downloads cache files for both transcription (audio hash) and
lyrics (artist+title hash) providers.
Args:
local_cache_dir: Local directory to download cache files to.
audio_hash: MD5 hash of audio file.
lyrics_hash: MD5 hash of artist+title.
Returns:
Dict with counts: {"downloaded": N, "not_found": M, "errors": E}
"""
os.makedirs(local_cache_dir, exist_ok=True)
stats = {"downloaded": 0, "not_found": 0, "errors": 0}
# Get all possible cache filenames
transcription_files = self._get_cache_filenames(
TRANSCRIPTION_PROVIDERS, audio_hash
)
lyrics_files = self._get_cache_filenames(LYRICS_PROVIDERS, lyrics_hash)
all_files = transcription_files + lyrics_files
for filename in all_files:
gcs_path = f"{self.GCS_CACHE_PREFIX}{filename}"
local_path = os.path.join(local_cache_dir, filename)
try:
if self.storage.file_exists(gcs_path):
self.storage.download_file(gcs_path, local_path)
logger.info(f"Cache hit: downloaded {filename}")
stats["downloaded"] += 1
else:
logger.debug(f"Cache miss: {filename} not in GCS")
stats["not_found"] += 1
except Exception as e:
logger.warning(f"Error downloading cache file {filename}: {e}")
stats["errors"] += 1
logger.info(
f"Cache sync from GCS complete: "
f"{stats['downloaded']} downloaded, "
f"{stats['not_found']} not found, "
f"{stats['errors']} errors"
)
return stats
def sync_cache_to_gcs(
self,
local_cache_dir: str,
audio_hash: str,
lyrics_hash: str,
) -> Dict[str, int]:
"""Upload new cache files from local directory to GCS.
Only uploads files that match expected cache patterns for the given
hashes and don't already exist in GCS.
Args:
local_cache_dir: Local directory with cache files.
audio_hash: MD5 hash of audio file.
lyrics_hash: MD5 hash of artist+title.
Returns:
Dict with counts: {"uploaded": N, "skipped": M, "errors": E}
"""
stats = {"uploaded": 0, "skipped": 0, "errors": 0}
if not os.path.exists(local_cache_dir):
logger.warning(f"Local cache dir does not exist: {local_cache_dir}")
return stats
# Get all possible cache filenames we're interested in
transcription_files = self._get_cache_filenames(
TRANSCRIPTION_PROVIDERS, audio_hash
)
lyrics_files = self._get_cache_filenames(LYRICS_PROVIDERS, lyrics_hash)
expected_files = set(transcription_files + lyrics_files)
# Check each file in local cache dir
for filename in os.listdir(local_cache_dir):
# Only process files we expect (matching our hash patterns)
if filename not in expected_files:
continue
local_path = os.path.join(local_cache_dir, filename)
if not os.path.isfile(local_path):
continue
gcs_path = f"{self.GCS_CACHE_PREFIX}{filename}"
try:
# Skip if already exists in GCS (same hash = same content)
if self.storage.file_exists(gcs_path):
logger.debug(f"Cache file already in GCS: {filename}")
stats["skipped"] += 1
continue
self.storage.upload_file(local_path, gcs_path)
logger.info(f"Uploaded cache file: {filename}")
stats["uploaded"] += 1
except Exception as e:
logger.warning(f"Error uploading cache file {filename}: {e}")
stats["errors"] += 1
logger.info(
f"Cache sync to GCS complete: "
f"{stats['uploaded']} uploaded, "
f"{stats['skipped']} skipped (already exist), "
f"{stats['errors']} errors"
)
return stats
+287
View File
@@ -0,0 +1,287 @@
"""
Packaging Service.
Provides CDG and TXT package generation functionality, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- CDG (CD+G) file generation from LRC files
- TXT lyric file generation from LRC files
- ZIP packaging of CDG/MP3 and TXT/MP3 pairs
"""
import logging
import os
import zipfile
from typing import Optional, Dict, Any, Tuple
logger = logging.getLogger(__name__)
class PackagingService:
"""
Service for creating CDG and TXT karaoke packages.
CDG (CD+Graphics) is a format used by karaoke machines.
TXT packages are used by software karaoke players.
Both formats are packaged as ZIP files with an MP3 audio track.
"""
def __init__(
self,
cdg_styles: Optional[Dict[str, Any]] = None,
dry_run: bool = False,
non_interactive: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the packaging service.
Args:
cdg_styles: CDG style configuration for CDG generation
dry_run: If True, log actions without performing them
non_interactive: If True, skip interactive prompts
logger: Optional logger instance
"""
self.cdg_styles = cdg_styles
self.dry_run = dry_run
self.non_interactive = non_interactive
self.logger = logger or logging.getLogger(__name__)
def create_cdg_package(
self,
lrc_file: str,
audio_file: str,
output_zip_path: str,
artist: str,
title: str,
output_mp3_path: Optional[str] = None,
output_cdg_path: Optional[str] = None,
) -> Tuple[str, Optional[str], Optional[str]]:
"""
Create a CDG package (ZIP containing CDG and MP3 files).
Args:
lrc_file: Path to the LRC lyrics file
audio_file: Path to the instrumental audio file
output_zip_path: Path for the output ZIP file
artist: Artist name for metadata
title: Song title for metadata
output_mp3_path: Optional path for the extracted MP3 file
output_cdg_path: Optional path for the extracted CDG file
Returns:
Tuple of (zip_path, mp3_path, cdg_path)
Raises:
ValueError: If CDG styles are not configured
FileNotFoundError: If input files are missing
Exception: If CDG generation fails
"""
self.logger.info(f"Creating CDG package for {artist} - {title}")
# Validate inputs
if not os.path.isfile(lrc_file):
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
if not os.path.isfile(audio_file):
raise FileNotFoundError(f"Audio file not found: {audio_file}")
# Check if ZIP already exists
if os.path.isfile(output_zip_path):
if self.non_interactive:
self.logger.info(
f"CDG ZIP exists, will be overwritten: {output_zip_path}"
)
else:
self.logger.info(f"CDG ZIP already exists: {output_zip_path}")
# Check if individual files exist (allows skipping generation)
if output_mp3_path and output_cdg_path:
if os.path.isfile(output_mp3_path) and os.path.isfile(output_cdg_path):
self.logger.info("Found existing MP3 and CDG files, creating ZIP directly")
if not self.dry_run:
self._create_zip_from_files(
output_zip_path,
[(output_mp3_path, os.path.basename(output_mp3_path)),
(output_cdg_path, os.path.basename(output_cdg_path))]
)
return output_zip_path, output_mp3_path, output_cdg_path
if self.dry_run:
self.logger.info(
f"DRY RUN: Would generate CDG package: {output_zip_path}"
)
return output_zip_path, output_mp3_path, output_cdg_path
# Generate CDG files
if self.cdg_styles is None:
raise ValueError(
"CDG styles configuration is required for CDG generation"
)
self.logger.info("Generating CDG and MP3 files")
from lyrics_transcriber.output.cdg import CDGGenerator
output_dir = os.path.dirname(output_zip_path) or os.getcwd()
generator = CDGGenerator(output_dir=output_dir, logger=self.logger)
cdg_file, mp3_file, zip_file = generator.generate_cdg_from_lrc(
lrc_file=lrc_file,
audio_file=audio_file,
title=title,
artist=artist,
cdg_styles=self.cdg_styles,
)
# Rename ZIP to expected output path if different
if os.path.isfile(zip_file) and zip_file != output_zip_path:
os.rename(zip_file, output_zip_path)
self.logger.info(f"Renamed CDG ZIP: {zip_file} -> {output_zip_path}")
if not os.path.isfile(output_zip_path):
raise Exception(f"Failed to create CDG ZIP file: {output_zip_path}")
# Extract the ZIP to get individual files if paths provided
extracted_mp3 = None
extracted_cdg = None
if output_mp3_path or output_cdg_path:
self.logger.info(f"Extracting CDG ZIP file: {output_zip_path}")
with zipfile.ZipFile(output_zip_path, "r") as zip_ref:
zip_ref.extractall(output_dir)
# Find extracted files
if output_mp3_path and os.path.isfile(output_mp3_path):
extracted_mp3 = output_mp3_path
self.logger.info(f"Extracted MP3: {extracted_mp3}")
if output_cdg_path and os.path.isfile(output_cdg_path):
extracted_cdg = output_cdg_path
self.logger.info(f"Extracted CDG: {extracted_cdg}")
self.logger.info(f"CDG package created: {output_zip_path}")
return output_zip_path, extracted_mp3, extracted_cdg
def create_txt_package(
self,
lrc_file: str,
mp3_file: str,
output_zip_path: str,
output_txt_path: Optional[str] = None,
) -> Tuple[str, Optional[str]]:
"""
Create a TXT package (ZIP containing TXT lyrics and MP3 files).
Args:
lrc_file: Path to the LRC lyrics file
mp3_file: Path to the MP3 audio file
output_zip_path: Path for the output ZIP file
output_txt_path: Optional path for the generated TXT file
Returns:
Tuple of (zip_path, txt_path)
Raises:
FileNotFoundError: If input files are missing
Exception: If TXT generation fails
"""
self.logger.info(f"Creating TXT package from {lrc_file}")
# Validate inputs
if not os.path.isfile(lrc_file):
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
if not os.path.isfile(mp3_file):
raise FileNotFoundError(f"MP3 file not found: {mp3_file}")
# Check if ZIP already exists
if os.path.isfile(output_zip_path):
if self.non_interactive:
self.logger.info(
f"TXT ZIP exists, will be overwritten: {output_zip_path}"
)
else:
self.logger.info(f"TXT ZIP already exists: {output_zip_path}")
if self.dry_run:
self.logger.info(
f"DRY RUN: Would create TXT package: {output_zip_path}"
)
return output_zip_path, output_txt_path
# Generate TXT from LRC
self.logger.info(f"Converting LRC to TXT format: {lrc_file}")
from lyrics_converter import LyricsConverter
txt_converter = LyricsConverter(output_format="txt", filepath=lrc_file)
converted_txt = txt_converter.convert_file()
# Write TXT file
if output_txt_path is None:
# Default to same name as ZIP but with .txt extension
output_txt_path = output_zip_path.replace(".zip", ".txt")
with open(output_txt_path, "w") as txt_file:
txt_file.write(converted_txt)
self.logger.info(f"TXT file written: {output_txt_path}")
# Create ZIP containing MP3 and TXT
self.logger.info(f"Creating TXT ZIP: {output_zip_path}")
self._create_zip_from_files(
output_zip_path,
[(mp3_file, os.path.basename(mp3_file)),
(output_txt_path, os.path.basename(output_txt_path))]
)
if not os.path.isfile(output_zip_path):
raise Exception(f"Failed to create TXT ZIP file: {output_zip_path}")
self.logger.info(f"TXT package created: {output_zip_path}")
return output_zip_path, output_txt_path
def _create_zip_from_files(
self,
zip_path: str,
files: list,
) -> None:
"""
Create a ZIP file from a list of files.
Args:
zip_path: Path for the output ZIP file
files: List of (file_path, archive_name) tuples
"""
with zipfile.ZipFile(zip_path, "w") as zipf:
for file_path, archive_name in files:
if os.path.isfile(file_path):
zipf.write(file_path, archive_name)
self.logger.debug(f"Added to ZIP: {archive_name}")
else:
self.logger.warning(f"File not found for ZIP: {file_path}")
# Singleton instance and factory function (following existing service pattern)
_packaging_service: Optional[PackagingService] = None
def get_packaging_service(
cdg_styles: Optional[Dict[str, Any]] = None,
**kwargs
) -> PackagingService:
"""
Get a packaging service instance.
Args:
cdg_styles: CDG style configuration
**kwargs: Additional arguments passed to PackagingService
Returns:
PackagingService instance
"""
global _packaging_service
# Create new instance if settings changed
if _packaging_service is None or cdg_styles:
_packaging_service = PackagingService(
cdg_styles=cdg_styles,
**kwargs
)
return _packaging_service
+1 -1
View File
@@ -32,7 +32,7 @@ logger = logging.getLogger(__name__)
# Collection names
USERS_COLLECTION = "users"
USERS_COLLECTION = "gen_users"
MAGIC_LINKS_COLLECTION = "magic_links"
SESSIONS_COLLECTION = "sessions"
PROCESSED_STRIPE_SESSIONS_COLLECTION = "processed_stripe_sessions"
+22 -1
View File
@@ -20,6 +20,7 @@ import os
import json
from typing import Optional
import httpx
from google.protobuf import duration_pb2
from backend.config import get_settings
from backend.services.tracing import inject_trace_context
@@ -38,6 +39,18 @@ WORKER_QUEUES = {
"video": "video-worker-queue",
}
# Dispatch deadlines for each worker type (in seconds)
# Cloud Tasks max is 1800s (30 min). These set how long Cloud Tasks waits
# for the HTTP handler to respond before considering it failed.
# Must be >= actual worker timeout + buffer for startup/upload.
WORKER_DISPATCH_DEADLINES = {
"audio": 1800, # 30 min - Modal separation can take 15-20 min
"lyrics": 1500, # 25 min - TRANSCRIPTION_TIMEOUT_SECONDS is 1200s (20 min)
"screens": 600, # 10 min - Screen generation is fast
"render-video": 1800, # 30 min - Video encoding can be slow
"video": 1800, # 30 min - Video encoding can be slow
}
class WorkerService:
"""
@@ -234,6 +247,9 @@ class WorkerService:
# This allows worker spans to link back to the original request trace
headers = inject_trace_context(headers)
# Get dispatch deadline for this worker type (how long Cloud Tasks waits for response)
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get(worker_type, 600)
# Build task payload
task = {
"http_request": {
@@ -247,6 +263,10 @@ class WorkerService:
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
},
},
# Set dispatch_deadline - how long Cloud Tasks waits for the handler to respond
# Default is 10 min, but audio separation can take 30+ min
# Max is 1800s (30 min) for Cloud Tasks
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
}
# Create task
@@ -254,7 +274,8 @@ class WorkerService:
# This prevents duplicate task errors on retries
response = self.tasks_client.create_task(parent=parent, task=task)
logger.info(
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name}"
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name} "
f"(dispatch_deadline={dispatch_deadline_seconds}s)"
)
return True
+445
View File
@@ -0,0 +1,445 @@
"""
YouTube Upload Service.
Provides video upload functionality for YouTube, extracted from KaraokeFinalise
for use by both the cloud backend (video_worker) and local CLI.
This service handles:
- Video uploads with metadata (title, description, tags)
- Thumbnail uploads
- Duplicate video detection and replacement
- Authentication via pre-stored credentials or client secrets file
"""
import logging
import os
from typing import Optional, Dict, Any, Tuple
from thefuzz import fuzz
logger = logging.getLogger(__name__)
class YouTubeUploadService:
"""
Service for uploading videos to YouTube.
Supports two authentication modes:
1. Pre-stored credentials (for server-side/non-interactive use)
2. Client secrets file (for interactive CLI use with browser OAuth)
"""
YOUTUBE_URL_PREFIX = "https://www.youtube.com/watch?v="
SCOPES = ["https://www.googleapis.com/auth/youtube"]
def __init__(
self,
credentials: Optional[Dict[str, Any]] = None,
client_secrets_file: Optional[str] = None,
non_interactive: bool = False,
server_side_mode: bool = False,
dry_run: bool = False,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the YouTube upload service.
Args:
credentials: Pre-stored OAuth credentials dict with keys:
- token: Access token
- refresh_token: Refresh token
- token_uri: Token endpoint URL
- client_id: OAuth client ID
- client_secret: OAuth client secret
- scopes: List of OAuth scopes
client_secrets_file: Path to OAuth client secrets JSON file
(for interactive authentication)
non_interactive: If True, skip interactive prompts
server_side_mode: If True, use exact matching for duplicate detection
dry_run: If True, log actions without performing them
logger: Optional logger instance
"""
self.credentials = credentials
self.client_secrets_file = client_secrets_file
self.non_interactive = non_interactive
self.server_side_mode = server_side_mode
self.dry_run = dry_run
self.logger = logger or logging.getLogger(__name__)
self._youtube_service = None
self._channel_id = None
def _authenticate(self):
"""
Authenticate with YouTube API and return service object.
Returns:
YouTube API service object
Raises:
Exception: If authentication fails or is required in non-interactive mode
"""
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
import pickle
# Check if we have pre-stored credentials (for non-interactive mode)
if self.credentials and self.non_interactive:
try:
creds = Credentials(
token=self.credentials.get('token'),
refresh_token=self.credentials.get('refresh_token'),
token_uri=self.credentials.get('token_uri'),
client_id=self.credentials.get('client_id'),
client_secret=self.credentials.get('client_secret'),
scopes=self.credentials.get('scopes', self.SCOPES)
)
# Refresh token if needed
if creds.expired and creds.refresh_token:
creds.refresh(Request())
youtube = build('youtube', 'v3', credentials=creds)
self.logger.info("Successfully authenticated with YouTube using pre-stored credentials")
return youtube
except Exception as e:
self.logger.error(f"Failed to authenticate with pre-stored credentials: {str(e)}")
# Fall through to original authentication if pre-stored credentials fail
# For non-interactive mode without pre-stored credentials, raise error
if self.non_interactive:
raise Exception(
"YouTube authentication required but running in non-interactive mode. "
"Please pre-authenticate or disable YouTube upload."
)
# Interactive authentication using client secrets file
if not self.client_secrets_file:
raise Exception("No YouTube credentials or client secrets file provided")
# Token file stores the user's access and refresh tokens
youtube_token_file = "/tmp/karaoke-finalise-youtube-token.pickle"
creds = None
# Check if we have saved credentials
if os.path.exists(youtube_token_file):
with open(youtube_token_file, "rb") as token:
creds = pickle.load(token)
# If there are no valid credentials, let the user log in
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
self.client_secrets_file, scopes=self.SCOPES
)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open(youtube_token_file, "wb") as token:
pickle.dump(creds, token)
return build("youtube", "v3", credentials=creds)
@property
def youtube_service(self):
"""Lazy-load YouTube service on first access."""
if self._youtube_service is None:
self._youtube_service = self._authenticate()
return self._youtube_service
def get_channel_id(self) -> Optional[str]:
"""
Get the authenticated user's YouTube channel ID.
Returns:
Channel ID string, or None if not found
"""
if self._channel_id:
return self._channel_id
request = self.youtube_service.channels().list(part="snippet", mine=True)
response = request.execute()
if "items" in response and len(response["items"]) > 0:
self._channel_id = response["items"][0]["id"]
return self._channel_id
return None
def check_duplicate(
self,
title: str,
exact_match: Optional[bool] = None
) -> Tuple[bool, Optional[str], Optional[str]]:
"""
Check if a video with the given title already exists on the channel.
Args:
title: Video title to search for
exact_match: If True, require exact title match. If None, uses
server_side_mode setting (exact in server mode, fuzzy in CLI)
Returns:
Tuple of (exists: bool, video_id: Optional[str], video_url: Optional[str])
"""
channel_id = self.get_channel_id()
if not channel_id:
self.logger.warning("Could not get channel ID, skipping duplicate check")
return False, None, None
use_exact_match = exact_match if exact_match is not None else self.server_side_mode
self.logger.info(f"Searching YouTube channel {channel_id} for title: {title}")
request = self.youtube_service.search().list(
part="snippet",
channelId=channel_id,
q=title,
type="video",
maxResults=10
)
response = request.execute()
if "items" not in response or len(response["items"]) == 0:
self.logger.info(f"No matching video found with title: {title}")
return False, None, None
for item in response["items"]:
# Verify the video actually belongs to our channel
result_channel_id = item["snippet"]["channelId"]
if result_channel_id != channel_id:
self.logger.debug(
f"Skipping video from different channel: {item['snippet']['title']} "
f"(channel: {result_channel_id})"
)
continue
found_title = item["snippet"]["title"]
# Determine if this is a match
if use_exact_match:
is_match = title.lower() == found_title.lower()
similarity_score = 100 if is_match else 0
else:
similarity_score = fuzz.ratio(title.lower(), found_title.lower())
is_match = similarity_score >= 70
if is_match:
video_id = item["id"]["videoId"]
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
self.logger.info(
f"Potential match found on YouTube channel with ID: {video_id} "
f"and title: {found_title} (similarity: {similarity_score}%)"
)
# In non-interactive mode, return the match directly
if self.non_interactive:
self.logger.info("Non-interactive mode, found a match.")
return True, video_id, video_url
# Interactive confirmation
confirmation = input(
f"Is '{found_title}' the video you are finalising? (y/n): "
).strip().lower()
if confirmation == "y":
return True, video_id, video_url
self.logger.info(f"No matching video found with title: {title}")
return False, None, None
def delete_video(self, video_id: str) -> bool:
"""
Delete a YouTube video by its ID.
Args:
video_id: The YouTube video ID to delete
Returns:
True if successful, False otherwise
"""
self.logger.info(f"Deleting YouTube video with ID: {video_id}")
if self.dry_run:
self.logger.info(f"DRY RUN: Would delete YouTube video with ID: {video_id}")
return True
try:
self.youtube_service.videos().delete(id=video_id).execute()
self.logger.info(f"Successfully deleted YouTube video with ID: {video_id}")
return True
except Exception as e:
self.logger.error(f"Failed to delete YouTube video with ID {video_id}: {e}")
return False
@staticmethod
def truncate_title(title: str, max_length: int = 95) -> str:
"""
Truncate title to the nearest whole word within max_length.
Args:
title: Title to truncate
max_length: Maximum length (default 95 for YouTube)
Returns:
Truncated title with "..." if needed
"""
if len(title) <= max_length:
return title
truncated_title = title[:max_length].rsplit(" ", 1)[0]
if len(truncated_title) < max_length:
truncated_title += " ..."
return truncated_title
def upload_video(
self,
video_path: str,
title: str,
description: str,
thumbnail_path: Optional[str] = None,
tags: Optional[list] = None,
category_id: str = "10", # Music category
privacy_status: str = "public",
replace_existing: bool = False,
) -> Tuple[Optional[str], Optional[str]]:
"""
Upload a video to YouTube with metadata and optional thumbnail.
Args:
video_path: Path to the video file to upload
title: Video title (will be truncated to 95 chars if needed)
description: Video description
thumbnail_path: Optional path to thumbnail image
tags: Optional list of tags/keywords
category_id: YouTube category ID (default "10" for Music)
privacy_status: "public", "private", or "unlisted"
replace_existing: If True, delete existing video with same title
Returns:
Tuple of (video_id, video_url) or (None, None) if upload failed/skipped
"""
from googleapiclient.http import MediaFileUpload
# Truncate title if needed
youtube_title = self.truncate_title(title)
self.logger.info(f"Uploading video to YouTube: {youtube_title}")
if self.dry_run:
self.logger.info(
f"DRY RUN: Would upload {video_path} to YouTube with title: {youtube_title}"
)
return "dry_run_video_id", f"{self.YOUTUBE_URL_PREFIX}dry_run_video_id"
# Check for existing video
should_replace = True if self.server_side_mode else replace_existing
exists, existing_id, existing_url = self.check_duplicate(youtube_title)
if exists:
if should_replace:
self.logger.info(f"Video already exists on YouTube, deleting before re-upload: {existing_url}")
if self.delete_video(existing_id):
self.logger.info("Successfully deleted existing video, proceeding with upload")
else:
self.logger.error("Failed to delete existing video, aborting upload")
return None, None
else:
self.logger.warning(f"Video already exists on YouTube, skipping upload: {existing_url}")
return existing_id, existing_url
# Prepare video metadata
body = {
"snippet": {
"title": youtube_title,
"description": description,
"tags": tags or [],
"categoryId": category_id,
},
"status": {"privacyStatus": privacy_status},
}
# Determine MIME type based on file extension
ext = os.path.splitext(video_path)[1].lower()
mime_type = {
".mkv": "video/x-matroska",
".mp4": "video/mp4",
".mov": "video/quicktime",
".avi": "video/x-msvideo",
}.get(ext, "video/*")
# Upload video
self.logger.info(f"Authenticating with YouTube...")
media_file = MediaFileUpload(video_path, mimetype=mime_type, resumable=True)
self.logger.info(f"Uploading video to YouTube...")
request = self.youtube_service.videos().insert(
part="snippet,status",
body=body,
media_body=media_file
)
response = request.execute()
video_id = response.get("id")
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
self.logger.info(f"Uploaded video to YouTube: {video_url}")
# Upload thumbnail if provided
if thumbnail_path and os.path.isfile(thumbnail_path):
try:
self.logger.info(f"Uploading thumbnail from: {thumbnail_path}")
media_thumbnail = MediaFileUpload(thumbnail_path, mimetype="image/jpeg")
self.youtube_service.thumbnails().set(
videoId=video_id,
media_body=media_thumbnail
).execute()
self.logger.info(f"Uploaded thumbnail for video ID {video_id}")
except Exception as e:
self.logger.error(f"Failed to upload thumbnail: {e}")
self.logger.warning(
"Video uploaded but thumbnail not set. "
"You may need to set it manually on YouTube."
)
elif thumbnail_path:
self.logger.warning(f"Thumbnail file not found, skipping: {thumbnail_path}")
return video_id, video_url
# Singleton instance and factory function (following existing service pattern)
_youtube_upload_service: Optional[YouTubeUploadService] = None
def get_youtube_upload_service(
credentials: Optional[Dict[str, Any]] = None,
client_secrets_file: Optional[str] = None,
**kwargs
) -> YouTubeUploadService:
"""
Get a YouTube upload service instance.
For server-side use, pass credentials from YouTubeService.
For CLI use, pass client_secrets_file.
Args:
credentials: Pre-stored OAuth credentials dict
client_secrets_file: Path to OAuth client secrets JSON file
**kwargs: Additional arguments passed to YouTubeUploadService
Returns:
YouTubeUploadService instance
"""
global _youtube_upload_service
# Create new instance if credentials/settings changed
if _youtube_upload_service is None or credentials or client_secrets_file:
_youtube_upload_service = YouTubeUploadService(
credentials=credentials,
client_secrets_file=client_secrets_file,
**kwargs
)
return _youtube_upload_service
+15 -9
View File
@@ -6,23 +6,29 @@ Provides common mocks and test utilities across all test modules.
NOTE: Module-level mocks are NOT applied here to allow emulator tests to work.
Individual tests must mock dependencies as needed.
"""
# IMPORTANT: Set environment variables FIRST, before ANY imports that might
# trigger backend module loading and Settings singleton creation.
import os
import pytest
from unittest.mock import Mock, MagicMock, AsyncMock, patch
from datetime import datetime, UTC
from fastapi.testclient import TestClient
# Mock google.auth.default AND firestore for unit tests if not using emulator
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
# Set up test environment variables BEFORE importing any backend modules
# This must happen before Settings is instantiated, which occurs on first import
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
from unittest.mock import MagicMock
# Set up test environment variables BEFORE importing any backend modules
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
os.environ.setdefault('ENVIRONMENT', 'test')
import pytest
from unittest.mock import Mock, MagicMock, AsyncMock, patch
from datetime import datetime, UTC
from fastapi.testclient import TestClient
# Mock google.auth.default AND firestore for unit tests if not using emulator
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
from unittest.mock import MagicMock
# Mock google.auth.default - prevents credential errors
try:
@@ -0,0 +1,443 @@
"""
Integration tests for worker log subcollection storage.
Tests the subcollection approach using real Firestore emulator to verify:
- Logs are stored in subcollection (jobs/{job_id}/logs)
- TTL expiry field is set correctly
- Large log volumes work without 1MB limit issues
- Concurrent writes are handled correctly
- Logs can be queried efficiently
Run with: ./scripts/run-emulator-tests.sh
"""
import pytest
import time
import requests
import os
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone, timedelta
def emulators_running() -> bool:
"""Check if GCP emulators are running."""
try:
requests.get("http://127.0.0.1:8080", timeout=1)
return True
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
return False
# Skip all tests in this module if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Set up environment for emulator
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
class TestWorkerLogsSubcollectionDirect:
"""Direct Firestore tests for worker logs subcollection."""
@pytest.fixture(autouse=True)
def setup_firestore(self):
"""Set up Firestore client for each test."""
from google.cloud import firestore
self.db = firestore.Client(project="test-project")
self.collection = "test-subcollection-jobs"
yield
def _create_test_job(self):
"""Create a test job document."""
job_id = f"test-sub-{int(time.time() * 1000)}"
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.set({
"job_id": job_id,
"status": "pending",
"created_at": datetime.now(timezone.utc),
"worker_logs": [] # Legacy field - kept empty for new jobs
})
return job_id
def _append_log_to_subcollection(self, job_id: str, worker: str, message: str, ttl_days: int = 30):
"""Add log to subcollection at jobs/{job_id}/logs."""
import uuid
log_id = str(uuid.uuid4())
now = datetime.now(timezone.utc)
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
logs_ref.document(log_id).set({
"id": log_id,
"job_id": job_id,
"timestamp": now,
"level": "INFO",
"worker": worker,
"message": message,
"ttl_expiry": now + timedelta(days=ttl_days)
})
return log_id
def _get_logs_from_subcollection(self, job_id: str, worker: str = None, limit: int = 500):
"""Get logs from subcollection."""
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
query = logs_ref.order_by("timestamp")
if worker:
from google.cloud.firestore_v1 import FieldFilter
query = query.where(filter=FieldFilter("worker", "==", worker))
query = query.limit(limit)
return [doc.to_dict() for doc in query.stream()]
def _count_logs_in_subcollection(self, job_id: str) -> int:
"""Count logs in subcollection."""
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
count_query = logs_ref.count()
result = count_query.get()
if result and len(result) > 0:
return result[0][0].value
return 0
def _delete_logs_subcollection(self, job_id: str, batch_size: int = 100) -> int:
"""Delete all logs in subcollection."""
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
deleted_count = 0
while True:
docs = list(logs_ref.limit(batch_size).stream())
if not docs:
break
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_count += 1
batch.commit()
return deleted_count
def test_subcollection_single_write(self):
"""Test writing single log to subcollection."""
job_id = self._create_test_job()
log_id = self._append_log_to_subcollection(job_id, "test", "Single log message")
time.sleep(0.1)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == 1
assert logs[0]["message"] == "Single log message"
assert logs[0]["worker"] == "test"
assert logs[0]["id"] == log_id
print("Single write to subcollection works")
def test_subcollection_ttl_field_is_set(self):
"""Test TTL expiry field is set correctly."""
job_id = self._create_test_job()
self._append_log_to_subcollection(job_id, "test", "Log with TTL", ttl_days=7)
time.sleep(0.1)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == 1
ttl_expiry = logs[0]["ttl_expiry"]
assert ttl_expiry is not None
# TTL should be ~7 days from now
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
if hasattr(ttl_expiry, 'timestamp'):
# Firestore datetime object
diff = abs((ttl_expiry.replace(tzinfo=timezone.utc) - expected_ttl).total_seconds())
else:
diff = 0 # If already datetime
assert diff < 60, f"TTL expiry should be ~7 days from now, diff was {diff}s"
print("TTL field is set correctly")
def test_subcollection_sequential_writes(self):
"""Test multiple sequential writes to subcollection."""
job_id = self._create_test_job()
for i in range(20):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.2)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == 20, f"Expected 20 logs, got {len(logs)}"
print(f"Sequential writes: {len(logs)} logs created")
def test_subcollection_concurrent_writes(self):
"""Test concurrent writes to subcollection - no race conditions."""
job_id = self._create_test_job()
num_writes = 50
def write_log(index):
self._append_log_to_subcollection(job_id, "worker", f"Concurrent Log {index}")
with ThreadPoolExecutor(max_workers=10) as executor:
futures = [executor.submit(write_log, i) for i in range(num_writes)]
for future in as_completed(futures):
future.result()
time.sleep(0.5)
logs = self._get_logs_from_subcollection(job_id)
assert len(logs) == num_writes, f"Expected {num_writes} logs, got {len(logs)}"
print(f"Concurrent writes: All {num_writes} logs preserved")
def test_subcollection_large_volume(self):
"""Test large volume of logs - demonstrates no 1MB limit."""
job_id = self._create_test_job()
num_logs = 500 # Would exceed 1MB in embedded array
# Write logs in batches
batch_size = 50
for batch_start in range(0, num_logs, batch_size):
with ThreadPoolExecutor(max_workers=10) as executor:
futures = []
for i in range(batch_start, min(batch_start + batch_size, num_logs)):
futures.append(executor.submit(
self._append_log_to_subcollection,
job_id,
"stress",
f"Large volume test log {i} - " + "x" * 500 # ~500 bytes per log
))
for future in as_completed(futures):
future.result()
time.sleep(1)
count = self._count_logs_in_subcollection(job_id)
assert count == num_logs, f"Expected {num_logs} logs, got {count}"
print(f"Large volume: {count} logs created (~{count * 500 / 1024:.1f}KB would exceed 1MB if embedded)")
def test_subcollection_filter_by_worker(self):
"""Test filtering logs by worker type."""
job_id = self._create_test_job()
# Add logs from different workers
for i in range(10):
self._append_log_to_subcollection(job_id, "audio", f"Audio log {i}")
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics log {i}")
self._append_log_to_subcollection(job_id, "video", f"Video log {i}")
time.sleep(0.3)
# Query by worker
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
all_logs = self._get_logs_from_subcollection(job_id)
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
assert len(all_logs) == 30, f"Expected 30 total logs, got {len(all_logs)}"
print("Worker filtering works correctly")
def test_subcollection_ordered_by_timestamp(self):
"""Test logs are returned in timestamp order."""
job_id = self._create_test_job()
# Add logs with slight delays to ensure different timestamps
for i in range(5):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.02)
logs = self._get_logs_from_subcollection(job_id)
# Verify order (timestamps should be ascending)
for i in range(len(logs) - 1):
assert logs[i]["timestamp"] <= logs[i + 1]["timestamp"], \
f"Logs not in order at index {i}"
print("Logs are ordered by timestamp")
def test_subcollection_delete_all_logs(self):
"""Test deleting all logs in subcollection."""
job_id = self._create_test_job()
# Add logs
for i in range(25):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.2)
count_before = self._count_logs_in_subcollection(job_id)
assert count_before == 25
# Delete all logs
deleted = self._delete_logs_subcollection(job_id)
time.sleep(0.2)
count_after = self._count_logs_in_subcollection(job_id)
assert deleted == 25, f"Expected to delete 25 logs, deleted {deleted}"
assert count_after == 0, f"Expected 0 logs after delete, got {count_after}"
print(f"Deleted {deleted} logs from subcollection")
def test_subcollection_concurrent_workers_interleaved(self):
"""Test simulating audio and lyrics workers writing concurrently."""
job_id = self._create_test_job()
logs_per_worker = 20
def audio_worker():
for i in range(logs_per_worker):
self._append_log_to_subcollection(job_id, "audio", f"Audio processing step {i}")
time.sleep(0.005)
def lyrics_worker():
for i in range(logs_per_worker):
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics processing step {i}")
time.sleep(0.005)
def video_worker():
for i in range(logs_per_worker):
self._append_log_to_subcollection(job_id, "video", f"Video encoding step {i}")
time.sleep(0.005)
# Start all workers
threads = [
threading.Thread(target=audio_worker),
threading.Thread(target=lyrics_worker),
threading.Thread(target=video_worker)
]
for t in threads:
t.start()
for t in threads:
t.join()
time.sleep(0.5)
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
video_logs = self._get_logs_from_subcollection(job_id, worker="video")
total = self._count_logs_in_subcollection(job_id)
assert len(audio_logs) == logs_per_worker
assert len(lyrics_logs) == logs_per_worker
assert len(video_logs) == logs_per_worker
assert total == logs_per_worker * 3
print(f"Concurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics + {len(video_logs)} video = {total}")
def test_subcollection_job_deletion_cleans_up_logs(self):
"""Test that deleting job document doesn't orphan logs."""
job_id = self._create_test_job()
# Add logs
for i in range(10):
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
time.sleep(0.2)
assert self._count_logs_in_subcollection(job_id) == 10
# Delete logs first (must happen before parent doc deletion in real code)
deleted = self._delete_logs_subcollection(job_id)
assert deleted == 10
# Now delete job document
self.db.collection(self.collection).document(job_id).delete()
# Verify subcollection is empty (Firestore doesn't cascade delete subcollections)
time.sleep(0.2)
assert self._count_logs_in_subcollection(job_id) == 0
print("Job deletion with log cleanup works correctly")
class TestSubcollectionVsEmbeddedArray:
"""Compare subcollection approach with embedded array."""
@pytest.fixture(autouse=True)
def setup_firestore(self):
"""Set up Firestore client for each test."""
from google.cloud import firestore
self.db = firestore.Client(project="test-project")
self.collection = "test-comparison-jobs"
yield
def _create_test_job(self, use_array: bool):
"""Create test job."""
job_id = f"test-cmp-{int(time.time() * 1000)}-{'arr' if use_array else 'sub'}"
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.set({
"job_id": job_id,
"status": "pending",
"created_at": datetime.now(timezone.utc),
"worker_logs": []
})
return job_id
def _append_log_array(self, job_id: str, message: str):
"""Add log to embedded array."""
from google.cloud import firestore
doc_ref = self.db.collection(self.collection).document(job_id)
doc_ref.update({
"worker_logs": firestore.ArrayUnion([{
"timestamp": datetime.now(timezone.utc).isoformat() + "Z",
"level": "INFO",
"worker": "test",
"message": message
}])
})
def _append_log_subcollection(self, job_id: str, message: str):
"""Add log to subcollection."""
import uuid
now = datetime.now(timezone.utc)
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
logs_ref.document(str(uuid.uuid4())).set({
"timestamp": now,
"level": "INFO",
"worker": "test",
"message": message,
"ttl_expiry": now + timedelta(days=30)
})
def test_document_size_comparison(self):
"""Compare document sizes between approaches."""
array_job_id = self._create_test_job(use_array=True)
sub_job_id = self._create_test_job(use_array=False)
num_logs = 100
large_message = "x" * 5000 # 5KB per log
# Add logs to both
print("\nAdding logs to both approaches...")
for i in range(num_logs):
self._append_log_array(array_job_id, f"Array log {i}: {large_message}")
self._append_log_subcollection(sub_job_id, f"Sub log {i}: {large_message}")
time.sleep(0.5)
# Get document sizes (approximate via raw data)
array_doc = self.db.collection(self.collection).document(array_job_id).get()
sub_doc = self.db.collection(self.collection).document(sub_job_id).get()
array_data = array_doc.to_dict()
sub_data = sub_doc.to_dict()
array_logs = array_data.get("worker_logs", [])
sub_logs = sub_data.get("worker_logs", [])
print(f"\nEmbedded array: {len(array_logs)} logs in document")
print(f"Subcollection: {len(sub_logs)} logs in document (0 expected)")
# Embedded array should have all logs in document
assert len(array_logs) == num_logs
# Subcollection should have empty array in main document
assert len(sub_logs) == 0
# Subcollection logs are in subcollection
sub_count = self.db.collection(self.collection).document(sub_job_id).collection("logs").count().get()[0][0].value
assert sub_count == num_logs
print(f"Subcollection logs stored separately: {sub_count}")
print("Subcollection approach keeps main document small")
print("Worker logs subcollection integration tests ready")
+6 -4
View File
@@ -460,16 +460,18 @@ class TestAudioSearchServiceDownload:
url="https://youtube.com/watch?v=abc123",
index=0,
)
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
service._fetcher.search.return_value = [mock_result]
# Ensure no remote client so we test the local cache path
service._remote_client = None
service.search("ABBA", "Waterloo")
with pytest.raises(DownloadError) as exc_info:
service.download(99, "/tmp")
assert "No cached result for index 99" in str(exc_info.value)
+244
View File
@@ -0,0 +1,244 @@
"""
Tests for DiscordNotificationService.
Tests cover:
- Service initialization
- Webhook URL validation
- Message posting
- Video notification posting
- Dry run mode
"""
import pytest
from unittest.mock import MagicMock, patch
from backend.services.discord_service import (
DiscordNotificationService,
get_discord_notification_service,
)
class TestDiscordNotificationServiceInit:
"""Test service initialization."""
def test_init_with_webhook_url(self):
"""Test initialization with valid webhook URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef123456"
service = DiscordNotificationService(webhook_url=url)
assert service.webhook_url == url
assert service.is_enabled() is True
def test_init_without_webhook_url(self):
"""Test initialization without webhook URL."""
service = DiscordNotificationService()
assert service.webhook_url is None
assert service.is_enabled() is False
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = DiscordNotificationService(dry_run=True)
assert service.dry_run is True
def test_init_with_invalid_webhook_url_raises(self):
"""Test that invalid webhook URL raises ValueError."""
with pytest.raises(ValueError) as exc_info:
DiscordNotificationService(webhook_url="https://example.com/webhook")
assert "Invalid Discord webhook URL" in str(exc_info.value)
def test_init_with_non_discord_url_raises(self):
"""Test that non-Discord URL raises ValueError."""
with pytest.raises(ValueError):
DiscordNotificationService(
webhook_url="https://slack.com/api/webhooks/123"
)
class TestDiscordNotificationServiceValidation:
"""Test webhook URL validation."""
def test_validate_valid_webhook_url(self):
"""Test that valid webhook URL passes validation."""
service = DiscordNotificationService()
# Should not raise
service._validate_webhook_url(
"https://discord.com/api/webhooks/123456789/abcdef"
)
def test_validate_invalid_webhook_url(self):
"""Test that invalid webhook URL raises ValueError."""
service = DiscordNotificationService()
with pytest.raises(ValueError) as exc_info:
service._validate_webhook_url("https://example.com/webhook")
assert "Invalid Discord webhook URL" in str(exc_info.value)
def test_validate_strips_whitespace(self):
"""Test that whitespace is stripped from URL."""
service = DiscordNotificationService()
# Should not raise - whitespace is stripped
service._validate_webhook_url(
" https://discord.com/api/webhooks/123456789/abcdef "
)
class TestDiscordNotificationServicePostMessage:
"""Test message posting."""
@patch("backend.services.discord_service.requests.post")
def test_post_message_success(self, mock_post):
"""Test successful message posting."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_message("Test message")
assert result is True
mock_post.assert_called_once_with(
url,
json={"content": "Test message"},
timeout=30
)
@patch("backend.services.discord_service.requests.post")
def test_post_message_with_custom_webhook(self, mock_post):
"""Test message posting with custom webhook URL."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
service = DiscordNotificationService()
custom_url = "https://discord.com/api/webhooks/987654321/fedcba"
result = service.post_message("Test message", webhook_url=custom_url)
assert result is True
mock_post.assert_called_once_with(
custom_url,
json={"content": "Test message"},
timeout=30
)
def test_post_message_no_webhook_raises(self):
"""Test that posting without webhook URL raises ValueError."""
service = DiscordNotificationService()
with pytest.raises(ValueError) as exc_info:
service.post_message("Test message")
assert "No Discord webhook URL provided" in str(exc_info.value)
def test_post_message_dry_run(self):
"""Test message posting in dry run mode."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url, dry_run=True)
result = service.post_message("Test message")
assert result is True
# In dry run mode, no actual request should be made
@patch("backend.services.discord_service.requests.post")
def test_post_message_http_error(self, mock_post):
"""Test handling of HTTP errors."""
import requests as req
mock_response = MagicMock()
mock_response.raise_for_status.side_effect = req.HTTPError("404 Not Found")
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
with pytest.raises(req.HTTPError):
service.post_message("Test message")
class TestDiscordNotificationServiceVideoNotification:
"""Test video notification posting."""
@patch("backend.services.discord_service.requests.post")
def test_post_video_notification_success(self, mock_post):
"""Test successful video notification."""
mock_response = MagicMock()
mock_response.raise_for_status = MagicMock()
mock_post.return_value = mock_response
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification(
"https://www.youtube.com/watch?v=abc123"
)
assert result is True
mock_post.assert_called_once()
call_args = mock_post.call_args
assert "New upload:" in call_args[1]["json"]["content"]
assert "abc123" in call_args[1]["json"]["content"]
def test_post_video_notification_no_youtube_url(self):
"""Test video notification with empty YouTube URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification("")
assert result is False
def test_post_video_notification_none_youtube_url(self):
"""Test video notification with None YouTube URL."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url)
result = service.post_video_notification(None)
assert result is False
def test_post_video_notification_dry_run(self):
"""Test video notification in dry run mode."""
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = DiscordNotificationService(webhook_url=url, dry_run=True)
result = service.post_video_notification(
"https://www.youtube.com/watch?v=abc123"
)
assert result is True
class TestGetDiscordNotificationService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.discord_service as module
module._discord_notification_service = None
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = get_discord_notification_service(webhook_url=url)
assert service is not None
assert isinstance(service, DiscordNotificationService)
assert service.webhook_url == url
def test_get_service_without_webhook(self):
"""Test factory function without webhook URL."""
import backend.services.discord_service as module
module._discord_notification_service = None
service = get_discord_notification_service()
assert service is not None
assert service.webhook_url is None
assert service.is_enabled() is False
def test_get_service_with_dry_run(self):
"""Test factory function with dry run mode."""
import backend.services.discord_service as module
module._discord_notification_service = None
url = "https://discord.com/api/webhooks/123456789/abcdef"
service = get_discord_notification_service(webhook_url=url, dry_run=True)
assert service.dry_run is True
+412
View File
@@ -0,0 +1,412 @@
"""
Tests for Encoding Interface.
Tests cover:
- EncodingInput and EncodingOutput dataclasses
- LocalEncodingBackend implementation
- Backend factory function
"""
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from backend.services.encoding_interface import (
EncodingInput,
EncodingOutput,
EncodingBackend,
LocalEncodingBackend,
GCEEncodingBackend,
get_encoding_backend,
)
class TestEncodingInput:
"""Test EncodingInput dataclass."""
def test_required_fields(self):
"""Test that required fields must be provided."""
input_config = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert input_config.title_video_path == "/path/title.mov"
assert input_config.end_video_path is None
def test_all_fields(self):
"""Test all fields including optional ones."""
input_config = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
end_video_path="/path/end.mov",
artist="Test Artist",
title="Test Title",
brand_code="NOMAD-1234",
output_dir="/output",
options={"quality": "high"}
)
assert input_config.artist == "Test Artist"
assert input_config.brand_code == "NOMAD-1234"
assert input_config.options["quality"] == "high"
class TestEncodingOutput:
"""Test EncodingOutput dataclass."""
def test_success_output(self):
"""Test successful output."""
output = EncodingOutput(
success=True,
lossless_4k_mp4_path="/output/video.mp4",
encoding_backend="local"
)
assert output.success is True
assert output.error_message is None
def test_failure_output(self):
"""Test failure output."""
output = EncodingOutput(
success=False,
error_message="Encoding failed",
encoding_backend="gce"
)
assert output.success is False
assert output.error_message == "Encoding failed"
def test_output_files_dict(self):
"""Test output_files dictionary."""
output = EncodingOutput(
success=True,
output_files={
"lossless_4k_mp4": "/output/lossless.mp4",
"720p_mp4": "/output/720p.mp4"
}
)
assert "lossless_4k_mp4" in output.output_files
assert output.output_files["720p_mp4"] == "/output/720p.mp4"
class TestLocalEncodingBackend:
"""Test LocalEncodingBackend implementation."""
def test_name(self):
"""Test backend name."""
backend = LocalEncodingBackend()
assert backend.name == "local"
def test_init_with_dry_run(self):
"""Test initialization with dry run."""
backend = LocalEncodingBackend(dry_run=True)
assert backend.dry_run is True
@pytest.mark.asyncio
@patch("subprocess.run")
async def test_is_available_success(self, mock_run):
"""Test availability check when FFmpeg is installed."""
mock_run.return_value = MagicMock(returncode=0)
backend = LocalEncodingBackend()
available = await backend.is_available()
assert available is True
@pytest.mark.asyncio
@patch("subprocess.run")
async def test_is_available_not_found(self, mock_run):
"""Test availability check when FFmpeg is not installed."""
mock_run.side_effect = FileNotFoundError()
backend = LocalEncodingBackend()
available = await backend.is_available()
assert available is False
@pytest.mark.asyncio
@patch.object(LocalEncodingBackend, "is_available")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_get_status(self, mock_get_service, mock_is_available):
"""Test status retrieval."""
mock_is_available.return_value = True
mock_service = MagicMock()
mock_service.hwaccel_available = True
mock_service.video_encoder = "h264_nvenc"
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
status = await backend.get_status()
assert status["backend"] == "local"
assert status["available"] is True
@pytest.mark.asyncio
@patch("asyncio.to_thread")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_encode_success(self, mock_get_service, mock_to_thread):
"""Test successful encoding."""
from backend.services.local_encoding_service import EncodingResult
mock_result = EncodingResult(
success=True,
output_files={"key": "/path/file.mp4"}
)
mock_to_thread.return_value = mock_result
mock_service = MagicMock()
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test Artist",
title="Test Title",
output_dir="/output"
)
output = await backend.encode(input_config)
assert output.success is True
assert output.encoding_backend == "local"
assert output.encoding_time_seconds is not None
@pytest.mark.asyncio
@patch("asyncio.to_thread")
@patch.object(LocalEncodingBackend, "_get_service")
async def test_encode_failure(self, mock_get_service, mock_to_thread):
"""Test encoding failure."""
from backend.services.local_encoding_service import EncodingResult
mock_result = EncodingResult(
success=False,
output_files={},
error="FFmpeg failed"
)
mock_to_thread.return_value = mock_result
mock_service = MagicMock()
mock_get_service.return_value = mock_service
backend = LocalEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test",
title="Test",
)
output = await backend.encode(input_config)
assert output.success is False
assert output.error_message == "FFmpeg failed"
class TestGCEEncodingBackend:
"""Test GCEEncodingBackend implementation."""
def test_name(self):
"""Test backend name."""
backend = GCEEncodingBackend()
assert backend.name == "gce"
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_is_available_enabled(self, mock_get_service):
"""Test availability when GCE is enabled."""
mock_service = MagicMock()
mock_service.is_enabled = True
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
available = await backend.is_available()
assert available is True
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_is_available_disabled(self, mock_get_service):
"""Test availability when GCE is disabled."""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
available = await backend.is_available()
assert available is False
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_missing_gcs_paths(self, mock_get_service):
"""Test encoding fails without GCS paths."""
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
# Missing GCS paths in options
)
output = await backend.encode(input_config)
assert output.success is False
assert "gcs_path" in output.error_message.lower()
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_success(self, mock_get_service):
"""Test successful GCE encoding."""
mock_service = MagicMock()
mock_service.encode_videos = AsyncMock(return_value={
"status": "complete",
"output_files": {
"mp4_4k_lossless": "gs://bucket/output/lossless.mp4",
"mp4_720p": "gs://bucket/output/720p.mp4",
}
})
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
artist="Test Artist",
title="Test Title",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
output = await backend.encode(input_config)
assert output.success is True
assert output.encoding_backend == "gce"
mock_service.encode_videos.assert_called_once()
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_failure(self, mock_get_service):
"""Test GCE encoding failure handling."""
mock_service = MagicMock()
mock_service.encode_videos = AsyncMock(side_effect=Exception("GCE worker error"))
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
output = await backend.encode(input_config)
assert output.success is False
assert "GCE worker error" in output.error_message
@patch.object(GCEEncodingBackend, "_get_service")
@pytest.mark.asyncio
async def test_encode_handles_list_result(self, mock_get_service):
"""Test GCE encoding handles list response gracefully.
This would have caught: 'list' object has no attribute 'get' error
when GCE worker returns a list instead of a dict.
"""
mock_service = MagicMock()
# Simulate GCE worker returning a list instead of dict
mock_service.encode_videos = AsyncMock(return_value=[
{"output_files": {"mp4_4k_lossless": "gs://bucket/output/lossless.mp4"}}
])
mock_get_service.return_value = mock_service
backend = GCEEncodingBackend()
input_config = EncodingInput(
title_video_path="/input/title.mov",
karaoke_video_path="/input/karaoke.mov",
instrumental_audio_path="/input/audio.flac",
options={
"job_id": "test-job",
"input_gcs_path": "gs://bucket/input/",
"output_gcs_path": "gs://bucket/output/",
}
)
# This should not raise an error
output = await backend.encode(input_config)
# Should still succeed by extracting from the list
assert output.success is True
assert output.lossless_4k_mp4_path == "gs://bucket/output/lossless.mp4"
class TestGetEncodingBackend:
"""Test encoding backend factory function."""
def test_get_local_backend(self):
"""Test getting local backend."""
backend = get_encoding_backend("local")
assert isinstance(backend, LocalEncodingBackend)
assert backend.name == "local"
@patch.object(GCEEncodingBackend, "_get_service")
def test_get_auto_backend_gce_disabled(self, mock_get_service):
"""Test getting auto backend falls back to local when GCE disabled."""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
backend = get_encoding_backend("auto")
assert isinstance(backend, LocalEncodingBackend)
def test_get_local_backend_with_options(self):
"""Test getting local backend with options."""
backend = get_encoding_backend("local", dry_run=True)
assert backend.dry_run is True
def test_get_gce_backend(self):
"""Test getting GCE backend."""
backend = get_encoding_backend("gce")
assert isinstance(backend, GCEEncodingBackend)
assert backend.name == "gce"
def test_get_unknown_backend_raises(self):
"""Test that unknown backend raises ValueError."""
with pytest.raises(ValueError) as exc_info:
get_encoding_backend("unknown")
assert "Unknown encoding backend type" in str(exc_info.value)
def test_get_gce_backend_with_options(self):
"""Test getting GCE backend with common options like dry_run.
This ensures all backends accept the same kwargs, preventing
errors when get_encoding_backend() passes **kwargs to different backends.
"""
# This would have caught: GCEEncodingBackend.__init__() got an unexpected keyword argument 'dry_run'
backend = get_encoding_backend("gce", dry_run=True)
assert isinstance(backend, GCEEncodingBackend)
assert backend.dry_run is True
@patch.object(GCEEncodingBackend, "_get_service")
def test_all_backends_accept_dry_run(self, mock_get_service):
"""Test that all backend types accept dry_run parameter.
This is an integration test to ensure the factory function
can pass dry_run to any backend without TypeError.
"""
mock_service = MagicMock()
mock_service.is_enabled = False
mock_get_service.return_value = mock_service
for backend_type in ["local", "gce", "auto"]:
# This should not raise TypeError
backend = get_encoding_backend(backend_type, dry_run=True)
assert backend is not None
+2 -1
View File
@@ -1128,7 +1128,8 @@ class TestFinaliseOnlyModels:
assert request.enable_txt is None
assert request.brand_prefix is None
assert request.keep_brand_code is None
assert request.enable_youtube_upload is False
# YouTube upload default is None (server applies default_enable_youtube_upload)
assert request.enable_youtube_upload is None
assert request.dropbox_path is None
assert request.gdrive_folder_id is None
@@ -0,0 +1,583 @@
"""
Regression tests for job creation issues discovered 2024-12-31.
These tests ensure:
1. user_email is properly extracted from AuthResult and set on jobs
2. URL jobs trigger workers sequentially (audio first, then lyrics)
3. Audio search downloads work without relying on in-memory cache
4. Transcription has proper timeout handling
See docs/archive/2024-12-31-job-failure-investigation.md for details.
"""
import asyncio
import pytest
from unittest.mock import Mock, AsyncMock, patch, MagicMock
from datetime import datetime, timezone
from backend.services.auth_service import AuthResult, UserType
class TestUserEmailExtraction:
"""
Issue 1: user_email not being set on jobs.
Root cause: Job creation endpoints used require_auth but didn't extract
auth_result.user_email and set it on the job.
These tests verify that authenticated user's email is properly set on jobs.
"""
@pytest.fixture
def mock_auth_result_with_email(self):
"""AuthResult for a logged-in user with email."""
return AuthResult(
is_valid=True,
user_type=UserType.UNLIMITED, # Session-based users typically have UNLIMITED
remaining_uses=-1,
message="Valid session",
user_email="testuser@example.com",
is_admin=False,
)
@pytest.fixture
def mock_auth_result_admin_token(self):
"""AuthResult for admin token without email."""
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=-1,
message="Valid admin token",
user_email=None,
is_admin=True,
)
def test_auth_result_has_user_email_field(self):
"""AuthResult must have user_email field for job association."""
result = AuthResult(
is_valid=True,
user_type=UserType.UNLIMITED,
remaining_uses=-1,
message="test",
user_email="user@example.com",
)
assert hasattr(result, 'user_email')
assert result.user_email == "user@example.com"
def test_auth_result_user_email_can_be_none(self):
"""AuthResult user_email can be None for token-based auth."""
result = AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=-1,
message="test",
user_email=None,
)
assert result.user_email is None
def test_upload_endpoint_extracts_user_email_from_auth(self):
"""
/jobs/upload endpoint must set user_email from authenticated user.
This verifies the code pattern by inspecting the source.
Integration tests verify the full behavior.
"""
from backend.api.routes import file_upload
import inspect
# Get the source code of upload_and_create_job
source = inspect.getsource(file_upload.upload_and_create_job)
# Verify the endpoint extracts user_email from auth_result
assert 'auth_result.user_email' in source, \
"upload_and_create_job must extract user_email from auth_result"
assert 'effective_user_email' in source, \
"upload_and_create_job must use effective_user_email pattern"
assert 'user_email=effective_user_email' in source or 'user_email=' in source, \
"upload_and_create_job must pass user_email to JobCreate"
def test_all_job_creation_endpoints_extract_user_email(self):
"""
All job creation endpoints must extract user_email from AuthResult.
This is a regression guard for Issue 1 (user_email not being set).
"""
from backend.api.routes import file_upload, audio_search
import inspect
endpoints_to_check = [
(file_upload, 'upload_and_create_job'),
(file_upload, 'create_job_from_url'),
(audio_search, 'search_audio'),
]
for module, func_name in endpoints_to_check:
func = getattr(module, func_name)
source = inspect.getsource(func)
assert 'auth_result.user_email' in source, \
f"{func_name} must extract user_email from auth_result"
@pytest.mark.asyncio
async def test_create_from_url_endpoint_sets_user_email(self, mock_auth_result_with_email):
"""
/jobs/create-from-url endpoint must set user_email from authenticated user.
"""
from backend.api.routes.file_upload import create_job_from_url, CreateJobFromUrlRequest
mock_request = Mock()
mock_request.headers = {}
mock_request.client = Mock(host="127.0.0.1")
mock_request.url = Mock(path="/api/jobs/create-from-url")
mock_background_tasks = Mock()
body = CreateJobFromUrlRequest(
url="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
artist="Rick Astley",
title="Never Gonna Give You Up",
)
with patch('backend.api.routes.file_upload.job_manager') as mock_jm, \
patch('backend.api.routes.file_upload.worker_service') as mock_worker, \
patch('backend.api.routes.file_upload.get_credential_manager') as mock_cred:
mock_job = Mock()
mock_job.job_id = "test-job-456"
mock_jm.create_job.return_value = mock_job
mock_cred.return_value.check_youtube_credentials.return_value = Mock(status=Mock(value="valid"))
await create_job_from_url(
request=mock_request,
background_tasks=mock_background_tasks,
body=body,
auth_result=mock_auth_result_with_email,
)
mock_jm.create_job.assert_called_once()
job_create_arg = mock_jm.create_job.call_args[0][0]
assert job_create_arg.user_email == "testuser@example.com", \
"create-from-url must set user_email from AuthResult"
@pytest.mark.asyncio
async def test_audio_search_endpoint_sets_user_email(self, mock_auth_result_with_email):
"""
/audio-search/search endpoint must set user_email from authenticated user.
"""
from backend.api.routes.audio_search import search_audio, AudioSearchRequest
mock_request = Mock()
mock_request.headers = {}
mock_request.client = Mock(host="127.0.0.1")
mock_request.url = Mock(path="/api/audio-search/search")
mock_background_tasks = Mock()
body = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
)
with patch('backend.api.routes.audio_search.job_manager') as mock_jm, \
patch('backend.api.routes.audio_search.get_audio_search_service') as mock_search, \
patch('backend.api.routes.audio_search.get_credential_manager') as mock_cred:
mock_job = Mock()
mock_job.job_id = "test-job-789"
mock_job.state_data = {}
mock_jm.create_job.return_value = mock_job
mock_jm.get_job.return_value = mock_job
mock_cred.return_value.check_youtube_credentials.return_value = Mock(status=Mock(value="valid"))
mock_search.return_value.search.return_value = []
await search_audio(
request=mock_request,
background_tasks=mock_background_tasks,
body=body,
auth_result=mock_auth_result_with_email,
)
mock_jm.create_job.assert_called_once()
job_create_arg = mock_jm.create_job.call_args[0][0]
assert job_create_arg.user_email == "testuser@example.com", \
"audio-search must set user_email from AuthResult"
def test_effective_user_email_prefers_auth_over_form(self):
"""
When both auth_result.user_email and form user_email are provided,
the authenticated user's email should take precedence.
"""
auth_email = "authenticated@example.com"
form_email = "form@example.com"
# This is the logic used in the endpoints
effective_user_email = auth_email or form_email
assert effective_user_email == auth_email, \
"Authenticated user's email should take precedence over form parameter"
def test_effective_user_email_falls_back_to_form(self):
"""
When auth_result.user_email is None (e.g., admin token),
form user_email should be used as fallback.
"""
auth_email = None
form_email = "form@example.com"
effective_user_email = auth_email or form_email
assert effective_user_email == form_email, \
"Form parameter should be used when auth has no email"
class TestUrlJobWorkerSequencing:
"""
Issue 2: YouTube URL download race condition.
Root cause: Both audio and lyrics workers were triggered in parallel.
For URL jobs, lyrics worker would timeout waiting for audio to download.
These tests verify that URL jobs trigger workers sequentially.
"""
@pytest.mark.asyncio
async def test_url_job_triggers_only_audio_worker_initially(self):
"""
create-from-url must only trigger audio worker, not lyrics worker.
The audio worker will trigger lyrics after download completes.
"""
from backend.api.routes.file_upload import _trigger_audio_worker_only
with patch('backend.api.routes.file_upload.worker_service') as mock_ws:
mock_ws.trigger_audio_worker = AsyncMock()
mock_ws.trigger_lyrics_worker = AsyncMock()
await _trigger_audio_worker_only("test-job-id")
mock_ws.trigger_audio_worker.assert_called_once_with("test-job-id")
mock_ws.trigger_lyrics_worker.assert_not_called()
@pytest.mark.asyncio
async def test_parallel_worker_triggers_both_workers(self):
"""
_trigger_workers_parallel should trigger both workers (for uploaded files).
"""
from backend.api.routes.file_upload import _trigger_workers_parallel
with patch('backend.api.routes.file_upload.worker_service') as mock_ws:
mock_ws.trigger_audio_worker = AsyncMock()
mock_ws.trigger_lyrics_worker = AsyncMock()
await _trigger_workers_parallel("test-job-id")
mock_ws.trigger_audio_worker.assert_called_once_with("test-job-id")
mock_ws.trigger_lyrics_worker.assert_called_once_with("test-job-id")
@pytest.mark.asyncio
async def test_audio_worker_triggers_lyrics_after_url_download(self):
"""
Audio worker must trigger lyrics worker after successful URL download.
"""
from backend.workers.audio_worker import _trigger_lyrics_worker_after_url_download
# Mock at the source module where it's imported from
with patch('backend.services.worker_service.get_worker_service') as mock_get_ws:
mock_ws = Mock()
mock_ws.trigger_lyrics_worker = AsyncMock()
mock_get_ws.return_value = mock_ws
await _trigger_lyrics_worker_after_url_download("test-job-id")
mock_ws.trigger_lyrics_worker.assert_called_once_with("test-job-id")
@pytest.mark.asyncio
async def test_audio_worker_lyrics_trigger_handles_errors_gracefully(self):
"""
If lyrics worker trigger fails, audio processing should continue.
"""
from backend.workers.audio_worker import _trigger_lyrics_worker_after_url_download
# Mock at the source module where it's imported from
with patch('backend.services.worker_service.get_worker_service') as mock_get_ws:
mock_ws = Mock()
mock_ws.trigger_lyrics_worker = AsyncMock(side_effect=Exception("Network error"))
mock_get_ws.return_value = mock_ws
# Should not raise exception
await _trigger_lyrics_worker_after_url_download("test-job-id")
class TestAudioSearchCacheIndependence:
"""
Issue 3: Audio search cache not persisting across Cloud Run instances.
Root cause: AudioSearchService used in-memory cache that doesn't persist
across horizontally scaled instances.
These tests verify downloads work using state_data, not in-memory cache.
"""
def test_search_results_stored_in_job_state_data(self):
"""
Search results must be stored in job.state_data for persistence.
This tests the Job model's ability to store search results.
"""
from backend.models.job import Job, JobStatus
job = Job(
job_id="test-123",
artist="Test Artist",
title="Test Song",
status=JobStatus.PENDING,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
state_data={
'audio_search_results': [
{
'provider': 'YouTube',
'title': 'Test Song',
'artist': 'Test Artist',
'url': 'https://youtube.com/watch?v=abc123',
'source_id': 'abc123',
}
]
}
)
assert 'audio_search_results' in job.state_data
assert len(job.state_data['audio_search_results']) == 1
assert job.state_data['audio_search_results'][0]['url'] == 'https://youtube.com/watch?v=abc123'
def test_audio_search_endpoint_stores_results_in_state_data(self):
"""
Verify that the search_audio code path stores results in job.state_data.
This tests that the audio_search route has the correct pattern for
persisting search results. The actual integration is tested elsewhere.
"""
# Verify the import and module structure exists
from backend.api.routes import audio_search
import inspect
# Get the source code of search_audio
source = inspect.getsource(audio_search.search_audio)
# Verify the endpoint stores results in state_data
# The code uses job_manager.update_job with state_data dict
assert 'audio_search_results' in source, \
"search_audio must store results under 'audio_search_results' key"
assert 'state_data' in source, \
"search_audio must use state_data for persistence"
def test_download_code_has_youtube_direct_download_branch(self):
"""
Verify that _download_and_start_processing has the YouTube direct download path.
This checks that the code path exists for downloading YouTube audio directly
using the URL from state_data, avoiding the in-memory cache dependency.
"""
from backend.api.routes import audio_search
import inspect
# Get the source of _download_and_start_processing
source = inspect.getsource(audio_search._download_and_start_processing)
# Verify the YouTube direct download branch exists
assert "source_name == 'YouTube'" in source, \
"_download_and_start_processing must check for YouTube source"
assert 'download_url' in source, \
"_download_and_start_processing must use download_url from state_data"
assert 'download_from_url' in source, \
"_download_and_start_processing must call download_from_url for YouTube"
def test_youtube_download_branch_avoids_cache_dependency(self):
"""
Verify that the YouTube download path uses URL from state_data,
not the in-memory cache that doesn't persist across Cloud Run instances.
"""
from backend.api.routes import audio_search
import inspect
source = inspect.getsource(audio_search._download_and_start_processing)
# The fix specifically uses selected.get('url') which comes from state_data
# rather than audio_search_service cache
assert "selected.get('url')" in source or "download_url" in source, \
"YouTube download must use URL from selected result (state_data)"
# Verify it doesn't rely on cache for YouTube
# The branch is: elif source_name == 'YouTube' and download_url:
assert "YouTube" in source and "download_url" in source, \
"YouTube branch must check for download_url availability"
class TestTranscriptionTimeout:
"""
Issue 4: Jobs stuck in downloading state.
Root cause: Lyrics worker's AudioShake transcription could hang forever
without proper timeout, leaving jobs stuck.
These tests verify timeout handling for transcription.
"""
def test_transcription_timeout_constant_exists(self):
"""
Verify transcription timeout constant is defined in lyrics_worker.
"""
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
assert TRANSCRIPTION_TIMEOUT_SECONDS > 0
assert TRANSCRIPTION_TIMEOUT_SECONDS >= 300, "Timeout should be at least 5 minutes for long songs"
assert TRANSCRIPTION_TIMEOUT_SECONDS <= 1500, "Timeout shouldn't be more than 25 minutes"
def test_transcription_timeout_value(self):
"""
Verify the specific timeout value (20 minutes = 1200 seconds).
This accounts for:
- Cloud Run cold start / worker initialization (1-5 min)
- AudioShake transcription (1-2 min)
- spaCy model loading for correction (2-3 min on cold start)
- Agentic AI correction (1-3 min)
"""
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
assert TRANSCRIPTION_TIMEOUT_SECONDS == 1200, "Transcription timeout should be 20 minutes (1200 seconds)"
@pytest.mark.asyncio
async def test_asyncio_wait_for_raises_timeout_error(self):
"""
Verify asyncio.wait_for properly raises TimeoutError.
This is a sanity check for the timeout mechanism we use.
"""
async def slow_operation():
await asyncio.sleep(10)
return "completed"
with pytest.raises(asyncio.TimeoutError):
await asyncio.wait_for(slow_operation(), timeout=0.1)
@pytest.mark.asyncio
async def test_lyrics_worker_timeout_converts_to_descriptive_error(self):
"""
Verify that timeout is converted to a descriptive exception message.
Tests the actual error conversion logic pattern used in lyrics_worker.
"""
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
# This simulates what happens in lyrics_worker when timeout occurs
error_message = None
try:
# Simulate the timeout handling pattern from lyrics_worker
raise asyncio.TimeoutError()
except asyncio.TimeoutError:
error_message = f"Transcription timed out after {TRANSCRIPTION_TIMEOUT_SECONDS} seconds"
assert error_message is not None
assert "timed out" in error_message.lower()
assert "1200" in error_message
def test_lyrics_worker_exception_marks_job_failed(self):
"""
Any exception in lyrics worker should mark job as failed.
"""
from backend.models.job import JobStatus
# Simulate the exception handling in lyrics_worker
job_status = JobStatus.DOWNLOADING
error_occurred = True
if error_occurred:
# This is what happens in the except block
job_status = JobStatus.FAILED
error_message = "Lyrics transcription failed: Transcription timed out after 1200 seconds"
assert job_status == JobStatus.FAILED
assert "timed out" in error_message.lower()
class TestJobOwnershipFiltering:
"""
Additional tests for job ownership and filtering.
These verify that jobs are properly associated with users
and can be filtered by user_email.
"""
def test_job_model_has_user_email_field(self):
"""Job model must have user_email field."""
from backend.models.job import Job, JobStatus
job = Job(
job_id="test-123",
artist="Test",
title="Test",
status=JobStatus.PENDING,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
user_email="user@example.com",
)
assert hasattr(job, 'user_email')
assert job.user_email == "user@example.com"
def test_job_create_model_has_user_email_field(self):
"""JobCreate model must have user_email field."""
from backend.models.job import JobCreate
job_create = JobCreate(
artist="Test",
title="Test",
user_email="user@example.com",
)
assert hasattr(job_create, 'user_email')
assert job_create.user_email == "user@example.com"
def test_job_create_user_email_is_optional(self):
"""JobCreate user_email should be optional for backward compatibility."""
from backend.models.job import JobCreate
job_create = JobCreate(
artist="Test",
title="Test",
)
assert job_create.user_email is None
def test_jobs_can_be_filtered_by_user_email(self):
"""Jobs should be filterable by user_email."""
from backend.models.job import Job, JobStatus
jobs = [
Job(
job_id="job-1",
artist="Test",
title="Test 1",
status=JobStatus.COMPLETE,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
user_email="user1@example.com",
),
Job(
job_id="job-2",
artist="Test",
title="Test 2",
status=JobStatus.COMPLETE,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
user_email="user2@example.com",
),
Job(
job_id="job-3",
artist="Test",
title="Test 3",
status=JobStatus.COMPLETE,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
user_email="user1@example.com",
),
]
user1_jobs = [j for j in jobs if j.user_email == "user1@example.com"]
assert len(user1_jobs) == 2
assert all(j.user_email == "user1@example.com" for j in user1_jobs)
@@ -0,0 +1,423 @@
"""
Tests for LocalEncodingService.
Tests cover:
- Service initialization
- Hardware acceleration detection
- FFmpeg command execution
- Individual encoding methods
- Full encoding pipeline
- Dry run mode
"""
import os
import tempfile
import pytest
from unittest.mock import MagicMock, patch, call
from backend.services.local_encoding_service import (
LocalEncodingService,
EncodingConfig,
EncodingResult,
get_local_encoding_service,
)
class TestLocalEncodingServiceInit:
"""Test service initialization."""
def test_init_default_values(self):
"""Test default initialization."""
service = LocalEncodingService()
assert service.dry_run is False
assert "ffmpeg" in service._ffmpeg_base_command
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = LocalEncodingService(dry_run=True)
assert service.dry_run is True
def test_init_with_debug_logging(self):
"""Test initialization with debug logging level."""
import logging
service = LocalEncodingService(log_level=logging.DEBUG)
assert "-loglevel verbose" in service._ffmpeg_base_command
def test_init_with_info_logging(self):
"""Test initialization with info logging level."""
import logging
service = LocalEncodingService(log_level=logging.INFO)
assert "-loglevel fatal" in service._ffmpeg_base_command
class TestLocalEncodingServiceHWAccel:
"""Test hardware acceleration detection."""
@patch("subprocess.run")
def test_detect_nvenc_available(self, mock_run):
"""Test NVENC detection when available."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalEncodingService()
# Force detection
service._detect_and_set_hwaccel()
assert service._hwaccel_available is True
assert service._video_encoder == "h264_nvenc"
assert service._scale_filter == "scale_cuda"
@patch("subprocess.run")
def test_detect_no_hwaccel(self, mock_run):
"""Test fallback when no hardware acceleration available."""
import subprocess
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
service = LocalEncodingService()
# Force detection
service._detect_and_set_hwaccel()
assert service._hwaccel_available is False
assert service._video_encoder == "libx264"
assert service._scale_filter == "scale"
def test_nvenc_quality_settings(self):
"""Test NVENC quality settings for different presets."""
service = LocalEncodingService()
service._hwaccel_available = True
lossless = service._get_nvenc_quality_settings("lossless")
assert "cq 0" in lossless
medium = service._get_nvenc_quality_settings("medium")
assert "p4" in medium
def test_nvenc_quality_settings_disabled(self):
"""Test NVENC quality settings when hwaccel is disabled."""
service = LocalEncodingService()
service._hwaccel_available = False
settings = service._get_nvenc_quality_settings("lossless")
assert settings == ""
class TestLocalEncodingServiceExecuteCommand:
"""Test command execution."""
@patch("subprocess.run")
def test_execute_command_success(self, mock_run):
"""Test successful command execution."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command")
assert result is True
mock_run.assert_called_once()
@patch("subprocess.run")
def test_execute_command_failure(self, mock_run):
"""Test command execution failure."""
import subprocess
mock_run.side_effect = subprocess.CalledProcessError(
1, "test", stderr="Error message"
)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command")
assert result is False
@patch("subprocess.run")
def test_execute_command_timeout(self, mock_run):
"""Test command execution timeout."""
import subprocess
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
service = LocalEncodingService()
result = service._execute_command("echo test", "Test command", timeout=30)
assert result is False
def test_execute_command_dry_run(self):
"""Test command execution in dry run mode."""
service = LocalEncodingService(dry_run=True)
result = service._execute_command("echo test", "Test command")
assert result is True
# No actual subprocess should be called in dry run mode
class TestLocalEncodingServiceEncodingMethods:
"""Test individual encoding methods."""
@patch.object(LocalEncodingService, "_execute_command")
def test_remux_with_instrumental(self, mock_execute):
"""Test remuxing with instrumental audio."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.remux_with_instrumental(
"/input/video.mov",
"/input/audio.flac",
"/output/video.mp4"
)
assert result is True
mock_execute.assert_called_once()
call_args = mock_execute.call_args[0][0]
assert "/input/video.mov" in call_args
assert "/input/audio.flac" in call_args
assert "-map 0:v -map 1:a" in call_args
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_convert_mov_to_mp4(self, mock_execute):
"""Test MOV to MP4 conversion."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.convert_mov_to_mp4(
"/input/video.mov",
"/output/video.mp4"
)
assert result is True
mock_execute.assert_called_once()
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_lossless_mp4_without_end(self, mock_execute):
"""Test lossless 4K MP4 encoding without end credits."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossless_mp4(
"/input/title.mov",
"/input/karaoke.mp4",
"/output/lossless.mp4"
)
assert result is True
mock_execute.assert_called_once()
call_args = mock_execute.call_args
assert "concat=n=2" in call_args[0][0] # GPU command
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_lossless_mp4_with_end(self, mock_execute):
"""Test lossless 4K MP4 encoding with end credits."""
mock_execute.return_value = True
with tempfile.TemporaryDirectory() as tmpdir:
end_file = os.path.join(tmpdir, "end.mov")
with open(end_file, "w") as f:
f.write("fake video")
service = LocalEncodingService()
result = service.encode_lossless_mp4(
"/input/title.mov",
"/input/karaoke.mp4",
"/output/lossless.mp4",
end_video=end_file
)
assert result is True
call_args = mock_execute.call_args
assert "concat=n=3" in call_args[0][0] # 3 videos
@patch.object(LocalEncodingService, "_execute_command")
def test_encode_lossy_mp4(self, mock_execute):
"""Test lossy 4K MP4 encoding."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossy_mp4(
"/input/lossless.mp4",
"/output/lossy.mp4"
)
assert result is True
call_args = mock_execute.call_args[0][0]
assert "-c:v copy" in call_args
assert "aac" in call_args.lower()
@patch.object(LocalEncodingService, "_execute_command")
def test_encode_lossless_mkv(self, mock_execute):
"""Test MKV encoding with FLAC audio."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_lossless_mkv(
"/input/lossless.mp4",
"/output/video.mkv"
)
assert result is True
call_args = mock_execute.call_args[0][0]
assert "-c:v copy" in call_args
assert "-c:a flac" in call_args
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
def test_encode_720p(self, mock_execute):
"""Test 720p encoding."""
mock_execute.return_value = True
service = LocalEncodingService()
result = service.encode_720p(
"/input/lossless.mp4",
"/output/720p.mp4"
)
assert result is True
call_args = mock_execute.call_args
# Should contain scale filter
assert "1280:720" in call_args[0][0] or "1280:720" in call_args[0][1]
class TestLocalEncodingServiceFullPipeline:
"""Test full encoding pipeline."""
@patch.object(LocalEncodingService, "remux_with_instrumental")
@patch.object(LocalEncodingService, "convert_mov_to_mp4")
@patch.object(LocalEncodingService, "encode_lossless_mp4")
@patch.object(LocalEncodingService, "encode_lossy_mp4")
@patch.object(LocalEncodingService, "encode_lossless_mkv")
@patch.object(LocalEncodingService, "encode_720p")
def test_encode_all_formats_success(
self, mock_720p, mock_mkv, mock_lossy, mock_lossless, mock_convert, mock_remux
):
"""Test successful full encoding pipeline."""
mock_remux.return_value = True
mock_convert.return_value = True
mock_lossless.return_value = True
mock_lossy.return_value = True
mock_mkv.return_value = True
mock_720p.return_value = True
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_with_vocals_mp4="/output/with_vocals.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
output_lossy_4k_mp4="/output/lossy_4k.mp4",
output_lossless_mkv="/output/lossless.mkv",
output_720p_mp4="/output/720p.mp4",
)
result = service.encode_all_formats(config)
assert result.success is True
assert "karaoke_mp4" in result.output_files
assert "lossless_4k_mp4" in result.output_files
assert "720p_mp4" in result.output_files
@patch.object(LocalEncodingService, "remux_with_instrumental")
def test_encode_all_formats_failure_early(self, mock_remux):
"""Test encoding pipeline failure at early step."""
mock_remux.return_value = False
service = LocalEncodingService()
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mov",
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
)
result = service.encode_all_formats(config)
assert result.success is False
assert "Failed to remux" in result.error
def test_encode_all_formats_dry_run(self):
"""Test encoding pipeline in dry run mode."""
service = LocalEncodingService(dry_run=True)
config = EncodingConfig(
title_video="/input/title.mov",
karaoke_video="/input/karaoke.mp4", # Already MP4
instrumental_audio="/input/instrumental.flac",
output_karaoke_mp4="/output/karaoke.mp4",
output_lossless_4k_mp4="/output/lossless_4k.mp4",
output_lossy_4k_mp4="/output/lossy_4k.mp4",
output_lossless_mkv="/output/lossless.mkv",
output_720p_mp4="/output/720p.mp4",
)
result = service.encode_all_formats(config)
# In dry run mode, all operations should "succeed"
assert result.success is True
class TestEncodingConfig:
"""Test EncodingConfig dataclass."""
def test_config_required_fields(self):
"""Test that required fields must be provided."""
config = EncodingConfig(
title_video="/path/title.mov",
karaoke_video="/path/karaoke.mov",
instrumental_audio="/path/audio.flac",
)
assert config.title_video == "/path/title.mov"
assert config.end_video is None # Optional field
def test_config_all_fields(self):
"""Test config with all fields."""
config = EncodingConfig(
title_video="/path/title.mov",
karaoke_video="/path/karaoke.mov",
instrumental_audio="/path/audio.flac",
end_video="/path/end.mov",
output_karaoke_mp4="/output/karaoke.mp4",
output_720p_mp4="/output/720p.mp4",
)
assert config.end_video == "/path/end.mov"
assert config.output_karaoke_mp4 == "/output/karaoke.mp4"
class TestEncodingResult:
"""Test EncodingResult dataclass."""
def test_result_success(self):
"""Test successful result."""
result = EncodingResult(
success=True,
output_files={"key": "/path/file.mp4"}
)
assert result.success is True
assert result.error is None
def test_result_failure(self):
"""Test failure result."""
result = EncodingResult(
success=False,
output_files={},
error="Something went wrong"
)
assert result.success is False
assert result.error == "Something went wrong"
class TestGetLocalEncodingService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.local_encoding_service as module
module._local_encoding_service = None
service = get_local_encoding_service()
assert service is not None
assert isinstance(service, LocalEncodingService)
def test_get_service_with_dry_run(self):
"""Test factory function with dry run option."""
import backend.services.local_encoding_service as module
module._local_encoding_service = None
service = get_local_encoding_service(dry_run=True)
assert service.dry_run is True
+382
View File
@@ -0,0 +1,382 @@
"""
Tests for PackagingService.
Tests cover:
- Service initialization
- CDG package creation
- TXT package creation
- Dry run mode
- Error handling
"""
import os
import zipfile
import tempfile
import pytest
from unittest.mock import MagicMock, patch
from backend.services.packaging_service import (
PackagingService,
get_packaging_service,
)
class TestPackagingServiceInit:
"""Test service initialization."""
def test_init_default_values(self):
"""Test default initialization."""
service = PackagingService()
assert service.cdg_styles is None
assert service.dry_run is False
assert service.non_interactive is False
def test_init_with_cdg_styles(self):
"""Test initialization with CDG styles."""
styles = {"font_size": 24, "bg_color": "black"}
service = PackagingService(cdg_styles=styles)
assert service.cdg_styles == styles
def test_init_with_dry_run(self):
"""Test initialization with dry run mode."""
service = PackagingService(dry_run=True)
assert service.dry_run is True
def test_init_with_non_interactive(self):
"""Test initialization with non-interactive mode."""
service = PackagingService(non_interactive=True)
assert service.non_interactive is True
class TestPackagingServiceCreateZipFromFiles:
"""Test internal ZIP creation."""
def test_create_zip_from_files(self):
"""Test creating a ZIP file from multiple files."""
with tempfile.TemporaryDirectory() as tmpdir:
# Create test files
file1 = os.path.join(tmpdir, "test1.txt")
file2 = os.path.join(tmpdir, "test2.txt")
zip_path = os.path.join(tmpdir, "output.zip")
with open(file1, "w") as f:
f.write("content1")
with open(file2, "w") as f:
f.write("content2")
service = PackagingService()
service._create_zip_from_files(
zip_path,
[(file1, "test1.txt"), (file2, "test2.txt")]
)
assert os.path.isfile(zip_path)
with zipfile.ZipFile(zip_path, "r") as zf:
names = zf.namelist()
assert "test1.txt" in names
assert "test2.txt" in names
class TestPackagingServiceCDGPackage:
"""Test CDG package creation."""
def test_create_cdg_package_missing_lrc_file(self):
"""Test that missing LRC file raises FileNotFoundError."""
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_cdg_package(
lrc_file="/nonexistent/file.lrc",
audio_file="/some/audio.flac",
output_zip_path="/output/test.zip",
artist="Test Artist",
title="Test Title",
)
assert "LRC file not found" in str(exc_info.value)
def test_create_cdg_package_missing_audio_file(self):
"""Test that missing audio file raises FileNotFoundError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_cdg_package(
lrc_file=lrc_file,
audio_file="/nonexistent/audio.flac",
output_zip_path="/output/test.zip",
artist="Test Artist",
title="Test Title",
)
assert "Audio file not found" in str(exc_info.value)
def test_create_cdg_package_no_styles_raises(self):
"""Test that CDG generation without styles raises ValueError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
# Service without CDG styles
service = PackagingService(cdg_styles=None)
with pytest.raises(ValueError) as exc_info:
service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
assert "CDG styles configuration is required" in str(exc_info.value)
def test_create_cdg_package_dry_run(self):
"""Test CDG package creation in dry run mode."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
service = PackagingService(dry_run=True)
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
# In dry run, ZIP should not be created
assert not os.path.isfile(zip_path)
assert result[0] == zip_path
def test_create_cdg_package_existing_files(self):
"""Test CDG package creation with existing MP3 and CDG files."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
mp3_file = os.path.join(tmpdir, "test.mp3")
cdg_file = os.path.join(tmpdir, "test.cdg")
zip_path = os.path.join(tmpdir, "output.zip")
# Create all files
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
with open(mp3_file, "w") as f:
f.write("fake mp3")
with open(cdg_file, "w") as f:
f.write("fake cdg")
service = PackagingService()
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
output_mp3_path=mp3_file,
output_cdg_path=cdg_file,
)
# ZIP should be created from existing files
assert os.path.isfile(zip_path)
assert result[0] == zip_path
assert result[1] == mp3_file
assert result[2] == cdg_file
@patch("lyrics_transcriber.output.cdg.CDGGenerator")
def test_create_cdg_package_with_generator(self, mock_cdg_generator_class):
"""Test CDG package creation using CDGGenerator."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
generated_zip = os.path.join(tmpdir, "generated.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
# Create the zip file that would be generated
with zipfile.ZipFile(generated_zip, "w") as zf:
zf.writestr("test.mp3", "fake mp3")
zf.writestr("test.cdg", "fake cdg")
mock_generator = MagicMock()
mock_generator.generate_cdg_from_lrc.return_value = (
"test.cdg", "test.mp3", generated_zip
)
mock_cdg_generator_class.return_value = mock_generator
styles = {"font_size": 24}
service = PackagingService(cdg_styles=styles)
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
)
mock_generator.generate_cdg_from_lrc.assert_called_once_with(
lrc_file=lrc_file,
audio_file=audio_file,
title="Test Title",
artist="Test Artist",
cdg_styles=styles,
)
assert os.path.isfile(zip_path)
class TestPackagingServiceTXTPackage:
"""Test TXT package creation."""
def test_create_txt_package_missing_lrc_file(self):
"""Test that missing LRC file raises FileNotFoundError."""
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_txt_package(
lrc_file="/nonexistent/file.lrc",
mp3_file="/some/audio.mp3",
output_zip_path="/output/test.zip",
)
assert "LRC file not found" in str(exc_info.value)
def test_create_txt_package_missing_mp3_file(self):
"""Test that missing MP3 file raises FileNotFoundError."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
service = PackagingService()
with pytest.raises(FileNotFoundError) as exc_info:
service.create_txt_package(
lrc_file=lrc_file,
mp3_file="/nonexistent/audio.mp3",
output_zip_path="/output/test.zip",
)
assert "MP3 file not found" in str(exc_info.value)
def test_create_txt_package_dry_run(self):
"""Test TXT package creation in dry run mode."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
mp3_file = os.path.join(tmpdir, "test.mp3")
zip_path = os.path.join(tmpdir, "output.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(mp3_file, "w") as f:
f.write("fake mp3")
service = PackagingService(dry_run=True)
result = service.create_txt_package(
lrc_file=lrc_file,
mp3_file=mp3_file,
output_zip_path=zip_path,
)
# In dry run, ZIP should not be created
assert not os.path.isfile(zip_path)
assert result[0] == zip_path
@patch("lyrics_converter.LyricsConverter")
def test_create_txt_package_success(self, mock_converter_class):
"""Test successful TXT package creation."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
mp3_file = os.path.join(tmpdir, "test.mp3")
zip_path = os.path.join(tmpdir, "output.zip")
txt_path = os.path.join(tmpdir, "output.txt")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(mp3_file, "w") as f:
f.write("fake mp3")
mock_converter = MagicMock()
mock_converter.convert_file.return_value = "Converted lyrics text"
mock_converter_class.return_value = mock_converter
service = PackagingService()
result = service.create_txt_package(
lrc_file=lrc_file,
mp3_file=mp3_file,
output_zip_path=zip_path,
output_txt_path=txt_path,
)
mock_converter_class.assert_called_once_with(
output_format="txt",
filepath=lrc_file
)
mock_converter.convert_file.assert_called_once()
assert os.path.isfile(zip_path)
assert os.path.isfile(txt_path)
assert result[0] == zip_path
assert result[1] == txt_path
# Verify ZIP contents
with zipfile.ZipFile(zip_path, "r") as zf:
names = zf.namelist()
assert "test.mp3" in names
assert "output.txt" in names
class TestGetPackagingService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.packaging_service as module
module._packaging_service = None
service = get_packaging_service()
assert service is not None
assert isinstance(service, PackagingService)
def test_get_service_with_cdg_styles(self):
"""Test factory function with CDG styles."""
import backend.services.packaging_service as module
module._packaging_service = None
styles = {"font_size": 24}
service = get_packaging_service(cdg_styles=styles)
assert service.cdg_styles == styles
def test_get_service_with_options(self):
"""Test factory function with additional options."""
import backend.services.packaging_service as module
module._packaging_service = None
service = get_packaging_service(dry_run=True, non_interactive=True)
assert service.dry_run is True
assert service.non_interactive is True
+2 -1
View File
@@ -149,7 +149,8 @@ class TestUploadSubmissionRequest:
# CDG/TXT disabled by default (requires style config)
assert request.enable_cdg is False
assert request.enable_txt is False
assert request.enable_youtube_upload is False
# YouTube upload default is None (use server default)
assert request.enable_youtube_upload is None
class TestStartReviewRequest:
+3 -3
View File
@@ -99,7 +99,7 @@ class TestStylesConfigRequirements:
"karaoke": {
"background_color": "#000000",
"font_path": "", # MUST be string, NOT None
"font": "Arial",
"font": "Noto Sans",
"ass_name": "Default", # REQUIRED
"primary_color": "112, 112, 247, 255",
"secondary_color": "255, 255, 255, 255",
@@ -241,7 +241,7 @@ class TestPreviewStyleLoading:
"background_image": "/original/path/background.png",
"font_path": "/original/path/font.ttf",
"background_color": "#000000",
"font": "Arial",
"font": "Noto Sans",
"ass_name": "Default",
}
}
@@ -308,7 +308,7 @@ class TestPreviewStyleLoading:
# Should have karaoke section with minimal/default values
assert "karaoke" in result_styles
assert result_styles["karaoke"]["background_color"] == "#000000"
assert result_styles["karaoke"]["font"] == "Arial"
assert result_styles["karaoke"]["font"] == "Noto Sans"
# Minimal styles have background_image as None (default)
assert result_styles["karaoke"].get("background_image") is None
@@ -0,0 +1,847 @@
"""
Tests for Video Worker Orchestrator.
Tests cover:
- OrchestratorConfig and OrchestratorResult dataclasses
- VideoWorkerOrchestrator initialization
- Individual stage methods
- Full pipeline execution
- Error handling and recovery
"""
import os
import tempfile
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from dataclasses import asdict
from backend.workers.video_worker_orchestrator import (
OrchestratorConfig,
OrchestratorResult,
VideoWorkerOrchestrator,
create_orchestrator_config_from_job,
)
class TestOrchestratorConfig:
"""Test OrchestratorConfig dataclass."""
def test_required_fields(self):
"""Test that required fields must be provided."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert config.job_id == "test-job"
assert config.artist == "Test Artist"
assert config.encoding_backend == "auto"
def test_default_values(self):
"""Test default values for optional fields."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert config.enable_cdg is False
assert config.enable_txt is False
assert config.enable_youtube_upload is False
assert config.dry_run is False
assert config.non_interactive is True
assert config.end_video_path is None
def test_all_fields(self):
"""Test all fields including optional ones."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
end_video_path="/path/end.mov",
enable_cdg=True,
enable_txt=True,
enable_youtube_upload=True,
brand_prefix="TEST",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
encoding_backend="gce",
)
assert config.enable_cdg is True
assert config.encoding_backend == "gce"
assert config.discord_webhook_url is not None
class TestOrchestratorResult:
"""Test OrchestratorResult dataclass."""
def test_success_result(self):
"""Test successful result."""
result = OrchestratorResult(
success=True,
final_video="/output/video.mp4",
brand_code="TEST-1234",
youtube_url="https://youtube.com/watch?v=test",
)
assert result.success is True
assert result.error_message is None
assert result.brand_code == "TEST-1234"
def test_failure_result(self):
"""Test failure result."""
result = OrchestratorResult(
success=False,
error_message="Encoding failed",
)
assert result.success is False
assert result.error_message == "Encoding failed"
assert result.final_video is None
def test_all_output_files(self):
"""Test all output file fields."""
result = OrchestratorResult(
success=True,
final_video="/output/lossless.mp4",
final_video_mkv="/output/lossless.mkv",
final_video_lossy="/output/lossy.mp4",
final_video_720p="/output/720p.mp4",
final_karaoke_cdg_zip="/output/cdg.zip",
final_karaoke_txt_zip="/output/txt.zip",
)
assert result.final_video == "/output/lossless.mp4"
assert result.final_video_mkv == "/output/lossless.mkv"
class TestVideoWorkerOrchestratorInit:
"""Test VideoWorkerOrchestrator initialization."""
def test_init_with_config(self):
"""Test initialization with config."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
assert orchestrator.config == config
assert orchestrator.result.success is False
def test_init_with_job_manager(self):
"""Test initialization with job manager."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
job_manager = MagicMock()
orchestrator = VideoWorkerOrchestrator(config, job_manager=job_manager)
assert orchestrator.job_manager == job_manager
class TestVideoWorkerOrchestratorServices:
"""Test service lazy-loading."""
def test_get_encoding_backend(self):
"""Test encoding backend lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
encoding_backend="local",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.encoding_interface.get_encoding_backend") as mock_get:
mock_backend = MagicMock()
mock_backend.name = "local"
mock_get.return_value = mock_backend
backend = orchestrator._get_encoding_backend()
assert backend == mock_backend
mock_get.assert_called_once()
def test_get_packaging_service(self):
"""Test packaging service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
cdg_styles={"background_color": "black"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.packaging_service.PackagingService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_packaging_service()
assert service == mock_service
MockService.assert_called_once()
def test_get_youtube_service(self):
"""Test YouTube service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
youtube_credentials={"token": "test"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.youtube_upload_service.YouTubeUploadService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_youtube_service()
assert service == mock_service
MockService.assert_called_once()
def test_get_discord_service(self):
"""Test Discord service lazy loading."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.discord_service.DiscordNotificationService") as MockService:
mock_service = MagicMock()
MockService.return_value = mock_service
service = orchestrator._get_discord_service()
assert service == mock_service
MockService.assert_called_once()
class TestVideoWorkerOrchestratorPackaging:
"""Test packaging stage."""
@pytest.mark.asyncio
async def test_run_packaging_no_lrc(self):
"""Test packaging stage with no LRC file."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=True,
lrc_file_path=None,
)
orchestrator = VideoWorkerOrchestrator(config)
# Should not raise, just skip
await orchestrator._run_packaging()
assert orchestrator.result.final_karaoke_cdg_zip is None
@pytest.mark.asyncio
async def test_run_packaging_cdg(self):
"""Test CDG packaging."""
with tempfile.TemporaryDirectory() as temp_dir:
lrc_file = os.path.join(temp_dir, "test.lrc")
audio_file = os.path.join(temp_dir, "test.flac")
# Create dummy files
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("dummy audio")
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path=audio_file,
lrc_file_path=lrc_file,
output_dir=temp_dir,
enable_cdg=True,
cdg_styles={"background_color": "black"},
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_packaging_service") as mock_get:
mock_service = MagicMock()
mock_service.create_cdg_package.return_value = (
f"{temp_dir}/cdg.zip",
f"{temp_dir}/test.mp3",
f"{temp_dir}/test.cdg",
)
mock_get.return_value = mock_service
await orchestrator._run_packaging()
mock_service.create_cdg_package.assert_called_once()
assert orchestrator.result.final_karaoke_cdg_zip == f"{temp_dir}/cdg.zip"
class TestVideoWorkerOrchestratorEncoding:
"""Test encoding stage."""
@pytest.mark.asyncio
async def test_run_encoding_success(self):
"""Test successful encoding."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
output_dir="/output",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
mock_backend = MagicMock()
mock_backend.name = "local"
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path="/output/lossless.mp4",
lossy_4k_mp4_path="/output/lossy.mp4",
lossy_720p_mp4_path="/output/720p.mp4",
lossless_mkv_path="/output/lossless.mkv",
encoding_time_seconds=120.5,
encoding_backend="local",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
mock_backend.encode.assert_called_once()
assert orchestrator.result.final_video == "/output/lossless.mp4"
assert orchestrator.result.final_video_720p == "/output/720p.mp4"
assert orchestrator.result.encoding_time_seconds == 120.5
def test_encoding_input_gcs_paths_pattern(self):
"""Test that EncodingInput.options contains proper GCS paths structure.
This test verifies the GCS path construction pattern that GCE encoding
requires. It tests the structure without running _run_encoding().
This would have caught: 'GCE encoding requires input_gcs_path and
output_gcs_path in options' error when the orchestrator didn't pass
the required paths for GCE encoding.
"""
from backend.services.encoding_interface import EncodingInput
# Test that EncodingInput can hold GCS paths in options
# This mirrors what the orchestrator should build
job_id = "test-job-123"
bucket = "test-bucket"
input_gcs_path = f"gs://{bucket}/jobs/{job_id}/"
output_gcs_path = f"gs://{bucket}/jobs/{job_id}/finals/"
encoding_input = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
artist="Test Artist",
title="Test Title",
brand_code="TEST-001",
output_dir="/output",
options={
"input_gcs_path": input_gcs_path,
"output_gcs_path": output_gcs_path,
},
)
# Verify the structure that GCEEncodingBackend expects
assert "input_gcs_path" in encoding_input.options, \
"EncodingInput.options must include input_gcs_path for GCE encoding"
assert "output_gcs_path" in encoding_input.options, \
"EncodingInput.options must include output_gcs_path for GCE encoding"
# Verify path format
assert encoding_input.options["input_gcs_path"].startswith("gs://")
assert encoding_input.options["output_gcs_path"].startswith("gs://")
assert job_id in encoding_input.options["input_gcs_path"]
assert job_id in encoding_input.options["output_gcs_path"]
@pytest.mark.asyncio
async def test_run_encoding_failure(self):
"""Test encoding failure."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
mock_backend = MagicMock()
mock_backend.name = "local"
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=False,
error_message="FFmpeg failed",
encoding_backend="local",
))
mock_get.return_value = mock_backend
with pytest.raises(Exception) as exc_info:
await orchestrator._run_encoding()
assert "Encoding failed" in str(exc_info.value)
@pytest.mark.asyncio
async def test_run_encoding_gce_downloads_files(self):
"""Test that GCE encoding downloads files from GCS to local directory.
This test verifies the fix for YouTube upload failure when using GCE encoding.
GCE encoding returns GCS blob paths, which need to be downloaded locally
before YouTube upload can access them.
"""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path=os.path.join(temp_dir, "title.mov"),
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
output_dir=temp_dir,
)
# Mock storage service
mock_storage = MagicMock()
mock_storage.download_file = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config,
storage=mock_storage,
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
# GCE backend returns GCS blob paths (not local paths)
mock_backend = MagicMock()
mock_backend.name = "gce" # Important: must be "gce" to trigger download
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path="jobs/test-job/finals/output_4k_lossless.mp4",
lossy_4k_mp4_path="jobs/test-job/finals/output_4k_lossy.mp4",
lossy_720p_mp4_path="jobs/test-job/finals/output_720p.mp4",
lossless_mkv_path="jobs/test-job/finals/output_4k.mkv",
encoding_time_seconds=60.0,
encoding_backend="gce",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# Verify download_file was called for each output file
assert mock_storage.download_file.call_count == 4
# Verify the result paths were updated to local paths
assert orchestrator.result.final_video.startswith(temp_dir)
assert orchestrator.result.final_video_mkv.startswith(temp_dir)
assert orchestrator.result.final_video_lossy.startswith(temp_dir)
assert orchestrator.result.final_video_720p.startswith(temp_dir)
@pytest.mark.asyncio
async def test_run_encoding_local_does_not_download(self):
"""Test that local encoding does NOT trigger GCS download.
Local encoding produces files directly in the output directory,
so no download is needed.
"""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path=os.path.join(temp_dir, "title.mov"),
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
output_dir=temp_dir,
)
# Mock storage service
mock_storage = MagicMock()
mock_storage.download_file = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config,
storage=mock_storage,
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
# Local backend returns local paths directly
mock_backend = MagicMock()
mock_backend.name = "local" # Local backend
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path=os.path.join(temp_dir, "lossless.mp4"),
lossy_4k_mp4_path=os.path.join(temp_dir, "lossy.mp4"),
lossy_720p_mp4_path=os.path.join(temp_dir, "720p.mp4"),
lossless_mkv_path=os.path.join(temp_dir, "lossless.mkv"),
encoding_time_seconds=120.0,
encoding_backend="local",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# Verify download_file was NOT called for local encoding
mock_storage.download_file.assert_not_called()
class TestVideoWorkerOrchestratorOrganization:
"""Test organization stage."""
@pytest.mark.asyncio
async def test_run_organization_keep_brand_code(self):
"""Test organization with existing brand code."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
keep_brand_code="NOMAD-9999",
)
orchestrator = VideoWorkerOrchestrator(config)
await orchestrator._run_organization()
assert orchestrator.result.brand_code == "NOMAD-9999"
@pytest.mark.asyncio
async def test_run_organization_generate_brand_code(self):
"""Test brand code generation from Dropbox."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
dropbox_path="/Karaoke/Tracks",
brand_prefix="TEST",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
mock_dropbox = MagicMock()
mock_dropbox.is_configured = True
mock_dropbox.get_next_brand_code.return_value = "TEST-0001"
mock_get.return_value = mock_dropbox
await orchestrator._run_organization()
assert orchestrator.result.brand_code == "TEST-0001"
class TestVideoWorkerOrchestratorDistribution:
"""Test distribution stage."""
@pytest.mark.asyncio
async def test_upload_to_youtube(self):
"""Test YouTube upload."""
with tempfile.TemporaryDirectory() as temp_dir:
video_file = os.path.join(temp_dir, "test.mp4")
with open(video_file, "w") as f:
f.write("dummy video")
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_youtube_upload=True,
youtube_credentials={"token": "test"},
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.final_video_lossy = video_file
with patch.object(orchestrator, "_get_youtube_service") as mock_get:
mock_service = MagicMock()
mock_service.upload_video.return_value = (
"video123",
"https://youtube.com/watch?v=video123"
)
mock_get.return_value = mock_service
await orchestrator._upload_to_youtube()
mock_service.upload_video.assert_called_once()
assert orchestrator.result.youtube_url == "https://youtube.com/watch?v=video123"
@pytest.mark.asyncio
async def test_upload_to_dropbox(self):
"""Test Dropbox upload."""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
output_dir=temp_dir,
dropbox_path="/Karaoke/Tracks",
brand_prefix="TEST",
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.brand_code = "TEST-0001"
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
mock_dropbox = MagicMock()
mock_dropbox.is_configured = True
mock_dropbox.create_shared_link.return_value = "https://dropbox.com/link"
mock_get.return_value = mock_dropbox
await orchestrator._upload_to_dropbox()
mock_dropbox.upload_folder.assert_called_once()
assert orchestrator.result.dropbox_link == "https://dropbox.com/link"
class TestVideoWorkerOrchestratorNotifications:
"""Test notifications stage."""
@pytest.mark.asyncio
async def test_run_notifications_with_youtube_url(self):
"""Test Discord notification with YouTube URL."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
orchestrator.result.youtube_url = "https://youtube.com/watch?v=test"
with patch.object(orchestrator, "_get_discord_service") as mock_get:
mock_service = MagicMock()
mock_service.post_video_notification.return_value = True
mock_get.return_value = mock_service
await orchestrator._run_notifications()
mock_service.post_video_notification.assert_called_once_with(
"https://youtube.com/watch?v=test"
)
@pytest.mark.asyncio
async def test_run_notifications_no_youtube_url(self):
"""Test notification skipped without YouTube URL."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
)
orchestrator = VideoWorkerOrchestrator(config)
# No youtube_url set
with patch.object(orchestrator, "_get_discord_service") as mock_get:
mock_service = MagicMock()
mock_get.return_value = mock_service
await orchestrator._run_notifications()
mock_service.post_video_notification.assert_not_called()
class TestVideoWorkerOrchestratorFullPipeline:
"""Test full pipeline execution."""
@pytest.mark.asyncio
async def test_run_full_pipeline_success(self):
"""Test successful full pipeline."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=True, # Enable to trigger packaging stage
)
orchestrator = VideoWorkerOrchestrator(config)
# Mock all stages
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org, \
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock) as mock_dist, \
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock) as mock_notify:
result = await orchestrator.run()
assert result.success is True
mock_packaging.assert_called_once()
mock_encoding.assert_called_once()
mock_org.assert_called_once()
mock_dist.assert_called_once()
mock_notify.assert_called_once()
@pytest.mark.asyncio
async def test_run_pipeline_skips_packaging_when_disabled(self):
"""Test pipeline skips packaging when CDG/TXT disabled."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
enable_cdg=False,
enable_txt=False,
)
orchestrator = VideoWorkerOrchestrator(config)
# Mock all stages
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock):
result = await orchestrator.run()
assert result.success is True
mock_packaging.assert_not_called()
mock_encoding.assert_called_once()
@pytest.mark.asyncio
async def test_run_full_pipeline_encoding_failure(self):
"""Test pipeline failure during encoding."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock), \
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org:
mock_encoding.side_effect = Exception("Encoding failed")
result = await orchestrator.run()
assert result.success is False
assert "Encoding failed" in result.error_message
mock_org.assert_not_called()
class TestCreateOrchestratorConfigFromJob:
"""Test helper function for creating config from job."""
def test_create_config_from_job(self):
"""Test creating config from a job object."""
job = MagicMock()
job.job_id = "test-123"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {"instrumental_selection": "clean"}
job.enable_cdg = True
job.enable_txt = False
job.enable_youtube_upload = True
job.brand_prefix = "NOMAD"
job.discord_webhook_url = "https://discord.com/api/webhooks/123/abc"
job.youtube_description_template = "Test description"
job.dropbox_path = "/Karaoke"
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = None
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
youtube_credentials={"token": "test"},
cdg_styles={"background": "black"},
)
assert config.job_id == "test-123"
assert config.artist == "Test Artist"
assert config.title == "Test Title"
assert config.enable_cdg is True
assert config.enable_youtube_upload is True
assert config.title_video_path == "/tmp/test/Test Artist - Test Title (Title).mov"
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental Clean).flac"
def test_create_config_from_job_with_existing_instrumental(self):
"""Test config with user-provided instrumental."""
job = MagicMock()
job.job_id = "test-123"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {"instrumental_selection": "custom"}
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = False
job.brand_prefix = None
job.discord_webhook_url = None
job.youtube_description_template = None
job.dropbox_path = None
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = "gs://bucket/instrumental.mp3"
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
)
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental User).mp3"
@@ -0,0 +1,509 @@
"""
Unit tests for worker log subcollection functionality.
Tests the WorkerLogEntry model, FirestoreService subcollection methods,
and JobManager log operations with the feature flag.
"""
import pytest
from unittest.mock import Mock, MagicMock, patch
from datetime import datetime, timezone, timedelta
import uuid
# Mock Firestore before imports
import sys
sys.modules['google.cloud.firestore'] = MagicMock()
sys.modules['google.cloud.firestore_v1'] = MagicMock()
from backend.models.worker_log import WorkerLogEntry, DEFAULT_LOG_TTL_DAYS
class TestWorkerLogEntry:
"""Tests for WorkerLogEntry dataclass."""
def test_create_log_entry(self):
"""Test creating a log entry with factory method."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
assert entry.job_id == "job123"
assert entry.worker == "audio"
assert entry.level == "INFO"
assert entry.message == "Test message"
assert entry.id is not None
assert entry.timestamp is not None
assert entry.ttl_expiry is not None
# TTL should be ~30 days from now
expected_ttl = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
def test_create_log_entry_truncates_long_message(self):
"""Test that messages longer than 1000 chars are truncated."""
long_message = "x" * 2000
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message=long_message
)
assert len(entry.message) == 1000
def test_create_log_entry_normalizes_level(self):
"""Test that log level is normalized to uppercase."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="info",
message="Test"
)
assert entry.level == "INFO"
def test_create_log_entry_custom_ttl(self):
"""Test creating entry with custom TTL."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test",
ttl_days=7
)
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
def test_create_log_entry_with_metadata(self):
"""Test creating entry with metadata."""
metadata = {"file_size": 1024, "duration": 300}
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test",
metadata=metadata
)
assert entry.metadata == metadata
def test_to_dict(self):
"""Test converting entry to dict for Firestore."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message",
metadata={"key": "value"}
)
d = entry.to_dict()
assert d["job_id"] == "job123"
assert d["worker"] == "audio"
assert d["level"] == "INFO"
assert d["message"] == "Test message"
assert d["metadata"] == {"key": "value"}
assert "id" in d
assert "timestamp" in d
assert "ttl_expiry" in d
def test_to_dict_without_metadata(self):
"""Test to_dict doesn't include metadata when None."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test"
)
d = entry.to_dict()
assert "metadata" not in d
def test_to_legacy_dict(self):
"""Test converting to legacy format for API compatibility."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
d = entry.to_legacy_dict()
assert "timestamp" in d
assert d["level"] == "INFO"
assert d["worker"] == "audio"
assert d["message"] == "Test message"
# Should not include id, job_id, ttl_expiry, metadata
assert "id" not in d
assert "job_id" not in d
assert "ttl_expiry" not in d
def test_from_dict(self):
"""Test creating entry from Firestore document."""
timestamp = datetime.now(timezone.utc)
ttl_expiry = timestamp + timedelta(days=30)
data = {
"id": "log123",
"job_id": "job123",
"timestamp": timestamp,
"level": "WARNING",
"worker": "video",
"message": "Warning message",
"metadata": {"error_code": 500},
"ttl_expiry": ttl_expiry
}
entry = WorkerLogEntry.from_dict(data)
assert entry.id == "log123"
assert entry.job_id == "job123"
assert entry.level == "WARNING"
assert entry.worker == "video"
assert entry.message == "Warning message"
assert entry.metadata == {"error_code": 500}
def test_from_dict_with_iso_strings(self):
"""Test creating entry from dict with ISO format strings."""
data = {
"timestamp": "2026-01-04T12:00:00Z",
"level": "INFO",
"worker": "audio",
"message": "Test"
}
entry = WorkerLogEntry.from_dict(data)
assert entry.timestamp.tzinfo == timezone.utc
assert entry.message == "Test"
def test_from_dict_missing_fields(self):
"""Test creating entry with missing fields uses defaults."""
data = {
"message": "Minimal log"
}
entry = WorkerLogEntry.from_dict(data)
assert entry.message == "Minimal log"
assert entry.level == "INFO" # Default
assert entry.worker == "unknown" # Default
assert entry.id is not None # Generated
class TestFirestoreServiceSubcollection:
"""Tests for FirestoreService log subcollection methods."""
@pytest.fixture
def mock_db(self):
"""Create mock Firestore database."""
return MagicMock()
@pytest.fixture
def firestore_service(self, mock_db):
"""Create FirestoreService with mocked DB."""
with patch('backend.services.firestore_service.firestore') as mock_firestore:
mock_firestore.Client.return_value = mock_db
from backend.services.firestore_service import FirestoreService
service = FirestoreService()
service.db = mock_db
return service
def test_append_log_to_subcollection(self, firestore_service, mock_db):
"""Test appending log to subcollection."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
# Setup mock chain
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_log_doc = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_logs_ref.document.return_value = mock_log_doc
firestore_service.append_log_to_subcollection("job123", entry)
# Verify correct path: jobs/{job_id}/logs/{log_id}
mock_db.collection.assert_called_with("jobs")
mock_collection.document.assert_called_with("job123")
mock_doc.collection.assert_called_with("logs")
mock_logs_ref.document.assert_called_with(entry.id)
mock_log_doc.set.assert_called_once()
def test_get_logs_from_subcollection(self, firestore_service, mock_db):
"""Test getting logs from subcollection."""
# Setup mock chain
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_query = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_logs_ref.order_by.return_value = mock_query
mock_query.limit.return_value = mock_query
# Mock document data
mock_doc1 = MagicMock()
mock_doc1.to_dict.return_value = {
"timestamp": datetime.now(timezone.utc),
"level": "INFO",
"worker": "audio",
"message": "Log 1"
}
mock_query.stream.return_value = iter([mock_doc1])
logs = firestore_service.get_logs_from_subcollection("job123")
assert len(logs) == 1
assert logs[0].message == "Log 1"
mock_logs_ref.order_by.assert_called_once()
def test_get_logs_from_subcollection_with_worker_filter(self, firestore_service, mock_db):
"""Test filtering logs by worker."""
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_query = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_logs_ref.order_by.return_value = mock_query
mock_query.where.return_value = mock_query
mock_query.limit.return_value = mock_query
mock_query.stream.return_value = iter([])
firestore_service.get_logs_from_subcollection("job123", worker="audio")
# Verify where clause was added
mock_query.where.assert_called()
def test_delete_logs_subcollection(self, firestore_service, mock_db):
"""Test deleting all logs in subcollection."""
mock_collection = MagicMock()
mock_doc = MagicMock()
mock_logs_ref = MagicMock()
mock_batch = MagicMock()
mock_db.collection.return_value = mock_collection
mock_collection.document.return_value = mock_doc
mock_doc.collection.return_value = mock_logs_ref
mock_db.batch.return_value = mock_batch
# First call returns 2 docs, second call returns empty
mock_log_doc1 = MagicMock()
mock_log_doc2 = MagicMock()
call_count = [0]
def stream_side_effect(*args, **kwargs):
call_count[0] += 1
if call_count[0] == 1:
return iter([mock_log_doc1, mock_log_doc2])
return iter([])
mock_logs_ref.limit.return_value = mock_logs_ref
mock_logs_ref.stream.side_effect = stream_side_effect
deleted = firestore_service.delete_logs_subcollection("job123")
assert deleted == 2
mock_batch.commit.assert_called()
class TestJobManagerLogging:
"""Tests for JobManager log methods with feature flag."""
@pytest.fixture
def mock_firestore_service(self):
"""Mock FirestoreService."""
with patch('backend.services.job_manager.FirestoreService') as mock:
service = MagicMock()
mock.return_value = service
yield service
@pytest.fixture
def mock_storage_service(self):
"""Mock StorageService."""
with patch('backend.services.job_manager.StorageService') as mock:
service = MagicMock()
mock.return_value = service
yield service
def test_append_worker_log_uses_subcollection_when_enabled(
self, mock_firestore_service, mock_storage_service
):
"""Test that logs go to subcollection when feature is enabled."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = True
from backend.services.job_manager import JobManager
manager = JobManager()
manager.append_worker_log(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
# Should call subcollection method
mock_firestore_service.append_log_to_subcollection.assert_called_once()
# Should NOT call legacy method
mock_firestore_service.append_worker_log.assert_not_called()
def test_append_worker_log_uses_array_when_disabled(
self, mock_firestore_service, mock_storage_service
):
"""Test that logs go to embedded array when feature is disabled."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = False
from backend.services.job_manager import JobManager
manager = JobManager()
manager.append_worker_log(
job_id="job123",
worker="audio",
level="INFO",
message="Test message"
)
# Should call legacy method
mock_firestore_service.append_worker_log.assert_called_once()
# Should NOT call subcollection method
mock_firestore_service.append_log_to_subcollection.assert_not_called()
def test_get_worker_logs_from_subcollection(
self, mock_firestore_service, mock_storage_service
):
"""Test getting logs from subcollection."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = True
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Test"
)
mock_firestore_service.get_logs_from_subcollection.return_value = [entry]
from backend.services.job_manager import JobManager
manager = JobManager()
logs = manager.get_worker_logs("job123")
assert len(logs) == 1
assert logs[0]["message"] == "Test"
mock_firestore_service.get_logs_from_subcollection.assert_called_once()
def test_get_worker_logs_falls_back_to_array(
self, mock_firestore_service, mock_storage_service
):
"""Test fallback to embedded array for legacy jobs."""
with patch('backend.services.job_manager.settings') as mock_settings:
mock_settings.use_log_subcollection = True
# Subcollection returns empty
mock_firestore_service.get_logs_from_subcollection.return_value = []
# Mock job with legacy logs
from backend.models.job import Job, JobStatus
mock_job = Job(
job_id="job123",
status=JobStatus.PENDING,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
worker_logs=[
{"timestamp": "2026-01-04T12:00:00Z", "level": "INFO", "worker": "audio", "message": "Legacy log"}
]
)
mock_firestore_service.get_job.return_value = mock_job
from backend.services.job_manager import JobManager
manager = JobManager()
logs = manager.get_worker_logs("job123")
assert len(logs) == 1
assert logs[0]["message"] == "Legacy log"
def test_delete_job_deletes_logs_subcollection(
self, mock_firestore_service, mock_storage_service
):
"""Test that deleting a job also deletes its logs subcollection."""
mock_firestore_service.get_job.return_value = None
mock_firestore_service.delete_logs_subcollection.return_value = 5
from backend.services.job_manager import JobManager
manager = JobManager()
manager.delete_job("job123", delete_files=False)
mock_firestore_service.delete_logs_subcollection.assert_called_once_with("job123")
mock_firestore_service.delete_job.assert_called_once_with("job123")
class TestWorkerLogEntryEdgeCases:
"""Edge case tests for WorkerLogEntry."""
def test_create_with_empty_message(self):
"""Test creating entry with empty message."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message=""
)
assert entry.message == ""
def test_create_with_unicode_message(self):
"""Test creating entry with Unicode characters."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="INFO",
message="Processing song: 日本語の曲 - アーティスト"
)
assert "日本語" in entry.message
def test_create_with_newlines_in_message(self):
"""Test creating entry with newlines."""
entry = WorkerLogEntry.create(
job_id="job123",
worker="audio",
level="ERROR",
message="Error occurred:\nLine 1\nLine 2"
)
assert "\n" in entry.message
def test_from_dict_handles_missing_ttl_expiry(self):
"""Test from_dict creates default TTL when missing."""
data = {
"timestamp": datetime.now(timezone.utc),
"level": "INFO",
"worker": "audio",
"message": "Test"
}
entry = WorkerLogEntry.from_dict(data)
assert entry.ttl_expiry is not None
# Should be ~30 days from now
expected = datetime.now(timezone.utc) + timedelta(days=30)
assert abs((entry.ttl_expiry - expected).total_seconds()) < 10
@@ -0,0 +1,568 @@
"""
Tests for YouTubeUploadService.
Tests cover:
- Authentication with pre-stored credentials
- Authentication with client secrets file
- Duplicate video detection (exact and fuzzy matching)
- Video deletion
- Video upload with metadata and thumbnail
- Title truncation
- Dry run mode
"""
import os
import pytest
from unittest.mock import MagicMock, patch, mock_open
from backend.services.youtube_upload_service import YouTubeUploadService, get_youtube_upload_service
class TestYouTubeUploadServiceInit:
"""Test service initialization."""
def test_init_with_credentials(self):
"""Test initialization with pre-stored credentials."""
creds = {
"token": "test_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
}
service = YouTubeUploadService(
credentials=creds,
non_interactive=True
)
assert service.credentials == creds
assert service.non_interactive is True
def test_init_with_client_secrets(self):
"""Test initialization with client secrets file."""
service = YouTubeUploadService(
client_secrets_file="/path/to/secrets.json"
)
assert service.client_secrets_file == "/path/to/secrets.json"
assert service.non_interactive is False
def test_init_default_values(self):
"""Test default values on initialization."""
service = YouTubeUploadService()
assert service.credentials is None
assert service.client_secrets_file is None
assert service.non_interactive is False
assert service.server_side_mode is False
assert service.dry_run is False
class TestYouTubeUploadServiceAuthentication:
"""Test authentication methods."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_authenticate_with_prestored_credentials(
self, mock_request, mock_credentials_class, mock_build
):
"""Test authentication using pre-stored credentials."""
# Setup mocks
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_build.return_value = mock_youtube
creds = {
"token": "test_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
"scopes": ["https://www.googleapis.com/auth/youtube"],
}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
# Access youtube_service to trigger authentication
result = service.youtube_service
assert result == mock_youtube
mock_credentials_class.assert_called_once()
mock_build.assert_called_once_with('youtube', 'v3', credentials=mock_creds)
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_authenticate_refreshes_expired_token(
self, mock_request, mock_credentials_class, mock_build
):
"""Test that expired tokens are refreshed."""
mock_creds = MagicMock()
mock_creds.expired = True
mock_creds.refresh_token = "test_refresh"
mock_credentials_class.return_value = mock_creds
creds = {
"token": "expired_token",
"refresh_token": "test_refresh",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "test_client_id",
"client_secret": "test_secret",
}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
service.youtube_service
mock_creds.refresh.assert_called_once()
def test_authenticate_non_interactive_no_credentials_raises(self):
"""Test that non-interactive mode without credentials raises error."""
service = YouTubeUploadService(non_interactive=True)
with pytest.raises(Exception) as exc_info:
service._authenticate()
assert "non-interactive mode" in str(exc_info.value).lower()
class TestYouTubeUploadServiceChannelId:
"""Test channel ID retrieval."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_get_channel_id_success(
self, mock_request, mock_credentials_class, mock_build
):
"""Test successful channel ID retrieval."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_list = MagicMock()
mock_list.execute.return_value = {
"items": [{"id": "UC123456"}]
}
mock_channels.list.return_value = mock_list
mock_youtube.channels.return_value = mock_channels
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
channel_id = service.get_channel_id()
assert channel_id == "UC123456"
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_get_channel_id_no_items(
self, mock_request, mock_credentials_class, mock_build
):
"""Test channel ID returns None when no items found."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_list = MagicMock()
mock_list.execute.return_value = {"items": []}
mock_channels.list.return_value = mock_list
mock_youtube.channels.return_value = mock_channels
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
channel_id = service.get_channel_id()
assert channel_id is None
class TestYouTubeUploadServiceDuplicateCheck:
"""Test duplicate video detection."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_exact_match_found(
self, mock_request, mock_credentials_class, mock_build
):
"""Test duplicate detection with exact match in server mode."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
# Mock channels().list() for get_channel_id
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Mock search().list() for duplicate check
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {
"items": [{
"id": {"videoId": "VIDEO123"},
"snippet": {
"channelId": "UC123456",
"title": "Test Artist - Test Song (Karaoke)"
}
}]
}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(
credentials=creds,
non_interactive=True,
server_side_mode=True
)
exists, video_id, video_url = service.check_duplicate(
"Test Artist - Test Song (Karaoke)"
)
assert exists is True
assert video_id == "VIDEO123"
assert "VIDEO123" in video_url
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_no_match(
self, mock_request, mock_credentials_class, mock_build
):
"""Test duplicate detection when no match found."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
exists, video_id, video_url = service.check_duplicate("Some Title")
assert exists is False
assert video_id is None
assert video_url is None
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_check_duplicate_skips_other_channels(
self, mock_request, mock_credentials_class, mock_build
):
"""Test that videos from other channels are skipped."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Return video from different channel
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {
"items": [{
"id": {"videoId": "VIDEO123"},
"snippet": {
"channelId": "UC_DIFFERENT", # Different channel
"title": "Test Title"
}
}]
}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
exists, video_id, video_url = service.check_duplicate("Test Title")
assert exists is False
class TestYouTubeUploadServiceDelete:
"""Test video deletion."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_delete_video_success(
self, mock_request, mock_credentials_class, mock_build
):
"""Test successful video deletion."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_videos = MagicMock()
mock_delete = MagicMock()
mock_delete.execute.return_value = None
mock_videos.delete.return_value = mock_delete
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
result = service.delete_video("VIDEO123")
assert result is True
mock_videos.delete.assert_called_once_with(id="VIDEO123")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_delete_video_failure(
self, mock_request, mock_credentials_class, mock_build
):
"""Test video deletion failure handling."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_videos = MagicMock()
mock_delete = MagicMock()
mock_delete.execute.side_effect = Exception("API Error")
mock_videos.delete.return_value = mock_delete
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
result = service.delete_video("VIDEO123")
assert result is False
def test_delete_video_dry_run(self):
"""Test video deletion in dry run mode."""
service = YouTubeUploadService(dry_run=True)
result = service.delete_video("VIDEO123")
assert result is True
class TestYouTubeUploadServiceTitleTruncation:
"""Test title truncation."""
def test_truncate_title_short_title(self):
"""Test that short titles are not truncated."""
title = "Short Title"
result = YouTubeUploadService.truncate_title(title)
assert result == title
def test_truncate_title_exact_length(self):
"""Test title at exact max length."""
title = "A" * 95
result = YouTubeUploadService.truncate_title(title, max_length=95)
assert result == title
assert len(result) == 95
def test_truncate_title_long_title(self):
"""Test that long titles are truncated at word boundary."""
title = "This is a very long title that exceeds the maximum length and needs to be truncated properly at a word boundary"
result = YouTubeUploadService.truncate_title(title, max_length=50)
assert len(result) <= 50
assert result.endswith("...")
def test_truncate_title_no_space(self):
"""Test truncation of title without spaces."""
title = "A" * 100
result = YouTubeUploadService.truncate_title(title, max_length=50)
assert len(result) <= 50
class TestYouTubeUploadServiceUpload:
"""Test video upload."""
@patch("googleapiclient.http.MediaFileUpload")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
def test_upload_video_success(
self, mock_request, mock_credentials_class, mock_build, mock_media_upload
):
"""Test successful video upload."""
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
# Mock channels for get_channel_id
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
# Mock search for duplicate check (no duplicates)
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
# Mock video insert
mock_videos = MagicMock()
mock_insert = MagicMock()
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
mock_videos.insert.return_value = mock_insert
mock_youtube.videos.return_value = mock_videos
mock_build.return_value = mock_youtube
mock_media_upload.return_value = MagicMock()
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Artist - Test Song (Karaoke)",
description="Test description",
tags=["karaoke", "test"]
)
assert video_id == "NEW_VIDEO_ID"
assert "NEW_VIDEO_ID" in video_url
def test_upload_video_dry_run(self):
"""Test video upload in dry run mode."""
service = YouTubeUploadService(dry_run=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Title",
description="Test description"
)
assert video_id == "dry_run_video_id"
assert "dry_run_video_id" in video_url
@patch("googleapiclient.http.MediaFileUpload")
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("google.auth.transport.requests.Request")
@patch("os.path.isfile")
def test_upload_video_with_thumbnail(
self, mock_isfile, mock_request, mock_credentials_class,
mock_build, mock_media_upload
):
"""Test video upload with thumbnail."""
mock_isfile.return_value = True
mock_creds = MagicMock()
mock_creds.expired = False
mock_credentials_class.return_value = mock_creds
mock_youtube = MagicMock()
mock_channels = MagicMock()
mock_channels_list = MagicMock()
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
mock_channels.list.return_value = mock_channels_list
mock_youtube.channels.return_value = mock_channels
mock_search = MagicMock()
mock_search_list = MagicMock()
mock_search_list.execute.return_value = {"items": []}
mock_search.list.return_value = mock_search_list
mock_youtube.search.return_value = mock_search
mock_videos = MagicMock()
mock_insert = MagicMock()
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
mock_videos.insert.return_value = mock_insert
mock_youtube.videos.return_value = mock_videos
# Mock thumbnail upload
mock_thumbnails = MagicMock()
mock_set = MagicMock()
mock_set.execute.return_value = None
mock_thumbnails.set.return_value = mock_set
mock_youtube.thumbnails.return_value = mock_thumbnails
mock_build.return_value = mock_youtube
mock_media_upload.return_value = MagicMock()
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
"client_id": "id", "client_secret": "secret"}
service = YouTubeUploadService(credentials=creds, non_interactive=True)
video_id, video_url = service.upload_video(
video_path="/path/to/video.mkv",
title="Test Title",
description="Test description",
thumbnail_path="/path/to/thumbnail.jpg"
)
assert video_id == "NEW_VIDEO_ID"
mock_thumbnails.set.assert_called_once()
class TestGetYouTubeUploadService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
# Reset global
import backend.services.youtube_upload_service as module
module._youtube_upload_service = None
service = get_youtube_upload_service(
credentials={"token": "test"},
non_interactive=True
)
assert service is not None
assert isinstance(service, YouTubeUploadService)
def test_get_service_with_client_secrets(self):
"""Test factory function with client secrets file."""
import backend.services.youtube_upload_service as module
module._youtube_upload_service = None
service = get_youtube_upload_service(
client_secrets_file="/path/to/secrets.json"
)
assert service is not None
assert service.client_secrets_file == "/path/to/secrets.json"
+36 -8
View File
@@ -36,6 +36,29 @@ from karaoke_gen.audio_processor import AudioProcessor
logger = logging.getLogger(__name__)
async def _trigger_lyrics_worker_after_url_download(job_id: str) -> None:
"""
Trigger lyrics worker after URL audio download completes.
For URL jobs, we use sequential triggering:
1. Audio worker downloads and uploads audio to GCS
2. Audio worker triggers lyrics worker (this function)
3. Both workers then proceed in parallel (audio separation + lyrics transcription)
This prevents the race condition where lyrics worker times out waiting for audio.
"""
from backend.services.worker_service import get_worker_service
try:
worker_service = get_worker_service()
await worker_service.trigger_lyrics_worker(job_id)
logger.info(f"Job {job_id}: Triggered lyrics worker after URL download")
except Exception as e:
# Log but don't fail - audio processing can still continue
# The job will eventually timeout if lyrics worker doesn't run
logger.error(f"Job {job_id}: Failed to trigger lyrics worker: {e}")
# Default model names - used by create_audio_processor and stored in state_data
DEFAULT_CLEAN_MODEL = "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
DEFAULT_BACKING_MODELS = ["mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"]
@@ -445,29 +468,34 @@ async def download_audio(
# Case 3: Fresh URL that needs downloading
if job.url:
logger.info(f"Job {job_id}: Downloading from URL: {job.url}")
# Use provided job_manager or create new one
jm = job_manager_instance or JobManager()
local_path = await download_from_url(
job.url,
temp_dir,
job.artist,
job.url,
temp_dir,
job.artist,
job.title,
job_manager=jm,
job_id=job_id
)
if local_path and os.path.exists(local_path):
# Upload to GCS and update job
gcs_path = f"jobs/{job_id}/input/{os.path.basename(local_path)}"
url = storage.upload_file(local_path, gcs_path)
# Update job with GCS path for lyrics worker
jm.update_job(job_id, {'input_media_gcs_path': gcs_path})
jm.update_file_url(job_id, 'input', 'audio', url)
logger.info(f"Job {job_id}: Downloaded and uploaded audio to GCS: {gcs_path}")
# For URL jobs, trigger lyrics worker now that audio is available
# This is the sequential trigger pattern - audio first, then lyrics
await _trigger_lyrics_worker_after_url_download(job_id)
return local_path
else:
logger.error(f"Job {job_id}: Failed to download from URL: {job.url}")
+129 -18
View File
@@ -32,6 +32,7 @@ from backend.models.job import JobStatus
from karaoke_gen.utils import sanitize_filename
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.lyrics_cache_service import LyricsCacheService
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.workers.style_helper import load_style_config
from backend.services.tracing import job_span, add_span_event, add_span_attribute
@@ -44,6 +45,20 @@ from backend.config import get_settings
logger = logging.getLogger(__name__)
# Timeout for entire transcription process including agentic AI correction (20 minutes)
# This needs to account for:
# - Cloud Run cold start / worker initialization (1-5 min)
# - AudioShake transcription (1-2 min)
# - spaCy model loading for correction (2-3 min on cold start)
# - Agentic AI correction (1-3 min)
TRANSCRIPTION_TIMEOUT_SECONDS = 1200
# Default agentic correction timeout (3 minutes)
# Configurable via AGENTIC_CORRECTION_TIMEOUT_SECONDS environment variable
# If agentic AI takes longer than this, abort and use uncorrected transcription
# Human review will correct any issues
DEFAULT_AGENTIC_TIMEOUT_SECONDS = 180
def _configure_agentic_ai():
"""Configure environment variables for agentic AI correction.
@@ -215,7 +230,40 @@ async def process_lyrics_transcription(job_id: str) -> bool:
raise Exception("Failed to download audio file")
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
download_span.set_attribute("audio_file", os.path.basename(audio_path))
# Set up LyricsTranscriber cache directory and sync from GCS
# This allows reusing cached AudioShake/lyrics API responses across Cloud Run instances
cache_dir = os.path.join(temp_dir, "lyrics-cache")
os.makedirs(cache_dir, exist_ok=True)
os.environ["LYRICS_TRANSCRIBER_CACHE_DIR"] = cache_dir
job_log.info(f"LyricsTranscriber cache dir: {cache_dir}")
# Sync cache from GCS (download any existing cached API responses)
# Note: Cache sync is best-effort - failures should not fail the job
cache_service = LyricsCacheService(storage)
audio_hash = None
lyrics_hash = None
with job_span("sync-cache-from-gcs", job_id):
try:
# Compute cache keys
audio_hash = cache_service.compute_audio_hash(audio_path)
# Use lyrics_artist/lyrics_title for lyrics hash (these are what LyricsTranscriber uses)
lyrics_search_artist = getattr(job, 'lyrics_artist', None) or job.artist
lyrics_search_title = getattr(job, 'lyrics_title', None) or job.title
lyrics_hash = cache_service.compute_lyrics_hash(lyrics_search_artist, lyrics_search_title)
job_log.info(f"Cache keys: audio_hash={audio_hash[:8]}..., lyrics_hash={lyrics_hash[:8]}...")
add_span_attribute("audio_hash", audio_hash[:8])
add_span_attribute("lyrics_hash", lyrics_hash[:8])
# Download relevant cache files from GCS
cache_stats = cache_service.sync_cache_from_gcs(cache_dir, audio_hash, lyrics_hash)
job_log.info(f"Cache sync from GCS: {cache_stats['downloaded']} downloaded, {cache_stats['not_found']} not found")
add_span_attribute("cache_hits", cache_stats["downloaded"])
except Exception as e:
job_log.warning(f"Cache sync from GCS failed (non-fatal): {e}", exc_info=True)
add_span_attribute("cache_sync_error", str(e)[:100])
# Update progress using state_data (don't change status during parallel processing)
# The status is managed at a higher level - workers just track their progress
job_manager.update_state_data(job_id, 'lyrics_progress', {
@@ -285,26 +333,73 @@ async def process_lyrics_transcription(job_id: str) -> bool:
if lyrics_search_title != job.title:
job_log.info(f" Lyrics search title override: {lyrics_search_title}")
logger.info(f"[job:{job_id}] Calling lyrics_processor.transcribe_lyrics()")
# Run transcription + correction
# Run transcription + correction with timeout
# AudioShake typically takes 1-2 minutes, but can hang.
with job_span("audioshake-transcription", job_id) as trans_span:
trans_start = time.time()
add_span_event("transcription_started")
# Run transcription in thread pool to avoid blocking the event loop
# (transcribe_lyrics is synchronous and takes 1-2 minutes)
# Note: transcribe_lyrics internally calls AudioShake and lyrics APIs
with metrics.time_external_api("audioshake", job_id):
result = await asyncio.to_thread(
lyrics_processor.transcribe_lyrics,
input_audio_wav=audio_path,
artist=job.artist, # Original artist for file naming
title=job.title, # Original title for file naming
track_output_dir=temp_dir,
lyrics_artist=lyrics_search_artist, # Override for lyrics search
lyrics_title=lyrics_search_title # Override for lyrics search
# Calculate deadline for agentic correction
# If agentic AI is enabled, the correction loop will check this deadline
# and abort early if exceeded, returning uncorrected transcription for human review
settings = get_settings()
agentic_deadline = None
timeout_seconds = settings.agentic_correction_timeout_seconds
if settings.use_agentic_ai:
agentic_deadline = time.time() + timeout_seconds
job_log.info(
f"Agentic AI enabled with {timeout_seconds}s timeout"
)
# Run transcription in thread pool
# When agentic AI is enabled, wrap with outer timeout as safety net
# Inner deadline check in corrector.py will break out of gap loop gracefully
# Outer timeout catches case where single LLM call hangs for too long
transcription_coro = asyncio.to_thread(
lyrics_processor.transcribe_lyrics,
input_audio_wav=audio_path,
artist=job.artist, # Original artist for file naming
title=job.title, # Original title for file naming
track_output_dir=temp_dir,
lyrics_artist=lyrics_search_artist, # Override for lyrics search
lyrics_title=lyrics_search_title, # Override for lyrics search
agentic_deadline=agentic_deadline, # Deadline for agentic timeout
)
with metrics.time_external_api("audioshake", job_id):
if settings.use_agentic_ai:
# Outer timeout must be generous to allow for AudioShake time
# (can take 2-5+ minutes for long songs) PLUS agentic correction.
# The inner deadline check in corrector.py handles the 3-minute
# correction limit gracefully by returning partial results.
# This outer timeout is a safety net for completely hung LLM calls.
outer_timeout = TRANSCRIPTION_TIMEOUT_SECONDS # Same as non-agentic
try:
result = await asyncio.wait_for(
transcription_coro,
timeout=outer_timeout
)
except asyncio.TimeoutError:
# This should rarely trigger since inner deadline check breaks first
# But provides hard stop if e.g. single LLM call hangs completely
job_log.error(
f"HARD TIMEOUT: Transcription exceeded {outer_timeout}s. "
"This indicates the inner deadline check failed to trigger."
)
raise RuntimeError(
f"Lyrics transcription timed out after {outer_timeout}s"
) from None
else:
# Non-agentic mode: use general timeout (10 minutes)
try:
result = await asyncio.wait_for(
transcription_coro,
timeout=TRANSCRIPTION_TIMEOUT_SECONDS
)
except asyncio.TimeoutError:
raise Exception(f"Transcription timed out after {TRANSCRIPTION_TIMEOUT_SECONDS} seconds")
trans_duration = time.time() - trans_start
trans_span.set_attribute("duration_seconds", trans_duration)
add_span_event("transcription_completed", {"duration_seconds": trans_duration})
@@ -319,7 +414,23 @@ async def process_lyrics_transcription(job_id: str) -> bool:
job_log.info("All lyrics data uploaded successfully")
logger.info(f"[job:{job_id}] All lyrics data uploaded successfully")
# Sync new cache files to GCS (upload any newly created cache entries)
# This persists cache across Cloud Run instances for future jobs
# Note: Cache upload is best-effort - failures should not fail the job
with job_span("sync-cache-to-gcs", job_id):
try:
if audio_hash and lyrics_hash:
upload_stats = cache_service.sync_cache_to_gcs(cache_dir, audio_hash, lyrics_hash)
job_log.info(f"Cache sync to GCS: {upload_stats['uploaded']} uploaded, {upload_stats['skipped']} already existed")
add_span_attribute("cache_uploads", upload_stats["uploaded"])
else:
job_log.warning("Skipping cache upload: missing audio_hash or lyrics_hash")
add_span_attribute("cache_upload_skipped", "missing_hashes")
except Exception as e:
job_log.warning(f"Cache upload to GCS failed (non-fatal): {e}")
add_span_attribute("cache_upload_error", str(e)[:100])
# Update progress using state_data (don't change status during parallel processing)
job_manager.update_state_data(job_id, 'lyrics_progress', {
'stage': 'lyrics_complete',
+473 -93
View File
@@ -1,20 +1,21 @@
"""
Video generation and finalization worker.
This worker uses KaraokeFinalise from the karaoke_gen package to handle:
1. Video encoding (4 formats)
2. CDG/TXT package generation
3. Discord notifications
4. Brand code generation
5. All other finalisation features
This worker has two code paths controlled by USE_NEW_ORCHESTRATOR:
The worker's job is simply to:
1. Download files from GCS and set up the directory structure KaraokeFinalise expects
2. Instantiate KaraokeFinalise with parameters from the job
3. Call process()
4. Upload results back to GCS
1. NEW PATH (USE_NEW_ORCHESTRATOR=true):
Uses VideoWorkerOrchestrator which provides a unified pipeline for all
encoding backends (GCE or local). This ensures features like YouTube upload,
Discord notifications, and CDG/TXT packaging work regardless of encoding backend.
This reuses 100% of the existing karaoke_gen functionality.
2. LEGACY PATH (USE_NEW_ORCHESTRATOR=false):
Uses KaraokeFinalise from the karaoke_gen package. This path has divergent
behavior between GCE and local encoding - GCE encoding bypasses many features.
The orchestrator-based approach is preferred as it:
- Eliminates GCE vs local code path divergence
- Ensures all features work consistently
- Provides better testability and maintainability
Observability:
- All operations wrapped in tracing spans for Cloud Trace visibility
@@ -35,6 +36,7 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.rclone_service import get_rclone_service
from backend.services.youtube_service import get_youtube_service
from backend.services.encoding_service import get_encoding_service
from backend.config import get_settings
from backend.workers.style_helper import load_style_config
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
@@ -47,6 +49,12 @@ from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
logger = logging.getLogger(__name__)
# Feature flag for new orchestrator-based pipeline
# Set to True to use the new unified pipeline that works with any encoding backend
# Set to False to use the legacy KaraokeFinalise-based pipeline
USE_NEW_ORCHESTRATOR = os.environ.get("USE_NEW_ORCHESTRATOR", "true").lower() == "true"
# Loggers to capture for video worker
# Include the full module path to properly capture KaraokeFinalise logs
VIDEO_WORKER_LOGGERS = [
@@ -55,16 +63,363 @@ VIDEO_WORKER_LOGGERS = [
]
async def generate_video(job_id: str) -> bool:
async def _encode_via_gce(
job_id: str,
job,
job_manager: JobManager,
storage: StorageService,
temp_dir: str,
base_name: str,
job_log,
) -> Dict[str, Any]:
"""
Generate final karaoke videos using KaraokeFinalise.
This sets up the directory structure and calls KaraokeFinalise.process()
which handles all encoding, packaging, and notifications.
Encode videos using the high-performance GCE encoding worker.
This offloads FFmpeg encoding to a dedicated C4 GCE instance with
Intel Granite Rapids 3.9 GHz CPU for 2-3x faster encoding.
Args:
job_id: Job ID
job: Job object
job_manager: Job manager instance
storage: Storage service instance
temp_dir: Temporary directory for output files
base_name: Base name for output files (e.g., "Artist - Title")
job_log: Job-specific logger
Returns:
Result dict with paths to encoded files
"""
settings = get_settings()
encoding_service = get_encoding_service()
job_log.info("Using GCE encoding worker for high-performance encoding")
logger.info(f"[job:{job_id}] Using GCE encoding worker")
# Construct GCS paths for the encoding worker
bucket_name = settings.gcs_bucket_name
input_gcs_path = f"gs://{bucket_name}/jobs/{job_id}/"
output_gcs_path = f"gs://{bucket_name}/jobs/{job_id}/encoded/"
# Determine instrumental selection for encoding config
instrumental_selection = job.state_data.get('instrumental_selection', 'clean')
existing_instrumental = getattr(job, 'existing_instrumental_gcs_path', None)
encoding_config = {
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mp4_720p"],
"base_name": base_name,
"instrumental_selection": instrumental_selection,
"existing_instrumental": existing_instrumental,
"ffmpeg_threads": 8, # c4-standard-8 has 8 vCPUs
}
job_log.info(f"Submitting encoding job to GCE worker")
job_log.info(f" Input: {input_gcs_path}")
job_log.info(f" Output: {output_gcs_path}")
def progress_callback(progress: int):
# Update job progress (encoding is 75-95% of total)
scaled_progress = 75 + int(progress * 0.2) # Map 0-100 to 75-95
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.ENCODING,
progress=scaled_progress,
message=f"Encoding videos ({progress}%)"
)
try:
# Submit and wait for encoding
with job_span("gce-encoding", job_id) as encoding_span:
encode_start = time.time()
add_span_event("gce_encoding_started")
result = await encoding_service.encode_videos(
job_id=job_id,
input_gcs_path=input_gcs_path,
output_gcs_path=output_gcs_path,
encoding_config=encoding_config,
progress_callback=progress_callback,
)
encode_duration = time.time() - encode_start
encoding_span.set_attribute("duration_seconds", encode_duration)
add_span_event("gce_encoding_completed", {"duration_seconds": encode_duration})
job_log.info(f"GCE encoding complete in {encode_duration:.1f}s")
logger.info(f"[job:{job_id}] GCE encoding complete in {encode_duration:.1f}s")
# Download encoded files from GCS to temp_dir
output_files = result.get("output_files", [])
local_files = {}
for gcs_path in output_files:
filename = os.path.basename(gcs_path)
local_path = os.path.join(temp_dir, filename)
job_log.info(f"Downloading encoded file: {filename}")
storage.download_file(gcs_path, local_path)
# Map to result keys expected by _upload_results
if "4K Lossless" in filename or "4k_lossless" in filename.lower():
local_files["final_video"] = local_path
elif "4K Lossy" in filename or "4k_lossy" in filename.lower():
local_files["final_video_lossy"] = local_path
elif "720p" in filename:
local_files["final_video_720p"] = local_path
elif filename.endswith(".mkv"):
local_files["final_video_mkv"] = local_path
job_log.info(f"Downloaded {len(local_files)} encoded files")
return local_files
except Exception as e:
job_log.error(f"GCE encoding failed: {e}")
logger.error(f"[job:{job_id}] GCE encoding failed: {e}")
raise
async def generate_video_orchestrated(job_id: str) -> bool:
"""
Generate final karaoke videos using the new VideoWorkerOrchestrator.
This provides a unified pipeline that works with any encoding backend
(GCE or local), ensuring all features like YouTube upload, Discord
notifications, and CDG/TXT packaging work consistently.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
from backend.workers.video_worker_orchestrator import (
VideoWorkerOrchestrator,
create_orchestrator_config_from_job,
)
start_time = time.time()
job_manager = JobManager()
storage = StorageService()
# Create job logger for remote debugging
job_log = create_job_logger(job_id, "video")
# Log with structured markers for easy Cloud Logging queries
logger.info(f"[job:{job_id}] WORKER_START worker=video orchestrator=true")
# Set up log capture for KaraokeFinalise (still used by some services)
log_handler = setup_job_logging(job_id, "video", *VIDEO_WORKER_LOGGERS)
job = job_manager.get_job(job_id)
if not job:
logger.error(f"[job:{job_id}] Job not found")
return False
# Validate prerequisites
if not _validate_prerequisites(job):
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
return False
# Create temporary working directory
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_video_{job_id}_")
original_cwd = os.getcwd()
# Set up rclone config if needed (for legacy Dropbox upload path)
rclone_service = None
if getattr(job, 'organised_dir_rclone_root', None):
rclone_service = get_rclone_service()
if rclone_service.setup_rclone_config():
job_log.info("Rclone config loaded for Dropbox upload")
# Load YouTube credentials if needed
youtube_credentials = None
if getattr(job, 'enable_youtube_upload', False):
youtube_service = get_youtube_service()
if youtube_service.is_configured:
youtube_credentials = youtube_service.get_credentials_dict()
job_log.info("YouTube credentials loaded for video upload")
else:
job_log.warning("YouTube credentials not available - upload will be skipped")
try:
# Wrap entire worker in a tracing span
with job_span("video-worker-orchestrated", job_id, {"artist": job.artist, "title": job.title}) as root_span:
with job_logging_context(job_id):
job_log.info(f"Starting orchestrated video generation for {job.artist} - {job.title}")
logger.info(f"[job:{job_id}] Starting orchestrated video generation")
# Transition to GENERATING_VIDEO state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.GENERATING_VIDEO,
progress=70,
message="Preparing files for video generation"
)
# Download and set up files
base_name = f"{job.artist} - {job.title}"
with job_span("download-files", job_id):
job_log.info("Downloading files from GCS...")
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
job_log.info("All files downloaded successfully")
# Load style config for CDG styles
style_config = await load_style_config(job, storage, temp_dir)
cdg_styles = style_config.get_cdg_styles()
# Change to working directory
os.chdir(temp_dir)
# Create orchestrator config from job
config = create_orchestrator_config_from_job(
job=job,
temp_dir=temp_dir,
youtube_credentials=youtube_credentials,
cdg_styles=cdg_styles,
)
# Run the orchestrated pipeline
orchestrator = VideoWorkerOrchestrator(
config=config,
job_manager=job_manager,
storage=storage,
job_logger=job_log,
)
with job_span("orchestrator-run", job_id):
result = await orchestrator.run()
if not result.success:
raise Exception(result.error_message or "Orchestrator failed")
# Prepare distribution directory for native uploads
with job_span("distribution", job_id):
await _handle_native_distribution(
job_id=job_id,
job=job,
job_log=job_log,
job_manager=job_manager,
temp_dir=temp_dir,
result={
'brand_code': result.brand_code,
'youtube_url': result.youtube_url,
'final_video': result.final_video,
'final_video_lossy': result.final_video_lossy,
'final_video_720p': result.final_video_720p,
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
'dropbox_link': result.dropbox_link,
'gdrive_files': result.gdrive_files,
},
storage=storage,
)
# Upload generated files to GCS
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.PACKAGING,
progress=95,
message="Uploading final files"
)
with job_span("upload-results", job_id):
await _upload_results(job_id, job_manager, storage, temp_dir, {
'final_video': result.final_video,
'final_video_mkv': result.final_video_mkv,
'final_video_lossy': result.final_video_lossy,
'final_video_720p': result.final_video_720p,
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
'final_karaoke_txt_zip': result.final_karaoke_txt_zip,
})
# Mark job as complete
logger.info(f"[job:{job_id}] Video generation complete")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
progress=100,
message="Karaoke generation complete!"
)
# Store result metadata in job
job_manager.update_job(job_id, {
'state_data': {
**job.state_data,
'brand_code': result.brand_code,
'youtube_url': result.youtube_url,
'dropbox_link': result.dropbox_link,
'gdrive_files': result.gdrive_files,
}
})
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.brand_code or '')
logger.info(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=success duration={duration:.1f}s")
return True
except Exception as e:
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Video generation failed: {str(e)}",
error_details={"stage": "video_generation", "error": str(e)}
)
return False
finally:
# Restore original working directory
os.chdir(original_cwd)
# Remove log handler
for logger_name in VIDEO_WORKER_LOGGERS:
try:
logging.getLogger(logger_name).removeHandler(log_handler)
except Exception:
pass
# Cleanup rclone config file
if rclone_service:
rclone_service.cleanup()
# Cleanup temporary directory
if os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
logger.debug(f"Cleaned up temp directory: {temp_dir}")
async def generate_video(job_id: str) -> bool:
"""
Generate final karaoke videos.
Routes to either the new orchestrator-based pipeline or the legacy
KaraokeFinalise-based pipeline based on USE_NEW_ORCHESTRATOR flag.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
if USE_NEW_ORCHESTRATOR:
logger.info(f"[job:{job_id}] Using new orchestrator-based pipeline")
return await generate_video_orchestrated(job_id)
else:
logger.info(f"[job:{job_id}] Using legacy KaraokeFinalise pipeline")
return await generate_video_legacy(job_id)
async def generate_video_legacy(job_id: str) -> bool:
"""
Generate final karaoke videos using KaraokeFinalise (legacy path).
This is the original implementation that has divergent behavior between
GCE and local encoding paths. Kept for rollback purposes.
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
@@ -184,80 +539,105 @@ async def generate_video(job_id: str) -> bool:
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
job_log.info(f"Countdown padding detected: {countdown_padding_seconds}s - instrumental will be padded if needed")
# Log finalization parameters
job_log.info("Creating KaraokeFinalise with parameters:")
job_log.info(f" enable_cdg: {getattr(job, 'enable_cdg', False)}")
job_log.info(f" enable_txt: {getattr(job, 'enable_txt', False)}")
job_log.info(f" brand_prefix: {getattr(job, 'brand_prefix', None)}")
job_log.info(f" discord_webhook: {'configured' if getattr(job, 'discord_webhook_url', None) else 'not configured'}")
job_log.info(f" instrumental source: {instrumental_source}")
job_log.info(f" countdown_padding_seconds: {countdown_padding_seconds}")
if existing_instrumental_path:
job_log.info(f" using user-provided instrumental (selection was: {instrumental_selection})")
# Create KaraokeFinalise with ALL the parameters from the job
# This reuses all existing functionality!
# Set up YouTube description file if template is provided
youtube_desc_path = None
if youtube_credentials and getattr(job, 'youtube_description_template', None):
youtube_desc_path = os.path.join(temp_dir, "youtube_description.txt")
with open(youtube_desc_path, 'w') as f:
f.write(job.youtube_description_template)
job_log.info("YouTube description template written to temp file")
finalise = KaraokeFinalise(
logger=logger,
log_level=logging.INFO,
dry_run=False,
instrumental_format="flac",
# CDG/TXT generation
enable_cdg=getattr(job, 'enable_cdg', False),
enable_txt=getattr(job, 'enable_txt', False),
cdg_styles=cdg_styles,
# Brand code and organization (server-side mode uses rclone)
brand_prefix=getattr(job, 'brand_prefix', None),
organised_dir=None, # Not used in cloud - files stay in GCS
organised_dir_rclone_root=getattr(job, 'organised_dir_rclone_root', None),
public_share_dir=None, # Not used in cloud
# Notifications
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
# YouTube upload (server-side with pre-loaded credentials)
youtube_client_secrets_file=None, # Not used with pre-stored credentials
youtube_description_file=youtube_desc_path,
user_youtube_credentials=youtube_credentials, # Pre-loaded from Secret Manager
rclone_destination=None,
email_template_file=None,
# Server-side optimizations
non_interactive=True,
server_side_mode=True,
selected_instrumental_file=instrumental_file,
# Audio synchronization - ensure instrumental matches vocal padding
countdown_padding_seconds=countdown_padding_seconds,
)
# Call process() - this does ALL the work:
# - Encodes to 4 video formats
# - Generates CDG/TXT packages if enabled
# - Posts Discord notification if configured
# - Handles brand code generation
with job_span("karaoke-finalise", job_id) as finalise_span:
finalise_start = time.time()
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
logger.info(f"[job:{job_id}] Starting KaraokeFinalise.process()")
add_span_event("finalise_started")
result = finalise.process()
finalise_duration = time.time() - finalise_start
finalise_span.set_attribute("duration_seconds", finalise_duration)
add_span_event("finalise_completed", {"duration_seconds": finalise_duration})
job_log.info("KaraokeFinalise.process() complete!")
if result.get('brand_code'):
job_log.info(f"Brand code: {result.get('brand_code')}")
logger.info(f"[job:{job_id}] KaraokeFinalise.process() complete in {finalise_duration:.1f}s")
logger.info(f"[job:{job_id}] Brand code: {result.get('brand_code')}")
# Check if GCE encoding is enabled for high-performance encoding
encoding_service = get_encoding_service()
use_gce_encoding = encoding_service.is_enabled
if use_gce_encoding:
# ============ GCE ENCODING PATH ============
# Use dedicated C4 GCE instance for 2-3x faster FFmpeg encoding
job_log.info("GCE encoding enabled - using high-performance encoding worker")
job_log.info(f" instrumental source: {instrumental_source}")
result = await _encode_via_gce(
job_id=job_id,
job=job,
job_manager=job_manager,
storage=storage,
temp_dir=temp_dir,
base_name=base_name,
job_log=job_log,
)
# GCE encoding doesn't generate brand_code, CDG/TXT, or Discord notifications
# These will be handled by distribution if needed
job_log.info("GCE encoding complete - encoded files downloaded")
else:
# ============ STANDARD ENCODING PATH ============
# Use KaraokeFinalise for encoding (runs on Cloud Run)
# Log finalization parameters
job_log.info("Using KaraokeFinalise for encoding (standard path)")
job_log.info(f" enable_cdg: {getattr(job, 'enable_cdg', False)}")
job_log.info(f" enable_txt: {getattr(job, 'enable_txt', False)}")
job_log.info(f" brand_prefix: {getattr(job, 'brand_prefix', None)}")
job_log.info(f" discord_webhook: {'configured' if getattr(job, 'discord_webhook_url', None) else 'not configured'}")
job_log.info(f" instrumental source: {instrumental_source}")
job_log.info(f" countdown_padding_seconds: {countdown_padding_seconds}")
if existing_instrumental_path:
job_log.info(f" using user-provided instrumental (selection was: {instrumental_selection})")
# Set up YouTube description file if template is provided
youtube_desc_path = None
if youtube_credentials and getattr(job, 'youtube_description_template', None):
youtube_desc_path = os.path.join(temp_dir, "youtube_description.txt")
with open(youtube_desc_path, 'w') as f:
f.write(job.youtube_description_template)
job_log.info("YouTube description template written to temp file")
finalise = KaraokeFinalise(
logger=logger,
log_level=logging.INFO,
dry_run=False,
instrumental_format="flac",
# CDG/TXT generation
enable_cdg=getattr(job, 'enable_cdg', False),
enable_txt=getattr(job, 'enable_txt', False),
cdg_styles=cdg_styles,
# Brand code and organization (server-side mode uses rclone)
brand_prefix=getattr(job, 'brand_prefix', None),
organised_dir=None, # Not used in cloud - files stay in GCS
organised_dir_rclone_root=getattr(job, 'organised_dir_rclone_root', None),
public_share_dir=None, # Not used in cloud
# Notifications
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
# YouTube upload (server-side with pre-loaded credentials)
youtube_client_secrets_file=None, # Not used with pre-stored credentials
youtube_description_file=youtube_desc_path,
user_youtube_credentials=youtube_credentials, # Pre-loaded from Secret Manager
rclone_destination=None,
email_template_file=None,
# Server-side optimizations
non_interactive=True,
server_side_mode=True,
selected_instrumental_file=instrumental_file,
# Audio synchronization - ensure instrumental matches vocal padding
countdown_padding_seconds=countdown_padding_seconds,
)
# Call process() - this does ALL the work:
# - Encodes to 4 video formats
# - Generates CDG/TXT packages if enabled
# - Posts Discord notification if configured
# - Handles brand code generation
with job_span("karaoke-finalise", job_id) as finalise_span:
finalise_start = time.time()
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
logger.info(f"[job:{job_id}] Starting KaraokeFinalise.process()")
add_span_event("finalise_started")
result = finalise.process()
finalise_duration = time.time() - finalise_start
finalise_span.set_attribute("duration_seconds", finalise_duration)
add_span_event("finalise_completed", {"duration_seconds": finalise_duration})
job_log.info("KaraokeFinalise.process() complete!")
if result.get('brand_code'):
job_log.info(f"Brand code: {result.get('brand_code')}")
logger.info(f"[job:{job_id}] KaraokeFinalise.process() complete in {finalise_duration:.1f}s")
logger.info(f"[job:{job_id}] Brand code: {result.get('brand_code')}")
# Native API distribution uploads (used by remote CLI instead of rclone)
with job_span("distribution", job_id):
@@ -0,0 +1,694 @@
"""
Video Worker Orchestrator.
Coordinates the video generation pipeline stages in a unified way,
regardless of whether encoding happens locally or on GCE.
This resolves the code path divergence where GCE encoding bypassed
features like YouTube upload, Discord notifications, and CDG/TXT packaging.
Pipeline stages:
1. Setup - Download files, prepare directories
2. Packaging - CDG/TXT generation (if enabled)
3. Encoding - GCE or Local via EncodingBackend interface
4. Organization - Brand code, folder structure
5. Distribution - YouTube, Dropbox, Google Drive uploads
6. Notification - Discord notifications
"""
import logging
import os
import time
from dataclasses import dataclass, field
from typing import Optional, Dict, Any, List
from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.tracing import job_span, add_span_event
logger = logging.getLogger(__name__)
@dataclass
class OrchestratorConfig:
"""Configuration for the video worker orchestrator."""
job_id: str
artist: str
title: str
# Input file paths (in temp_dir)
title_video_path: str
karaoke_video_path: str
instrumental_audio_path: str
end_video_path: Optional[str] = None
lrc_file_path: Optional[str] = None
title_jpg_path: Optional[str] = None
# Output directory
output_dir: str = ""
# Feature flags
enable_cdg: bool = False
enable_txt: bool = False
enable_youtube_upload: bool = False
# Service configurations
brand_prefix: Optional[str] = None
discord_webhook_url: Optional[str] = None
youtube_credentials: Optional[Dict[str, Any]] = None
youtube_description_template: Optional[str] = None
cdg_styles: Optional[Dict[str, Any]] = None
# Dropbox/GDrive configuration
dropbox_path: Optional[str] = None
gdrive_folder_id: Optional[str] = None
# Keep existing brand code (for re-processing)
keep_brand_code: Optional[str] = None
# Encoding backend preference
encoding_backend: str = "auto" # "auto", "local", "gce"
# Additional options
dry_run: bool = False
non_interactive: bool = True
@dataclass
class OrchestratorResult:
"""Result from the video worker orchestrator."""
success: bool
error_message: Optional[str] = None
# Generated files
final_video: Optional[str] = None # Lossless 4K MP4
final_video_mkv: Optional[str] = None # Lossless 4K MKV
final_video_lossy: Optional[str] = None # Lossy 4K MP4
final_video_720p: Optional[str] = None # Lossy 720p MP4
final_karaoke_cdg_zip: Optional[str] = None
final_karaoke_txt_zip: Optional[str] = None
# Organization
brand_code: Optional[str] = None
# Distribution results
youtube_url: Optional[str] = None
dropbox_link: Optional[str] = None
gdrive_files: Optional[Dict[str, str]] = field(default_factory=dict)
# Timing
encoding_time_seconds: Optional[float] = None
total_time_seconds: Optional[float] = None
class VideoWorkerOrchestrator:
"""
Orchestrates the video generation pipeline.
This class coordinates all stages of video generation in a unified way,
ensuring that features like YouTube upload and Discord notifications
work regardless of whether GCE or local encoding is used.
"""
def __init__(
self,
config: OrchestratorConfig,
job_manager: Optional[JobManager] = None,
storage: Optional[StorageService] = None,
job_logger: Optional[logging.Logger] = None,
):
"""
Initialize the orchestrator.
Args:
config: Orchestrator configuration
job_manager: Job manager for state updates (optional)
storage: Storage service for file downloads (optional)
job_logger: Job-specific logger (optional)
"""
self.config = config
self.job_manager = job_manager
self.storage = storage
self.job_log = job_logger or logging.getLogger(__name__)
# Services (lazy-loaded)
self._encoding_backend = None
self._packaging_service = None
self._youtube_service = None
self._discord_service = None
# State
self.result = OrchestratorResult(success=False)
def _get_encoding_backend(self):
"""Lazy-load the encoding backend."""
if self._encoding_backend is None:
from backend.services.encoding_interface import get_encoding_backend
self._encoding_backend = get_encoding_backend(
backend_type=self.config.encoding_backend,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._encoding_backend
def _get_packaging_service(self):
"""Lazy-load the packaging service."""
if self._packaging_service is None:
from backend.services.packaging_service import PackagingService
self._packaging_service = PackagingService(
cdg_styles=self.config.cdg_styles,
dry_run=self.config.dry_run,
non_interactive=self.config.non_interactive,
logger=self.job_log,
)
return self._packaging_service
def _get_youtube_service(self):
"""Lazy-load the YouTube upload service."""
if self._youtube_service is None:
from backend.services.youtube_upload_service import YouTubeUploadService
self._youtube_service = YouTubeUploadService(
credentials=self.config.youtube_credentials,
non_interactive=self.config.non_interactive,
server_side_mode=True,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._youtube_service
def _get_discord_service(self):
"""Lazy-load the Discord notification service."""
if self._discord_service is None:
from backend.services.discord_service import DiscordNotificationService
self._discord_service = DiscordNotificationService(
webhook_url=self.config.discord_webhook_url,
dry_run=self.config.dry_run,
logger=self.job_log,
)
return self._discord_service
def _update_progress(self, status: JobStatus, progress: int, message: str):
"""Update job progress if job_manager is available."""
if self.job_manager:
self.job_manager.transition_to_state(
job_id=self.config.job_id,
new_status=status,
progress=progress,
message=message
)
async def run(self) -> OrchestratorResult:
"""
Run the full video generation pipeline.
Returns:
OrchestratorResult with generated files and metadata
"""
start_time = time.time()
try:
with job_span("orchestrator", self.config.job_id) as span:
self.job_log.info(f"Starting orchestrated video generation for {self.config.artist} - {self.config.title}")
# Stage 1: Packaging (CDG/TXT) - runs BEFORE encoding
# This depends on LRC file, not on encoded videos
if self.config.enable_cdg or self.config.enable_txt:
await self._run_packaging()
# Stage 2: Encoding
await self._run_encoding()
# Stage 3: Organization (brand code)
await self._run_organization()
# Stage 4: Distribution (YouTube, Dropbox, GDrive)
await self._run_distribution()
# Stage 5: Notifications (Discord)
await self._run_notifications()
self.result.success = True
self.result.total_time_seconds = time.time() - start_time
self.job_log.info(f"Orchestrated video generation complete in {self.result.total_time_seconds:.1f}s")
except Exception as e:
self.result.success = False
self.result.error_message = str(e)
self.result.total_time_seconds = time.time() - start_time
self.job_log.error(f"Orchestrated video generation failed: {e}")
logger.error(f"[job:{self.config.job_id}] Orchestration failed: {e}")
return self.result
async def _run_packaging(self):
"""Run the packaging stage (CDG/TXT generation)."""
self.job_log.info("Starting packaging stage (CDG/TXT)")
if not self.config.lrc_file_path or not os.path.isfile(self.config.lrc_file_path):
self.job_log.warning("No LRC file available, skipping CDG/TXT packaging")
return
base_name = f"{self.config.artist} - {self.config.title}"
packaging_service = self._get_packaging_service()
# Generate CDG package
if self.config.enable_cdg:
self.job_log.info("Generating CDG package")
try:
cdg_zip_path = os.path.join(
self.config.output_dir,
f"{base_name} (Final Karaoke CDG).zip"
)
mp3_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).mp3"
)
cdg_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).cdg"
)
zip_file, mp3_file, cdg_file = packaging_service.create_cdg_package(
lrc_file=self.config.lrc_file_path,
audio_file=self.config.instrumental_audio_path,
output_zip_path=cdg_zip_path,
artist=self.config.artist,
title=self.config.title,
output_mp3_path=mp3_path,
output_cdg_path=cdg_path,
)
self.result.final_karaoke_cdg_zip = zip_file
self.job_log.info(f"CDG package created: {zip_file}")
except Exception as e:
self.job_log.error(f"CDG generation failed: {e}")
# Don't fail the pipeline, CDG is optional
# Generate TXT package
if self.config.enable_txt:
self.job_log.info("Generating TXT package")
try:
# TXT package needs MP3 file (from CDG generation or create it)
mp3_path = os.path.join(
self.config.output_dir,
f"{base_name} (Karaoke).mp3"
)
if not os.path.isfile(mp3_path):
self.job_log.info("MP3 not found, CDG must be enabled first for TXT")
return
txt_zip_path = os.path.join(
self.config.output_dir,
f"{base_name} (Final Karaoke TXT).zip"
)
zip_file, txt_file = packaging_service.create_txt_package(
lrc_file=self.config.lrc_file_path,
mp3_file=mp3_path,
output_zip_path=txt_zip_path,
)
self.result.final_karaoke_txt_zip = zip_file
self.job_log.info(f"TXT package created: {zip_file}")
except Exception as e:
self.job_log.error(f"TXT generation failed: {e}")
# Don't fail the pipeline, TXT is optional
async def _run_encoding(self):
"""Run the encoding stage."""
self.job_log.info("Starting encoding stage")
self._update_progress(JobStatus.ENCODING, 75, "Encoding videos")
encoding_backend = self._get_encoding_backend()
self.job_log.info(f"Using encoding backend: {encoding_backend.name}")
# Build encoding input
from backend.services.encoding_interface import EncodingInput
from backend.config import settings
# For GCE encoding, we need to provide GCS paths
gcs_bucket = settings.gcs_bucket_name
input_gcs_path = f"gs://{gcs_bucket}/jobs/{self.config.job_id}/"
output_gcs_path = f"gs://{gcs_bucket}/jobs/{self.config.job_id}/finals/"
encoding_input = EncodingInput(
title_video_path=self.config.title_video_path,
karaoke_video_path=self.config.karaoke_video_path,
instrumental_audio_path=self.config.instrumental_audio_path,
end_video_path=self.config.end_video_path,
artist=self.config.artist,
title=self.config.title,
brand_code=self.config.keep_brand_code,
output_dir=self.config.output_dir,
options={
"job_id": self.config.job_id,
"input_gcs_path": input_gcs_path,
"output_gcs_path": output_gcs_path,
},
)
# Run encoding
with job_span("encoding", self.config.job_id) as span:
add_span_event("encoding_started", {"backend": encoding_backend.name})
output = await encoding_backend.encode(encoding_input)
add_span_event("encoding_completed", {
"success": output.success,
"duration": output.encoding_time_seconds or 0
})
if not output.success:
raise Exception(f"Encoding failed: {output.error_message}")
# Store results - for GCE backend, these are GCS blob paths that need to be downloaded
self.result.final_video = output.lossless_4k_mp4_path
self.result.final_video_mkv = output.lossless_mkv_path
self.result.final_video_lossy = output.lossy_4k_mp4_path
self.result.final_video_720p = output.lossy_720p_mp4_path
self.result.encoding_time_seconds = output.encoding_time_seconds
# For GCE encoding, download the encoded files from GCS to local directory
# This is required for YouTube upload and other local file operations
if encoding_backend.name == "gce" and self.storage:
await self._download_gce_encoded_files(output)
self.job_log.info(f"Encoding complete ({encoding_backend.name}) in {output.encoding_time_seconds:.1f}s")
async def _download_gce_encoded_files(self, output):
"""
Download GCE-encoded files from GCS to the local output directory.
GCE encoding stores files in GCS and returns blob paths like:
'jobs/{job_id}/finals/output_4k_lossless.mp4'
This method downloads those files locally so that subsequent stages
(YouTube upload, etc.) can access them as local files.
Args:
output: EncodingOutput from the GCE backend with GCS blob paths
"""
self.job_log.info("Downloading GCE-encoded files from GCS")
# Map of result attributes to download
file_mappings = [
('lossless_4k_mp4_path', 'final_video'),
('lossless_mkv_path', 'final_video_mkv'),
('lossy_4k_mp4_path', 'final_video_lossy'),
('lossy_720p_mp4_path', 'final_video_720p'),
]
downloaded_count = 0
for output_attr, result_attr in file_mappings:
gcs_path = getattr(output, output_attr, None)
if not gcs_path:
continue
# Extract filename from GCS path
filename = os.path.basename(gcs_path)
local_path = os.path.join(self.config.output_dir, filename)
try:
self.job_log.info(f"Downloading {filename} from GCS")
self.storage.download_file(gcs_path, local_path)
# Update the result to point to local file
setattr(self.result, result_attr, local_path)
downloaded_count += 1
self.job_log.info(f"Downloaded {filename} to {local_path}")
except Exception as e:
self.job_log.error(f"Failed to download {filename}: {e}")
# Clear the result attribute so downstream doesn't try to use invalid GCS path
setattr(self.result, result_attr, None)
# Don't fail - some formats might not be generated
self.job_log.info(f"Downloaded {downloaded_count} encoded files from GCS")
async def _run_organization(self):
"""Run the organization stage (brand code generation)."""
self.job_log.info("Starting organization stage")
# Use existing brand code if provided
if self.config.keep_brand_code:
self.result.brand_code = self.config.keep_brand_code
self.job_log.info(f"Using preserved brand code: {self.result.brand_code}")
return
# Generate brand code from Dropbox if configured
if self.config.dropbox_path and self.config.brand_prefix:
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if dropbox.is_configured:
brand_code = dropbox.get_next_brand_code(
self.config.dropbox_path,
self.config.brand_prefix
)
self.result.brand_code = brand_code
self.job_log.info(f"Generated brand code: {brand_code}")
else:
self.job_log.warning("Dropbox not configured, skipping brand code generation")
except Exception as e:
self.job_log.error(f"Brand code generation failed: {e}")
# Don't fail - brand code is optional
async def _run_distribution(self):
"""Run the distribution stage (YouTube, Dropbox, GDrive uploads)."""
self.job_log.info("Starting distribution stage")
self._update_progress(JobStatus.PACKAGING, 90, "Uploading files")
# YouTube upload
if self.config.enable_youtube_upload and self.config.youtube_credentials:
await self._upload_to_youtube()
# Dropbox upload
if self.config.dropbox_path and self.config.brand_prefix:
await self._upload_to_dropbox()
# Google Drive upload
if self.config.gdrive_folder_id:
await self._upload_to_gdrive()
async def _upload_to_youtube(self):
"""Upload video to YouTube."""
self.job_log.info("Uploading to YouTube")
# Find the best video file to upload (prefer MKV for FLAC audio, then lossless MP4)
video_to_upload = None
if self.result.final_video_mkv and os.path.isfile(self.result.final_video_mkv):
video_to_upload = self.result.final_video_mkv
elif self.result.final_video and os.path.isfile(self.result.final_video):
video_to_upload = self.result.final_video
elif self.result.final_video_lossy and os.path.isfile(self.result.final_video_lossy):
video_to_upload = self.result.final_video_lossy
if not video_to_upload:
self.job_log.warning("No video file available for YouTube upload")
return
try:
youtube_service = self._get_youtube_service()
# Build video title
title = f"{self.config.artist} - {self.config.title} (Karaoke)"
# Build description
description = self.config.youtube_description_template or ""
if self.result.brand_code:
description = f"{description}\n\nBrand Code: {self.result.brand_code}".strip()
# Upload
video_id, video_url = youtube_service.upload_video(
video_path=video_to_upload,
title=title,
description=description,
thumbnail_path=self.config.title_jpg_path,
tags=["karaoke", self.config.artist, self.config.title],
replace_existing=True, # Server-side always replaces
)
if video_url:
self.result.youtube_url = video_url
self.job_log.info(f"Uploaded to YouTube: {video_url}")
else:
self.job_log.warning("YouTube upload did not return a URL")
except Exception as e:
self.job_log.error(f"YouTube upload failed: {e}")
# Don't fail the pipeline - YouTube is optional
async def _upload_to_dropbox(self):
"""Upload files to Dropbox."""
self.job_log.info("Uploading to Dropbox")
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if not dropbox.is_configured:
self.job_log.warning("Dropbox not configured, skipping upload")
return
base_name = f"{self.config.artist} - {self.config.title}"
folder_name = f"{self.result.brand_code or 'TRACK-0000'} - {base_name}"
remote_folder = f"{self.config.dropbox_path}/{folder_name}"
# Upload entire output directory
dropbox.upload_folder(self.config.output_dir, remote_folder)
# Create sharing link
try:
sharing_link = dropbox.create_shared_link(remote_folder)
self.result.dropbox_link = sharing_link
self.job_log.info(f"Dropbox sharing link: {sharing_link}")
except Exception as e:
self.job_log.warning(f"Failed to create Dropbox sharing link: {e}")
self.job_log.info("Dropbox upload complete")
except Exception as e:
self.job_log.error(f"Dropbox upload failed: {e}")
# Don't fail the pipeline - Dropbox is optional
async def _upload_to_gdrive(self):
"""Upload files to Google Drive."""
self.job_log.info("Uploading to Google Drive")
try:
from backend.services.gdrive_service import get_gdrive_service
gdrive = get_gdrive_service()
if not gdrive.is_configured:
self.job_log.warning("Google Drive not configured, skipping upload")
return
base_name = f"{self.config.artist} - {self.config.title}"
brand_code = self.result.brand_code or f"{self.config.brand_prefix or 'TRACK'}-0000"
# Map result files to expected keys
output_files = {
'final_karaoke_lossy_mp4': self.result.final_video_lossy,
'final_karaoke_lossy_720p_mp4': self.result.final_video_720p,
'final_karaoke_cdg_zip': self.result.final_karaoke_cdg_zip,
}
uploaded = gdrive.upload_to_public_share(
root_folder_id=self.config.gdrive_folder_id,
brand_code=brand_code,
base_name=base_name,
output_files=output_files,
)
self.result.gdrive_files = uploaded
self.job_log.info(f"Google Drive upload complete: {len(uploaded)} files")
except Exception as e:
self.job_log.error(f"Google Drive upload failed: {e}")
# Don't fail the pipeline - GDrive is optional
async def _run_notifications(self):
"""Run the notifications stage (Discord)."""
self.job_log.info("Starting notifications stage")
if not self.config.discord_webhook_url:
self.job_log.debug("No Discord webhook configured, skipping notification")
return
if not self.result.youtube_url:
self.job_log.info("No YouTube URL available, skipping Discord notification")
return
try:
discord_service = self._get_discord_service()
discord_service.post_video_notification(self.result.youtube_url)
self.job_log.info("Discord notification sent")
except Exception as e:
self.job_log.error(f"Discord notification failed: {e}")
# Don't fail the pipeline - notifications are optional
def create_orchestrator_config_from_job(
job,
temp_dir: str,
youtube_credentials: Optional[Dict[str, Any]] = None,
cdg_styles: Optional[Dict[str, Any]] = None,
) -> OrchestratorConfig:
"""
Create an OrchestratorConfig from a job object.
This is a helper function to bridge the existing job structure
with the new orchestrator configuration.
Args:
job: Job object from Firestore
temp_dir: Temporary directory with downloaded files
youtube_credentials: Pre-loaded YouTube credentials
cdg_styles: CDG style configuration
Returns:
OrchestratorConfig for the orchestrator
"""
base_name = f"{job.artist} - {job.title}"
# Determine instrumental file path
instrumental_selection = job.state_data.get('instrumental_selection', 'clean')
existing_instrumental = getattr(job, 'existing_instrumental_gcs_path', None)
if existing_instrumental:
ext = Path(existing_instrumental).suffix.lower()
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
else:
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
return OrchestratorConfig(
job_id=job.job_id,
artist=job.artist,
title=job.title,
# Input files
title_video_path=os.path.join(temp_dir, f"{base_name} (Title).mov"),
karaoke_video_path=os.path.join(temp_dir, f"{base_name} (With Vocals).mov"),
instrumental_audio_path=instrumental_path,
end_video_path=os.path.join(temp_dir, f"{base_name} (End).mov"),
lrc_file_path=os.path.join(temp_dir, f"{base_name} (Karaoke).lrc"),
title_jpg_path=os.path.join(temp_dir, f"{base_name} (Title).jpg"),
# Output directory
output_dir=temp_dir,
# Feature flags
enable_cdg=getattr(job, 'enable_cdg', False),
enable_txt=getattr(job, 'enable_txt', False),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
# Service configurations
brand_prefix=getattr(job, 'brand_prefix', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_credentials=youtube_credentials,
youtube_description_template=getattr(job, 'youtube_description_template', None),
cdg_styles=cdg_styles,
# Dropbox/GDrive
dropbox_path=getattr(job, 'dropbox_path', None),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
# Keep existing brand code
keep_brand_code=getattr(job, 'keep_brand_code', None),
# Encoding backend - auto selects GCE if available
encoding_backend="auto",
# Server-side defaults
dry_run=False,
non_interactive=True,
)
+1 -1
View File
@@ -75,7 +75,7 @@ steps:
--concurrency 1 \
--max-instances 50 \
--min-instances 0 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
images:
+215
View File
@@ -272,6 +272,221 @@ Content-Type: application/json
Bonus credit for detailed feedback (50+ chars).
## Admin Endpoints
All admin endpoints require an admin-role session token.
### Admin Dashboard Stats
```http
GET /api/admin/stats/overview
Authorization: Bearer ADMIN_TOKEN
```
Returns platform statistics:
```json
{
"total_users": 100,
"active_users_7d": 25,
"active_users_30d": 60,
"total_jobs": 500,
"jobs_last_7d": 50,
"jobs_last_30d": 150,
"jobs_by_status": {
"pending": 5,
"processing": 3,
"awaiting_review": 10,
"complete": 400,
"failed": 20
},
"total_credits_issued_30d": 200,
"total_beta_testers": 30
}
```
### List Users (Admin)
```http
GET /api/users/admin/users
GET /api/users/admin/users?search=user@example.com
GET /api/users/admin/users?limit=20&offset=0&sort_by=created_at&sort_order=desc
Authorization: Bearer ADMIN_TOKEN
```
Returns paginated user list with total count.
### User Detail (Admin)
```http
GET /api/users/admin/users/{email}/detail
Authorization: Bearer ADMIN_TOKEN
```
Returns full user profile including:
- User info and stats
- Recent credit transactions (last 20)
- Recent jobs (last 10)
- Active sessions count
### Add Credits (Admin)
```http
POST /api/users/admin/credits
Authorization: Bearer ADMIN_TOKEN
Content-Type: application/json
{
"email": "user@example.com",
"amount": 5,
"reason": "Beta reward"
}
```
### Set User Role (Admin)
```http
POST /api/users/admin/users/{email}/role
Authorization: Bearer ADMIN_TOKEN
Content-Type: application/json
{"role": "admin"}
```
### Enable/Disable User (Admin)
```http
POST /api/users/admin/users/{email}/enable
POST /api/users/admin/users/{email}/disable
Authorization: Bearer ADMIN_TOKEN
```
### Beta Stats (Admin)
```http
GET /api/users/admin/beta/stats
Authorization: Bearer ADMIN_TOKEN
```
### Beta Feedback List (Admin)
```http
GET /api/users/admin/beta/feedback?limit=50
Authorization: Bearer ADMIN_TOKEN
```
### Delete Job (Admin)
```http
DELETE /api/jobs/{job_id}?delete_files=true
Authorization: Bearer ADMIN_TOKEN
```
Admins can delete any job. Regular users can only delete their own jobs.
### Audio Search Management (Admin)
#### List Audio Searches
```http
GET /api/admin/audio-searches
GET /api/admin/audio-searches?limit=50&status_filter=awaiting_audio_selection
Authorization: Bearer ADMIN_TOKEN
```
Returns jobs with cached audio search results. Useful for:
- Monitoring search activity
- Identifying stale cached results (YouTube-only when lossless should be available)
- Clearing cache for specific jobs
Response:
```json
{
"jobs": [
{
"job_id": "abc123",
"status": "awaiting_audio_selection",
"user_email": "user@example.com",
"audio_search_artist": "Artist Name",
"audio_search_title": "Song Title",
"created_at": "2026-01-03T12:00:00Z",
"results_count": 5,
"has_lossless": false,
"providers": ["YouTube"],
"results_summary": [...]
}
],
"total": 1
}
```
#### Clear Audio Search Cache
```http
POST /api/admin/audio-searches/{job_id}/clear-cache
Authorization: Bearer ADMIN_TOKEN
```
Clears cached search results and resets job to `pending` status, allowing a new search.
Use when:
- Cached results are stale (e.g., flacfetch was updated with new providers)
- User wants to search again
- Results appear incomplete (YouTube-only when lossless should exist)
Response:
```json
{
"status": "success",
"job_id": "abc123",
"message": "Cleared 5 cached search results. Job reset to pending. Flacfetch cache also cleared.",
"previous_status": "awaiting_audio_selection",
"new_status": "pending",
"results_cleared": 5,
"flacfetch_cache_cleared": true,
"flacfetch_error": null
}
```
Note: This endpoint now also clears the flacfetch GCS cache for the artist/title combination, ensuring the next search hits trackers fresh.
#### Clear All Flacfetch Cache
```http
DELETE /api/admin/cache
Authorization: Bearer ADMIN_TOKEN
```
Clears the entire flacfetch search cache (GCS-backed). Use when flacfetch has been updated and you want all subsequent searches to use fresh tracker results.
Response:
```json
{
"status": "success",
"message": "Cleared 15 cache entries from flacfetch.",
"deleted_count": 15
}
```
#### Get Flacfetch Cache Stats
```http
GET /api/admin/cache/stats
Authorization: Bearer ADMIN_TOKEN
```
Returns statistics about the flacfetch search cache.
Response:
```json
{
"count": 42,
"total_size_bytes": 128000,
"oldest_entry": "2025-12-15T10:30:00Z",
"newest_entry": "2026-01-03T15:45:00Z",
"configured": true
}
```
## Rate Limits
No rate limits currently implemented.
+65 -7
View File
@@ -23,12 +23,16 @@
│ • Video Worker - Final assembly and encoding │
└───────────────────────────┬─────────────────────────────────────┘
│
┌─────────────┼─────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────────┐
│ Firestore│ │ GCS │ │Secret Manager│
│ (jobs) │ │ (files) │ │ (API keys) │
└──────────┘ └──────────┘ └──────────────┘
┌─────────────┼─────────────┬───────────────┐
▼ ▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────────┐
│ Firestore│ │ GCS │ │Secret Manager│ │GCE Encoding │
│ (jobs) │ │ (files) │ │ (API keys) │ │ Worker (*) │
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
(*) GCE Encoding Worker: C4-standard-8 VM with Intel Granite Rapids for
high-performance FFmpeg encoding. Used for both final video encoding and
preview video generation. Optional - falls back to Cloud Run.
```
## Processing Pipeline
@@ -54,7 +58,7 @@
▼
4. HUMAN REVIEW (AWAITING_REVIEW)
• React UI for lyrics correction
• Preview video generation
• Preview video generation (optionally via GCE for speed)
• Submit corrections
▼
5. VIDEO RENDERING
@@ -152,6 +156,60 @@ LyricsTranscriber LyricsTranscriber
| Genius | Reference lyrics | Yes |
| YouTube API | Video upload | Optional |
## Firestore Collections
karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses separate Firestore collections:
| Collection | Purpose | Key Fields |
|------------|---------|------------|
| `gen_users` | karaoke-gen user accounts | email, credits, role, is_active |
| `jobs` | Karaoke generation jobs | job_id, user_email, status, state_data |
| `jobs/{job_id}/logs` | Worker log entries (subcollection) | timestamp, level, worker, message, ttl_expiry |
| `sessions` | Magic link auth sessions | user_email, token, expires_at |
| `magic_links` | Passwordless auth tokens | email, token, expires_at, used |
| `beta_feedback` | Beta program feedback | user_email, ratings, comments |
**Note**: The `users` collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
**Worker Logs**: Stored in subcollection (`jobs/{job_id}/logs`) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: `USE_LOG_SUBCOLLECTION` (default: true). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
## Video Worker Orchestrator
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
```text
┌─────────────────────────────────────────────────────────────┐
│ VideoWorkerOrchestrator │
│ │
│ Stage 1: PACKAGING Stage 2: ENCODING │
│ ├─ CDG file generation ├─ Local (FFmpeg) │
│ └─ TXT lyrics export └─ GCE (remote VM) │
│ │
│ Stage 3: ORGANIZATION Stage 4: DISTRIBUTION │
│ ├─ Final path setup ├─ Dropbox upload │
│ └─ File organization ├─ Google Drive upload │
│ └─ YouTube upload │
│ │
│ Stage 5: NOTIFICATIONS │
│ └─ Discord webhooks │
└─────────────────────────────────────────────────────────────┘
```
**Key Design Decision**: The orchestrator coordinates stages rather than embedding logic in monolithic functions. This:
- Ensures all features work with both local and GCE encoding
- Makes each stage independently testable (139 new tests)
- Allows feature flag rollback via `USE_NEW_ORCHESTRATOR` env var
**Service Modules**:
| Module | Purpose |
|--------|---------|
| `youtube_upload_service.py` | YouTube OAuth flow and upload |
| `discord_service.py` | Discord webhook notifications |
| `packaging_service.py` | CDG/TXT package generation |
| `local_encoding_service.py` | FFmpeg encoding (Cloud Run) |
| `encoding_interface.py` | Abstract interface + GCE backend |
| `video_worker_orchestrator.py` | Stage coordination |
## Tech Stack
- **Backend**: FastAPI, Python 3.12, Cloud Run
+11 -1
View File
@@ -151,7 +151,11 @@ ADMIN_TOKENS=token1,token2
```bash
NEXT_PUBLIC_API_URL=http://localhost:8000
KARAOKE_ACCESS_TOKEN=... # For E2E tests
# For E2E tests (see frontend/.env.local.example)
KARAOKE_ACCESS_TOKEN=... # Skip email enrollment
TESTMAIL_API_KEY=... # testmail.app API key
TESTMAIL_NAMESPACE=... # testmail.app namespace
```
## Key Scripts
@@ -162,6 +166,12 @@ KARAOKE_ACCESS_TOKEN=... # For E2E tests
| `./scripts/run-backend-local.sh` | Start backend with emulators |
| `./scripts/start-emulators.sh` | Start emulators only |
| `./scripts/stop-emulators.sh` | Stop emulators |
| `./scripts/debug-job.sh <job_id>` | Debug a cloud job (status, logs, files) |
| `./scripts/get_job.py <job_id>` | Fetch job data from Firestore |
| `./scripts/fetch_job_logs.py <job_id>` | Fetch cloud job logs to local file |
| `./scripts/analyze_log_timing.py <log>` | Profile karaoke-gen performance from logs |
| `./scripts/benchmark_ffmpeg.py` | Isolated FFmpeg encoding benchmark |
| `./scripts/compare_local_vs_remote.py` | Compare local vs cloud karaoke-gen performance |
## Project Structure
+239
View File
@@ -0,0 +1,239 @@
# E2E Happy Path Test Specification
> **STATUS**: ✅ **PASSING** - Test completes all 12 steps in ~38 minutes. Last verified 2026-01-02.
>
> **IMPORTANT**: This document defines the requirements for a comprehensive end-to-end test that validates the entire karaoke generation flow with REAL user interactions only. NO API shortcuts allowed.
## Overview
**Goal**: A single Playwright test that validates the complete karaoke generation journey from first-time user signup through to distribution, using only browser interactions like a real user would.
**Environment**: Production (`gen.nomadkaraoke.com` / `api.nomadkaraoke.com`)
**Automation**: Runs daily via GitHub Actions and can be triggered on-demand.
## Test Requirements
### 1. New User Registration (testmail.app)
- [ ] Navigate to landing page (`gen.nomadkaraoke.com`)
- [ ] Click "Sign In" or "Get Started" button
- [ ] Generate a unique test email address (testmail.app namespace)
- [ ] Enter the test email in the beta enrollment form
- [ ] Submit the form
- [ ] Wait for welcome/magic-link email via testmail.app API
- [ ] Extract the magic link from the email
- [ ] Click/navigate to the magic link to authenticate
- [ ] Verify user is logged in and has credits
**Note**: When `use_test_token=true` workflow input is set, steps 1-9 above are skipped and the test uses a pre-authenticated token instead.
### 2. Create Karaoke Job (UI Only)
- [ ] Navigate to the karaoke creation page
- [ ] Enter artist name: `piri` (test song with cached results)
- [ ] Enter song title: `dog`
- [ ] Click "Search" or equivalent button
- [ ] Wait for search results to appear in the UI
### 3. Audio Selection (UI Only)
- [ ] Audio selection dialog/panel should appear
- [ ] View available audio options in the UI
- [ ] Click to select the first/best audio option
- [ ] Confirm the selection via UI button
- [ ] Wait for job to transition to processing state
### 4. Wait for Lyrics Transcription
- [ ] Monitor job status in the UI (not via API polling)
- [ ] Wait for job to reach "awaiting lyrics review" state
- [ ] This may take 10-15+ minutes for audio download + transcription
### 5. Lyrics Review (FULL UI Interaction)
> **CRITICAL**: This MUST be done through the UI, not via API calls
- [ ] Click to open the lyrics review interface
- [ ] Verify lyrics are displayed in the editor
- [ ] Interact with the lyrics editor:
- [ ] View synced lyrics with timestamps
- [ ] Optionally make a small edit to prove UI interaction works
- [ ] Preview the video if preview functionality exists
- [ ] Click "Approve" / "Submit" / "Complete Review" button in UI
- [ ] Wait for job to transition to next state
### 6. Instrumental Selection (FULL UI Interaction)
> **CRITICAL**: This MUST be done through the UI, not via API calls
- [ ] Wait for instrumental options to become available
- [ ] Instrumental selection dialog/panel should appear in UI
- [ ] View available instrumental options (e.g., "clean", "with backing vocals")
- [ ] Click to select the preferred instrumental option
- [ ] Confirm selection via UI button
- [ ] Wait for job to transition to video rendering state
### 7. Wait for Video Rendering & Encoding
- [ ] Monitor job progress in the UI
- [ ] Wait for video rendering to complete
- [ ] Wait for final encoding to complete
- [ ] Job should reach "complete" status
### 8. Verify Completion & Downloads
- [ ] Verify job shows as "complete" in the UI
- [ ] Verify download links are visible and clickable
- [ ] Click at least one download link to verify it works
- [ ] Verify the downloaded file is valid (non-zero size, correct format)
### 9. Distribution Verification & Cleanup
> **CRITICAL**: Test MUST clean up after itself to avoid polluting production distribution channels
- [ ] Verify job.state_data contains distribution results:
- [ ] `brand_code` - The generated brand code (e.g., "NOMAD-1234")
- [ ] `youtube_url` - YouTube video URL (if YouTube enabled)
- [ ] `dropbox_link` - Dropbox shared link (if Dropbox enabled)
- [ ] `gdrive_files` - Google Drive file IDs (if GDrive enabled)
- [ ] **CLEANUP** - After verification, delete all distributed content:
- [ ] Delete YouTube video via API (using stored youtube_url)
- [ ] Delete Dropbox folder via API (using brand_code + dropbox_path)
- [ ] Delete Google Drive files via API (using gdrive_files IDs)
- [ ] Delete the test job from the system (GCS + Firestore)
## Technical Requirements
### Timeouts
| Stage | Timeout |
|-------|---------|
| Page navigation | 30 seconds |
| Email arrival | 120 seconds |
| Audio search | 60 seconds |
| Lyrics transcription | 20 minutes |
| Video rendering | 15 minutes |
| Final encoding | 10 minutes |
| **Total test** | **60 minutes** |
### Environment Variables
```bash
TESTMAIL_NAMESPACE=xxx # Required - for email testing (testmail.app)
TESTMAIL_API_KEY=xxx # Required - for email testing (testmail.app)
E2E_TEST_TOKEN=xxx # Optional - skip signup, use existing account
# When use_test_token=true, skips email signup and uses pre-authenticated token
```
### Test Song
Use `piri - dog` because:
- Has cached flacfetch results (faster audio search)
- Known to work reliably
- Short song = faster processing
### Screenshots & Artifacts
Capture screenshots at each major milestone:
1. Landing page loaded
2. Beta form submitted
3. Email received
4. Logged in successfully
5. Job created
6. Audio selected
7. Lyrics review opened
8. Lyrics approved
9. Instrumental selected
10. Job completed
11. Download verified
### Logging
Log detailed progress at each step with timestamps for debugging.
## GitHub Actions Workflow
**File**: `.github/workflows/e2e-happy-path.yml`
### Triggers
1. **Daily schedule**: Run at 6 AM UTC every day
2. **Manual dispatch**: Can be triggered on-demand with optional parameters
3. **Label trigger**: When `e2e-full` label added to PR
### Secrets Required
- `TESTMAIL_API_KEY` - For email testing (testmail.app)
- `TESTMAIL_NAMESPACE` - testmail.app namespace for unique email addresses
### Workflow Steps
1. Checkout code
2. Setup Node.js
3. Install dependencies
4. Install Playwright browsers
5. Run the e2e happy path test
6. Upload artifacts (screenshots, videos, logs)
7. Report results
### Success Criteria
- All UI interactions complete successfully
- Job reaches "complete" status
- Download link works
- No JavaScript errors in console
- All screenshots captured
## What This Test Proves
When this test passes, it proves:
1. **User signup works** - New users can register via magic link
2. **Authentication works** - Email verification and login flow functional
3. **Job creation works** - Can search and create karaoke jobs
4. **Audio pipeline works** - Audio search, selection, and download functional
5. **Transcription works** - Lyrics transcription completes successfully
6. **Lyrics review UI works** - Users can review and approve lyrics
7. **Instrumental selection UI works** - Users can select instrumental versions
8. **Video rendering works** - Full video generation pipeline functional
9. **Downloads work** - Users can download their completed karaoke videos
10. **End-to-end integration** - All systems work together correctly
## Anti-Patterns to Avoid
1. **NO API calls to complete review** - Must click buttons in UI
2. **NO API calls to select instrumental** - Must click buttons in UI
3. **NO pre-authenticated tokens** - Must go through full signup
4. **NO skipping steps** - Every user-facing interaction must happen
5. **NO hardcoded waits** - Use proper Playwright waiting strategies
## File Location
**Test file**: `frontend/e2e/production/happy-path-real-user.spec.ts`
This is separate from the existing `full-user-journey.spec.ts` which uses API shortcuts.
## Iteration Plan
1. ~~Create initial test with all steps stubbed~~
2. ~~Implement each section one at a time~~
3. ~~Run via GHA workflow~~
4. ~~Debug failures using artifacts~~
5. ~~Iterate until fully passing~~ ✅ Completed 2026-01-02
6. Add to required CI checks once stable
## Key Fixes Applied
During iteration, three issues were fixed:
1. **STEP 8 status detection**: "rendering" status comes BEFORE instrumental selection, not after. Changed to only skip on "encoding".
2. **Instrumental page navigation**: Changed `waitUntil: 'networkidle'` to `'load'` because audio players prevent networkidle.
3. **Download verification**: Changed HEAD to GET request with `maxRedirects: 0` since download endpoint doesn't support HEAD.
See [LESSONS-LEARNED.md](LESSONS-LEARNED.md) for details.
---
*Last updated: 2026-01-02*
*Author: Claude (via Andrew's request)*
+764
View File
@@ -4,6 +4,25 @@ Key insights for future AI agents working on this codebase.
## Architecture Decisions
### Separate Collections for Multi-App Projects
**Problem**: karaoke-gen and karaoke-decide shared the same `users` Firestore collection with incompatible schemas. karaoke-gen users have `credits`, `role`, `is_active`; karaoke-decide users have `user_id`, `is_guest`, `quiz_*` fields.
**Symptoms**:
- Admin dashboard showed users from both apps mixed together
- Potential for data corruption if wrong app writes to wrong user
- Hard to run collection-wide queries (indexes, aggregations) for one app
**Solution**: Migrate karaoke-gen to dedicated `gen_users` collection. Created migration script that:
1. Identified karaoke-gen users by schema (has `credits` + `role` + `is_active`, lacks `user_id` + `is_guest`)
2. Copied 42 users to `gen_users` with same document IDs (email addresses)
3. Updated Pulumi indexes from `users` to `gen_users`
4. Left karaoke-decide's `users` collection untouched
**Key insight**: When multiple apps share a GCP project, use distinct collection names from day one. Prefixing collections with app name (e.g., `gen_users`, `decide_users`) avoids schema conflicts and makes ownership clear.
**Migration script**: `scripts/migrate_users_to_gen_users.py` - supports `--dry-run` and `--delete` flags.
### LyricsTranscriber: Library, Not Server
**Problem**: LyricsTranscriber's `ReviewServer` blocks waiting for human input - incompatible with async cloud architecture.
@@ -32,6 +51,31 @@ Audio separation and lyrics transcription run in parallel:
- Progress tracked separately in `state_data`
- Screens generation waits for both to complete
### Cloud Run Performance: Cold Starts and FFmpeg
**Problem**: Cloud karaoke generation was ~3x slower than local CLI (15 min vs 5 min for a typical song).
**Root causes identified**:
1. **Cold start penalty**: Cloud Run with `min-instances: 0` caused 77-second cold starts for FFmpeg-heavy operations (title screen generation). Warm instances take only 6 seconds.
2. **FFmpeg version**: Debian's package (5.1.8) is outdated and built with security-focused `--toolchain=hardened` which reduces performance. Static builds (7.0.2) from johnvansickle.com are ~20-40% faster.
3. **Generic x86_64 build**: Debian packages use generic CPU instructions, not optimized for Cloud Run's Cascade Lake/Ice Lake processors.
**Solutions applied**:
- Set `min-instances: 4` to keep workers warm (eliminates cold start entirely)
- Use John Van Sickle's static FFmpeg 7.0.2 instead of Debian package
- Added `/api/health/detailed` endpoint to expose FFmpeg version for debugging
**Key insight**: For CPU-bound encoding workloads on Cloud Run:
- GPU acceleration often doesn't help (filters like libass aren't GPU-accelerated)
- Newer FFmpeg versions have significant encoding optimizations
- Static builds with CPU-specific tuning outperform generic distribution packages
- Cold starts hurt more than expected for initialization-heavy binaries like FFmpeg
**Debugging**: Use `scripts/compare_local_vs_remote.py` to benchmark local vs cloud performance.
## Common Gotchas
### Cross-Domain localStorage Isolation
@@ -203,6 +247,43 @@ def _get_effective_distribution_settings(...) -> EffectiveDistributionSettings:
**Lesson**: When you see the same logic repeated in multiple places, refactor into a helper function immediately. Otherwise bugs will be fixed in one place but not others.
### Python `or` Operator Can't Override True Defaults with False
**Problem**: Using Python's `or` operator for default logic prevents explicit `false` from overriding a `true` server default.
```python
# BAD - can't explicitly disable if server default is true
effective_value = body.enable_feature or settings.default_enable_feature
# When body.enable_feature=False and settings.default=True:
# False or True = True (ignores explicit False!)
```
**Symptoms**:
- Feature can be enabled but never disabled via API
- Works fine when server default is `false` but breaks when default is `true`
- No errors - just ignores the explicit `false` value
**Solution**: Use `Optional[bool] = None` with explicit `is not None` check:
```python
# Request model
enable_feature: Optional[bool] = None # None = use server default
# Default logic
effective_value = (
body.enable_feature
if body.enable_feature is not None
else settings.default_enable_feature
)
```
This allows three states:
- `None` (not specified) → use server default
- `True` → explicitly enable
- `False` → explicitly disable (overrides server default)
**Lesson**: When a boolean field needs to support "use server default" behavior, use `Optional[bool] = None` instead of `bool = False`. The `or` operator conflates "not specified" with "explicitly false".
### Field Name Mismatches Between Endpoints
**Problem**: Different endpoints setting different field names for the same logical value, where consumers check only one field name.
@@ -295,6 +376,77 @@ await expect(page.getByText('Hello,')).toBeVisible({ timeout: 5000 });
**Why it matters**: Playwright assertions have built-in retry logic and fail fast when conditions aren't met. Arbitrary timeouts either waste time (waiting when condition is already met) or cause flaky tests (not waiting long enough on slow CI).
### Playwright: Use 'load' Instead of 'networkidle' for Audio-Heavy Pages
**Problem**: Pages with audio players or ongoing audio network requests never reach `networkidle` state.
```typescript
// BAD - hangs indefinitely on pages with audio players
await page.goto(url, { waitUntil: 'networkidle' });
```
**Symptom**: Test times out after 30-45 minutes waiting for navigation to complete, even though the page is fully rendered and usable.
**Solution**: Use `waitUntil: 'load'` plus a small fixed wait if needed:
```typescript
// GOOD - works with audio-heavy pages
await page.goto(url, { waitUntil: 'load' });
await page.waitForTimeout(3000); // Allow JS to initialize
```
**Root cause**: Audio players make continuous network requests for streaming/buffering, preventing the network from ever becoming "idle".
### Playwright: HEAD Requests May Not Be Supported
**Problem**: Using `page.request.head()` to verify download URLs returns 405 Method Not Allowed on some endpoints.
```typescript
// BAD - many download endpoints don't support HEAD
const response = await page.request.head(downloadUrl);
// Returns 405 instead of 200
```
**Solution**: Use GET with `maxRedirects: 0` to check accessibility without downloading:
```typescript
// GOOD - works with all endpoints, doesn't follow redirects
const response = await page.request.get(downloadUrl, { maxRedirects: 0 });
expect(response.status() >= 200 && response.status() < 400).toBe(true);
```
**Why maxRedirects: 0**: Download URLs often redirect to cloud storage (GCS, S3). A 302/307 redirect means the URL is valid. Following the redirect would download the entire file.
### E2E Tests: Understand Job State Machine Flow
**Problem**: E2E test skipped instrumental selection because it checked for "rendering" in status, but rendering happens BEFORE instrumental selection, not after.
```typescript
// BAD - assumes rendering means instrumental is done
if (status.includes('rendering') || status.includes('encoding')) {
console.log('Past instrumental stage');
break;
}
```
**Actual job flow**:
```text
downloading → transcribing → in_review → rendering_video →
awaiting_instrumental_selection → encoding → complete
```
**Solution**: Only skip on statuses that are definitively past the target stage:
```typescript
// GOOD - encoding is definitely past instrumental
if (status.includes('encoding') && !status.includes('awaiting')) {
console.log('Past instrumental stage');
break;
}
```
**Lesson**: Before writing status-checking logic, trace through the actual state machine to understand which states come before/after your target state.
## Deployment Notes
### Cloud Run Timeouts
@@ -304,6 +456,62 @@ Default 5-minute timeout may not be enough for video encoding. Consider:
- Breaking encoding into smaller chunks
- Increasing timeout for specific operations
### Cloud Tasks dispatch_deadline (Separate from Cloud Run Timeout)
**Problem**: Cloud Tasks HTTP handlers have a separate `dispatch_deadline` (default 10 minutes, max 30 minutes) that controls how long Cloud Tasks waits for the HTTP response. This is DIFFERENT from Cloud Run's service timeout.
**Symptom**: Worker killed silently after 10 minutes even though Cloud Run timeout was set to 30 minutes. Job appears stuck with no error.
**Solution**: Set explicit `dispatch_deadline` when creating Cloud Tasks:
```python
from google.protobuf import duration_pb2
task = {
"http_request": {...},
"dispatch_deadline": duration_pb2.Duration(seconds=1800), # 30 min max
}
```
**Key insight**: For tasks that might take >30 minutes, Cloud Tasks HTTP targets won't work. Use Cloud Run Jobs instead (supports up to 24-hour execution).
| Timeout Type | Default | Max | What It Controls |
|-------------|---------|-----|------------------|
| Cloud Tasks `dispatch_deadline` | 10 min | 30 min | How long Cloud Tasks waits for HTTP response |
| Cloud Run service timeout | 5 min | 60 min | How long a single HTTP request can run |
| Cloud Run Jobs timeout | 10 min | 24 hr | How long a batch job execution can run |
See `docs/archive/2026-01-01-worker-timeout-fixes.md` for full details.
### Two-Layer Timeout Pattern for Long-Running Loops
**Problem**: A loop processing many items (e.g., 74 LLM calls at 15s each = 18+ minutes) can block indefinitely. Single outer timeout isn't enough because it may interrupt mid-operation and leave state inconsistent.
**Solution**: Use two layers of timeout protection:
```python
# Layer 1: Inner deadline check (cooperative, clean exit)
deadline = time.time() + timeout_seconds
for item in items:
if time.time() > deadline:
logger.warning("Deadline exceeded, breaking early")
break # Exit gracefully with partial/no results
process(item)
# Layer 2: Outer asyncio.wait_for (safety net for hung operations)
result = await asyncio.wait_for(
asyncio.to_thread(run_loop_with_deadline),
timeout=timeout_seconds + 60 # Buffer for cleanup
)
```
**Why both layers**:
- Inner check: Handles normal case (many items). Exits cleanly between iterations.
- Outer timeout: Catches edge case (single operation hangs). Hard stop if inner check never runs.
**Design choice**: Use `break` not `raise` for inner timeout. This returns partial/uncorrected results rather than failing the job, allowing downstream human review to fix issues.
See `docs/archive/2025-12-31-agentic-timeout-implementation.md` for implementation details.
### Secret Manager Access
Cloud Run service account needs `Secret Manager Secret Accessor` role. This is managed via Pulumi in `infrastructure/`.
@@ -356,6 +564,47 @@ Code must handle both formats when invoking LangChain ChatVertexAI.
See `docs/archive/2025-12-30-gemini3-agentic-correction-fix.md` for full details.
### LangChain gRPC vs REST Providers
**Problem**: `langchain-google-vertexai` uses gRPC which can hang indefinitely during connection establishment with no timeout, causing cloud jobs to freeze silently.
**Symptoms**:
- Job hangs at "Initializing model..." with no further logs
- No timeout or error - just waits forever
- Hard to diagnose because no exception is raised
**Solution**: Use `langchain-google-genai` (REST-based) instead. This package supports BOTH Vertex AI (service account auth) and Google AI Studio (API key auth):
```python
# Before (gRPC, can hang)
from langchain_google_vertexai import ChatVertexAI
model = ChatVertexAI(model=model_name, project=project)
# After (REST, reliable) - Using Vertex AI with service account/ADC
from langchain_google_genai import ChatGoogleGenerativeAI
model = ChatGoogleGenerativeAI(model=model_name, project=project_id) # Uses ADC
# Alternative - Using Google AI Studio with API key
model = ChatGoogleGenerativeAI(model=model_name, google_api_key=api_key)
```
**Key insight**: When `project` parameter is provided, `ChatGoogleGenerativeAI` automatically uses the Vertex AI backend with Application Default Credentials (ADC). On Cloud Run, this uses the attached service account - no API key needed.
**Additional protection**: Wrap model initialization in ThreadPoolExecutor timeout:
```python
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(create_model)
try:
model = future.result(timeout=30) # Fail fast
except FuturesTimeoutError:
raise InitializationTimeoutError("Model init timed out")
```
**Why ThreadPoolExecutor**: Works cross-platform (Windows, Linux, macOS). Python's `signal.alarm` only works on Unix.
See `docs/archive/2025-12-30-langchain-vertexai-to-genai-migration.md` for full details.
### CI/CD: GitHub Actions
Migrated from Cloud Build to GitHub Actions for:
@@ -376,6 +625,85 @@ GitHub-hosted runners have ~14GB disk, which caused failures during Docker build
See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
### Docker Disk Space Management on Self-Hosted Runners
**Problem**: Even with 200GB SSDs, self-hosted runners can fill up with Docker images. Each CI build creates new dangling images (~15GB each), and without aggressive cleanup, disk fills in days.
**Symptoms**:
- CI jobs fail with `No space left on device`
- Jobs that require Docker (build, deploy, emulator tests) fail
- Runners 11-20 at 100% while 1-10 at 30% (newer runners had less cleanup history)
**Root cause**: Original cleanup cron used `docker system prune -af --filter "until=168h"` which only removes images older than 7 days. With frequent CI builds, most dangling images are < 7 days old and never get cleaned.
**Solution** (two-layer approach):
1. **Hourly threshold-based cleanup** (on runners):
```bash
# /etc/cron.hourly/docker-cleanup
THRESHOLD=70
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
if [ "$USAGE" -gt "$THRESHOLD" ]; then
docker system prune -af # No age filter!
docker builder prune -af
fi
```
2. **Pre-job disk check** (in CI workflow):
```yaml
- name: Check disk space
run: |
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
if [ "$USAGE" -gt 70 ]; then
docker system prune -af || true
docker builder prune -af || true
fi
if [ "$USAGE" -gt 85 ]; then
echo "::error::Disk usage exceeds 85%. Runner needs maintenance."
exit 1
fi
```
**Key insights**:
- Remove the age filter (`--filter "until=168h"`) - dangling images should always be cleaned
- Threshold-based cleanup is better than time-based (don't clean if disk is fine)
- Pre-job checks provide fail-fast behavior with clear error messages
- `docker builder prune -af` catches buildkit cache that `system prune` misses
### Fonts in Docker for Video Rendering
**Problem**: Video rendering showed replacement characters (squares with question marks) for special Unicode symbols like `♪` in intro/instrumental screens.
**Root cause**: The `python:3.11-slim` Docker base image has NO fonts installed. FFmpeg/libass falls back to whatever it can find, which often lacks support for musical symbols, emoji, or non-Latin scripts.
**Symptoms**:
- `♪ INTRO (19 seconds) ♪` displayed as `□ INTRO (19 seconds) □`
- Unicode characters render as tofu (replacement boxes)
- Only appears in cloud-generated videos, not local dev (which uses system fonts)
**Solution**: Install comprehensive Noto fonts in the Docker base image:
```dockerfile
RUN apt-get update && apt-get install -y \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
```
**Package breakdown**:
- `fonts-noto-core` (~10MB): Latin, Greek, Cyrillic, musical symbols
- `fonts-noto-cjk` (~150MB): Chinese, Japanese, Korean
- `fonts-noto-extra` (~50MB): Arabic, Hebrew, Thai, Hindi, other scripts
- `fonts-noto-color-emoji` (~10MB): Color emoji (limited libass support)
- `fontconfig`: Font configuration system (required for `fc-cache`)
**Note on color emoji**: FFmpeg's libass subtitle renderer doesn't fully support color emoji fonts - it renders text as vectors and can't display color bitmap glyphs. Color emoji appears as monochrome outlines or may not render at all in ASS subtitles.
**Lesson**: When building Docker images for video/subtitle rendering, install fonts explicitly. The default slim images have nothing, and font fallback behavior differs from local development machines.
## Performance Observations
| Operation | Duration |
@@ -464,6 +792,48 @@ export function getToken() {
**Lesson**: For values that can change (auth tokens, user preferences), read fresh from storage on each access rather than caching in module scope.
### Zustand Module-Level Initialization Doesn't Re-run on Client Navigation
**Problem**: Module-level initialization code in zustand stores only runs once when the module is first imported. Client-side navigation (e.g., `router.push()`) doesn't reload modules, so initialization code doesn't re-run.
**Example bug**: Auth module had initialization that fetched user data if a token existed:
```typescript
// auth.ts - module level (runs once at import)
if (typeof window !== 'undefined') {
const token = getAccessToken()
if (token) {
useAuth.getState().fetchUser() // Only runs on initial page load!
}
}
```
After beta enrollment:
1. User lands on `/` (landing page) - auth.ts imported, no token yet, `fetchUser()` not called
2. User completes enrollment, token stored via `setAccessToken()`
3. `router.push('/app')` - client-side navigation, modules already loaded
4. Auth module initialization doesn't re-run - `user` stays `null`
5. UI shows "Login" button instead of user credits
**Symptoms**:
- Feature works after page reload but not after client-side navigation
- Zustand store has correct persisted data (token) but not computed data (user profile)
- E2E tests that use reload pass; tests that use `router.push` fail
**Solution**: Components that need fresh data should trigger fetches in useEffect:
```typescript
// In /app page component
const { user, fetchUser } = useAuth()
useEffect(() => {
const token = getAccessToken()
if (token && !user) {
fetchUser() // Ensure user data is loaded regardless of how we got here
}
}, [user, fetchUser])
```
**Lesson**: Don't rely on module-level initialization for data that might need to be fetched after client-side navigation. Use useEffect in components to ensure data is loaded when needed.
### GCP Organization Policies Block Service Account Keys
**Problem**: GCP organization policies can block service account key creation even when you have Owner access to the project.
@@ -508,6 +878,400 @@ rm /tmp/policy.yaml key.json # After copying key contents
**Lesson**: Service account keys are a security risk and should be avoided when possible. Use Workload Identity Federation instead. But for third-party services that require JSON keys (like LangFuse), this temporary policy override is the safest approach.
### Cloud Run Horizontal Scaling and In-Memory Cache
**Problem**: In-memory caches (singletons, module-level dicts) don't persist across Cloud Run instances. When scaling horizontally, request 1 may hit instance A and populate the cache, but request 2 may hit instance B which has an empty cache.
**Example**: `AudioSearchService` cached search results in `self._cached_results`. Search hit instance A, but download hit instance B:
```
Error: No cached result for index 0. Available indices: 0--1. Run search() first.
```
**Solution**: Never rely on in-memory state across requests. Persist to Firestore/GCS:
```python
# BAD - cache lost if next request hits different instance
self._cached_results = search_results
result = self._cached_results[selection_index]
# GOOD - persist in job state_data
job_manager.update_job(job_id, {'state_data': {'search_results': results}})
# Later...
job = job_manager.get_job(job_id)
result = job.state_data['search_results'][selection_index]
```
**Lesson**: Design for stateless request handling. Any state that must survive between requests must be persisted externally.
### Firestore Cache Has No Automatic Invalidation
**Problem**: Audio search results are cached in Firestore (`job.state_data.audio_search_results`) with no expiration or invalidation. When a library like flacfetch is updated with new providers or bug fixes, existing cached results remain stale forever.
**Symptoms**:
- Job shows only 5 YouTube results when CLI shows 59 results including lossless sources
- User cannot get better search results without admin intervention
- No "re-search" option in the UI
**Example**: Job `6ddbbef1` was searched when flacfetch had a bug returning only YouTube results. After flacfetch was fixed, the job still showed the old 5-result cache.
**Solution**: Added admin endpoints and UI to manage cached searches:
- `GET /api/admin/audio-searches` - List jobs with cached results
- `POST /api/admin/audio-searches/{job_id}/clear-cache` - Clear both Firestore and flacfetch GCS cache
- `DELETE /api/admin/cache` - Clear entire flacfetch cache
- `GET /api/admin/cache/stats` - View cache statistics
- Admin UI at `/admin/searches` shows cache stats and "Clear All Cache" button
**Key insight**: There are TWO caches involved:
1. **Firestore** (`job.state_data.audio_search_results`) - stores results for user selection UI
2. **Flacfetch GCS** - caches tracker search responses (30-day TTL)
Both must be cleared for a truly fresh search. The clear-cache endpoint now clears both automatically.
**Future consideration**: Add automatic cache invalidation based on:
- Time (TTL)
- flacfetch version change
- User-initiated "re-search" button
**Lesson**: When caching external API results, consider how/when the cache should be invalidated. Permanent caches need admin tooling to manage stale data. When multiple layers cache the same data, clearing one layer isn't enough - you need to clear all of them.
### GCS Downloads Preserve Directory Structure
**Problem**: When downloading files from GCS, the directory structure is preserved. Code that uses non-recursive glob patterns won't find files in subdirectories.
**Example**: GCE encoding worker downloaded from `gs://bucket/jobs/{job_id}/` but files were stored as:
- `videos/with_vocals.mkv`
- `stems/instrumental_clean.flac`
The original code used `work_dir.glob("*.mkv")` which only searched the root directory, causing "No video files found" errors.
**Solution**: Use recursive glob patterns and filter results:
```python
# BAD - only searches root directory
input_files = list(work_dir.glob("*.mkv"))
# GOOD - searches all subdirectories
input_files = list(work_dir.glob("**/*.mkv"))
# Filter out outputs directory to avoid re-encoding
input_files = [f for f in input_files if "outputs" not in str(f)]
```
**Related gotcha**: Case sensitivity matters. GCS stores filenames as-is, so if files are uploaded as `instrumental_clean.flac` (lowercase), searching for `*Instrumental*.flac` (capital I) won't find them. Use case-insensitive patterns or search for both cases.
**Lesson**: When writing code that processes files downloaded from GCS, always use recursive glob patterns and verify the expected directory structure matches what's actually stored.
### GCS Blob Paths vs Full URIs
**Problem**: When uploading files to GCS and returning paths to other services, there's confusion between "blob paths" (just the path within a bucket) and "full GCS URIs" (gs://bucket/path).
**Symptoms**:
- 404 errors when downloading files that were just uploaded
- Paths look like: `bucket/gs://bucket/path/file.mp4` (doubled prefix)
- Storage service prepends bucket name to what's already a full URI
**Example of the bug**:
```python
# GCE worker uploads to gs://bucket/jobs/id/encoded/
# Then returns full URI in response: "gs://bucket/jobs/id/encoded/file.mp4"
# Backend tries: storage.download_file("gs://bucket/jobs/id/encoded/file.mp4")
# Storage service prepends bucket: "bucket/gs://bucket/jobs/id/encoded/file.mp4"
# Result: 404 - No such object
```
**Solution**: Return blob paths (path within bucket), not full GCS URIs:
```python
# output_gcs_path = "gs://bucket/jobs/id/encoded/"
gcs_path = output_gcs_path.replace("gs://", "")
parts = gcs_path.split("/", 1)
bucket = parts[0]
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
# Return blob paths like: "jobs/id/encoded/file.mp4"
blob_path = f"{prefix}/{filename}" if prefix else filename
```
**Related gotcha**: Double slashes in paths. When the prefix already ends with "/" and you concatenate with another "/":
```python
# BAD - creates "encoded//file.mp4"
prefix = "jobs/id/encoded/" # Has trailing slash
blob_name = f"{prefix}/{filename}" # Adds another slash
# GOOD - strip trailing slash first
prefix = parts[1].rstrip("/")
blob_name = f"{prefix}/{filename}" if prefix else filename
```
**Lesson**: When building systems that pass GCS paths between services, establish a clear contract: either always use full URIs (`gs://bucket/path`) or always use blob paths (`path/within/bucket`). Document which format each API expects.
### Library Caches Need GCS Sync in Cloud Run
**Problem**: Libraries like LyricsTranscriber cache API responses to local directories (e.g., `~/lyrics-transcriber-cache`). In Cloud Run, each container is ephemeral - the cache is lost on every restart or scale-up, causing redundant API calls.
**Symptoms**:
- Same song processed multiple times always hits external APIs
- Higher API costs than expected (AudioShake charges per transcription)
- Slower processing even for repeated jobs
**Solution**: Sync library cache to/from GCS before/after each job:
```python
# Set cache dir via environment variable (library reads this)
cache_dir = os.path.join(temp_dir, "lyrics-cache")
os.environ["LYRICS_TRANSCRIBER_CACHE_DIR"] = cache_dir
# Download relevant cache files from GCS before processing
cache_service.sync_cache_from_gcs(cache_dir, audio_hash, lyrics_hash)
# ... run processing ...
# Upload new cache files to GCS after processing
cache_service.sync_cache_to_gcs(cache_dir, audio_hash, lyrics_hash)
```
**Key design decisions**:
1. Download only relevant files (by hash) - don't sync entire cache directory
2. Skip files that already exist in GCS on upload (same input = same output)
3. Use environment variable to configure library cache dir (avoids modifying library code)
4. Cache indefinitely (no TTL) - API responses don't change for same inputs
**Lesson**: When using third-party libraries with local caching in Cloud Run, check if they support configurable cache directories. If so, sync to GCS for persistence across instances.
### Sequential vs Parallel Worker Triggering
**Problem**: For URL-based jobs (YouTube), triggering both audio and lyrics workers in parallel caused failures. Lyrics worker needs the audio file, but audio worker hadn't downloaded it yet.
```python
# BAD - for URL jobs, audio isn't ready yet
background_tasks.add_task(_trigger_workers_parallel, job_id) # Both start together
# Lyrics worker times out waiting for audio
```
**Solution**: For URL jobs, trigger workers sequentially:
```python
# GOOD - audio downloads first, then triggers lyrics
if job.url and not job.input_media_gcs_path:
# Only trigger audio worker; it will trigger lyrics after download
background_tasks.add_task(worker_service.trigger_audio_worker, job_id)
else:
# For uploaded files, parallel is fine
background_tasks.add_task(_trigger_workers_parallel, job_id)
```
**Lesson**: Understand data dependencies between workers. If worker B needs output from worker A, don't run them in parallel.
### Always Extract Auth Context Into Jobs
**Problem**: Job creation endpoints used `require_auth` but never extracted `auth_result.user_email` to store on the job. Jobs were created with `user_email=None`, so users couldn't see their own jobs.
```python
# BAD - auth used for access control but email not captured
@router.post("/jobs/upload")
async def upload(auth_result = Depends(require_auth)):
job = JobCreate(...) # user_email not set!
```
**Solution**: Extract user identity from auth and set on job:
```python
# GOOD - capture authenticated user's identity
@router.post("/jobs/upload")
async def upload(auth_result: AuthResult = Depends(require_auth)):
effective_user_email = auth_result.user_email or form_user_email
job = JobCreate(user_email=effective_user_email, ...)
```
**Lesson**: Auth provides identity, not just access control. If jobs need owner association, explicitly extract and store the user identifier.
See `docs/archive/2024-12-31-job-failure-investigation.md` for full details.
### Alternative Code Paths Must Implement All Features
**Problem**: Adding a GCE encoding option created a code path that completely bypassed the existing `KaraokeFinalise` class, which contained YouTube upload, Discord notifications, and CDG/TXT packaging. Jobs using GCE encoding silently missed these features.
```python
# BAD - two divergent code paths
if use_gce_encoding:
await gce_encoding_worker.encode(job_id) # Returns, does nothing else
return # YouTube upload, Discord, CDG never happen!
else:
finalise = KaraokeFinalise(...) # Has all the features
await finalise.run()
```
**Symptoms**:
- Jobs complete successfully but YouTube URL is never set
- No Discord notifications for GCE-encoded jobs
- CDG/TXT files never generated for GCE jobs
- No errors in logs (the code path simply doesn't include those features)
**Solution**: Create an orchestrator that runs all stages regardless of encoding backend:
```python
# GOOD - unified pipeline with swappable encoding
orchestrator = VideoWorkerOrchestrator(config)
result = await orchestrator.run()
# Orchestrator always runs: packaging → encoding → organization → distribution → notifications
# The encoding step uses either local or GCE backend based on config
```
**Key insight**: When adding an alternative implementation (like GCE encoding), audit ALL side effects of the original code path. The new path must replicate them or delegate to shared code.
**Architecture pattern**: The orchestrator pattern separates "what stages run" from "how each stage runs". Stages are always executed; only the implementation varies.
See `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` for full details.
### Firestore Document 1MB Limit with Embedded Arrays
**Problem**: Firestore documents have a 1MB size limit. Storing logs as an embedded array (`worker_logs: [{...}, {...}, ...]`) within job documents caused job failures when logs exceeded the limit during long-running operations like video encoding.
**Example**: Job `501258e1` failed when logs reached 1.26MB (2,547 log entries), representing 98.6% of the maximum document size. The job update failed with:
```
Document cannot be written because its size exceeds the maximum allowed size
```
**Root cause**: Video encoding jobs generate thousands of log entries (FFmpeg output, progress updates). Each log entry is ~500 bytes. 2,000+ entries easily exceed 1MB.
**Solution**: Store logs in a Firestore subcollection instead of embedded array:
```python
# BAD - embedded array hits 1MB limit
doc_ref.update({
"worker_logs": firestore.ArrayUnion([{"message": "..."}])
})
# GOOD - subcollection has no size limit per parent doc
logs_ref = db.collection("jobs").document(job_id).collection("logs")
logs_ref.document(log_id).set({
"timestamp": now,
"level": "INFO",
"worker": "video",
"message": "...",
"ttl_expiry": now + timedelta(days=30)
})
```
**Key benefits of subcollection approach**:
1. Each log entry is its own document (no size limit concerns)
2. Natural association with parent job (path: `jobs/{job_id}/logs/{log_id}`)
3. Can add TTL via Firestore TTL policies for automatic cleanup
4. Efficient queries with composite indexes (worker + timestamp)
5. Concurrent workers can write without race conditions (unlike ArrayUnion which can conflict)
**Migration strategy**: Used feature flag (`USE_LOG_SUBCOLLECTION=true`) for rollback capability. New jobs use subcollection; old jobs with embedded arrays continue to work via fallback read logic.
**Lesson**: Never use embedded arrays for unbounded data. If an array can grow indefinitely based on operation duration or user input, use a subcollection instead. The 1MB limit applies to the entire document, including all fields.
### Splitting Fast Generation from Slow Encoding
**Problem**: Preview video generation took 60+ seconds on Cloud Run, causing poor UX during lyrics review. The bottleneck was FFmpeg encoding (CPU-bound), not the subtitle generation (fast text processing).
**Solution**: Split the operation into two phases with optional offloading:
1. **ASS Generation (Cloud Run)**: Generate ASS subtitle file locally (~2s). This is pure text processing with no CPU-intensive encoding.
2. **Video Encoding (GCE)**: Upload ASS to GCS and offload FFmpeg encoding to the high-performance GCE worker (~15-20s on C4-standard-8).
```python
# Generate ASS only (fast, local)
result = CorrectionOperations.generate_preview_video(
...,
ass_only=True, # Skip video encoding
)
# Upload ASS to GCS
storage.upload_file(result["ass_path"], f"jobs/{job_id}/previews/{preview_hash}.ass")
# Offload encoding to GCE
gce_result = await encoding_service.encode_preview_video(
job_id=preview_hash,
ass_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.ass",
audio_gcs_path=audio_gcs_path,
output_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.mp4",
)
```
**Key insight**: When a pipeline has both fast (text generation) and slow (media encoding) steps, design for separability:
- Add `_only` parameters to skip slow steps
- Return intermediate outputs (paths to generated files)
- Allow external services to handle the slow part
- Keep fallback to local processing for reliability
**Configuration**: Controlled by `USE_GCE_PREVIEW_ENCODING` env var. Falls back to local encoding if GCE fails or is disabled.
**Result**: Preview generation dropped from 60+ seconds to ~15-20 seconds.
### External Service Response Format Mismatches
**Problem**: The GCE encoding worker returned responses in unexpected formats, causing `'list' object has no attribute 'get'` errors that were hard to debug through the async pipeline.
**Issues encountered**:
1. **Status endpoint returning list instead of dict**: The `/status/{job_id}` endpoint sometimes returned a list `[{...}]` instead of a dict `{...}`. Calling `.get()` on the list caused failures.
2. **output_files as list of paths instead of dict**: The worker returned:
```json
"output_files": ["jobs/id/finals/output_4k_lossless.mp4", "jobs/id/finals/output_720p.mp4"]
```
But the backend expected:
```json
"output_files": {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
```
**Symptoms**:
- Job fails late in pipeline (during encoding completion handling)
- Error message is cryptic (`'list' object has no attribute 'get'`)
- Hard to reproduce locally (depends on GCE worker state)
- Multiple similar errors from different root causes
**Solution**: Add defensive type checking for all external service responses:
```python
# Always check response type before accessing
if isinstance(status, list):
logger.warning(f"GCE returned list instead of dict: {status}")
status = status[0] if status and isinstance(status[0], dict) else {}
if not isinstance(status, dict):
status = {}
# Convert list of paths to expected dict format
if isinstance(output_files, list):
output_dict = {}
for path in output_files:
filename = path.split("/")[-1]
if "4k_lossless" in filename:
output_dict["mp4_4k_lossless"] = path
# ... etc
output_files = output_dict
```
**Key insight**: When integrating with external services, never assume the response format. Add explicit type checks and conversions, with logging when unexpected formats are encountered. This is especially important for services you don't control (internal microservices, third-party APIs).
**Debugging tip**: When hitting format errors in async pipelines, check the raw response from the external service directly (curl with API key) before assuming the bug is in your code.
### GCE Encoding Returns GCS Paths, Not Local Files
**Problem**: When using GCE encoding (remote high-performance encoder), the orchestrator received GCS blob paths like `jobs/{id}/finals/output_4k_lossless.mp4` instead of local file paths. Subsequent operations like YouTube upload checked `os.path.isfile()` which returned `False`, causing silent failures.
**Symptoms**:
- YouTube upload logged "No video file available for YouTube upload"
- Other file-based operations silently skipped
- Dropbox/GDrive uploads succeeded (they use different code paths)
- No errors in logs - just warnings that were easy to miss
**Root cause**: The `VideoWorkerOrchestrator._run_encoding()` method stored the paths returned by `GCEEncodingBackend` directly without downloading the files. For local encoding, the paths are already local. For GCE encoding, the paths are GCS blob paths that need to be downloaded first.
**Solution**: Added `_download_gce_encoded_files()` method that:
1. Detects when GCE backend was used (`encoding_backend.name == "gce"`)
2. Downloads each encoded file from GCS to the local temp directory
3. Updates result paths to point to local files
```python
# After encoding completes
if encoding_backend.name == "gce" and self.storage:
await self._download_gce_encoded_files(output)
```
**Key insight**: When an operation can run on multiple backends (local vs remote), ensure the outputs are normalized to a common format before subsequent stages consume them. The orchestrator pattern should abstract away backend differences, not leak them to downstream consumers.
**Testing tip**: Test the full pipeline with each backend type, not just the encoding step in isolation. The bug only manifested when YouTube upload ran after GCE encoding - unit tests of each stage passed.
## What We'd Do Differently
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
+34 -2
View File
@@ -7,6 +7,7 @@
- **Web App**: https://gen.nomadkaraoke.com
- `/` - Landing page with pricing and beta enrollment
- `/app` - Main app (upload audio, review lyrics, download videos)
- `/admin` - Admin dashboard (user management, job monitoring, beta program)
- **Backend API**: https://api.nomadkaraoke.com - FastAPI on Cloud Run
- **CLI Tools**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
@@ -16,7 +17,7 @@
|-----------|--------|
| Audio upload & separation | Working |
| Lyrics transcription | Working |
| Agentic AI correction | Working (Gemini 3 Flash via Vertex AI) |
| Agentic AI correction | Working (Gemini 2 Flash via Vertex AI) |
| Human lyrics review | Working |
| Preview video generation | Working |
| Instrumental selection | Working |
@@ -25,15 +26,46 @@
| Magic link auth | Working |
| Payment flow (Stripe) | Working |
| Beta tester program | Working |
| Admin dashboard | Working |
| CI/CD self-hosted runner | Working (GCP) |
| E2E happy path test | Working (38 min full pipeline) |
## Known Issues
- CDG format generation requires additional style configuration
- Long audio files (>10 min) may timeout on some workers
## Recent Changes
- **Worker Logs Subcollection** (2026-01-04): Moved `worker_logs` from embedded array in job documents to Firestore subcollection (`jobs/{job_id}/logs`). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flag `USE_LOG_SUBCOLLECTION=true` (default). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
- **Video Worker Orchestrator** (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag `USE_NEW_ORCHESTRATOR` (default: true) enables rollback. 139 new tests across 6 new service modules. See [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator) and [archive/2026-01-04-video-worker-orchestrator-refactor.md](archive/2026-01-04-video-worker-orchestrator-refactor.md).
- **Full Unicode Font Support** (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fonts-in-docker-for-video-rendering).
- **Flacfetch Cache API Integration** (2026-01-03): Enhanced audio search cache management to also clear flacfetch's GCS cache. When admins clear a job's cache, both Firestore (job state) and flacfetch (GCS search results) are cleared simultaneously. Added "Clear All Cache" button and cache stats display to admin UI. New endpoints: `DELETE /api/admin/cache`, `GET /api/admin/cache/stats`. See [API.md](API.md#audio-search-management-admin).
- **Audio Search Cache Management** (2026-01-03): Added admin UI at `/admin/searches` to view and manage cached audio search results. Admins can clear stale caches (e.g., when flacfetch is updated with new providers) and allow users to re-search. Fixes issue where jobs showed only YouTube results when lossless sources should be available. See [API.md](API.md#audio-search-management-admin) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-cache-has-no-automatic-invalidation).
- **User Database Separation** (2026-01-03): Migrated karaoke-gen users from shared `users` collection to dedicated `gen_users` collection. This separates karaoke-gen user data from karaoke-decide user data (both apps share the same GCP project/Firestore instance). Migration script in `scripts/migrate_users_to_gen_users.py`. See [ARCHITECTURE.md](ARCHITECTURE.md#firestore-collections) for collection details.
- **Admin Dashboard** (2026-01-03): Added comprehensive admin dashboard at `/admin` with: stats overview (users, jobs, credits), user management (search, sort, add credits, toggle role, enable/disable), job browser (filter by status/user, view details, delete), and beta program monitoring. Query-param based routing for static export compatibility. See [API.md](API.md#admin-endpoints) for backend endpoints.
- **GCE Encoding Worker** (2026-01-03): Added optional high-performance encoding on dedicated GCE VM (C4-standard-8 with Intel Granite Rapids 3.9 GHz). Uses `USE_GCE_ENCODING=true` env var. Falls back to Cloud Run if unavailable. See [ARCHITECTURE.md](ARCHITECTURE.md).
- **Runner Disk Space Auto-Cleanup** (2026-01-03): Fixed CI failures caused by self-hosted runners filling up with Docker images. Changed cleanup from daily/7-day-old to hourly/threshold-based (70%). Added pre-job disk checks to all self-hosted CI jobs. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#docker-disk-space-management-on-self-hosted-runners)
- **E2E Happy Path Test** (2026-01-02): Added comprehensive E2E test that validates the complete karaoke generation journey using ONLY browser UI interactions (no API shortcuts). Test covers all 12 steps from landing page through cleanup. Runs daily via GitHub Actions and takes ~38 minutes. See [E2E-HAPPY-PATH-TEST-SPEC.md](E2E-HAPPY-PATH-TEST-SPEC.md)
- **LyricsTranscriber Cache Persistence** (2026-01-02): Added GCS-backed cache for LyricsTranscriber to avoid redundant API calls. AudioShake transcription and lyrics provider responses (Genius, Spotify, LRCLib, Musixmatch) are now cached to `gs://karaoke-gen-storage/lyrics-transcriber-cache/`. Cache is synced before/after each job, persisting across Cloud Run instances. Significantly reduces API costs and speeds up repeated processing of same songs.
- **Worker Timeout Fixes** (2026-01-01): Fixed 3 timeout issues blocking job completion: (1) Lyrics transcription timeout increased to 20 min (PR #153), (2) Cloud Tasks dispatch_deadline added for audio worker - default 10 min was killing Modal API calls (PR #154), (3) Enabled Cloud Run Jobs for video encoding - 1-hour timeout vs 30-min Cloud Run service limit (PR #155). Full E2E pipeline now verified working. See [archive/2026-01-01-worker-timeout-fixes.md](archive/2026-01-01-worker-timeout-fixes.md)
- **Agentic Correction Timeout** (2025-12-31): Added 3-minute configurable timeout for agentic AI lyrics correction. Prevents stuck jobs when songs have many gaps (74+ gaps could take 30+ minutes). On timeout, skips correction and proceeds to human review with raw transcription. See [archive/2025-12-31-agentic-timeout-implementation.md](archive/2025-12-31-agentic-timeout-implementation.md)
- **Job Failure Fixes** (2025-12-31): Fixed 4 critical issues causing job failures: (1) user_email not set on jobs - users couldn't see their jobs, (2) YouTube URL race condition - lyrics worker started before audio downloaded, (3) Audio search cache not persisting across Cloud Run instances, (4) Jobs stuck in downloading state - added 10min transcription timeout. See [archive/2024-12-31-job-failure-investigation.md](archive/2024-12-31-job-failure-investigation.md)
- **Vertex AI Auth Fix** (2025-12-31): Fixed `ChatGoogleGenerativeAI` to use Vertex AI backend with service account auth on Cloud Run. PR #145 broke cloud deployments by requiring `GOOGLE_API_KEY`. Solution: pass `project` parameter to trigger Vertex AI backend with ADC. See [archive/2025-12-31-vertex-ai-auth-fix.md](archive/2025-12-31-vertex-ai-auth-fix.md)
- **LangChain Provider Migration** (2025-12-30): Migrated from `langchain-google-vertexai` (gRPC) to `langchain-google-genai` (REST) to fix silent hangs during model initialization. Added initialization and warm-up timeouts using ThreadPoolExecutor for fail-fast behavior. See [archive/2025-12-30-langchain-vertexai-to-genai-migration.md](archive/2025-12-30-langchain-vertexai-to-genai-migration.md)
- **LangFuse Prompt Management** (2025-12-30): Replaced hardcoded LLM prompts with LangFuse-managed prompts for dynamic iteration without redeploying. Added migration script, LangFuse Vertex AI service account for GCP credits. See [archive/2025-12-30-langfuse-prompt-management.md](archive/2025-12-30-langfuse-prompt-management.md)
- **Gemini 3 Flash Agentic Correction Fix** (2025-12-30): Fixed agentic correction hanging due to Gemini 3 requiring `global` location instead of regional endpoints, and multimodal response format handling. Added local test script for faster iteration. See [archive/2025-12-30-gemini3-agentic-correction-fix.md](archive/2025-12-30-gemini3-agentic-correction-fix.md)
- **Cold Start UX** (2025-12-30): Frontend now shows friendly "Warming up the generator..." messages when backend is slow to respond (cold start after deployment or inactivity). Progressive messages appear after 5s and 15s to reassure users during 30s+ cold starts.
@@ -0,0 +1,291 @@
# Job Failure Investigation Report
**Date**: 2024-12-31
**Investigated Jobs**: 76296c31, 67bc8ca8, 02207418, cadd7153, 6a461e4e, 14035e2b, 643983e3, 9726a99b
**Affected User**: [redacted]@gmail.com (plus E2E test jobs)
## Executive Summary
Investigation revealed **4 distinct issues** causing job failures and the user complaint that "jobs don't show up in my account":
1. **CRITICAL: user_email not set on jobs** - Jobs aren't associated with users
2. **YouTube URL download race condition** - Lyrics worker starts before audio is downloaded
3. **Audio search cache not shared across instances** - Cloud Run scaling breaks download
4. **Lyrics worker stalling silently** - Jobs stuck in "downloading" state forever
---
## Issue 1: user_email Not Being Set on Jobs (CRITICAL)
### Symptoms
- User reports: "jobs polling is always returning a []"
- User has active account with 2 credits, active session
- User investigation shows `Jobs Created: 0` despite creating jobs
- All 8 investigated jobs have `user_email: null`
### Root Cause
The job creation endpoints use `require_auth` dependency which returns `AuthResult` containing `user_email`. However, the endpoints **do not extract this email and set it on the job**.
**In `backend/api/routes/file_upload.py`:**
- `user_email` is an optional form field (line 397) that clients must explicitly pass
- The `auth_result.user_email` from authentication is ignored
- Jobs are created with `user_email=None` unless client explicitly sends it
**Same issue in:**
- `/jobs/upload` endpoint
- `/jobs/create-from-url` endpoint
- `/jobs/create-with-upload-urls` endpoint
- `/api/audio-search/search` endpoint
### Fix Required
Modify all job creation endpoints to:
1. Extract `user_email` from `AuthResult` when available
2. Prefer authenticated user's email over client-provided value
3. Set this on the `JobCreate` model
**Example fix for `create-from-url`:**
```python
@router.post("/jobs/create-from-url")
async def create_job_from_url(
request: Request,
body: CreateJobFromUrlRequest,
auth_result: AuthResult = Depends(require_auth) # Use AuthResult, not tuple
):
# Extract user email from authenticated session
effective_user_email = auth_result.user_email or body.user_email
job_create = JobCreate(
...
user_email=effective_user_email, # Now properly set
...
)
```
### Priority
**CRITICAL** - This is blocking all authenticated users from seeing their jobs.
---
## Issue 2: YouTube URL Download Race Condition
### Symptoms
- Jobs cadd7153, 6a461e4e, 14035e2b failed with: `Lyrics transcription failed: Failed to download audio file`
- All have `input_media_gcs_path: null`
- All were created from YouTube URLs
### Root Cause
In `file_upload.py` at `/jobs/create-from-url`:
1. Job is created with `url` but no `input_media_gcs_path`
2. Job immediately transitions to `DOWNLOADING` state
3. **Both audio and lyrics workers are triggered in parallel** (`_trigger_workers_parallel`)
The problem:
- Audio worker is responsible for downloading from YouTube and uploading to GCS
- Lyrics worker needs the audio file from GCS to transcribe
- Lyrics worker polls for `input_media_gcs_path` for up to 5 minutes (max_wait_seconds=300)
- If YouTube download is slow or fails, lyrics worker times out
**Additional YouTube download issues:**
- `YOUTUBE_COOKIES` env var may not be set, causing "Sign in to confirm you're not a bot" errors
- YouTube rate limiting can cause downloads to fail
### Evidence in `lyrics_worker.py:401-427`
```python
# For URL jobs, wait for audio worker to download and upload to GCS
if job.url:
while time.time() - start_time < max_wait_seconds:
# Poll for input_media_gcs_path
if updated_job.input_media_gcs_path:
return local_path
await asyncio.sleep(poll_interval)
logger.error(f"Timed out waiting for audio download")
return None
```
### Fix Required
Option A (Recommended): **Sequential triggering for URL jobs**
```python
# In create-from-url endpoint:
if job.url and not job.input_media_gcs_path:
# Only trigger audio worker first for URL jobs
# Lyrics worker triggered after audio is downloaded
background_tasks.add_task(worker_service.trigger_audio_worker, job_id)
else:
# For uploaded files, parallel is fine
background_tasks.add_task(_trigger_workers_parallel, job_id)
```
Option B: **Audio worker triggers lyrics worker after download**
- Have `audio_worker.download_audio()` trigger lyrics worker upon successful URL download
### Priority
**HIGH** - YouTube/URL-based jobs are failing consistently.
---
## Issue 3: Audio Search Cache Not Persisting
### Symptoms
- Job 76296c31 failed with: `Audio download failed: No cached result for index 0. Available indices: 0--1. Run search() first.`
- Job was created from `audio_search` mode
### Root Cause
`AudioSearchService` is a singleton with in-memory cache (`self._cached_results`). Cloud Run uses horizontal scaling with multiple instances, so:
1. Request 1 (search): Hits instance A, caches results in A's memory
2. Request 2 (select/download): May hit instance B, which has empty cache
3. Download fails because `len(self._cached_results) = 0`
**In `audio_search_service.py:279-284`:**
```python
if result_index < 0 or result_index >= len(self._cached_results):
raise DownloadError(
f"No cached result for index {result_index}. "
f"Available indices: 0-{len(self._cached_results) - 1}. " # Shows "0--1" when empty
"Run search() first."
)
```
### Partial Fix Already in Place
The code stores search results in `job.state_data['audio_search_results']` and attempts to use `download_by_id()` which doesn't require cache. However, for YouTube sources, it falls back to cache-based download.
### Fix Required
**Never rely on in-memory cache across requests.** Always use job.state_data.
In `audio_search.py` `_download_and_start_processing()`:
```python
# Get the selected result from job state_data (not service cache)
search_results = job.state_data.get('audio_search_results', [])
selected = search_results[selection_index]
# For YouTube, construct a fake AudioSearchResult from state_data
# and call download_by_id with video URL
if selected['provider'] == 'YouTube':
result = audio_search_service.download_by_id(
source_name='YouTube',
source_id=selected.get('source_id'), # Video ID
output_dir=temp_dir,
download_url=selected.get('url'), # YouTube URL
)
```
### Priority
**HIGH** - Audio search mode is broken in multi-instance environments.
---
## Issue 4: Jobs Stuck in Downloading State
### Symptoms
- Jobs 643983e3, 9726a99b have status `downloading` indefinitely
- `state_data.audio_complete: true` - audio worker finished
- `state_data.lyrics_progress.stage: "transcribing"` - lyrics worker started
- NO `lyrics_complete` flag - lyrics worker never finished
- Status never transitions beyond `downloading`
### Root Cause
The lyrics worker appears to crash or timeout during AudioShake transcription without properly failing the job. Possible causes:
1. **AudioShake API timeout** - transcription takes too long
2. **Cloud Run instance killed** - worker running longer than timeout allows
3. **Unhandled exception** - error during transcription not caught properly
4. **Memory exhaustion** - large audio files causing OOM
The lyrics worker's exception handling (`lyrics_worker.py:375-383`) should mark job as failed:
```python
except Exception as e:
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Lyrics transcription failed: {str(e)}",
)
```
But if the Cloud Run instance is killed (e.g., timeout, OOM), this code never runs.
### Fix Required
1. **Add job-level timeouts with cleanup**
```python
# In job_manager.py - scheduled task to clean up stale jobs
async def cleanup_stale_jobs():
# Find jobs in 'downloading' state for more than X hours
# Check if workers are actually running
# Mark as failed if stuck
```
2. **Add heartbeat mechanism**
```python
# Workers periodically update timestamp
job_manager.update_state_data(job_id, 'worker_heartbeat', datetime.now())
# Background job detects stale heartbeats and fails job
```
3. **Improve AudioShake timeout handling**
```python
# In lyrics_worker.py, add explicit timeout:
async with asyncio.timeout(600): # 10 minute timeout
result = await asyncio.to_thread(lyrics_processor.transcribe_lyrics, ...)
```
### Priority
**MEDIUM** - Jobs are stuck but not actively failing. Need monitoring/cleanup mechanism.
---
## Summary of Fixes
| Issue | Priority | Fix Complexity | Affected Code |
|-------|----------|---------------|---------------|
| 1. user_email not set | CRITICAL | Low | `file_upload.py`, `audio_search.py` |
| 2. URL download race | HIGH | Medium | `file_upload.py`, `audio_worker.py` |
| 3. Audio search cache | HIGH | Medium | `audio_search.py`, `audio_search_service.py` |
| 4. Stuck jobs cleanup | MEDIUM | Medium | `job_manager.py`, workers |
---
## Implemented Fixes
### Issue 1: user_email - FIXED
**Changes:**
- Updated all job creation endpoints to use `AuthResult` type instead of tuple
- Extract `auth_result.user_email` and set it on `JobCreate`
- Prefer authenticated user's email over client-provided value
**Files modified:**
- `backend/api/routes/file_upload.py` - All job creation endpoints
- `backend/api/routes/audio_search.py` - Audio search endpoint
### Issue 2: URL Download Race Condition - FIXED
**Changes:**
- For URL jobs, only trigger audio worker initially (not lyrics)
- Audio worker triggers lyrics worker after successful URL download
- Added `_trigger_audio_worker_only()` function for URL jobs
- Added `_trigger_lyrics_worker_after_url_download()` in audio_worker
**Files modified:**
- `backend/api/routes/file_upload.py` - Changed create-from-url to use sequential triggering
- `backend/workers/audio_worker.py` - Trigger lyrics worker after URL download
### Issue 3: Audio Search Cache - FIXED
**Changes:**
- For YouTube sources without remote flacfetch, download directly using URL from `state_data`
- Avoids relying on in-memory cache which doesn't persist across Cloud Run instances
- Added fallback warning for cache-based downloads
**Files modified:**
- `backend/api/routes/audio_search.py` - Added direct YouTube download path
### Issue 4: Stuck Jobs Timeout - FIXED
**Changes:**
- Added 10-minute timeout around AudioShake transcription call
- Uses `asyncio.wait_for` to ensure transcription doesn't hang forever
- On timeout, raises exception which marks job as failed
**Files modified:**
- `backend/workers/lyrics_worker.py` - Added timeout handling
---
## Testing
All 812 backend tests pass after fixes.
@@ -0,0 +1,178 @@
# LangChain Vertex AI Migration Plan
**Date**: 2025-12-30
**Status**: Planned
**Priority**: Medium (deprecation warning, not yet breaking)
## Background
The `ChatVertexAI` class from `langchain-google-vertexai` was deprecated in version 3.2.0 and will be removed in 4.0.0. The recommended replacement is `ChatGoogleGenerativeAI` from the `langchain-google-genai` package.
Deprecation warning observed in logs:
```
LangChainDeprecationWarning: The class `ChatVertexAI` was deprecated in LangChain 3.2.0
and will be removed in 4.0.0. An updated version of the class exists in the
`langchain-google-genai` package and should be used instead.
```
## Current Implementation
**File**: `lyrics_transcriber_temp/lyrics_transcriber/correction/agentic/providers/model_factory.py`
```python
from langchain_google_vertexai import ChatVertexAI
model = ChatVertexAI(
model=model_name,
project=config.gcp_project_id,
location=config.gcp_location,
timeout=config.request_timeout_seconds,
max_retries=config.max_retries,
callbacks=callbacks,
)
```
## Target Implementation
```python
from langchain_google_genai import ChatGoogleGenerativeAI
model = ChatGoogleGenerativeAI(
model=model_name,
vertexai=True, # Keep using Vertex AI authentication
project=config.gcp_project_id,
location=config.gcp_location,
timeout=config.request_timeout_seconds,
max_retries=config.max_retries,
callbacks=callbacks,
)
```
## Migration Steps
### 1. Update Dependencies
In `pyproject.toml` (or wherever dependencies are managed):
```toml
# Remove
langchain-google-vertexai = "^x.x.x"
# Add
langchain-google-genai = "^4.0.0"
```
### 2. Update model_factory.py
**File**: `lyrics_transcriber_temp/lyrics_transcriber/correction/agentic/providers/model_factory.py`
```python
# Change import
# Old:
from langchain_google_vertexai import ChatVertexAI
# New:
from langchain_google_genai import ChatGoogleGenerativeAI
```
Update the `_create_vertexai_model` method:
```python
def _create_vertexai_model(
self, model_name: str, callbacks: List[Any], config: ProviderConfig
) -> Any:
"""Create ChatGoogleGenerativeAI model for Google Gemini via Vertex AI.
Uses Application Default Credentials (ADC) for authentication.
In Cloud Run, this uses the service account automatically.
Locally, run: gcloud auth application-default login
"""
from langchain_google_genai import ChatGoogleGenerativeAI
model = ChatGoogleGenerativeAI(
model=model_name,
vertexai=True, # Use Vertex AI backend (not Google AI Studio)
project=config.gcp_project_id,
location=config.gcp_location,
timeout=config.request_timeout_seconds,
max_retries=config.max_retries,
callbacks=callbacks,
)
logger.debug(f"🤖 Created Vertex AI model: {model_name} (project={config.gcp_project_id})")
return model
```
### 3. Alternative: Environment Variable Approach
Instead of passing `vertexai=True`, you can set environment variables:
```bash
export GOOGLE_GENAI_USE_VERTEXAI=true
export GOOGLE_CLOUD_PROJECT=your-project-id
export GOOGLE_CLOUD_LOCATION=global # or us-central1, etc.
```
Then the code simplifies to:
```python
model = ChatGoogleGenerativeAI(
model=model_name,
timeout=config.request_timeout_seconds,
max_retries=config.max_retries,
callbacks=callbacks,
)
```
## Key Differences
| Aspect | ChatVertexAI (old) | ChatGoogleGenerativeAI (new) |
|--------|-------------------|------------------------------|
| Package | `langchain-google-vertexai` | `langchain-google-genai` |
| Protocol | gRPC | REST |
| Vertex AI Support | Native | Via `vertexai=True` param |
| Auth Methods | ADC, Service Account | Same + API keys |
| Multimodal | Yes | Yes |
## Potential Benefits
1. **REST instead of gRPC**: May resolve the `fork_posix.cc` gRPC warnings we observed:
```
I0000 00:00:1767135828.790289 30025640 fork_posix.cc:71] Other threads are
currently calling into gRPC, skipping fork() handlers
```
2. **Unified SDK**: The `google-genai` SDK is the consolidated approach for all Google AI services.
3. **Future-proof**: Avoids breaking changes when `langchain-google-vertexai` 4.0 removes `ChatVertexAI`.
## Potential Risks
1. **REST vs gRPC latency**: Some users reported slightly slower calls with REST. Monitor LangFuse traces after migration.
2. **Behavior differences**: Test thoroughly to ensure identical behavior for:
- Multimodal response format handling
- Timeout behavior
- Retry logic
- Error messages
## Testing Plan
1. Run existing agentic correction unit tests
2. Run a full local CLI job with `USE_AGENTIC_AI=1`
3. Compare LangFuse traces before/after for:
- Response times
- Success rates
- Error types
4. Verify Cloud Run deployment works with new package
## References
- [langchain-google-genai 4.0.0 Announcement](https://github.com/langchain-ai/langchain-google/discussions/1422)
- [ChatGoogleGenerativeAI Docs](https://reference.langchain.com/python/integrations/langchain_google_genai/)
- [LangChain Google Releases](https://github.com/langchain-ai/langchain-google/releases)
## Notes
- This migration was identified during investigation of 499 timeout errors in agentic correction (see PR #143)
- The gRPC → REST change might actually help with the connection issues we observed
- Authentication via Vertex AI (ADC/service accounts) is fully supported in the new package
@@ -0,0 +1,95 @@
# LangChain Vertex AI to Google GenAI Migration
**Date**: 2025-12-30
**PR**: #145
## Problem
Cloud job `b1b2648c` hung silently during agentic correction. No logs appeared after "LangFuse prompt service initialized". Investigation revealed `ChatVertexAI` (langchain-google-vertexai) was hanging during gRPC connection establishment with no timeout.
## Solution
Migrated from `langchain-google-vertexai` to `langchain-google-genai`:
| Before | After |
|--------|-------|
| `ChatVertexAI` (gRPC-based) | `ChatGoogleGenerativeAI` (REST-based) |
| Vertex AI service account auth | GOOGLE_API_KEY (AI Studio) |
| No initialization timeout | 30s init + 15s warm-up timeouts |
| DEBUG-level logs | INFO-level logs with timing |
## Key Changes
### Dependencies
```toml
# Before
langchain-google-vertexai = ">=3.1.1"
# After
langchain-google-genai = ">=2.0.0"
```
### Model Creation
```python
# Before (gRPC, can hang)
from langchain_google_vertexai import ChatVertexAI
model = ChatVertexAI(model=model_name, project=project, location=location)
# After (REST, explicit API key)
from langchain_google_genai import ChatGoogleGenerativeAI
model = ChatGoogleGenerativeAI(
model=model_name,
google_api_key=api_key,
convert_system_message_to_human=True, # Gemini doesn't support system messages
)
```
### Timeout Implementation
Used `ThreadPoolExecutor` for cross-platform timeout (works on Windows, Linux, macOS):
```python
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(self._factory.create_chat_model, model, config)
try:
self._chat_model = future.result(timeout=30)
except FuturesTimeoutError:
raise InitializationTimeoutError("Model initialization timed out")
```
### Warm-up Pattern
Added lightweight "warm-up" request after model creation to establish connection before real work:
```python
warm_up_prompt = 'Respond with exactly: {"status":"ready"}'
response = model.invoke([HumanMessage(content=warm_up_prompt)])
```
## Files Modified
| File | Changes |
|------|---------|
| `pyproject.toml` | Replaced dependency, bumped version |
| `lyrics_transcriber_temp/pyproject.toml` | Same |
| `.../providers/config.py` | Added `initialization_timeout_seconds`, `warmup_timeout_seconds` |
| `.../providers/model_factory.py` | Migrated to `ChatGoogleGenerativeAI`, added timing logs |
| `.../providers/langchain_bridge.py` | Added ThreadPoolExecutor timeout wrapper, warm-up, INFO logs |
| `.../agent.py` | Added timing logs for classify_gap |
## Why REST over gRPC
1. **Simpler debugging** - HTTP requests are easier to trace than gRPC streams
2. **No connection pooling issues** - REST is stateless
3. **Better timeout behavior** - HTTP timeouts are well-understood
4. **Cloud-friendly** - Works better through proxies and load balancers
## Authentication
The new approach uses Google AI Studio API keys instead of Vertex AI service accounts:
- **Get key**: https://aistudio.google.com/app/apikey
- **Set env var**: `GOOGLE_API_KEY=<your-key>`
- **Benefits**: Simpler setup, no IAM configuration needed
Note: This uses Google AI Studio quotas, not GCP billing. For high-volume production use, consider keeping Vertex AI with proper timeout handling.
@@ -0,0 +1,68 @@
# Agentic AI Correction Timeout Implementation
**Date**: 2025-12-31
**PR**: #149
**Version**: 0.85.0
## Problem
Agentic AI lyrics correction was causing stuck jobs in production. Songs with many gaps (74+) could take 30+ minutes to process since each gap takes 10-30 seconds for LLM inference. This blocked the entire job pipeline.
Example from production logs:
```
Processing 22/74 gaps (~15s each) = ~18 minutes elapsed
Estimated total: 74 gaps × 15s = 18+ minutes
```
## Solution
Implemented a configurable 3-minute timeout (default) with two layers:
### Layer 1: Inner Deadline Check (Cooperative)
In `corrector.py`, check deadline before processing each gap:
```python
if deadline and time.time() > deadline:
self.logger.warning("AGENTIC TIMEOUT: Deadline exceeded...")
break # Return uncorrected transcription for human review
```
### Layer 2: Outer asyncio.wait_for (Safety Net)
In `lyrics_worker.py`, wrap the entire transcription call:
```python
result = await asyncio.wait_for(
asyncio.to_thread(lyrics_processor.transcribe_lyrics, ...),
timeout=timeout_seconds + 60 # Extra buffer
)
```
## Behavior
| Scenario | What Happens |
|----------|--------------|
| Completes in <3 min | Normal flow, corrections applied |
| Gap loop exceeds deadline | Inner check breaks, returns uncorrected |
| LLM hangs indefinitely | Outer timeout fires (rare), job fails |
| Any timeout | Job proceeds to human review with raw transcription |
## Files Changed
- `backend/config.py` - Added `agentic_correction_timeout_seconds` config
- `backend/workers/lyrics_worker.py` - Deadline calculation, outer timeout wrapper
- `karaoke_gen/lyrics_processor.py` - Pass deadline to controller
- `lyrics_transcriber_temp/lyrics_transcriber/core/controller.py` - Accept deadline parameter
- `lyrics_transcriber_temp/lyrics_transcriber/correction/corrector.py` - Deadline check in gap loop
## Configuration
Environment variable: `AGENTIC_CORRECTION_TIMEOUT_SECONDS` (default: 180)
## Key Design Decisions
1. **Break, don't raise**: Inner timeout uses `break` to exit the gap loop gracefully, returning whatever we have (uncorrected). This is better than raising an exception because:
- No retry logic needed
- Job continues to human review
- User can manually correct any issues
2. **Two-layer protection**: The inner check handles normal cases (many gaps). The outer timeout catches edge cases (single LLM call hanging). Both are needed for reliability.
3. **Configurable timeout**: Default 3 minutes balances correction quality vs. user experience. It can be adjusted via environment variable per deployment.
@@ -0,0 +1,72 @@
# ChatGoogleGenerativeAI Vertex AI Auth Fix
**Date**: 2025-12-31
**PR**: #147
**Related**: PR #145 (LangChain provider migration)
## Background
PR #145 migrated from `ChatVertexAI` (gRPC) to `ChatGoogleGenerativeAI` (REST) to fix silent hangs during model initialization. However, the migration only configured API key authentication, breaking Cloud Run deployments that use service account credentials.
## Problem
After deploying PR #145, Cloud Run lyrics worker failed with:
```
🤖 Failed to initialize chat model: GOOGLE_API_KEY environment variable is required for Google/Gemini models
```
The code required `GOOGLE_API_KEY` even though:
- Cloud Run already has `GOOGLE_CLOUD_PROJECT` set
- Service account credentials are available via ADC (Application Default Credentials)
- Vertex AI IAM permissions were already configured
## Root Cause
`ChatGoogleGenerativeAI` supports TWO authentication backends:
| Backend | Trigger | Auth Method |
|---------|---------|-------------|
| Vertex AI | `project` param set | Service account / ADC |
| Google AI Studio | Only `google_api_key` | API key |
PR #145 only passed `google_api_key`, forcing the Google AI Studio backend which doesn't support service accounts.
## Solution
Pass `project` parameter when `GOOGLE_CLOUD_PROJECT` is set to trigger Vertex AI backend:
```python
# Before (broken)
model = ChatGoogleGenerativeAI(model=model_name, google_api_key=api_key)
# After (works with service account)
model_kwargs = {"model": model_name, ...}
if project:
model_kwargs["project"] = project # Triggers Vertex AI backend
if api_key:
model_kwargs["google_api_key"] = api_key # Optional fallback
model = ChatGoogleGenerativeAI(**model_kwargs)
```
## Key Insight
`ChatGoogleGenerativeAI` from `langchain-google-genai` is a unified client that supports both Vertex AI and Google AI Studio APIs. The `project` parameter is the key discriminator - when present, it uses Vertex AI with ADC auth.
## Testing
Verified that on Cloud Run with `GOOGLE_CLOUD_PROJECT=nomadkaraoke`:
- No `GOOGLE_API_KEY` required
- Uses attached service account via ADC
- REST API (avoids gRPC hang issues from PR #145)
## Files Changed
- `lyrics_transcriber_temp/.../providers/model_factory.py` - Added project param logic
- `docs/LESSONS-LEARNED.md` - Updated documentation
## Timing Analysis Context
This fix was discovered during a timing analysis comparing local vs remote job performance. Key findings:
- Remote jobs had 38+ minute delays in lyrics worker
- Root cause: gRPC hangs (fixed by PR #145) + auth misconfiguration (fixed by this PR)
- With both fixes, remote processing should match local performance
@@ -0,0 +1,78 @@
# Worker Timeout Fixes
**Date**: 2026-01-01
**PRs**: #153, #154, #155
**Status**: Complete
## Problem
Jobs were failing at different stages due to worker timeouts:
1. **Lyrics worker** - Timeout after 600 seconds (10 minutes) during transcription
2. **Audio worker** - Killed after 10 minutes by Cloud Tasks default `dispatch_deadline`
3. **Video worker** - Timeout after 30 minutes (Cloud Run service timeout)
## Root Causes
### Lyrics Transcription Timeout (PR #153)
- `TRANSCRIPTION_TIMEOUT_SECONDS` was 600 (10 minutes)
- AudioShake transcription + agentic correction can take 15+ minutes for complex songs
- **Fix**: Increased to 1200 seconds (20 minutes)
### Cloud Tasks dispatch_deadline (PR #154)
- Cloud Tasks HTTP handlers have a default `dispatch_deadline` of 10 minutes
- Audio separation via Modal API takes 15-20+ minutes
- Worker was killed before Modal could complete, but no error surfaced
- **Symptom**: Job stuck at `downloading` with `audio_complete: false`
- **Fix**: Added explicit `dispatch_deadline` per worker type:
```python
WORKER_DISPATCH_DEADLINES = {
"audio": 1800, # 30 min
"lyrics": 1500, # 25 min
"screens": 600, # 10 min
"render-video": 1800, # 30 min
"video": 1800, # 30 min
}
```
### Video Encoding Timeout (PR #155)
- Cloud Run service timeout is 1800 seconds (30 minutes)
- `KaraokeFinalise.process()` takes 30-40+ minutes for average songs
- Video encoding includes: 4 format encodings, CDG generation, TXT packages
- **Symptom**: Job stuck at `encoding` with `updated_at` frozen
- **Fix**: Enabled `USE_CLOUD_RUN_JOBS_FOR_VIDEO=true` in `cloudbuild.yaml`
- Cloud Run Jobs support up to 1-hour execution (configured in `infrastructure/__main__.py`)
## Key Insight: dispatch_deadline vs Cloud Run Timeout
These are different timeouts:
| Timeout | Default | Max | Controls |
|---------|---------|-----|----------|
| Cloud Tasks `dispatch_deadline` | 10 min | 30 min | How long Cloud Tasks waits for HTTP response |
| Cloud Run service timeout | 5 min | 60 min | How long Cloud Run allows request to run |
| Cloud Run Jobs timeout | 10 min | 24 hr | How long a batch job can run |
For worker tasks that take >30 minutes, Cloud Tasks HTTP targets can't work. Use Cloud Run Jobs instead.
## Verification
Job `eb725b2c` completed successfully through the full pipeline:
- Audio separation: Complete
- Lyrics transcription: Complete
- Video encoding: Complete
- All 4 video formats generated
- All stems, lyrics, and packages available
## Files Changed
- `backend/services/worker_service.py` - Added `dispatch_deadline` to Cloud Tasks
- `cloudbuild.yaml` - Enabled `USE_CLOUD_RUN_JOBS_FOR_VIDEO=true`
- `pyproject.toml` - Version bumps (0.86.4, 0.86.5)
## Lessons Learned
1. **Cloud Tasks dispatch_deadline is separate from Cloud Run timeout** - Both need to be configured for long-running workers
2. **Default dispatch_deadline (10 min) is often insufficient** - Modal API, video encoding, and other heavy processing need more time
3. **Cloud Run Jobs for truly long tasks** - For anything that might take >30 minutes, Cloud Tasks HTTP targets won't work
4. **E2E tests reveal timeout issues** - Production E2E tests caught these issues that unit tests missed
@@ -0,0 +1,101 @@
# Video Worker Orchestrator Refactor - 2026-01-04
## Summary
Major refactor to unify video generation pipeline with a new VideoWorkerOrchestrator. This resolves the GCE vs KaraokeFinalise code path divergence that caused features like YouTube upload, Discord notifications, and CDG/TXT packaging to be missing from the GCE encoding path.
## Problem
Job 5b78ad46 completed successfully via GCE encoding but had no YouTube URL. Investigation revealed:
1. The GCE encoding path was a completely separate code branch
2. When `USE_GCE_ENCODING=true`, the worker triggered GCE encoding and returned immediately
3. This bypassed the entire `KaraokeFinalise` class which contained:
- YouTube upload logic
- Discord webhook notifications
- CDG/TXT package generation
- Dropbox/Google Drive uploads
The root cause wasn't a bug in any individual feature - it was an architectural issue where alternative implementations didn't share common post-processing stages.
## Solution
Created a stage-based orchestrator pattern:
```
VideoWorkerOrchestrator
├── Stage 1: Packaging (CDG, TXT)
├── Stage 2: Encoding (Local or GCE - swappable)
├── Stage 3: Organization (file paths, structure)
├── Stage 4: Distribution (Dropbox, GDrive, YouTube)
└── Stage 5: Notifications (Discord)
```
The orchestrator always runs all stages. Only the encoding implementation varies based on backend selection.
## Changes
### New Service Modules
| File | Lines | Purpose |
|------|-------|---------|
| `youtube_upload_service.py` | ~445 | YouTube OAuth and upload |
| `discord_service.py` | ~172 | Discord webhook notifications |
| `packaging_service.py` | ~287 | CDG/TXT package generation |
| `local_encoding_service.py` | ~590 | FFmpeg encoding on Cloud Run |
| `encoding_interface.py` | ~418 | Abstract interface + GCE backend |
| `video_worker_orchestrator.py` | ~628 | Stage coordination |
### Test Coverage
| Test File | Tests | Coverage |
|-----------|-------|----------|
| `test_youtube_upload_service.py` | 25 | OAuth, upload, error handling |
| `test_discord_service.py` | 18 | Webhook formatting, delivery |
| `test_packaging_service.py` | 22 | CDG/TXT generation |
| `test_local_encoding_service.py` | 24 | FFmpeg encoding variants |
| `test_encoding_interface.py` | 23 | Interface + GCE backend |
| `test_video_worker_orchestrator.py` | 27 | Full pipeline, stages |
| **Total** | **139** | |
### video_worker.py Updates
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: `true`)
- `generate_video()` routes to orchestrator or legacy path
- Legacy code preserved as `generate_video_legacy()` for rollback
- New `generate_video_orchestrated()` builds config and runs orchestrator
## Feature Flag
```bash
# Enable new orchestrator (default)
USE_NEW_ORCHESTRATOR=true
# Rollback to legacy behavior
USE_NEW_ORCHESTRATOR=false
```
## Key Design Decisions
1. **Stage-based orchestration**: Each stage is independent and testable
2. **Strategy pattern for encoding**: `EncodingBackend` interface with local and GCE implementations
3. **Lazy service loading**: Services instantiated only when needed via `_get_*()` methods
4. **Dataclass configs**: `OrchestratorConfig` and `OrchestratorResult` for clear contracts
5. **Feature flag default true**: New code active immediately since tests are comprehensive
## Testing Notes
- All 976 existing tests continue to pass
- 139 new tests added across 6 test files
- E2E verification: Jobs 501258e1 and d21e6ef0 successfully processed through new pipeline
- Both jobs reached `awaiting_instrumental_selection` as expected (requires user input to complete)
## PR
- PR #182: https://github.com/nomadkaraoke/karaoke-gen/pull/182
- Merged: 2026-01-04
- Version: 0.89.0
## Lessons Learned
See [LESSONS-LEARNED.md](../LESSONS-LEARNED.md#alternative-code-paths-must-implement-all-features) for the key architectural lesson about alternative code paths.
+19
View File
@@ -0,0 +1,19 @@
# Environment variables for local development and testing
# Copy this file to .env.local and fill in your values
# =============================================================================
# E2E Testing - Email (for beta enrollment flow)
# =============================================================================
# testmail.app credentials for email testing
# Get these from https://testmail.app dashboard
TESTMAIL_API_KEY=your-api-key-here
TESTMAIL_NAMESPACE=your-namespace-here
# =============================================================================
# E2E Testing - Authentication (optional)
# =============================================================================
# Skip email enrollment by providing an existing access token
# Useful for faster local testing when you already have credits
# KARAOKE_ACCESS_TOKEN=your-token-here
+286
View File
@@ -0,0 +1,286 @@
"use client"
import { useEffect, useState } from "react"
import { adminApi, AdminBetaStats, AdminBetaFeedback } from "@/lib/api"
import { StatsCard, StatsGrid } from "@/components/admin/stats-card"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import { Badge } from "@/components/ui/badge"
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table"
import {
Users,
MessageSquare,
Star,
RefreshCw,
Loader2,
ThumbsUp,
ThumbsDown,
} from "lucide-react"
import { Button } from "@/components/ui/button"
import { useToast } from "@/hooks/use-toast"
export default function AdminBetaPage() {
const { toast } = useToast()
const [stats, setStats] = useState<AdminBetaStats | null>(null)
const [feedback, setFeedback] = useState<AdminBetaFeedback[]>([])
const [loading, setLoading] = useState(true)
const loadData = async () => {
try {
setLoading(true)
const [statsData, feedbackData] = await Promise.all([
adminApi.getBetaStats(),
adminApi.getBetaFeedback(50),
])
setStats(statsData)
setFeedback(feedbackData.feedback || [])
} catch (err: any) {
console.error("Failed to load beta data:", err)
toast({
title: "Error",
description: err.message || "Failed to load beta program data",
variant: "destructive",
})
} finally {
setLoading(false)
}
}
useEffect(() => {
loadData()
}, [])
const renderStars = (rating?: number) => {
if (!rating) return "—"
return (
<div className="flex gap-0.5">
{[1, 2, 3, 4, 5].map((star) => (
<Star
key={star}
className={`w-4 h-4 ${
star <= rating ? "fill-yellow-400 text-yellow-400" : "text-muted"
}`}
/>
))}
</div>
)
}
const formatDate = (dateStr?: string) => {
if (!dateStr) return "—"
const date = new Date(dateStr)
if (isNaN(date.getTime())) return "—"
return date.toLocaleDateString()
}
return (
<div className="space-y-6">
{/* Header */}
<div className="flex items-center justify-between">
<div>
<h1 className="text-2xl font-bold tracking-tight">Beta Program</h1>
<p className="text-muted-foreground">
Manage beta testers and view feedback
</p>
</div>
<Button variant="outline" size="sm" onClick={loadData} disabled={loading}>
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
Refresh
</Button>
</div>
{/* Stats Grid */}
<StatsGrid>
<StatsCard
title="Total Beta Testers"
value={stats?.total_beta_testers ?? 0}
icon={Users}
loading={loading}
/>
<StatsCard
title="Active Testers"
value={stats?.active_testers ?? 0}
icon={Users}
loading={loading}
/>
<StatsCard
title="Pending Feedback"
value={stats?.pending_feedback ?? 0}
icon={MessageSquare}
loading={loading}
/>
<StatsCard
title="Total Feedback"
value={stats?.total_feedback_submissions ?? 0}
icon={MessageSquare}
loading={loading}
/>
</StatsGrid>
{/* Average Ratings */}
<Card>
<CardHeader>
<CardTitle className="text-base">Average Ratings</CardTitle>
<CardDescription>Overall feedback scores from beta testers</CardDescription>
</CardHeader>
<CardContent>
{loading ? (
<div className="flex items-center justify-center py-8">
<Loader2 className="w-6 h-6 animate-spin text-muted-foreground" />
</div>
) : stats?.average_ratings ? (
<div className="grid gap-4 md:grid-cols-4">
<div className="rounded-lg border p-4">
<p className="text-sm text-muted-foreground">Overall</p>
<div className="flex items-center gap-2 mt-1">
<span className="text-2xl font-bold">
{stats.average_ratings.overall?.toFixed(1) || "—"}
</span>
<span className="text-muted-foreground">/ 5</span>
</div>
</div>
<div className="rounded-lg border p-4">
<p className="text-sm text-muted-foreground">Ease of Use</p>
<div className="flex items-center gap-2 mt-1">
<span className="text-2xl font-bold">
{stats.average_ratings.ease_of_use?.toFixed(1) || "—"}
</span>
<span className="text-muted-foreground">/ 5</span>
</div>
</div>
<div className="rounded-lg border p-4">
<p className="text-sm text-muted-foreground">Lyrics Accuracy</p>
<div className="flex items-center gap-2 mt-1">
<span className="text-2xl font-bold">
{stats.average_ratings.lyrics_accuracy?.toFixed(1) || "—"}
</span>
<span className="text-muted-foreground">/ 5</span>
</div>
</div>
<div className="rounded-lg border p-4">
<p className="text-sm text-muted-foreground">Correction Experience</p>
<div className="flex items-center gap-2 mt-1">
<span className="text-2xl font-bold">
{stats.average_ratings.correction_experience?.toFixed(1) || "—"}
</span>
<span className="text-muted-foreground">/ 5</span>
</div>
</div>
</div>
) : (
<p className="text-muted-foreground">No ratings available yet</p>
)}
</CardContent>
</Card>
{/* Feedback List */}
<Card>
<CardHeader>
<CardTitle className="text-base">Recent Feedback</CardTitle>
<CardDescription>Feedback submissions from beta testers</CardDescription>
</CardHeader>
<CardContent>
{loading ? (
<div className="flex items-center justify-center py-8">
<Loader2 className="w-6 h-6 animate-spin text-muted-foreground" />
</div>
) : feedback.length === 0 ? (
<p className="text-muted-foreground text-center py-8">No feedback submissions yet</p>
) : (
<div className="space-y-4">
{feedback.map((item) => (
<div key={item.id} className="border rounded-lg p-4">
<div className="flex items-start justify-between gap-4">
<div className="flex-1">
<div className="flex items-center gap-2 mb-2">
<span className="font-medium text-sm">{item.user_email}</span>
{item.job_id && (
<Badge variant="outline" className="text-xs">
Job: {item.job_id}
</Badge>
)}
<span className="text-xs text-muted-foreground">
{formatDate(item.created_at)}
</span>
</div>
{/* Ratings */}
<div className="flex flex-wrap gap-4 mb-3 text-sm">
<div className="flex items-center gap-1">
<span className="text-muted-foreground">Overall:</span>
{renderStars(item.overall_rating)}
</div>
{item.ease_of_use_rating && (
<div className="flex items-center gap-1">
<span className="text-muted-foreground">Ease:</span>
{renderStars(item.ease_of_use_rating)}
</div>
)}
{item.lyrics_accuracy_rating && (
<div className="flex items-center gap-1">
<span className="text-muted-foreground">Accuracy:</span>
{renderStars(item.lyrics_accuracy_rating)}
</div>
)}
</div>
{/* Comments */}
{item.what_went_well && (
<div className="mb-2">
<p className="text-xs text-muted-foreground">What went well:</p>
<p className="text-sm">{item.what_went_well}</p>
</div>
)}
{item.what_could_improve && (
<div className="mb-2">
<p className="text-xs text-muted-foreground">What could improve:</p>
<p className="text-sm">{item.what_could_improve}</p>
</div>
)}
{item.additional_comments && (
<div>
<p className="text-xs text-muted-foreground">Additional comments:</p>
<p className="text-sm">{item.additional_comments}</p>
</div>
)}
</div>
{/* Recommend/Use Again */}
<div className="flex flex-col gap-1 text-sm">
{item.would_recommend !== undefined && (
<div className="flex items-center gap-1">
{item.would_recommend ? (
<ThumbsUp className="w-4 h-4 text-green-500" />
) : (
<ThumbsDown className="w-4 h-4 text-red-500" />
)}
<span className="text-xs text-muted-foreground">Recommend</span>
</div>
)}
{item.would_use_again !== undefined && (
<div className="flex items-center gap-1">
{item.would_use_again ? (
<ThumbsUp className="w-4 h-4 text-green-500" />
) : (
<ThumbsDown className="w-4 h-4 text-red-500" />
)}
<span className="text-xs text-muted-foreground">Use Again</span>
</div>
)}
</div>
</div>
</div>
))}
</div>
)}
</CardContent>
</Card>
</div>
)
}
+428
View File
@@ -0,0 +1,428 @@
"use client"
import { useEffect, useState } from "react"
import { useSearchParams, useRouter } from "next/navigation"
import { api, adminApi, Job } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Badge } from "@/components/ui/badge"
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"
import {
Accordion,
AccordionContent,
AccordionItem,
AccordionTrigger,
} from "@/components/ui/accordion"
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@/components/ui/alert-dialog"
import {
ArrowLeft,
RefreshCw,
Loader2,
Trash2,
User,
Clock,
FileText,
XCircle,
} from "lucide-react"
import { useToast } from "@/hooks/use-toast"
export default function AdminJobDetailPage() {
const searchParams = useSearchParams()
const router = useRouter()
const { toast } = useToast()
const jobId = searchParams.get("id") || ""
const [job, setJob] = useState<Job | null>(null)
const [logs, setLogs] = useState<any[]>([])
const [loading, setLoading] = useState(true)
const [logsLoading, setLogsLoading] = useState(false)
// Delete dialog
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false)
const [deleting, setDeleting] = useState(false)
const loadJob = async () => {
if (!jobId) {
setLoading(false)
return
}
try {
setLoading(true)
const data = await api.getJob(jobId)
setJob(data)
} catch (err: any) {
console.error("Failed to load job:", err)
toast({
title: "Error",
description: err.message || "Failed to load job",
variant: "destructive",
})
} finally {
setLoading(false)
}
}
const loadLogs = async () => {
if (!jobId) return
try {
setLogsLoading(true)
const data = await api.getJobLogs(jobId, 200)
setLogs(data)
} catch (err: any) {
console.error("Failed to load logs:", err)
} finally {
setLogsLoading(false)
}
}
useEffect(() => {
loadJob()
loadLogs()
}, [jobId])
const handleDelete = async () => {
try {
setDeleting(true)
await adminApi.deleteJob(jobId)
toast({
title: "Job Deleted",
description: `Job ${jobId} has been deleted`,
})
router.push("/admin/jobs")
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to delete job",
variant: "destructive",
})
} finally {
setDeleting(false)
}
}
const getStatusVariant = (status: string) => {
if (status === "complete" || status === "prep_complete") return "default"
if (status === "failed") return "destructive"
if (status === "cancelled") return "outline"
if (status.includes("awaiting")) return "secondary"
return "secondary"
}
const getLogLevelColor = (level: string) => {
switch (level?.toUpperCase()) {
case "ERROR":
return "text-red-500"
case "WARNING":
return "text-yellow-500"
case "INFO":
return "text-blue-500"
case "DEBUG":
return "text-gray-500"
default:
return "text-foreground"
}
}
const formatDate = (dateStr?: string) => {
if (!dateStr) return "—"
return new Date(dateStr).toLocaleString()
}
if (!jobId) {
return (
<div className="text-center py-12">
<p className="text-muted-foreground">No job ID provided</p>
<Button variant="outline" className="mt-4" onClick={() => router.push("/admin/jobs")}>
<ArrowLeft className="w-4 h-4 mr-2" />
Back to Jobs
</Button>
</div>
)
}
if (loading) {
return (
<div className="flex items-center justify-center py-12">
<Loader2 className="w-8 h-8 animate-spin text-muted-foreground" />
</div>
)
}
if (!job) {
return (
<div className="text-center py-12">
<p className="text-muted-foreground">Job not found</p>
<Button variant="outline" className="mt-4" onClick={() => router.push("/admin/jobs")}>
<ArrowLeft className="w-4 h-4 mr-2" />
Back to Jobs
</Button>
</div>
)
}
return (
<div className="space-y-6">
{/* Header */}
<div className="flex items-center justify-between">
<div className="flex items-center gap-4">
<Button variant="ghost" size="icon" onClick={() => router.push("/admin/jobs")}>
<ArrowLeft className="w-4 h-4" />
</Button>
<div>
<h1 className="text-2xl font-bold tracking-tight flex items-center gap-2">
Job {job.job_id}
<Badge variant={getStatusVariant(job.status)}>
{job.status.replace(/_/g, " ")}
</Badge>
</h1>
{job.artist && job.title && (
<p className="text-muted-foreground">{job.artist} - {job.title}</p>
)}
</div>
</div>
<div className="flex gap-2">
<Button variant="outline" size="sm" onClick={() => { loadJob(); loadLogs(); }} disabled={loading}>
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
Refresh
</Button>
<Button
variant="destructive"
size="sm"
onClick={() => setDeleteDialogOpen(true)}
>
<Trash2 className="w-4 h-4 mr-2" />
Delete
</Button>
</div>
</div>
{/* Overview Cards */}
<div className="grid gap-4 md:grid-cols-4">
<Card>
<CardHeader className="pb-2">
<CardTitle className="text-sm font-medium flex items-center gap-2">
<User className="w-4 h-4" />
User
</CardTitle>
</CardHeader>
<CardContent>
<p className="text-sm truncate">
{job.user_email || "Unknown"}
</p>
</CardContent>
</Card>
<Card>
<CardHeader className="pb-2">
<CardTitle className="text-sm font-medium flex items-center gap-2">
<Clock className="w-4 h-4" />
Created
</CardTitle>
</CardHeader>
<CardContent>
<p className="text-sm">{formatDate(job.created_at)}</p>
</CardContent>
</Card>
<Card>
<CardHeader className="pb-2">
<CardTitle className="text-sm font-medium flex items-center gap-2">
<Clock className="w-4 h-4" />
Updated
</CardTitle>
</CardHeader>
<CardContent>
<p className="text-sm">{formatDate(job.updated_at)}</p>
</CardContent>
</Card>
<Card>
<CardHeader className="pb-2">
<CardTitle className="text-sm font-medium flex items-center gap-2">
Progress
</CardTitle>
</CardHeader>
<CardContent>
<div className="flex items-center gap-2">
<div className="flex-1 h-2 bg-muted rounded-full overflow-hidden">
<div
className="h-full bg-primary transition-all"
style={{ width: `${job.progress}%` }}
/>
</div>
<span className="text-sm font-medium">{job.progress}%</span>
</div>
</CardContent>
</Card>
</div>
{/* Error Message */}
{job.error_message && (
<Card className="border-destructive">
<CardHeader>
<CardTitle className="text-destructive flex items-center gap-2">
<XCircle className="w-5 h-5" />
Error
</CardTitle>
</CardHeader>
<CardContent>
<p className="text-sm">{job.error_message}</p>
</CardContent>
</Card>
)}
{/* Accordion Sections */}
<Accordion type="multiple" className="w-full" defaultValue={["timeline", "logs"]}>
{/* Timeline */}
<AccordionItem value="timeline">
<AccordionTrigger>
<div className="flex items-center gap-2">
<Clock className="w-4 h-4" />
Timeline ({job.timeline?.length || 0} events)
</div>
</AccordionTrigger>
<AccordionContent>
<div className="space-y-2 max-h-[400px] overflow-y-auto">
{job.timeline && job.timeline.length > 0 ? (
job.timeline.map((event, i) => (
<div
key={i}
className="flex items-start gap-3 text-sm border-l-2 border-muted pl-4 py-1"
>
<div className="flex-1">
<div className="flex items-center gap-2">
<Badge variant="outline" className="text-xs">
{event.status}
</Badge>
{event.progress !== undefined && (
<span className="text-muted-foreground">
{event.progress}%
</span>
)}
</div>
{event.message && (
<p className="text-muted-foreground mt-1">{event.message}</p>
)}
</div>
<span className="text-xs text-muted-foreground whitespace-nowrap">
{formatDate(event.timestamp)}
</span>
</div>
))
) : (
<p className="text-muted-foreground text-sm">No timeline events</p>
)}
</div>
</AccordionContent>
</AccordionItem>
{/* Worker Logs */}
<AccordionItem value="logs">
<AccordionTrigger>
<div className="flex items-center gap-2">
<FileText className="w-4 h-4" />
Worker Logs ({logs.length} entries)
{logsLoading && <Loader2 className="w-4 h-4 animate-spin ml-2" />}
</div>
</AccordionTrigger>
<AccordionContent>
<div className="space-y-1 max-h-[500px] overflow-y-auto font-mono text-xs">
{logs.length > 0 ? (
logs.map((log, i) => (
<div key={i} className="flex gap-2 py-0.5 border-b border-muted/50">
<span className="text-muted-foreground whitespace-nowrap">
{log.timestamp ? new Date(log.timestamp).toLocaleTimeString() : "—"}
</span>
<span className={`w-16 ${getLogLevelColor(log.level)}`}>
[{log.level}]
</span>
<span className="text-muted-foreground w-20">
{log.worker || "—"}
</span>
<span className="flex-1 break-all">{log.message}</span>
</div>
))
) : (
<p className="text-muted-foreground">No logs available</p>
)}
</div>
</AccordionContent>
</AccordionItem>
{/* State Data */}
<AccordionItem value="state">
<AccordionTrigger>
<div className="flex items-center gap-2">
<FileText className="w-4 h-4" />
State Data
</div>
</AccordionTrigger>
<AccordionContent>
<pre className="text-xs bg-muted p-4 rounded-md overflow-x-auto max-h-[400px]">
{JSON.stringify(job.state_data || {}, null, 2)}
</pre>
</AccordionContent>
</AccordionItem>
{/* File URLs */}
<AccordionItem value="files">
<AccordionTrigger>
<div className="flex items-center gap-2">
<FileText className="w-4 h-4" />
File URLs
</div>
</AccordionTrigger>
<AccordionContent>
<pre className="text-xs bg-muted p-4 rounded-md overflow-x-auto max-h-[400px]">
{JSON.stringify(job.file_urls || {}, null, 2)}
</pre>
</AccordionContent>
</AccordionItem>
{/* Raw Job Data */}
<AccordionItem value="raw">
<AccordionTrigger>
<div className="flex items-center gap-2">
<FileText className="w-4 h-4" />
Raw Job Data
</div>
</AccordionTrigger>
<AccordionContent>
<pre className="text-xs bg-muted p-4 rounded-md overflow-x-auto max-h-[500px]">
{JSON.stringify(job, null, 2)}
</pre>
</AccordionContent>
</AccordionItem>
</Accordion>
{/* Delete Confirmation Dialog */}
<AlertDialog open={deleteDialogOpen} onOpenChange={setDeleteDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Delete Job</AlertDialogTitle>
<AlertDialogDescription>
Are you sure you want to delete job {jobId}? This action cannot be
undone and will also delete all associated files.
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleDelete}
disabled={deleting}
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
>
{deleting && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
Delete
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</div>
)
}
+327
View File
@@ -0,0 +1,327 @@
"use client"
import { useEffect, useState, useCallback } from "react"
import { useRouter } from "next/navigation"
import { adminApi, Job } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Badge } from "@/components/ui/badge"
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table"
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select"
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@/components/ui/alert-dialog"
import {
Search,
RefreshCw,
Loader2,
Trash2,
ExternalLink,
} from "lucide-react"
import { useToast } from "@/hooks/use-toast"
const statusOptions = [
{ value: "all", label: "All Statuses" },
{ value: "pending", label: "Pending" },
{ value: "downloading", label: "Downloading" },
{ value: "separating_stage1", label: "Separating (Stage 1)" },
{ value: "separating_stage2", label: "Separating (Stage 2)" },
{ value: "transcribing", label: "Transcribing" },
{ value: "correcting", label: "Correcting" },
{ value: "awaiting_review", label: "Awaiting Review" },
{ value: "in_review", label: "In Review" },
{ value: "rendering_video", label: "Rendering Video" },
{ value: "awaiting_instrumental_selection", label: "Awaiting Instrumental" },
{ value: "generating_video", label: "Generating Video" },
{ value: "encoding", label: "Encoding" },
{ value: "complete", label: "Complete" },
{ value: "failed", label: "Failed" },
{ value: "cancelled", label: "Cancelled" },
]
export default function AdminJobsPage() {
const router = useRouter()
const { toast } = useToast()
const [jobs, setJobs] = useState<Job[]>([])
const [loading, setLoading] = useState(true)
const [statusFilter, setStatusFilter] = useState("all")
const [userEmailFilter, setUserEmailFilter] = useState("")
const [searchInput, setSearchInput] = useState("")
// Delete dialog
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false)
const [jobToDelete, setJobToDelete] = useState<Job | null>(null)
const [deleting, setDeleting] = useState(false)
const loadJobs = useCallback(async () => {
try {
setLoading(true)
const data = await adminApi.listAllJobs({
status: statusFilter !== "all" ? statusFilter : undefined,
user_email: userEmailFilter || undefined,
limit: 100,
})
setJobs(data)
} catch (err: any) {
console.error("Failed to load jobs:", err)
toast({
title: "Error",
description: err.message || "Failed to load jobs",
variant: "destructive",
})
} finally {
setLoading(false)
}
}, [statusFilter, userEmailFilter, toast])
useEffect(() => {
loadJobs()
}, [loadJobs])
const handleSearch = () => {
setUserEmailFilter(searchInput)
}
const handleKeyPress = (e: React.KeyboardEvent) => {
if (e.key === "Enter") {
handleSearch()
}
}
const handleDelete = async () => {
if (!jobToDelete) return
try {
setDeleting(true)
await adminApi.deleteJob(jobToDelete.job_id)
toast({
title: "Job Deleted",
description: `Job ${jobToDelete.job_id} has been deleted`,
})
setDeleteDialogOpen(false)
setJobToDelete(null)
loadJobs()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to delete job",
variant: "destructive",
})
} finally {
setDeleting(false)
}
}
const getStatusVariant = (status: string) => {
if (status === "complete" || status === "prep_complete") return "default"
if (status === "failed") return "destructive"
if (status === "cancelled") return "outline"
if (status.includes("awaiting")) return "secondary"
return "secondary"
}
const formatDate = (dateStr?: string) => {
if (!dateStr) return "—"
return new Date(dateStr).toLocaleString()
}
return (
<div className="space-y-6">
{/* Header */}
<div className="flex items-center justify-between">
<div>
<h1 className="text-2xl font-bold tracking-tight">Jobs</h1>
<p className="text-muted-foreground">
View and manage all jobs in the system
</p>
</div>
<Button variant="outline" size="sm" onClick={loadJobs} disabled={loading}>
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
Refresh
</Button>
</div>
{/* Filters */}
<div className="flex flex-col sm:flex-row gap-4">
<div className="flex-1 flex gap-2">
<Input
placeholder="Filter by user email..."
value={searchInput}
onChange={(e) => setSearchInput(e.target.value)}
onKeyDown={handleKeyPress}
className="max-w-sm"
/>
<Button variant="secondary" onClick={handleSearch}>
<Search className="w-4 h-4" />
</Button>
</div>
<Select value={statusFilter} onValueChange={setStatusFilter}>
<SelectTrigger className="w-[200px]">
<SelectValue placeholder="Filter by status" />
</SelectTrigger>
<SelectContent>
{statusOptions.map((option) => (
<SelectItem key={option.value} value={option.value}>
{option.label}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
{/* Table */}
<div className="rounded-md border">
<Table>
<TableHeader>
<TableRow>
<TableHead>Job ID</TableHead>
<TableHead>User</TableHead>
<TableHead>Artist / Title</TableHead>
<TableHead>Status</TableHead>
<TableHead>Progress</TableHead>
<TableHead>Created</TableHead>
<TableHead className="text-right">Actions</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{loading ? (
<TableRow>
<TableCell colSpan={7} className="text-center py-8">
<Loader2 className="w-6 h-6 animate-spin mx-auto text-muted-foreground" />
</TableCell>
</TableRow>
) : jobs.length === 0 ? (
<TableRow>
<TableCell colSpan={7} className="text-center py-8 text-muted-foreground">
No jobs found
</TableCell>
</TableRow>
) : (
jobs.map((job) => (
<TableRow
key={job.job_id}
className="cursor-pointer hover:bg-muted/50"
onClick={() => router.push(`/admin/jobs/${job.job_id}`)}
>
<TableCell className="font-mono text-sm">{job.job_id}</TableCell>
<TableCell className="text-sm text-muted-foreground max-w-[150px] truncate">
{job.user_email || "—"}
</TableCell>
<TableCell>
{job.artist && job.title ? (
<span className="max-w-[200px] truncate block">
{job.artist} - {job.title}
</span>
) : job.filename ? (
<span className="max-w-[200px] truncate block text-muted-foreground">
{job.filename}
</span>
) : (
"—"
)}
</TableCell>
<TableCell>
<Badge variant={getStatusVariant(job.status)}>
{job.status.replace(/_/g, " ")}
</Badge>
</TableCell>
<TableCell>
<div className="flex items-center gap-2">
<div className="w-16 h-2 bg-muted rounded-full overflow-hidden">
<div
className="h-full bg-primary transition-all"
style={{ width: `${job.progress}%` }}
/>
</div>
<span className="text-xs text-muted-foreground">
{job.progress}%
</span>
</div>
</TableCell>
<TableCell className="text-sm">
{formatDate(job.created_at)}
</TableCell>
<TableCell className="text-right">
<div className="flex justify-end gap-1" onClick={(e) => e.stopPropagation()}>
<Button
variant="ghost"
size="icon"
onClick={() => router.push(`/admin/jobs/${job.job_id}`)}
title="View details"
>
<ExternalLink className="w-4 h-4" />
</Button>
<Button
variant="ghost"
size="icon"
onClick={() => {
setJobToDelete(job)
setDeleteDialogOpen(true)
}}
title="Delete job"
className="text-destructive hover:text-destructive"
>
<Trash2 className="w-4 h-4" />
</Button>
</div>
</TableCell>
</TableRow>
))
)}
</TableBody>
</Table>
</div>
{/* Summary */}
<p className="text-sm text-muted-foreground">
Showing {jobs.length} jobs
{statusFilter !== "all" && ` with status "${statusFilter}"`}
{userEmailFilter && ` for user "${userEmailFilter}"`}
</p>
{/* Delete Confirmation Dialog */}
<AlertDialog open={deleteDialogOpen} onOpenChange={setDeleteDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Delete Job</AlertDialogTitle>
<AlertDialogDescription>
Are you sure you want to delete job {jobToDelete?.job_id}? This action
cannot be undone and will also delete all associated files.
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleDelete}
disabled={deleting}
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
>
{deleting && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
Delete
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</div>
)
}
+140
View File
@@ -0,0 +1,140 @@
"use client"
import React, { useEffect, useState } from "react"
import { useRouter } from "next/navigation"
import { getAccessToken } from "@/lib/api"
import { useAuth } from "@/lib/auth"
import { Loader2 } from "lucide-react"
import { SidebarProvider, SidebarInset, SidebarTrigger } from "@/components/ui/sidebar"
import { AdminSidebar } from "@/components/admin/admin-sidebar"
import { Separator } from "@/components/ui/separator"
import {
Breadcrumb,
BreadcrumbItem,
BreadcrumbLink,
BreadcrumbList,
BreadcrumbPage,
BreadcrumbSeparator,
} from "@/components/ui/breadcrumb"
import { usePathname } from "next/navigation"
function AdminBreadcrumb() {
const pathname = usePathname()
const getBreadcrumbs = () => {
const segments = pathname.split("/").filter(Boolean)
const breadcrumbs: { label: string; href?: string }[] = []
// Always start with Admin
if (segments[0] === "admin") {
if (segments.length === 1) {
breadcrumbs.push({ label: "Dashboard" })
} else {
breadcrumbs.push({ label: "Admin", href: "/admin" })
if (segments[1] === "users") {
if (segments.length === 2) {
breadcrumbs.push({ label: "Users" })
} else {
breadcrumbs.push({ label: "Users", href: "/admin/users" })
breadcrumbs.push({ label: decodeURIComponent(segments[2]) })
}
} else if (segments[1] === "jobs") {
if (segments.length === 2) {
breadcrumbs.push({ label: "Jobs" })
} else if (segments[2]) {
breadcrumbs.push({ label: "Jobs", href: "/admin/jobs" })
breadcrumbs.push({ label: decodeURIComponent(segments[2]) })
}
} else if (segments[1] === "searches") {
breadcrumbs.push({ label: "Audio Searches" })
} else if (segments[1] === "beta") {
breadcrumbs.push({ label: "Beta Program" })
}
}
}
return breadcrumbs
}
const breadcrumbs = getBreadcrumbs()
return (
<Breadcrumb>
<BreadcrumbList>
{breadcrumbs.map((crumb, index) => (
<React.Fragment key={`${crumb.href || ''}-${crumb.label}`}>
{index > 0 && <BreadcrumbSeparator />}
<BreadcrumbItem>
{crumb.href ? (
<BreadcrumbLink href={crumb.href}>{crumb.label}</BreadcrumbLink>
) : (
<BreadcrumbPage>{crumb.label}</BreadcrumbPage>
)}
</BreadcrumbItem>
</React.Fragment>
))}
</BreadcrumbList>
</Breadcrumb>
)
}
export default function AdminLayout({
children,
}: {
children: React.ReactNode
}) {
const router = useRouter()
const { user } = useAuth()
const [isChecking, setIsChecking] = useState(true)
useEffect(() => {
const token = getAccessToken()
if (!token) {
router.replace("/")
return
}
// Check if user is admin (either by role or @nomadkaraoke.com email)
if (user) {
const isAdmin = user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")
if (!isAdmin) {
router.replace("/app")
return
}
}
// If we have a token but no user yet, wait for auth to load
if (!user) {
return
}
setIsChecking(false)
}, [router, user])
// Show loading while checking auth
if (isChecking || !user) {
return (
<div
className="min-h-screen flex items-center justify-center"
style={{ background: "var(--bg)" }}
>
<Loader2 className="w-8 h-8 animate-spin" style={{ color: "var(--text-muted)" }} />
</div>
)
}
return (
<SidebarProvider>
<AdminSidebar />
<SidebarInset>
<header className="flex h-16 shrink-0 items-center gap-2 border-b px-4">
<SidebarTrigger className="-ml-1" />
<Separator orientation="vertical" className="mr-2 h-4" />
<AdminBreadcrumb />
</header>
<main className="flex-1 p-4 md:p-6">{children}</main>
</SidebarInset>
</SidebarProvider>
)
}
+223
View File
@@ -0,0 +1,223 @@
"use client"
import { useEffect, useState } from "react"
import { adminApi, AdminStatsOverview } from "@/lib/api"
import { StatsCard, StatsGrid } from "@/components/admin/stats-card"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import { Badge } from "@/components/ui/badge"
import {
Users,
Briefcase,
CreditCard,
TestTube2,
Clock,
CheckCircle,
XCircle,
AlertCircle,
Loader2,
RefreshCw,
} from "lucide-react"
import { Button } from "@/components/ui/button"
export default function AdminDashboardPage() {
const [stats, setStats] = useState<AdminStatsOverview | null>(null)
const [loading, setLoading] = useState(true)
const [error, setError] = useState<string | null>(null)
const loadStats = async () => {
try {
setLoading(true)
setError(null)
const data = await adminApi.getStats()
setStats(data)
} catch (err: unknown) {
const message = err instanceof Error ? err.message : "Failed to load statistics"
setError(message)
console.error("Failed to load admin stats:", err)
} finally {
setLoading(false)
}
}
useEffect(() => {
loadStats()
}, [])
if (error) {
return (
<div className="flex flex-col items-center justify-center py-12">
<XCircle className="w-12 h-12 text-destructive mb-4" />
<h2 className="text-lg font-semibold mb-2">Failed to load statistics</h2>
<p className="text-muted-foreground mb-4">{error}</p>
<Button onClick={loadStats}>
<RefreshCw className="w-4 h-4 mr-2" />
Retry
</Button>
</div>
)
}
return (
<div className="space-y-6">
{/* Header */}
<div className="flex items-center justify-between">
<div>
<h1 className="text-2xl font-bold tracking-tight">Dashboard</h1>
<p className="text-muted-foreground">
Overview of your karaoke platform
</p>
</div>
<Button variant="outline" size="sm" onClick={loadStats} disabled={loading}>
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
Refresh
</Button>
</div>
{/* Main Stats Grid */}
<StatsGrid>
<StatsCard
title="Total Users"
value={stats?.total_users ?? 0}
description={`${stats?.active_users_7d ?? 0} active in last 7 days`}
icon={Users}
loading={loading}
/>
<StatsCard
title="Total Jobs"
value={stats?.total_jobs ?? 0}
description={`${stats?.jobs_last_7d ?? 0} in last 7 days`}
icon={Briefcase}
loading={loading}
/>
<StatsCard
title="Credits Issued (30d)"
value={stats?.total_credits_issued_30d ?? 0}
description="Credits added to accounts"
icon={CreditCard}
loading={loading}
/>
<StatsCard
title="Beta Testers"
value={stats?.total_beta_testers ?? 0}
description="Active beta participants"
icon={TestTube2}
loading={loading}
/>
</StatsGrid>
{/* Secondary Stats */}
<div className="grid gap-4 md:grid-cols-2">
{/* Job Status Breakdown */}
<Card>
<CardHeader>
<CardTitle className="text-base">Job Status Breakdown</CardTitle>
<CardDescription>Current status of all jobs in the system</CardDescription>
</CardHeader>
<CardContent>
{loading ? (
<div className="flex items-center justify-center py-8">
<Loader2 className="w-6 h-6 animate-spin text-muted-foreground" />
</div>
) : (
<div className="space-y-3">
<div className="flex items-center justify-between">
<div className="flex items-center gap-2">
<Clock className="w-4 h-4 text-yellow-500" />
<span className="text-sm">Pending</span>
</div>
<Badge variant="secondary">{stats?.jobs_by_status?.pending ?? 0}</Badge>
</div>
<div className="flex items-center justify-between">
<div className="flex items-center gap-2">
<Loader2 className="w-4 h-4 text-blue-500" />
<span className="text-sm">Processing</span>
</div>
<Badge variant="secondary">{stats?.jobs_by_status?.processing ?? 0}</Badge>
</div>
<div className="flex items-center justify-between">
<div className="flex items-center gap-2">
<AlertCircle className="w-4 h-4 text-orange-500" />
<span className="text-sm">Awaiting Review</span>
</div>
<Badge variant="secondary">{stats?.jobs_by_status?.awaiting_review ?? 0}</Badge>
</div>
<div className="flex items-center justify-between">
<div className="flex items-center gap-2">
<AlertCircle className="w-4 h-4 text-purple-500" />
<span className="text-sm">Awaiting Instrumental</span>
</div>
<Badge variant="secondary">{stats?.jobs_by_status?.awaiting_instrumental ?? 0}</Badge>
</div>
<div className="flex items-center justify-between">
<div className="flex items-center gap-2">
<CheckCircle className="w-4 h-4 text-green-500" />
<span className="text-sm">Complete</span>
</div>
<Badge variant="secondary">{stats?.jobs_by_status?.complete ?? 0}</Badge>
</div>
<div className="flex items-center justify-between">
<div className="flex items-center gap-2">
<XCircle className="w-4 h-4 text-red-500" />
<span className="text-sm">Failed</span>
</div>
<Badge variant="destructive">{stats?.jobs_by_status?.failed ?? 0}</Badge>
</div>
<div className="flex items-center justify-between">
<div className="flex items-center gap-2">
<XCircle className="w-4 h-4 text-gray-500" />
<span className="text-sm">Cancelled</span>
</div>
<Badge variant="outline">{stats?.jobs_by_status?.cancelled ?? 0}</Badge>
</div>
</div>
)}
</CardContent>
</Card>
{/* Activity Summary */}
<Card>
<CardHeader>
<CardTitle className="text-base">Activity Summary</CardTitle>
<CardDescription>User and job activity over time</CardDescription>
</CardHeader>
<CardContent>
{loading ? (
<div className="flex items-center justify-center py-8">
<Loader2 className="w-6 h-6 animate-spin text-muted-foreground" />
</div>
) : (
<div className="space-y-4">
<div>
<h4 className="text-sm font-medium mb-2">Users</h4>
<div className="grid grid-cols-2 gap-4">
<div className="rounded-lg border p-3">
<p className="text-2xl font-bold">{stats?.active_users_7d ?? 0}</p>
<p className="text-xs text-muted-foreground">Active (7d)</p>
</div>
<div className="rounded-lg border p-3">
<p className="text-2xl font-bold">{stats?.active_users_30d ?? 0}</p>
<p className="text-xs text-muted-foreground">Active (30d)</p>
</div>
</div>
</div>
<div>
<h4 className="text-sm font-medium mb-2">Jobs</h4>
<div className="grid grid-cols-2 gap-4">
<div className="rounded-lg border p-3">
<p className="text-2xl font-bold">{stats?.jobs_last_7d ?? 0}</p>
<p className="text-xs text-muted-foreground">Created (7d)</p>
</div>
<div className="rounded-lg border p-3">
<p className="text-2xl font-bold">{stats?.jobs_last_30d ?? 0}</p>
<p className="text-xs text-muted-foreground">Created (30d)</p>
</div>
</div>
</div>
</div>
)}
</CardContent>
</Card>
</div>
</div>
)
}
+473
View File
@@ -0,0 +1,473 @@
"use client"
import { useEffect, useState, useCallback } from "react"
import { adminApi, AudioSearchJobSummary, CacheStatsResponse } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Badge } from "@/components/ui/badge"
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table"
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select"
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@/components/ui/alert-dialog"
import {
Tooltip,
TooltipContent,
TooltipProvider,
TooltipTrigger,
} from "@/components/ui/tooltip"
import {
RefreshCw,
Loader2,
Trash2,
Music,
CheckCircle,
XCircle,
ExternalLink,
Database,
HardDrive,
} from "lucide-react"
import { useToast } from "@/hooks/use-toast"
const statusOptions = [
{ value: "all", label: "All Statuses" },
{ value: "awaiting_audio_selection", label: "Awaiting Selection" },
{ value: "pending", label: "Pending" },
{ value: "complete", label: "Complete" },
{ value: "failed", label: "Failed" },
]
export default function AdminSearchesPage() {
const { toast } = useToast()
const [searches, setSearches] = useState<AudioSearchJobSummary[]>([])
const [loading, setLoading] = useState(true)
const [statusFilter, setStatusFilter] = useState("all")
// Clear cache dialog (single job)
const [clearDialogOpen, setClearDialogOpen] = useState(false)
const [jobToClear, setJobToClear] = useState<AudioSearchJobSummary | null>(null)
const [clearing, setClearing] = useState(false)
// Clear all cache dialog
const [clearAllDialogOpen, setClearAllDialogOpen] = useState(false)
const [clearingAll, setClearingAll] = useState(false)
// Cache stats
const [cacheStats, setCacheStats] = useState<CacheStatsResponse | null>(null)
const [statsLoading, setStatsLoading] = useState(false)
const loadSearches = useCallback(async () => {
try {
setLoading(true)
const data = await adminApi.listAudioSearches({
limit: 100,
status_filter: statusFilter !== "all" ? statusFilter : undefined,
})
setSearches(data.jobs)
} catch (err: any) {
console.error("Failed to load searches:", err)
toast({
title: "Error",
description: err.message || "Failed to load audio searches",
variant: "destructive",
})
} finally {
setLoading(false)
}
}, [statusFilter, toast])
const loadCacheStats = useCallback(async () => {
try {
setStatsLoading(true)
const stats = await adminApi.getCacheStats()
setCacheStats(stats)
} catch (err: any) {
// Silently fail - stats are optional
console.error("Failed to load cache stats:", err)
setCacheStats(null)
} finally {
setStatsLoading(false)
}
}, [])
useEffect(() => {
loadSearches()
loadCacheStats()
}, [loadSearches, loadCacheStats])
const handleClearCache = async () => {
if (!jobToClear) return
try {
setClearing(true)
const result = await adminApi.clearAudioSearchCache(jobToClear.job_id)
toast({
title: "Cache Cleared",
description: result.message,
})
setClearDialogOpen(false)
setJobToClear(null)
loadSearches()
loadCacheStats()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to clear cache",
variant: "destructive",
})
} finally {
setClearing(false)
}
}
const handleClearAllCache = async () => {
try {
setClearingAll(true)
const result = await adminApi.clearAllCache()
toast({
title: "All Cache Cleared",
description: result.message,
})
setClearAllDialogOpen(false)
loadCacheStats()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to clear all cache",
variant: "destructive",
})
} finally {
setClearingAll(false)
}
}
const formatBytes = (bytes: number): string => {
if (bytes === 0) return "0 B"
const k = 1024
const sizes = ["B", "KB", "MB", "GB"]
const i = Math.floor(Math.log(bytes) / Math.log(k))
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + " " + sizes[i]
}
const getStatusVariant = (status: string) => {
if (status === "complete" || status === "prep_complete") return "default"
if (status === "failed") return "destructive"
if (status === "cancelled") return "outline"
if (status.includes("awaiting")) return "secondary"
return "secondary"
}
const formatDate = (dateStr?: string) => {
if (!dateStr) return "—"
return new Date(dateStr).toLocaleString()
}
const getProviderBadgeClass = (provider: string) => {
switch (provider.toLowerCase()) {
case "youtube":
return "bg-red-500/20 text-red-400 border-red-500/30"
case "red":
return "bg-red-600/30 text-red-300 border-red-500/30"
case "ops":
return "bg-blue-500/20 text-blue-400 border-blue-500/30"
default:
return "bg-slate-500/20 text-slate-400 border-slate-500/30"
}
}
return (
<div className="space-y-6">
{/* Header */}
<div className="flex items-center justify-between">
<div>
<h1 className="text-2xl font-bold tracking-tight">Audio Searches</h1>
<p className="text-muted-foreground">
View and manage cached audio search results
</p>
</div>
<div className="flex items-center gap-2">
<Button variant="outline" size="sm" onClick={() => { loadSearches(); loadCacheStats() }} disabled={loading}>
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
Refresh
</Button>
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="outline"
size="sm"
className="text-red-500 hover:text-red-400 hover:bg-red-500/10 border-red-500/30"
onClick={() => setClearAllDialogOpen(true)}
>
<Database className="w-4 h-4 mr-2" />
Clear All Cache
</Button>
</TooltipTrigger>
<TooltipContent>
<p>Clear entire flacfetch search cache</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
</div>
</div>
{/* Cache Stats */}
{cacheStats && (
<div className="flex items-center gap-4 text-sm text-muted-foreground bg-muted/30 rounded-lg px-4 py-2">
<div className="flex items-center gap-2">
<HardDrive className="w-4 h-4" />
<span>Flacfetch Cache:</span>
</div>
<Badge variant="outline" className="font-mono">
{cacheStats.count} entries
</Badge>
<Badge variant="outline" className="font-mono">
{formatBytes(cacheStats.total_size_bytes)}
</Badge>
{cacheStats.newest_entry && (
<span className="text-xs">
Latest: {new Date(cacheStats.newest_entry).toLocaleDateString()}
</span>
)}
</div>
)}
{/* Filters */}
<div className="flex gap-4">
<Select value={statusFilter} onValueChange={setStatusFilter}>
<SelectTrigger className="w-[200px]">
<SelectValue placeholder="Filter by status" />
</SelectTrigger>
<SelectContent>
{statusOptions.map((opt) => (
<SelectItem key={opt.value} value={opt.value}>
{opt.label}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
{/* Table */}
<div className="rounded-md border">
<Table>
<TableHeader>
<TableRow>
<TableHead>Job ID</TableHead>
<TableHead>Search Query</TableHead>
<TableHead>Results</TableHead>
<TableHead>Providers</TableHead>
<TableHead>Status</TableHead>
<TableHead>User</TableHead>
<TableHead>Created</TableHead>
<TableHead className="text-right">Actions</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{loading ? (
<TableRow>
<TableCell colSpan={8} className="text-center py-8">
<Loader2 className="w-6 h-6 animate-spin mx-auto text-muted-foreground" />
</TableCell>
</TableRow>
) : searches.length === 0 ? (
<TableRow>
<TableCell colSpan={8} className="text-center py-8 text-muted-foreground">
<Music className="w-8 h-8 mx-auto mb-2 opacity-50" />
No audio searches found
</TableCell>
</TableRow>
) : (
searches.map((search) => (
<TableRow key={search.job_id}>
<TableCell className="font-mono text-xs">
<a
href={`/admin/jobs?id=${search.job_id}`}
className="text-blue-500 hover:underline"
>
{search.job_id.slice(0, 8)}...
</a>
</TableCell>
<TableCell>
<div className="max-w-[200px]">
<div className="font-medium truncate">
{search.audio_search_artist || "—"}
</div>
<div className="text-sm text-muted-foreground truncate">
{search.audio_search_title || "—"}
</div>
</div>
</TableCell>
<TableCell>
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<div className="flex items-center gap-2">
<span className="font-medium">{search.results_count}</span>
{search.has_lossless ? (
<CheckCircle className="w-4 h-4 text-green-500" />
) : (
<XCircle className="w-4 h-4 text-red-400" />
)}
</div>
</TooltipTrigger>
<TooltipContent>
<p>{search.has_lossless ? "Has lossless sources" : "No lossless sources (YouTube only)"}</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
</TableCell>
<TableCell>
<div className="flex flex-wrap gap-1">
{search.providers.map((provider) => (
<Badge
key={provider}
variant="outline"
className={`text-[10px] ${getProviderBadgeClass(provider)}`}
>
{provider}
</Badge>
))}
</div>
</TableCell>
<TableCell>
<Badge variant={getStatusVariant(search.status)}>
{search.status.replace(/_/g, " ")}
</Badge>
</TableCell>
<TableCell className="text-xs text-muted-foreground">
{search.user_email?.split("@")[0] || "—"}
</TableCell>
<TableCell className="text-xs text-muted-foreground">
{formatDate(search.created_at)}
</TableCell>
<TableCell className="text-right">
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="ghost"
size="icon"
className="h-8 w-8 text-orange-500 hover:text-orange-400 hover:bg-orange-500/10"
onClick={() => {
setJobToClear(search)
setClearDialogOpen(true)
}}
>
<Trash2 className="w-4 h-4" />
</Button>
</TooltipTrigger>
<TooltipContent>
<p>Clear cache & allow re-search</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
</TableCell>
</TableRow>
))
)}
</TableBody>
</Table>
</div>
{/* Summary */}
{!loading && searches.length > 0 && (
<div className="text-sm text-muted-foreground">
Showing {searches.length} jobs with cached search results.
{searches.filter(s => !s.has_lossless).length > 0 && (
<span className="ml-2 text-orange-400">
{searches.filter(s => !s.has_lossless).length} have YouTube-only results (may need cache refresh).
</span>
)}
</div>
)}
{/* Clear Cache Dialog (single job) */}
<AlertDialog open={clearDialogOpen} onOpenChange={setClearDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Clear Search Cache?</AlertDialogTitle>
<AlertDialogDescription>
This will clear the cached search results for job{" "}
<span className="font-mono">{jobToClear?.job_id}</span> and reset it to pending
status, allowing a new search to be performed.
{jobToClear && !jobToClear.has_lossless && (
<span className="block mt-2 text-orange-400">
This job only has YouTube results - clearing the cache may fix this if flacfetch has been updated.
</span>
)}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleClearCache}
disabled={clearing}
className="bg-orange-600 hover:bg-orange-500"
>
{clearing ? (
<Loader2 className="w-4 h-4 animate-spin mr-2" />
) : (
<Trash2 className="w-4 h-4 mr-2" />
)}
Clear Cache
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
{/* Clear All Cache Dialog */}
<AlertDialog open={clearAllDialogOpen} onOpenChange={setClearAllDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Clear Entire Flacfetch Cache?</AlertDialogTitle>
<AlertDialogDescription>
This will delete all cached search results from flacfetch&apos;s GCS cache
{cacheStats && (
<span className="font-medium"> ({cacheStats.count} entries, {formatBytes(cacheStats.total_size_bytes)})</span>
)}.
<span className="block mt-2 text-orange-400">
All subsequent searches will hit the trackers fresh. This does NOT affect
the Firestore job caches - use individual &quot;Clear Cache&quot; buttons for those.
</span>
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleClearAllCache}
disabled={clearingAll}
className="bg-red-600 hover:bg-red-500"
>
{clearingAll ? (
<Loader2 className="w-4 h-4 animate-spin mr-2" />
) : (
<Database className="w-4 h-4 mr-2" />
)}
Clear All Cache
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</div>
)
}
+485
View File
@@ -0,0 +1,485 @@
"use client"
import { useEffect, useState } from "react"
import { useSearchParams, useRouter } from "next/navigation"
import { adminApi, AdminUserDetail } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Badge } from "@/components/ui/badge"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table"
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog"
import { Input } from "@/components/ui/input"
import { Label } from "@/components/ui/label"
import {
ArrowLeft,
RefreshCw,
Loader2,
Plus,
Shield,
ShieldOff,
UserX,
UserCheck,
CreditCard,
Briefcase,
Clock,
} from "lucide-react"
import { useToast } from "@/hooks/use-toast"
export default function AdminUserDetailPage() {
const searchParams = useSearchParams()
const router = useRouter()
const { toast } = useToast()
const email = searchParams.get("email") || ""
const [user, setUser] = useState<AdminUserDetail | null>(null)
const [loading, setLoading] = useState(true)
const [actionLoading, setActionLoading] = useState(false)
// Credit dialog
const [creditDialogOpen, setCreditDialogOpen] = useState(false)
const [creditAmount, setCreditAmount] = useState("")
const [creditReason, setCreditReason] = useState("")
const loadUser = async () => {
if (!email) {
setLoading(false)
return
}
try {
setLoading(true)
const data = await adminApi.getUserDetail(email)
setUser(data)
} catch (err: any) {
console.error("Failed to load user:", err)
toast({
title: "Error",
description: err.message || "Failed to load user",
variant: "destructive",
})
} finally {
setLoading(false)
}
}
useEffect(() => {
loadUser()
}, [email])
const handleAddCredits = async () => {
if (!creditAmount || !creditReason) return
const amount = parseInt(creditAmount.trim(), 10)
if (!Number.isInteger(amount) || amount < 1 || amount > 100000) {
toast({
title: "Invalid Amount",
description: "Please enter a valid amount between 1 and 100,000",
variant: "destructive",
})
return
}
try {
setActionLoading(true)
await adminApi.addCredits(email, amount, creditReason)
toast({
title: "Credits Added",
description: `Added ${creditAmount} credits`,
})
setCreditDialogOpen(false)
setCreditAmount("")
setCreditReason("")
loadUser()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to add credits",
variant: "destructive",
})
} finally {
setActionLoading(false)
}
}
const handleToggleRole = async () => {
if (!user) return
const newRole = user.role === "admin" ? "user" : "admin"
try {
setActionLoading(true)
await adminApi.setUserRole(email, newRole)
toast({
title: "Role Updated",
description: `User is now ${newRole}`,
})
loadUser()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to update role",
variant: "destructive",
})
} finally {
setActionLoading(false)
}
}
const handleToggleStatus = async () => {
if (!user) return
try {
setActionLoading(true)
if (user.is_active) {
await adminApi.disableUser(email)
toast({
title: "User Disabled",
description: "User has been disabled",
})
} else {
await adminApi.enableUser(email)
toast({
title: "User Enabled",
description: "User has been enabled",
})
}
loadUser()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to update user status",
variant: "destructive",
})
} finally {
setActionLoading(false)
}
}
const formatDate = (dateStr?: string) => {
if (!dateStr) return "Never"
return new Date(dateStr).toLocaleString()
}
if (!email) {
return (
<div className="text-center py-12">
<p className="text-muted-foreground">No user email provided</p>
<Button variant="outline" className="mt-4" onClick={() => router.push("/admin/users")}>
<ArrowLeft className="w-4 h-4 mr-2" />
Back to Users
</Button>
</div>
)
}
if (loading) {
return (
<div className="flex items-center justify-center py-12">
<Loader2 className="w-8 h-8 animate-spin text-muted-foreground" />
</div>
)
}
if (!user) {
return (
<div className="text-center py-12">
<p className="text-muted-foreground">User not found</p>
<Button variant="outline" className="mt-4" onClick={() => router.push("/admin/users")}>
<ArrowLeft className="w-4 h-4 mr-2" />
Back to Users
</Button>
</div>
)
}
return (
<div className="space-y-6">
{/* Header */}
<div className="flex items-center justify-between">
<div className="flex items-center gap-4">
<Button variant="ghost" size="icon" onClick={() => router.push("/admin/users")}>
<ArrowLeft className="w-4 h-4" />
</Button>
<div>
<h1 className="text-2xl font-bold tracking-tight flex items-center gap-2">
{user.display_name || user.email}
<Badge variant={user.role === "admin" ? "default" : "secondary"}>
{user.role}
</Badge>
{!user.is_active && <Badge variant="destructive">Disabled</Badge>}
</h1>
<p className="text-muted-foreground">{user.email}</p>
</div>
</div>
<div className="flex gap-2">
<Button variant="outline" size="sm" onClick={loadUser} disabled={loading}>
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
Refresh
</Button>
</div>
</div>
{/* Actions */}
<div className="flex flex-wrap gap-2">
<Button onClick={() => setCreditDialogOpen(true)} disabled={actionLoading}>
<Plus className="w-4 h-4 mr-2" />
Add Credits
</Button>
<Button
variant="outline"
onClick={handleToggleRole}
disabled={actionLoading}
>
{user.role === "admin" ? (
<>
<ShieldOff className="w-4 h-4 mr-2" />
Remove Admin
</>
) : (
<>
<Shield className="w-4 h-4 mr-2" />
Make Admin
</>
)}
</Button>
<Button
variant={user.is_active ? "destructive" : "outline"}
onClick={handleToggleStatus}
disabled={actionLoading}
>
{user.is_active ? (
<>
<UserX className="w-4 h-4 mr-2" />
Disable User
</>
) : (
<>
<UserCheck className="w-4 h-4 mr-2" />
Enable User
</>
)}
</Button>
</div>
{/* Stats Grid */}
<div className="grid gap-4 md:grid-cols-4">
<Card>
<CardHeader className="pb-2">
<CardTitle className="text-sm font-medium flex items-center gap-2">
<CreditCard className="w-4 h-4" />
Credits
</CardTitle>
</CardHeader>
<CardContent>
<p className="text-2xl font-bold">{user.credits}</p>
</CardContent>
</Card>
<Card>
<CardHeader className="pb-2">
<CardTitle className="text-sm font-medium flex items-center gap-2">
<Briefcase className="w-4 h-4" />
Jobs Created
</CardTitle>
</CardHeader>
<CardContent>
<p className="text-2xl font-bold">{user.total_jobs_created}</p>
</CardContent>
</Card>
<Card>
<CardHeader className="pb-2">
<CardTitle className="text-sm font-medium flex items-center gap-2">
<Briefcase className="w-4 h-4" />
Jobs Completed
</CardTitle>
</CardHeader>
<CardContent>
<p className="text-2xl font-bold">{user.total_jobs_completed}</p>
</CardContent>
</Card>
<Card>
<CardHeader className="pb-2">
<CardTitle className="text-sm font-medium flex items-center gap-2">
<Clock className="w-4 h-4" />
Active Sessions
</CardTitle>
</CardHeader>
<CardContent>
<p className="text-2xl font-bold">{user.active_sessions_count}</p>
</CardContent>
</Card>
</div>
<div className="grid gap-6 md:grid-cols-2">
{/* Account Info */}
<Card>
<CardHeader>
<CardTitle>Account Information</CardTitle>
</CardHeader>
<CardContent className="space-y-3">
<div className="flex justify-between">
<span className="text-muted-foreground">Email Verified</span>
<Badge variant={user.email_verified ? "default" : "secondary"}>
{user.email_verified ? "Yes" : "No"}
</Badge>
</div>
<div className="flex justify-between">
<span className="text-muted-foreground">Beta Tester</span>
<Badge variant={user.is_beta_tester ? "default" : "secondary"}>
{user.is_beta_tester ? user.beta_tester_status || "Yes" : "No"}
</Badge>
</div>
<div className="flex justify-between">
<span className="text-muted-foreground">Created</span>
<span className="text-sm">{formatDate(user.created_at)}</span>
</div>
<div className="flex justify-between">
<span className="text-muted-foreground">Last Login</span>
<span className="text-sm">{formatDate(user.last_login_at)}</span>
</div>
<div className="flex justify-between">
<span className="text-muted-foreground">Last Updated</span>
<span className="text-sm">{formatDate(user.updated_at)}</span>
</div>
</CardContent>
</Card>
{/* Credit Transactions */}
<Card>
<CardHeader>
<CardTitle>Recent Credit Transactions</CardTitle>
<CardDescription>Last 20 transactions</CardDescription>
</CardHeader>
<CardContent>
{user.credit_transactions.length === 0 ? (
<p className="text-muted-foreground text-sm">No transactions</p>
) : (
<div className="space-y-2 max-h-[300px] overflow-y-auto">
{user.credit_transactions.slice().reverse().map((txn, i) => (
<div key={txn.id || i} className="flex justify-between items-center text-sm border-b pb-2">
<div>
<span className={txn.amount > 0 ? "text-green-500" : "text-red-500"}>
{txn.amount > 0 ? "+" : ""}{txn.amount}
</span>
<span className="text-muted-foreground ml-2">{txn.reason}</span>
</div>
<span className="text-xs text-muted-foreground">
{new Date(txn.created_at).toLocaleDateString()}
</span>
</div>
))}
</div>
)}
</CardContent>
</Card>
</div>
{/* Recent Jobs */}
<Card>
<CardHeader>
<CardTitle>Recent Jobs</CardTitle>
<CardDescription>Last 10 jobs by this user</CardDescription>
</CardHeader>
<CardContent>
{user.recent_jobs.length === 0 ? (
<p className="text-muted-foreground text-sm">No jobs</p>
) : (
<Table>
<TableHeader>
<TableRow>
<TableHead>Job ID</TableHead>
<TableHead>Artist / Title</TableHead>
<TableHead>Status</TableHead>
<TableHead>Created</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{user.recent_jobs.map((job) => (
<TableRow
key={job.job_id}
className="cursor-pointer hover:bg-muted/50"
onClick={() => router.push(`/admin/jobs/detail?id=${job.job_id}`)}
>
<TableCell className="font-mono text-sm">{job.job_id}</TableCell>
<TableCell>
{job.artist && job.title ? `${job.artist} - ${job.title}` : "—"}
</TableCell>
<TableCell>
<Badge variant={
job.status === "complete" ? "default" :
job.status === "failed" ? "destructive" :
"secondary"
}>
{job.status}
</Badge>
</TableCell>
<TableCell className="text-sm">
{job.created_at ? new Date(job.created_at).toLocaleString() : "—"}
</TableCell>
</TableRow>
))}
</TableBody>
</Table>
)}
</CardContent>
</Card>
{/* Add Credits Dialog */}
<Dialog open={creditDialogOpen} onOpenChange={setCreditDialogOpen}>
<DialogContent>
<DialogHeader>
<DialogTitle>Add Credits</DialogTitle>
<DialogDescription>
Add credits to {user.email}
</DialogDescription>
</DialogHeader>
<div className="space-y-4 py-4">
<div className="space-y-2">
<Label htmlFor="amount">Amount</Label>
<Input
id="amount"
type="number"
min="1"
placeholder="Enter amount"
value={creditAmount}
onChange={(e) => setCreditAmount(e.target.value)}
/>
</div>
<div className="space-y-2">
<Label htmlFor="reason">Reason</Label>
<Input
id="reason"
placeholder="e.g., Beta reward, Support credit"
value={creditReason}
onChange={(e) => setCreditReason(e.target.value)}
/>
</div>
</div>
<DialogFooter>
<Button variant="outline" onClick={() => setCreditDialogOpen(false)}>
Cancel
</Button>
<Button
onClick={handleAddCredits}
disabled={!creditAmount || !creditReason || actionLoading}
>
{actionLoading && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
Add Credits
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
</div>
)
}
+375
View File
@@ -0,0 +1,375 @@
"use client"
import { useEffect, useState, useCallback } from "react"
import { useRouter } from "next/navigation"
import { adminApi, AdminUser, AdminUserListResponse } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Badge } from "@/components/ui/badge"
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table"
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select"
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
DialogTrigger,
} from "@/components/ui/dialog"
import { Label } from "@/components/ui/label"
import {
Search,
RefreshCw,
ChevronLeft,
ChevronRight,
Plus,
Loader2,
CreditCard,
Shield,
ShieldOff,
UserX,
UserCheck,
} from "lucide-react"
import { useToast } from "@/hooks/use-toast"
export default function AdminUsersPage() {
const router = useRouter()
const { toast } = useToast()
const [users, setUsers] = useState<AdminUser[]>([])
const [loading, setLoading] = useState(true)
const [total, setTotal] = useState(0)
const [offset, setOffset] = useState(0)
const [limit] = useState(20)
const [hasMore, setHasMore] = useState(false)
const [search, setSearch] = useState("")
const [searchInput, setSearchInput] = useState("")
const [sortBy, setSortBy] = useState<"created_at" | "last_login_at" | "credits" | "email">("created_at")
const [sortOrder, setSortOrder] = useState<"asc" | "desc">("desc")
// Credit dialog state
const [creditDialogOpen, setCreditDialogOpen] = useState(false)
const [selectedUser, setSelectedUser] = useState<AdminUser | null>(null)
const [creditAmount, setCreditAmount] = useState("")
const [creditReason, setCreditReason] = useState("")
const [addingCredits, setAddingCredits] = useState(false)
const loadUsers = useCallback(async () => {
try {
setLoading(true)
const data = await adminApi.listUsers({
limit,
offset,
search: search || undefined,
sort_by: sortBy,
sort_order: sortOrder,
})
setUsers(data.users)
setTotal(data.total)
setHasMore(data.has_more)
} catch (err: any) {
console.error("Failed to load users:", err)
toast({
title: "Error",
description: err.message || "Failed to load users",
variant: "destructive",
})
} finally {
setLoading(false)
}
}, [limit, offset, search, sortBy, sortOrder, toast])
useEffect(() => {
loadUsers()
}, [loadUsers])
const handleSearch = () => {
setOffset(0)
setSearch(searchInput)
}
const handleKeyPress = (e: React.KeyboardEvent) => {
if (e.key === "Enter") {
handleSearch()
}
}
const handleAddCredits = async () => {
if (!selectedUser || !creditAmount || !creditReason) return
const amount = parseInt(creditAmount.trim(), 10)
if (!Number.isInteger(amount) || amount < 1 || amount > 100000) {
toast({
title: "Invalid Amount",
description: "Please enter a valid amount between 1 and 100,000",
variant: "destructive",
})
return
}
try {
setAddingCredits(true)
await adminApi.addCredits(selectedUser.email, amount, creditReason)
toast({
title: "Credits Added",
description: `Added ${creditAmount} credits to ${selectedUser.email}`,
})
setCreditDialogOpen(false)
setCreditAmount("")
setCreditReason("")
setSelectedUser(null)
loadUsers()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to add credits",
variant: "destructive",
})
} finally {
setAddingCredits(false)
}
}
const handleToggleRole = async (user: AdminUser) => {
const newRole = user.role === "admin" ? "user" : "admin"
try {
await adminApi.setUserRole(user.email, newRole)
toast({
title: "Role Updated",
description: `${user.email} is now ${newRole}`,
})
loadUsers()
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to update role",
variant: "destructive",
})
}
}
return (
<div className="space-y-6">
{/* Header */}
<div className="flex items-center justify-between">
<div>
<h1 className="text-2xl font-bold tracking-tight">Users</h1>
<p className="text-muted-foreground">
Manage user accounts and credits
</p>
</div>
<Button variant="outline" size="sm" onClick={loadUsers} disabled={loading}>
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
Refresh
</Button>
</div>
{/* Filters */}
<div className="flex flex-col sm:flex-row gap-4">
<div className="flex-1 flex gap-2">
<Input
placeholder="Search by email..."
value={searchInput}
onChange={(e) => setSearchInput(e.target.value)}
onKeyPress={handleKeyPress}
className="max-w-sm"
/>
<Button variant="secondary" onClick={handleSearch}>
<Search className="w-4 h-4" />
</Button>
</div>
<div className="flex gap-2">
<Select value={sortBy} onValueChange={(v) => setSortBy(v as any)}>
<SelectTrigger className="w-[140px]">
<SelectValue placeholder="Sort by" />
</SelectTrigger>
<SelectContent>
<SelectItem value="created_at">Created</SelectItem>
<SelectItem value="last_login_at">Last Login</SelectItem>
<SelectItem value="credits">Credits</SelectItem>
<SelectItem value="email">Email</SelectItem>
</SelectContent>
</Select>
<Select value={sortOrder} onValueChange={(v) => setSortOrder(v as any)}>
<SelectTrigger className="w-[100px]">
<SelectValue placeholder="Order" />
</SelectTrigger>
<SelectContent>
<SelectItem value="desc">Desc</SelectItem>
<SelectItem value="asc">Asc</SelectItem>
</SelectContent>
</Select>
</div>
</div>
{/* Table */}
<div className="rounded-md border">
<Table>
<TableHeader>
<TableRow>
<TableHead>Email</TableHead>
<TableHead>Role</TableHead>
<TableHead className="text-right">Credits</TableHead>
<TableHead className="text-right">Jobs</TableHead>
<TableHead className="text-right">Actions</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{loading ? (
<TableRow>
<TableCell colSpan={5} className="text-center py-8">
<Loader2 className="w-6 h-6 animate-spin mx-auto text-muted-foreground" />
</TableCell>
</TableRow>
) : users.length === 0 ? (
<TableRow>
<TableCell colSpan={5} className="text-center py-8 text-muted-foreground">
No users found
</TableCell>
</TableRow>
) : (
users.map((user) => (
<TableRow
key={user.email}
className="cursor-pointer hover:bg-muted/50"
onClick={() => router.push(`/admin/users/detail?email=${encodeURIComponent(user.email)}`)}
>
<TableCell className="font-medium">
{user.display_name || user.email}
{user.display_name && (
<span className="text-muted-foreground text-sm block">
{user.email}
</span>
)}
</TableCell>
<TableCell>
<Badge variant={user.role === "admin" ? "default" : "secondary"}>
{user.role}
</Badge>
</TableCell>
<TableCell className="text-right">{user.credits}</TableCell>
<TableCell className="text-right">
{user.total_jobs_created ?? 0} / {user.total_jobs_completed ?? 0}
</TableCell>
<TableCell className="text-right">
<div className="flex justify-end gap-1" onClick={(e) => e.stopPropagation()}>
<Button
variant="ghost"
size="icon"
onClick={() => {
setSelectedUser(user)
setCreditDialogOpen(true)
}}
title="Add credits"
>
<Plus className="w-4 h-4" />
</Button>
<Button
variant="ghost"
size="icon"
onClick={() => handleToggleRole(user)}
title={user.role === "admin" ? "Remove admin" : "Make admin"}
>
{user.role === "admin" ? (
<ShieldOff className="w-4 h-4" />
) : (
<Shield className="w-4 h-4" />
)}
</Button>
</div>
</TableCell>
</TableRow>
))
)}
</TableBody>
</Table>
</div>
{/* Pagination */}
<div className="flex items-center justify-between">
<p className="text-sm text-muted-foreground">
{total === 0 ? "No users found" : `Showing ${offset + 1}-${Math.min(offset + users.length, total)} of ${total} users`}
</p>
<div className="flex gap-2">
<Button
variant="outline"
size="sm"
onClick={() => setOffset(Math.max(0, offset - limit))}
disabled={offset === 0}
>
<ChevronLeft className="w-4 h-4" />
Previous
</Button>
<Button
variant="outline"
size="sm"
onClick={() => setOffset(offset + limit)}
disabled={!hasMore}
>
Next
<ChevronRight className="w-4 h-4" />
</Button>
</div>
</div>
{/* Add Credits Dialog */}
<Dialog open={creditDialogOpen} onOpenChange={setCreditDialogOpen}>
<DialogContent>
<DialogHeader>
<DialogTitle>Add Credits</DialogTitle>
<DialogDescription>
Add credits to {selectedUser?.email}
</DialogDescription>
</DialogHeader>
<div className="space-y-4 py-4">
<div className="space-y-2">
<Label htmlFor="amount">Amount</Label>
<Input
id="amount"
type="number"
min="1"
placeholder="Enter amount"
value={creditAmount}
onChange={(e) => setCreditAmount(e.target.value)}
/>
</div>
<div className="space-y-2">
<Label htmlFor="reason">Reason</Label>
<Input
id="reason"
placeholder="e.g., Beta reward, Support credit"
value={creditReason}
onChange={(e) => setCreditReason(e.target.value)}
/>
</div>
</div>
<DialogFooter>
<Button variant="outline" onClick={() => setCreditDialogOpen(false)}>
Cancel
</Button>
<Button
onClick={handleAddCredits}
disabled={!creditAmount || !creditReason || addingCredits}
>
{addingCredits && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
Add Credits
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
</div>
)
}
+11 -60
View File
@@ -3,11 +3,11 @@
import { useState, useEffect, useCallback } from "react"
import { useRouter } from "next/navigation"
import { api, Job, getAccessToken } from "@/lib/api"
import { useAutoMode, getAutoModeFromUrl } from "@/lib/auto-mode"
import { useAuth } from "@/lib/auth"
import { useJobNotifications, useVisibilityRefresh } from "@/hooks/use-notifications"
import { Button } from "@/components/ui/button"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import { Music2, RefreshCw, Loader2, Zap, ZapOff, Moon, Sun } from "lucide-react"
import { Music2, RefreshCw, Loader2, Moon, Sun } from "lucide-react"
import { WarmingUpLoader } from "@/components/WarmingUpLoader"
import { JobCard } from "@/components/job"
import { JobSubmission } from "@/components/job/JobSubmission"
@@ -28,8 +28,8 @@ export default function AppPage() {
const [isLoadingJobs, setIsLoadingJobs] = useState(true)
const [isInitialLoad, setIsInitialLoad] = useState(true)
const [isAuthenticated, setIsAuthenticated] = useState<boolean | null>(null)
const { enabled: autoModeEnabled, setEnabled: setAutoMode, toggle: toggleAutoMode } = useAutoMode()
const { isDarkMode, toggleTheme, mounted } = useTheme()
const { user, fetchUser } = useAuth()
// Memoize loadJobs for use with visibility refresh
const loadJobs = useCallback(async () => {
@@ -69,13 +69,15 @@ export default function AppPage() {
setIsAuthenticated(true)
}, [router])
// Initialize auto-mode from URL parameter on mount
// Ensure user data is loaded after authentication
// This handles the case where user navigates via client-side navigation
// (e.g., after beta enrollment) and the module-level fetchUser() didn't run
useEffect(() => {
const urlAutoMode = getAutoModeFromUrl()
if (urlAutoMode) {
setAutoMode(true)
const token = getAccessToken()
if (token && !user) {
fetchUser()
}
}, [setAutoMode])
}, [user, fetchUser])
// Load jobs on mount (only if authenticated)
useEffect(() => {
@@ -109,7 +111,7 @@ export default function AppPage() {
return (
<div className="min-h-screen" style={{ background: 'var(--bg)' }}>
{/* AutoProcessor - handles non-interactive mode */}
{/* AutoProcessor - handles non-interactive mode for jobs with that flag */}
<AutoProcessor jobs={jobs} onJobsChanged={loadJobs} />
{/* Header */}
@@ -122,36 +124,6 @@ export default function AppPage() {
</div>
</div>
<div className="flex items-center gap-1 sm:gap-2 shrink-0">
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="ghost"
size="sm"
onClick={toggleAutoMode}
className={`min-h-[40px] px-2 sm:px-3 ${autoModeEnabled ? "text-amber-400 hover:text-amber-300 bg-amber-500/10" : ""}`}
style={!autoModeEnabled ? { color: 'var(--text-muted)' } : undefined}
>
{autoModeEnabled ? (
<Zap className="w-4 h-4 sm:mr-2" />
) : (
<ZapOff className="w-4 h-4 sm:mr-2" />
)}
<span className="hidden sm:inline">Auto</span>
</Button>
</TooltipTrigger>
<TooltipContent side="bottom" className="max-w-xs">
<p className="font-medium">
{autoModeEnabled ? "Auto Mode Enabled" : "Auto Mode Disabled"}
</p>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
{autoModeEnabled
? "Jobs will auto-complete review and select clean instrumental (like -y flag)"
: "Click to enable non-interactive mode"}
</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
<Button
variant="ghost"
size="sm"
@@ -193,27 +165,6 @@ export default function AppPage() {
</header>
<main className="px-4 py-8 space-y-6">
{/* Auto Mode Banner */}
{autoModeEnabled && (
<div className="bg-amber-500/10 border border-amber-500/30 rounded-lg p-4 flex items-center gap-3">
<Zap className="w-5 h-5 text-amber-400 flex-shrink-0" />
<div className="flex-1">
<p className="text-sm text-amber-200 font-medium">Non-Interactive Mode Active</p>
<p className="text-xs text-amber-200/70 mt-0.5">
Jobs will automatically accept lyrics and select clean instrumental (equivalent to -y flag)
</p>
</div>
<Button
variant="ghost"
size="sm"
onClick={toggleAutoMode}
className="text-amber-200 hover:text-amber-100 hover:bg-amber-500/20"
>
Disable
</Button>
</div>
)}
<div className="grid lg:grid-cols-2 gap-8">
{/* Submit Job Card */}
<Card className="backdrop-blur" style={{ borderColor: 'var(--card-border)', backgroundColor: 'var(--card)' }}>
+24 -17
View File
@@ -2,7 +2,6 @@
import { useEffect, useCallback, useRef } from "react"
import { api, Job } from "@/lib/api"
import { useAutoMode } from "@/lib/auto-mode"
interface AutoProcessorProps {
/** List of jobs to monitor */
@@ -13,27 +12,27 @@ interface AutoProcessorProps {
/**
* AutoProcessor component that automatically handles interactive states
* when auto-mode is enabled.
* for jobs that have non_interactive mode enabled.
*
* This is equivalent to the -y flag in karaoke-gen-remote CLI:
* - Auto-completes lyrics review (accepts auto-corrected lyrics)
* - Auto-selects "clean" instrumental
* - Auto-selects instrumental based on backing vocals analyzer recommendation
* - Auto-selects first/best audio search result
*/
export function AutoProcessor({ jobs, onJobsChanged }: AutoProcessorProps) {
const { enabled, markProcessing, unmarkProcessing, isProcessing } = useAutoMode()
const processedRef = useRef<Set<string>>(new Set())
const processingRef = useRef<Set<string>>(new Set())
const processJob = useCallback(async (job: Job, action: string) => {
const processingKey = `${job.job_id}-${action}`
// Skip if already being processed
if (isProcessing(processingKey) || processedRef.current.has(processingKey)) {
// Skip if already being processed or already processed
if (processingRef.current.has(processingKey) || processedRef.current.has(processingKey)) {
return
}
// Mark as processing
markProcessing(processingKey)
processingRef.current.add(processingKey)
processedRef.current.add(processingKey)
console.log(`[AutoProcessor] Auto-processing ${action} for job ${job.job_id}`)
@@ -46,11 +45,16 @@ export function AutoProcessor({ jobs, onJobsChanged }: AutoProcessorProps) {
console.log(`[AutoProcessor] Auto-completed review for job ${job.job_id}`)
break
case 'instrumental':
// Auto-select "clean" instrumental
await api.selectInstrumental(job.job_id, 'clean')
console.log(`[AutoProcessor] Auto-selected clean instrumental for job ${job.job_id}`)
case 'instrumental': {
// Use backing vocals analyzer recommendation (default to clean if review needed or missing)
const analysis = job.state_data?.backing_vocals_analysis
const recommendation = analysis?.recommended_selection
// Only use 'with_backing' if explicitly recommended by analyzer, otherwise clean (safe default)
const selection = recommendation === 'with_backing' ? 'with_backing' : 'clean'
await api.selectInstrumental(job.job_id, selection)
console.log(`[AutoProcessor] Auto-selected ${selection} instrumental for job ${job.job_id} (analyzer recommendation: ${recommendation || 'none'})`)
break
}
case 'audio':
// Auto-select first audio search result (index 0)
@@ -66,15 +70,18 @@ export function AutoProcessor({ jobs, onJobsChanged }: AutoProcessorProps) {
// Remove from processed so it can be retried
processedRef.current.delete(processingKey)
} finally {
unmarkProcessing(processingKey)
processingRef.current.delete(processingKey)
}
}, [isProcessing, markProcessing, unmarkProcessing, onJobsChanged])
}, [onJobsChanged])
useEffect(() => {
if (!enabled) return
// Find jobs that need auto-processing
// Find jobs that need auto-processing (only jobs with non_interactive flag)
for (const job of jobs) {
// Only process jobs with non_interactive mode enabled
if (!job.non_interactive) {
continue
}
switch (job.status) {
case 'awaiting_review':
processJob(job, 'review')
@@ -87,7 +94,7 @@ export function AutoProcessor({ jobs, onJobsChanged }: AutoProcessorProps) {
break
}
}
}, [enabled, jobs, processJob])
}, [jobs, processJob])
// This component doesn't render anything - it's purely for side effects
return null
@@ -45,7 +45,7 @@ describe('AuthStatus', () => {
// Should show user email and credits
expect(screen.getByText('test@example.com')).toBeInTheDocument()
expect(screen.getByText('5')).toBeInTheDocument()
expect(screen.getByText('5 credits')).toBeInTheDocument()
})
it('opens auth dialog when Login is clicked', () => {
+18 -44
View File
@@ -7,14 +7,6 @@
import { render, screen } from '@testing-library/react'
import { JobCard } from '../job/JobCard'
import { Job } from '@/lib/api'
import { useAutoMode } from '@/lib/auto-mode'
// Mock the auto-mode hook
jest.mock('@/lib/auto-mode', () => ({
useAutoMode: jest.fn()
}))
const mockUseAutoMode = useAutoMode as jest.Mock
// Mock the child components
jest.mock('../job/JobActions', () => ({
@@ -48,11 +40,6 @@ describe('JobCard', () => {
beforeEach(() => {
jest.clearAllMocks()
// Default mock: auto-mode disabled
mockUseAutoMode.mockReturnValue({
enabled: false,
isProcessing: jest.fn().mockReturnValue(false)
})
})
it('renders job information', () => {
@@ -104,14 +91,7 @@ describe('JobCard', () => {
expect(card).toHaveClass('border-red-500/30')
})
describe('when auto-mode is disabled', () => {
beforeEach(() => {
mockUseAutoMode.mockReturnValue({
enabled: false,
isProcessing: jest.fn().mockReturnValue(false)
})
})
describe('regular jobs (non_interactive: false)', () => {
it('shows lyrics review link for awaiting_review status', () => {
const reviewJob = { ...mockJob, status: 'awaiting_review', review_token: 'test-token-123' }
render(<JobCard job={reviewJob} onRefresh={mockOnRefresh} />)
@@ -146,18 +126,24 @@ describe('JobCard', () => {
expect(href).toContain('baseApiUrl=')
expect(href).toContain('instrumentalToken=test-instrumental-token')
})
it('does not show Auto badge for regular jobs', () => {
render(<JobCard job={mockJob} onRefresh={mockOnRefresh} />)
expect(screen.queryByText('Auto')).not.toBeInTheDocument()
})
})
describe('when auto-mode is enabled', () => {
beforeEach(() => {
mockUseAutoMode.mockReturnValue({
enabled: true,
isProcessing: jest.fn().mockReturnValue(false)
})
describe('non-interactive jobs (non_interactive: true)', () => {
it('shows Auto badge for non-interactive jobs', () => {
const nonInteractiveJob = { ...mockJob, non_interactive: true }
render(<JobCard job={nonInteractiveJob} onRefresh={mockOnRefresh} />)
expect(screen.getByText('Auto')).toBeInTheDocument()
})
it('shows auto-accept message for awaiting_review status', () => {
const reviewJob = { ...mockJob, status: 'awaiting_review' }
const reviewJob = { ...mockJob, status: 'awaiting_review', non_interactive: true }
render(<JobCard job={reviewJob} onRefresh={mockOnRefresh} />)
expect(screen.getByText(/Will auto-accept/)).toBeInTheDocument()
@@ -165,32 +151,20 @@ describe('JobCard', () => {
})
it('shows auto-select message for awaiting_instrumental_selection status', () => {
const instrumentalJob = { ...mockJob, status: 'awaiting_instrumental_selection' }
const instrumentalJob = { ...mockJob, status: 'awaiting_instrumental_selection', non_interactive: true }
render(<JobCard job={instrumentalJob} onRefresh={mockOnRefresh} />)
expect(screen.getByText(/Will auto-select clean/)).toBeInTheDocument()
expect(screen.getByText(/Will auto-select/)).toBeInTheDocument()
expect(screen.queryByText('Select Instrumental')).not.toBeInTheDocument()
})
it('shows auto-select message for awaiting_audio_selection status', () => {
const audioJob = { ...mockJob, status: 'awaiting_audio_selection' }
it('shows auto-select first message for awaiting_audio_selection status', () => {
const audioJob = { ...mockJob, status: 'awaiting_audio_selection', non_interactive: true }
render(<JobCard job={audioJob} onRefresh={mockOnRefresh} />)
expect(screen.getByText(/Will auto-select first/)).toBeInTheDocument()
expect(screen.queryByRole('button', { name: 'Select Audio' })).not.toBeInTheDocument()
})
it('shows auto-processing indicator when processing', () => {
mockUseAutoMode.mockReturnValue({
enabled: true,
isProcessing: jest.fn().mockReturnValue(true)
})
const reviewJob = { ...mockJob, status: 'awaiting_review' }
render(<JobCard job={reviewJob} onRefresh={mockOnRefresh} />)
expect(screen.getByText(/Auto-processing.../)).toBeInTheDocument()
})
})
it('shows output links for complete jobs', () => {
+112
View File
@@ -0,0 +1,112 @@
"use client"
import Link from "next/link"
import { usePathname } from "next/navigation"
import {
LayoutDashboard,
Users,
Briefcase,
TestTube2,
ArrowLeft,
Settings,
Search,
} from "lucide-react"
import {
Sidebar,
SidebarContent,
SidebarFooter,
SidebarGroup,
SidebarGroupContent,
SidebarGroupLabel,
SidebarHeader,
SidebarMenu,
SidebarMenuButton,
SidebarMenuItem,
SidebarSeparator,
} from "@/components/ui/sidebar"
const navItems = [
{
title: "Dashboard",
href: "/admin",
icon: LayoutDashboard,
},
{
title: "Users",
href: "/admin/users",
icon: Users,
},
{
title: "Jobs",
href: "/admin/jobs",
icon: Briefcase,
},
{
title: "Audio Searches",
href: "/admin/searches",
icon: Search,
},
{
title: "Beta Program",
href: "/admin/beta",
icon: TestTube2,
},
]
export function AdminSidebar() {
const pathname = usePathname()
const isActive = (href: string) => {
if (href === "/admin") {
return pathname === "/admin"
}
return pathname.startsWith(href)
}
return (
<Sidebar>
<SidebarHeader className="border-b border-sidebar-border">
<div className="flex items-center gap-2 px-2 py-2">
<Settings className="w-6 h-6 text-primary" />
<span className="font-semibold text-lg">Admin Panel</span>
</div>
</SidebarHeader>
<SidebarContent>
<SidebarGroup>
<SidebarGroupLabel>Navigation</SidebarGroupLabel>
<SidebarGroupContent>
<SidebarMenu>
{navItems.map((item) => (
<SidebarMenuItem key={item.href}>
<SidebarMenuButton asChild isActive={isActive(item.href)}>
<Link href={item.href}>
<item.icon className="w-4 h-4" />
<span>{item.title}</span>
</Link>
</SidebarMenuButton>
</SidebarMenuItem>
))}
</SidebarMenu>
</SidebarGroupContent>
</SidebarGroup>
</SidebarContent>
<SidebarSeparator />
<SidebarFooter>
<SidebarMenu>
<SidebarMenuItem>
<SidebarMenuButton asChild>
<Link href="/app">
<ArrowLeft className="w-4 h-4" />
<span>Back to App</span>
</Link>
</SidebarMenuButton>
</SidebarMenuItem>
</SidebarMenu>
</SidebarFooter>
</Sidebar>
)
}
+2
View File
@@ -0,0 +1,2 @@
export { AdminSidebar } from "./admin-sidebar"
export { StatsCard, StatsGrid } from "./stats-card"
+87
View File
@@ -0,0 +1,87 @@
"use client"
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"
import { Skeleton } from "@/components/ui/skeleton"
import { cn } from "@/lib/utils"
import { LucideIcon } from "lucide-react"
interface StatsCardProps {
title: string
value: string | number
description?: string
icon?: LucideIcon
trend?: {
value: number
label: string
positive?: boolean
}
loading?: boolean
className?: string
}
export function StatsCard({
title,
value,
description,
icon: Icon,
trend,
loading,
className,
}: StatsCardProps) {
if (loading) {
return (
<Card className={cn("", className)}>
<CardHeader className="flex flex-row items-center justify-between space-y-0 pb-2">
<CardTitle className="text-sm font-medium">
<Skeleton className="h-4 w-24" />
</CardTitle>
</CardHeader>
<CardContent>
<Skeleton className="h-8 w-16" />
<Skeleton className="h-3 w-32 mt-2" />
</CardContent>
</Card>
)
}
return (
<Card className={cn("", className)}>
<CardHeader className="flex flex-row items-center justify-between space-y-0 pb-2">
<CardTitle className="text-sm font-medium text-muted-foreground">
{title}
</CardTitle>
{Icon && <Icon className="h-4 w-4 text-muted-foreground" />}
</CardHeader>
<CardContent>
<div className="text-2xl font-bold">{value}</div>
{description && (
<p className="text-xs text-muted-foreground mt-1">{description}</p>
)}
{trend && (
<p className={cn(
"text-xs mt-1",
trend.positive === true ? "text-green-500" : trend.positive === false ? "text-red-500" : "text-muted-foreground"
)}>
{trend.positive === true ? "+" : ""}{trend.value}% {trend.label}
</p>
)}
</CardContent>
</Card>
)
}
interface StatsGridProps {
children: React.ReactNode
className?: string
}
export function StatsGrid({ children, className }: StatsGridProps) {
return (
<div className={cn(
"grid gap-4 md:grid-cols-2 lg:grid-cols-4",
className
)}>
{children}
</div>
)
}
+24 -2
View File
@@ -12,7 +12,7 @@ import {
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "./ui/dropdown-menu"
import { User, LogOut, Settings } from "lucide-react"
import { User, LogOut, Settings, Shield, Mail } from "lucide-react"
import Link from "next/link"
export function AppHeader() {
@@ -40,7 +40,7 @@ export function AppHeader() {
<Link href="/jobs" className="text-sm font-medium hover:text-primary transition-colors">
My Jobs
</Link>
{user.role === "admin" && (
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
<Link href="/admin" className="text-sm font-medium hover:text-primary transition-colors">
Admin
</Link>
@@ -48,6 +48,14 @@ export function AppHeader() {
</nav>
<div className="flex items-center gap-4">
<a
href="mailto:andrew@nomadkaraoke.com"
className="hidden lg:flex items-center gap-1.5 text-xs text-muted-foreground hover:text-foreground transition-colors"
>
<Mail className="w-3.5 h-3.5" />
<span>Need help?</span>
</a>
<div className="hidden sm:flex items-center gap-2 px-4 py-2 rounded-lg bg-primary/10 border border-primary/20">
<div className="w-2 h-2 rounded-full bg-primary animate-pulse" />
<span className="text-sm font-medium text-primary">{user.credits} credits</span>
@@ -73,10 +81,24 @@ export function AppHeader() {
<span>{user.credits} credits</span>
</div>
</DropdownMenuItem>
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
<DropdownMenuItem asChild>
<Link href="/admin">
<Shield className="w-4 h-4 mr-2" />
Admin Dashboard
</Link>
</DropdownMenuItem>
)}
<DropdownMenuItem>
<Settings className="w-4 h-4 mr-2" />
Settings
</DropdownMenuItem>
<DropdownMenuItem asChild>
<a href="mailto:andrew@nomadkaraoke.com">
<Mail className="w-4 h-4 mr-2" />
Need help? Contact us
</a>
</DropdownMenuItem>
<DropdownMenuSeparator />
<DropdownMenuItem onClick={handleLogout} className="text-destructive focus:text-destructive">
<LogOut className="w-4 h-4 mr-2" />
+14 -2
View File
@@ -1,7 +1,8 @@
"use client"
import { useState, useEffect } from "react"
import { User, LogOut, CreditCard, Coins, KeyRound } from "lucide-react"
import { User, LogOut, CreditCard, Coins, KeyRound, Shield } from "lucide-react"
import Link from "next/link"
import { Button } from "@/components/ui/button"
import {
DropdownMenu,
@@ -69,7 +70,7 @@ export function AuthStatus({ onAuthChange }: AuthStatusProps) {
</span>
<span className="flex items-center gap-1 text-amber-400 font-medium">
<Coins className="w-3 h-3" />
{user.credits}
{user.credits} {user.credits === 1 ? 'credit' : 'credits'}
</span>
</Button>
</DropdownMenuTrigger>
@@ -97,6 +98,17 @@ export function AuthStatus({ onAuthChange }: AuthStatusProps) {
<CreditCard className="w-4 h-4 mr-2" />
<span>Buy More Credits</span>
</DropdownMenuItem>
{(user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")) && (
<>
<DropdownMenuSeparator className="bg-slate-700" />
<DropdownMenuItem asChild className="text-slate-300 focus:text-white focus:bg-slate-800">
<Link href="/admin">
<Shield className="w-4 h-4 mr-2" />
<span>Admin Dashboard</span>
</Link>
</DropdownMenuItem>
</>
)}
<DropdownMenuSeparator className="bg-slate-700" />
<DropdownMenuItem
onClick={handleLogout}
+19 -30
View File
@@ -2,9 +2,8 @@
import { useState } from "react"
import { Job } from "@/lib/api"
import { useAutoMode } from "@/lib/auto-mode"
import { Button } from "@/components/ui/button"
import { Loader2, ExternalLink, Zap } from "lucide-react"
import { ExternalLink, Zap } from "lucide-react"
import { JobActions } from "./JobActions"
import { JobLogs } from "./JobLogs"
import { OutputLinks } from "./OutputLinks"
@@ -77,7 +76,6 @@ interface JobCardProps {
export function JobCard({ job, onRefresh }: JobCardProps) {
const [showLogs, setShowLogs] = useState(false)
const [showAudioSearch, setShowAudioSearch] = useState(false)
const { enabled: autoModeEnabled, isProcessing: isAutoProcessing } = useAutoMode()
const createdAt = new Date(job.created_at).toLocaleString()
const isInteractive = job.status === "awaiting_review" ||
@@ -86,14 +84,6 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
job.status === "awaiting_audio_selection"
const isComplete = job.status === "complete"
const isFailed = job.status === "failed"
const isProcessing = !isComplete && !isFailed && job.status !== "cancelled"
// Check if this job is being auto-processed
const isBeingAutoProcessed = autoModeEnabled && isInteractive && (
isAutoProcessing(`${job.job_id}-review`) ||
isAutoProcessing(`${job.job_id}-instrumental`) ||
isAutoProcessing(`${job.job_id}-audio`)
)
// Build review URL for awaiting_review status
const getReviewUrl = () => {
@@ -125,22 +115,14 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
// Render primary action button based on status
const renderPrimaryAction = () => {
if (autoModeEnabled && isInteractive) {
// For non-interactive jobs in interactive states, show auto-processing message
if (job.non_interactive && isInteractive) {
return (
<div className="flex items-center gap-1 text-xs text-amber-400">
{isBeingAutoProcessed ? (
<>
<Loader2 className="w-3 h-3 animate-spin" />
Auto-processing...
</>
) : (
<>
<Zap className="w-3 h-3" />
Will auto-{job.status === "awaiting_review" || job.status === "in_review" ? "accept" :
job.status === "awaiting_instrumental_selection" ? "select clean" :
"select first"}
</>
)}
<Zap className="w-3 h-3" />
Will auto-{job.status === "awaiting_review" || job.status === "in_review" ? "accept" :
job.status === "awaiting_instrumental_selection" ? "select" :
"select first"}
</div>
)
}
@@ -198,10 +180,18 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
backgroundColor: 'var(--card)',
}}
>
{/* Header row with title */}
<p className="font-medium truncate" style={{ color: 'var(--text)' }}>
{job.artist || "Unknown"} - {job.title || "Unknown"}
</p>
{/* Header row with title and non-interactive badge */}
<div className="flex items-start justify-between gap-2">
<p className="font-medium truncate" style={{ color: 'var(--text)' }}>
{job.artist || "Unknown"} - {job.title || "Unknown"}
</p>
{job.non_interactive && (
<span className="shrink-0 inline-flex items-center gap-1 text-[10px] px-1.5 py-0.5 rounded bg-amber-500/20 text-amber-400 font-medium">
<Zap className="w-2.5 h-2.5" />
Auto
</span>
)}
</div>
{/* Meta row: ID, date, status */}
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
@@ -254,4 +244,3 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
</div>
)
}
+75
View File
@@ -39,6 +39,9 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
const [selectedTheme, setSelectedTheme] = useState<string | undefined>()
const [colorOverrides, setColorOverrides] = useState<ColorOverrides>({})
// Non-interactive mode (shared across all tabs)
const [nonInteractive, setNonInteractive] = useState(false)
async function handleUploadSubmit(e: React.FormEvent) {
e.preventDefault()
setError("")
@@ -57,6 +60,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
await api.uploadJob(uploadFile, uploadArtist.trim(), uploadTitle.trim(), {
theme_id: selectedTheme,
color_overrides: cleanColorOverrides(colorOverrides),
non_interactive: nonInteractive,
})
setUploadFile(null)
setUploadArtist("")
@@ -91,6 +95,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
{
theme_id: selectedTheme,
color_overrides: cleanColorOverrides(colorOverrides),
non_interactive: nonInteractive,
}
)
setYoutubeUrl("")
@@ -122,6 +127,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
await api.searchAudio(searchArtist.trim(), searchTitle.trim(), false, {
theme_id: selectedTheme,
color_overrides: cleanColorOverrides(colorOverrides),
non_interactive: nonInteractive,
})
setSearchArtist("")
setSearchTitle("")
@@ -250,6 +256,29 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
/>
</div>
{/* Non-interactive mode */}
<div className="space-y-2 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
<div className="flex items-start gap-3">
<input
type="checkbox"
id="non-interactive-upload"
checked={nonInteractive}
onChange={(e) => setNonInteractive(e.target.checked)}
disabled={isSubmitting}
className="mt-1 w-4 h-4 rounded border-gray-600 bg-gray-700 text-amber-500 focus:ring-amber-500 focus:ring-offset-gray-800"
/>
<div>
<Label htmlFor="non-interactive-upload" className="cursor-pointer" style={{ color: 'var(--text)' }}>
Skip lyrics review (non-interactive)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn&apos;t perfect.
Skipping review will likely result in incorrect words in your karaoke video.
</p>
</div>
</div>
</div>
{error && activeTab === "upload" && (
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
)}
@@ -332,6 +361,29 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
/>
</div>
{/* Non-interactive mode */}
<div className="space-y-2 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
<div className="flex items-start gap-3">
<input
type="checkbox"
id="non-interactive-url"
checked={nonInteractive}
onChange={(e) => setNonInteractive(e.target.checked)}
disabled={isSubmitting}
className="mt-1 w-4 h-4 rounded border-gray-600 bg-gray-700 text-amber-500 focus:ring-amber-500 focus:ring-offset-gray-800"
/>
<div>
<Label htmlFor="non-interactive-url" className="cursor-pointer" style={{ color: 'var(--text)' }}>
Skip lyrics review (non-interactive)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn&apos;t perfect.
Skipping review will likely result in incorrect words in your karaoke video.
</p>
</div>
</div>
</div>
{error && activeTab === "url" && (
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
)}
@@ -397,6 +449,29 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
/>
</div>
{/* Non-interactive mode */}
<div className="space-y-2 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
<div className="flex items-start gap-3">
<input
type="checkbox"
id="non-interactive-search"
checked={nonInteractive}
onChange={(e) => setNonInteractive(e.target.checked)}
disabled={isSubmitting}
className="mt-1 w-4 h-4 rounded border-gray-600 bg-gray-700 text-amber-500 focus:ring-amber-500 focus:ring-offset-gray-800"
/>
<div>
<Label htmlFor="non-interactive-search" className="cursor-pointer" style={{ color: 'var(--text)' }}>
Skip lyrics review (non-interactive)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn&apos;t perfect.
Skipping review will likely result in incorrect words in your karaoke video.
</p>
</div>
</div>
</div>
{error && activeTab === "search" && (
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
)}
+93 -26
View File
@@ -1,9 +1,9 @@
"use client"
import { useEffect, useState } from "react"
import { useEffect, useState, useRef, useCallback } from "react"
import { api } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Download, Loader2, ExternalLink, FolderOpen } from "lucide-react"
import { Download, Loader2, ExternalLink, FolderOpen, Copy } from "lucide-react"
interface OutputLinksProps {
jobId: string
@@ -14,17 +14,28 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
const [youtubeUrl, setYoutubeUrl] = useState<string | null>(null)
const [dropboxUrl, setDropboxUrl] = useState<string | null>(null)
const [isLoading, setIsLoading] = useState(false)
const [copiedUrl, setCopiedUrl] = useState<string | null>(null)
const copyTimeoutRef = useRef<NodeJS.Timeout | null>(null)
useEffect(() => {
loadOutputLinks()
}, [jobId])
// Cleanup timeout on unmount
useEffect(() => {
return () => {
if (copyTimeoutRef.current) {
clearTimeout(copyTimeoutRef.current)
}
}
}, [])
async function loadOutputLinks() {
setIsLoading(true)
try {
const result = await api.getDownloadUrls(jobId)
setDownloadUrls(result.download_urls)
// Extract YouTube and Dropbox URLs from state_data if available
const job = await api.getJob(jobId)
if (job.state_data?.youtube_url) {
@@ -40,6 +51,31 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
}
}
const copyToClipboard = useCallback(async (url: string) => {
// Check if clipboard API is available
if (!navigator?.clipboard?.writeText) {
console.error("Clipboard API not available")
return
}
try {
await navigator.clipboard.writeText(url)
setCopiedUrl(url)
// Clear any existing timeout
if (copyTimeoutRef.current) {
clearTimeout(copyTimeoutRef.current)
}
// Set new timeout
copyTimeoutRef.current = setTimeout(() => {
setCopiedUrl(null)
}, 2000)
} catch (err) {
console.error("Failed to copy to clipboard:", err)
}
}, [])
if (isLoading) {
return (
<div className="flex items-center gap-2 text-xs" style={{ color: 'var(--text-muted)' }}>
@@ -60,33 +96,55 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
<div className="flex flex-wrap gap-2">
{/* Distribution Links */}
{youtubeUrl && (
<a
href={youtubeUrl}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-red-600 hover:bg-red-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<ExternalLink className="w-3 h-3" />
View on YouTube
</a>
<div className="flex gap-1">
<a
href={youtubeUrl}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded-l bg-red-600 hover:bg-red-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<ExternalLink className="w-3 h-3" />
View on YouTube
</a>
<button
type="button"
onClick={(e) => { e.stopPropagation(); copyToClipboard(youtubeUrl) }}
className="inline-flex items-center text-xs px-2 py-1.5 rounded-r bg-red-700 hover:bg-red-600 text-white"
title="Copy YouTube URL"
aria-label={copiedUrl === youtubeUrl ? "YouTube URL copied" : "Copy YouTube URL"}
>
{copiedUrl === youtubeUrl ? <span className="text-[10px]">Copied!</span> : <Copy className="w-3 h-3" />}
</button>
</div>
)}
{dropboxUrl && (
<a
href={dropboxUrl}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-blue-600 hover:bg-blue-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<FolderOpen className="w-3 h-3" />
Dropbox Folder
</a>
<div className="flex gap-1">
<a
href={dropboxUrl}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded-l bg-blue-600 hover:bg-blue-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<FolderOpen className="w-3 h-3" />
Dropbox Folder
</a>
<button
type="button"
onClick={(e) => { e.stopPropagation(); copyToClipboard(dropboxUrl) }}
className="inline-flex items-center text-xs px-2 py-1.5 rounded-r bg-blue-700 hover:bg-blue-600 text-white"
title="Copy Dropbox URL"
aria-label={copiedUrl === dropboxUrl ? "Dropbox URL copied" : "Copy Dropbox URL"}
>
{copiedUrl === dropboxUrl ? <span className="text-[10px]">Copied!</span> : <Copy className="w-3 h-3" />}
</button>
</div>
)}
{/* Download Links */}
{downloadUrls?.finals?.lossy_720p && (
{downloadUrls?.finals?.lossy_720p_mp4 && (
<a
href={api.getDownloadUrl(jobId, "finals", "lossy_720p")}
href={api.getDownloadUrl(jobId, "finals", "lossy_720p_mp4")}
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-green-600 hover:bg-green-500 text-white"
onClick={(e) => e.stopPropagation()}
>
@@ -104,6 +162,16 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
4K Video
</a>
)}
{downloadUrls?.videos?.with_vocals && (
<a
href={api.getDownloadUrl(jobId, "videos", "with_vocals")}
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-purple-600 hover:bg-purple-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<Download className="w-3 h-3" />
With Vocals
</a>
)}
{downloadUrls?.packages?.cdg_zip && (
<a
href={api.getDownloadUrl(jobId, "packages", "cdg_zip")}
@@ -134,4 +202,3 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
</div>
)
}
+129 -28
View File
@@ -1,13 +1,13 @@
/**
* Email Testing Utilities using MailSlurp
* Email Testing Utilities using testmail.app
*
* MailSlurp provides disposable email inboxes for e2e testing.
* Free tier: 50 emails/month - perfect for CI/CD testing.
* testmail.app provides namespace-based email addresses for e2e testing.
* Emails arrive at: {namespace}.{tag}@inbox.testmail.app
*
* Setup:
* 1. Create account at https://app.mailslurp.com
* 2. Get API key from dashboard
* 3. Set MAILSLURP_API_KEY environment variable
* 1. Create account at https://testmail.app
* 2. Get API key and namespace from dashboard
* 3. Set TESTMAIL_API_KEY and TESTMAIL_NAMESPACE environment variables
*
* Usage in tests:
* const emailHelper = await createEmailHelper();
@@ -17,9 +17,27 @@
* const magicLink = emailHelper.extractMagicLink(email);
*/
import { MailSlurp, InboxDto, Email } from 'mailslurp-client';
const TESTMAIL_API_KEY = process.env.TESTMAIL_API_KEY;
const TESTMAIL_NAMESPACE = process.env.TESTMAIL_NAMESPACE;
const TESTMAIL_API_URL = 'https://api.testmail.app/api/json';
const MAILSLURP_API_KEY = process.env.MAILSLURP_API_KEY;
/**
* Inbox object compatible with existing test code
*/
export interface InboxDto {
id: string;
emailAddress: string;
}
/**
* Email object compatible with existing test code
*/
export interface Email {
subject?: string;
body?: string;
from?: string;
timestamp?: number;
}
export interface EmailHelper {
isAvailable: boolean;
@@ -30,40 +48,122 @@ export interface EmailHelper {
deleteInbox: (inboxId: string) => Promise<void>;
}
/**
* testmail.app API response structure
*/
interface TestmailResponse {
result: string;
message?: string;
count: number;
limit: number;
offset: number;
emails: TestmailEmail[];
}
interface TestmailEmail {
from: string;
subject: string;
html: string;
text: string;
timestamp: number;
tag: string;
}
/**
* Generate a unique tag for this test run
*/
function generateUniqueTag(): string {
const timestamp = Date.now();
const random = Math.random().toString(36).substring(2, 8);
return `test-${timestamp}-${random}`;
}
/**
* Creates an email testing helper.
* Returns a helper with isAvailable=false if MAILSLURP_API_KEY is not set.
* Returns a helper with isAvailable=false if testmail.app env vars are not set.
*/
export async function createEmailHelper(): Promise<EmailHelper> {
if (!MAILSLURP_API_KEY) {
console.warn('MAILSLURP_API_KEY not set - email testing disabled');
if (!TESTMAIL_API_KEY || !TESTMAIL_NAMESPACE) {
console.warn('TESTMAIL_API_KEY or TESTMAIL_NAMESPACE not set - email testing disabled');
return createDisabledHelper();
}
const mailslurp = new MailSlurp({ apiKey: MAILSLURP_API_KEY });
return {
isAvailable: true,
/**
* Create a new disposable inbox for testing
* Create a new "inbox" for testing.
* In testmail.app, this just generates a unique tag - no API call needed.
*/
async createInbox(): Promise<InboxDto> {
const inbox = await mailslurp.createInbox();
console.log(`Created test inbox: ${inbox.emailAddress}`);
return inbox;
const tag = generateUniqueTag();
const emailAddress = `${TESTMAIL_NAMESPACE}.${tag}@inbox.testmail.app`;
console.log(`Created test inbox: ${emailAddress}`);
return { id: tag, emailAddress };
},
/**
* Wait for an email to arrive in the inbox
* @param inboxId - The inbox ID to check
* @param inboxId - The tag to check (returned from createInbox as id)
* @param timeout - Timeout in ms (default: 60000 = 1 minute)
*/
async waitForEmail(inboxId: string, timeout = 60000): Promise<Email> {
console.log(`Waiting for email in inbox ${inboxId}...`);
const email = await mailslurp.waitForLatestEmail(inboxId, timeout);
console.log(`Received email: ${email.subject}`);
return email;
console.log(`Waiting for email with tag ${inboxId}...`);
// testmail.app livequery timeout is in seconds, max ~60s per request
// For longer timeouts, we poll multiple times
const pollInterval = Math.min(timeout, 30000); // 30s max per poll
const startTime = Date.now();
while (Date.now() - startTime < timeout) {
const remainingTime = timeout - (Date.now() - startTime);
const waitTime = Math.min(pollInterval, remainingTime);
const waitSeconds = Math.ceil(waitTime / 1000);
const url = new URL(TESTMAIL_API_URL);
url.searchParams.set('apikey', TESTMAIL_API_KEY!);
url.searchParams.set('namespace', TESTMAIL_NAMESPACE!);
url.searchParams.set('tag', inboxId);
url.searchParams.set('livequery', 'true');
url.searchParams.set('wait', waitSeconds.toString());
try {
const response = await fetch(url.toString());
if (!response.ok) {
throw new Error(`testmail.app API error: ${response.status} ${response.statusText}`);
}
const data: TestmailResponse = await response.json();
if (data.result !== 'success') {
throw new Error(`testmail.app API error: ${data.message || 'Unknown error'}`);
}
if (data.emails && data.emails.length > 0) {
// Return the most recent email
const latestEmail = data.emails.sort((a, b) => b.timestamp - a.timestamp)[0];
console.log(`Received email: ${latestEmail.subject}`);
// Convert to our Email format (use html as body, fallback to text)
return {
subject: latestEmail.subject,
body: latestEmail.html || latestEmail.text,
from: latestEmail.from,
timestamp: latestEmail.timestamp,
};
}
} catch (error) {
// If it's a timeout or network error, continue polling
if (Date.now() - startTime < timeout) {
console.log(`Polling for email... (${Math.round((Date.now() - startTime) / 1000)}s elapsed)`);
continue;
}
throw error;
}
}
throw new Error(`Timed out waiting for email after ${timeout}ms`);
},
/**
@@ -135,21 +235,22 @@ export async function createEmailHelper(): Promise<EmailHelper> {
},
/**
* Delete an inbox after testing
* Delete an inbox after testing.
* In testmail.app, emails auto-expire - this is a no-op.
*/
async deleteInbox(inboxId: string): Promise<void> {
await mailslurp.deleteInbox(inboxId);
console.log(`Deleted inbox ${inboxId}`);
// testmail.app emails auto-expire, no cleanup needed
console.log(`Inbox ${inboxId} cleanup skipped (testmail.app auto-expires)`);
},
};
}
/**
* Creates a disabled helper for when MailSlurp is not configured
* Creates a disabled helper for when testmail.app is not configured
*/
function createDisabledHelper(): EmailHelper {
const notAvailable = () => {
throw new Error('Email testing not available - set MAILSLURP_API_KEY');
throw new Error('Email testing not available - set TESTMAIL_API_KEY and TESTMAIL_NAMESPACE');
};
return {
@@ -166,5 +267,5 @@ function createDisabledHelper(): EmailHelper {
* Check if email testing is available
*/
export function isEmailTestingAvailable(): boolean {
return !!MAILSLURP_API_KEY;
return !!(TESTMAIL_API_KEY && TESTMAIL_NAMESPACE);
}
@@ -0,0 +1,219 @@
import { test, expect, Page } from "@playwright/test";
/**
* Admin Dashboard Production Tests
*
* Tests the admin dashboard functionality on live production.
* Requires KARAOKE_ADMIN_TOKEN environment variable with admin privileges.
* Admin tokens must belong to a user with @nomadkaraoke.com email domain.
*
* Run with:
* KARAOKE_ADMIN_TOKEN=xxx npx playwright test e2e/production/admin-dashboard.spec.ts --reporter=list
*/
const PROD_URL = "https://gen.nomadkaraoke.com";
const API_URL = "https://api.nomadkaraoke.com";
interface UserResponse {
user: {
email: string;
role: string;
};
}
async function validateAdminToken(
token: string
): Promise<{ isAdmin: boolean; email?: string; error?: string }> {
try {
const response = await fetch(`${API_URL}/api/users/me`, {
headers: { Authorization: `Bearer ${token}` },
});
if (!response.ok) {
return { isAdmin: false, error: `API returned ${response.status}` };
}
const data = (await response.json()) as UserResponse;
const isAdmin =
data.user?.role === "admin" ||
data.user?.email?.endsWith("@nomadkaraoke.com");
return {
isAdmin,
email: data.user?.email,
error: isAdmin ? undefined : `User ${data.user?.email} has role "${data.user?.role}", not admin`,
};
} catch (err) {
return { isAdmin: false, error: `Failed to validate token: ${err}` };
}
}
function getAdminToken(): string | null {
// Prefer KARAOKE_ADMIN_TOKEN, fall back to KARAOKE_ACCESS_TOKEN
return process.env.KARAOKE_ADMIN_TOKEN || process.env.KARAOKE_ACCESS_TOKEN || null;
}
async function authenticatePage(page: Page, token: string): Promise<void> {
await page.addInitScript((t) => {
localStorage.setItem("karaoke_access_token", t);
}, token);
}
// Check if we have a valid admin token before running tests
let adminToken: string | null = null;
let tokenValidation: { isAdmin: boolean; email?: string; error?: string } | null = null;
test.beforeAll(async () => {
adminToken = getAdminToken();
if (adminToken) {
tokenValidation = await validateAdminToken(adminToken);
if (!tokenValidation.isAdmin) {
console.log(`⚠️ Token validation failed: ${tokenValidation.error}`);
console.log(" Admin tests require a token for a @nomadkaraoke.com user");
console.log(" Set KARAOKE_ADMIN_TOKEN environment variable with an admin token");
} else {
console.log(`✓ Admin token validated for ${tokenValidation.email}`);
}
} else {
console.log("⚠️ No admin token found (KARAOKE_ADMIN_TOKEN or KARAOKE_ACCESS_TOKEN)");
}
});
test.describe("Admin Dashboard - Production", () => {
test.beforeEach(async ({ page }) => {
test.skip(!adminToken || !tokenValidation?.isAdmin, "Skipping: No valid admin token available");
await authenticatePage(page, adminToken!);
});
test("admin dashboard loads and shows stats", async ({ page }) => {
await page.goto(`${PROD_URL}/admin`);
// Wait for page to load
await expect(page.locator("h1")).toContainText("Admin Dashboard", {
timeout: 30000,
});
// Check stats cards are present
await expect(page.getByText("Total Users")).toBeVisible();
await expect(page.getByText("Total Jobs")).toBeVisible();
await expect(page.getByText("Active Users")).toBeVisible();
// Check job status breakdown card is present
await expect(page.getByText("Jobs by Status")).toBeVisible();
});
test("admin sidebar navigation works", async ({ page }) => {
await page.goto(`${PROD_URL}/admin`);
// Wait for dashboard to load
await expect(page.locator("h1")).toContainText("Admin Dashboard", {
timeout: 30000,
});
// Click Users in sidebar
await page.getByRole("link", { name: /Users/i }).first().click();
await expect(page).toHaveURL(/\/admin\/users/);
await expect(page.locator("h1")).toContainText("Users");
// Click Jobs in sidebar
await page.getByRole("link", { name: /Jobs/i }).first().click();
await expect(page).toHaveURL(/\/admin\/jobs/);
await expect(page.locator("h1")).toContainText("Jobs");
// Click Beta Program in sidebar
await page.getByRole("link", { name: /Beta/i }).first().click();
await expect(page).toHaveURL(/\/admin\/beta/);
await expect(page.locator("h1")).toContainText("Beta Program");
// Navigate back to dashboard
await page.getByRole("link", { name: /Dashboard/i }).first().click();
await expect(page).toHaveURL(/\/admin$/);
});
test("users page loads with user list", async ({ page }) => {
await page.goto(`${PROD_URL}/admin/users`);
// Wait for page to load
await expect(page.locator("h1")).toContainText("Users", { timeout: 30000 });
// Check table headers are present
await expect(page.getByRole("columnheader", { name: "Email" })).toBeVisible();
await expect(page.getByRole("columnheader", { name: "Role" })).toBeVisible();
await expect(page.getByRole("columnheader", { name: "Credits" })).toBeVisible();
// Check search input is present
await expect(page.getByPlaceholder(/Search/i)).toBeVisible();
// Check pagination controls
await expect(page.getByText(/Showing/)).toBeVisible();
});
test("jobs page loads with job list", async ({ page }) => {
await page.goto(`${PROD_URL}/admin/jobs`);
// Wait for page to load
await expect(page.locator("h1")).toContainText("Jobs", { timeout: 30000 });
// Check table headers are present
await expect(page.getByRole("columnheader", { name: "Job ID" })).toBeVisible();
await expect(page.getByRole("columnheader", { name: "Status" })).toBeVisible();
// Check filter controls
await expect(page.getByPlaceholder(/Filter by user/i)).toBeVisible();
});
test("beta page loads with stats", async ({ page }) => {
await page.goto(`${PROD_URL}/admin/beta`);
// Wait for page to load
await expect(page.locator("h1")).toContainText("Beta Program", {
timeout: 30000,
});
// Check stats cards
await expect(page.getByText("Total Beta Testers")).toBeVisible();
await expect(page.getByText("Feedback Submissions")).toBeVisible();
});
});
test.describe("Admin API - Production", () => {
test.beforeEach(async () => {
test.skip(!adminToken || !tokenValidation?.isAdmin, "Skipping: No valid admin token available");
});
test("API stats endpoint returns data", async ({ request }) => {
const response = await request.get(`${API_URL}/api/admin/stats/overview`, {
headers: {
Authorization: `Bearer ${adminToken}`,
},
});
expect(response.status()).toBe(200);
const data = await response.json();
expect(data).toHaveProperty("total_users");
expect(data).toHaveProperty("total_jobs");
expect(data).toHaveProperty("jobs_by_status");
expect(typeof data.total_users).toBe("number");
expect(typeof data.total_jobs).toBe("number");
console.log(`Stats: ${data.total_users} users, ${data.total_jobs} jobs`);
});
test("API users list endpoint returns data", async ({ request }) => {
const response = await request.get(`${API_URL}/api/users/admin/users?limit=5`, {
headers: {
Authorization: `Bearer ${adminToken}`,
},
});
expect(response.status()).toBe(200);
const data = await response.json();
expect(data).toHaveProperty("users");
expect(data).toHaveProperty("total");
expect(Array.isArray(data.users)).toBe(true);
console.log(`Users: ${data.total} total, fetched ${data.users.length}`);
});
});
@@ -25,7 +25,8 @@ import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-tes
* 10. Verify Outputs - Check download URLs
*
* Environment Variables:
* - MAILSLURP_API_KEY: For email testing (required for full flow)
* - TESTMAIL_API_KEY: For email testing (required for full flow)
* - TESTMAIL_NAMESPACE: Your testmail.app namespace
* - KARAOKE_ACCESS_TOKEN: Skip enrollment and use existing token
*/
@@ -47,8 +48,9 @@ const TIMEOUTS = {
action: 30_000, // 30s for UI actions
expect: 60_000, // 60s for assertions
apiCall: 120_000, // 2min for API calls
download: 900_000, // 15min for remote flacfetch downloads (torrents can be slow)
jobProcessing: 600_000, // 10min for job processing
fullTest: 900_000, // 15min for full test
fullTest: 2400_000, // 40min for full test (download + processing + video generation)
} as const;
// =============================================================================
@@ -188,11 +190,11 @@ test.describe('Production E2E - Full User Journey', () => {
});
console.log(' Authenticated successfully with existing token');
} else if (isEmailTestingAvailable()) {
console.log(' Starting beta enrollment with MailSlurp');
console.log(' Starting beta enrollment with testmail.app');
const emailHelper = await createEmailHelper();
if (!emailHelper.isAvailable) {
throw new Error('MailSlurp not available');
throw new Error('testmail.app not available');
}
// Create test inbox
@@ -248,7 +250,7 @@ test.describe('Production E2E - Full User Journey', () => {
}
} else {
throw new Error(
'No KARAOKE_ACCESS_TOKEN and no MAILSLURP_API_KEY - cannot authenticate'
'No KARAOKE_ACCESS_TOKEN and no TESTMAIL_API_KEY/TESTMAIL_NAMESPACE - cannot authenticate'
);
}
@@ -388,73 +390,48 @@ test.describe('Production E2E - Full User Journey', () => {
console.log('STEP 5 COMPLETE: Audio selection handled');
// =========================================================================
// STEP 6: Wait for Processing (Lyrics)
// STEP 6: Wait for Processing (Download + Audio Separation + Lyrics)
// =========================================================================
console.log('\n========================================');
console.log('STEP 6: Wait for Processing');
console.log('========================================');
console.log(' Polling job status...');
console.log(' Polling job status (this may take 15-20 minutes for remote downloads)...');
const { status: statusAfterProcessing, job: jobAfterProcessing } = await pollJobStatus(
request,
jobId,
['in_review', 'awaiting_instrumental_selection', 'completed'],
accessToken!
['awaiting_review', 'in_review', 'awaiting_instrumental_selection', 'complete'],
accessToken!,
TIMEOUTS.download // Use longer timeout for downloads
);
console.log(`STEP 6 COMPLETE: Job reached ${statusAfterProcessing}`);
// =========================================================================
// STEP 7: Lyrics Review
// STEP 7: Complete Lyrics Review (via API)
// =========================================================================
console.log('\n========================================');
console.log('STEP 7: Lyrics Review');
console.log('========================================');
if (statusAfterProcessing === 'in_review') {
await page.reload();
await page.waitForLoadState('networkidle');
if (statusAfterProcessing === 'awaiting_review' || statusAfterProcessing === 'in_review') {
console.log(' Completing lyrics review via API (accepting AI-generated lyrics)...');
const reviewBtn = page.getByRole('button', { name: /review.*lyrics/i }).first();
if (await reviewBtn.isVisible({ timeout: 10000 }).catch(() => false)) {
console.log(' Opening lyrics review...');
// This opens a new tab
const [popup] = await Promise.all([
page.context().waitForEvent('page', { timeout: 15000 }).catch(() => null),
reviewBtn.click(),
]);
if (popup) {
await popup.waitForLoadState('networkidle');
await popup.waitForTimeout(5000);
await popup.screenshot({
path: 'test-results/07a-review-ui.png',
fullPage: true,
});
// Click Preview Video
const previewBtn = popup.getByRole('button', { name: /preview.*video/i }).first();
if (await previewBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
console.log(' Clicking Preview Video...');
await previewBtn.click();
await popup.waitForTimeout(30000); // Preview generation
// Click Complete Review
const completeBtn = popup.getByRole('button', { name: /complete.*review/i }).first();
if (await completeBtn.isVisible({ timeout: 60000 }).catch(() => false)) {
console.log(' Clicking Complete Review...');
await completeBtn.click();
await popup.waitForTimeout(3000);
console.log(' Review completed');
}
}
if (!popup.isClosed()) {
await popup.close();
}
// Use the API to complete the review - this accepts the current lyrics as-is
const completeReviewResponse = await request.post(
`${API_URL}/api/jobs/${jobId}/complete-review`,
{
headers: { Authorization: `Bearer ${accessToken}` },
timeout: TIMEOUTS.apiCall,
}
);
if (completeReviewResponse.ok()) {
const result = await completeReviewResponse.json();
console.log(` Review completed: ${result.message}`);
} else {
const errorText = await completeReviewResponse.text();
throw new Error(`Failed to complete review: ${completeReviewResponse.status()} - ${errorText}`);
}
} else {
console.log(` Skipping - job status is ${statusAfterProcessing}`);
@@ -463,43 +440,44 @@ test.describe('Production E2E - Full User Journey', () => {
console.log('STEP 7 COMPLETE: Lyrics review handled');
// =========================================================================
// STEP 8: Instrumental Selection
// STEP 8: Wait for Video Rendering and Instrumental Selection (via API)
// =========================================================================
console.log('\n========================================');
console.log('STEP 8: Instrumental Selection');
console.log('========================================');
// Wait for job to reach instrumental selection
// Wait for job to reach instrumental selection (after video rendering)
console.log(' Waiting for video rendering to complete...');
const { status: statusBeforeInstrumental } = await pollJobStatus(
request,
jobId,
['awaiting_instrumental_selection', 'completed'],
['awaiting_instrumental_selection', 'complete'],
accessToken!,
120000 // 2 minutes
).catch(() => ({ status: 'unknown', job: null }));
TIMEOUTS.download // Video rendering can take 10-15 minutes with orchestrator
);
if (statusBeforeInstrumental === 'awaiting_instrumental_selection') {
await page.reload();
await page.waitForLoadState('networkidle');
console.log(' Selecting clean instrumental via API...');
const instrumentalBtn = page.getByRole('button', { name: /select.*instrumental/i }).first();
if (await instrumentalBtn.isVisible({ timeout: 10000 }).catch(() => false)) {
console.log(' Opening instrumental selection...');
await instrumentalBtn.click();
await page.waitForTimeout(2000);
const dialog = page.locator('[role="dialog"]');
if (await dialog.isVisible()) {
await page.screenshot({ path: 'test-results/08-instrumental-dialog.png' });
// Select first instrumental (usually "Clean")
const selectBtn = dialog.getByRole('button', { name: /select/i }).first();
if (await selectBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
await selectBtn.click();
console.log(' Instrumental selected');
await page.waitForTimeout(3000);
}
// Use the API to select the clean instrumental
const selectInstrumentalResponse = await request.post(
`${API_URL}/api/jobs/${jobId}/select-instrumental`,
{
headers: {
Authorization: `Bearer ${accessToken}`,
'Content-Type': 'application/json',
},
data: { selection: 'clean' },
timeout: TIMEOUTS.apiCall,
}
);
if (selectInstrumentalResponse.ok()) {
const result = await selectInstrumentalResponse.json();
console.log(` Instrumental selected: ${result.message}`);
} else {
const errorText = await selectInstrumentalResponse.text();
throw new Error(`Failed to select instrumental: ${selectInstrumentalResponse.status()} - ${errorText}`);
}
} else {
console.log(` Skipping - job status is ${statusBeforeInstrumental}`);
@@ -508,17 +486,17 @@ test.describe('Production E2E - Full User Journey', () => {
console.log('STEP 8 COMPLETE: Instrumental selection handled');
// =========================================================================
// STEP 9: Wait for Completion
// STEP 9: Wait for Final Video Generation and Completion
// =========================================================================
console.log('\n========================================');
console.log('STEP 9: Wait for Completion');
console.log('========================================');
console.log(' Waiting for job to complete...');
console.log(' Waiting for final video generation (encoding 4 formats)...');
const { status: finalStatus, job: finalJob } = await pollJobStatus(
request,
jobId,
['completed'],
['complete'], // Note: status is 'complete' not 'completed'
accessToken!,
TIMEOUTS.jobProcessing
);
@@ -526,29 +504,57 @@ test.describe('Production E2E - Full User Journey', () => {
console.log('STEP 9 COMPLETE: Job completed');
// =========================================================================
// STEP 10: Verify Outputs
// STEP 10: Verify Outputs and Download URLs
// =========================================================================
console.log('\n========================================');
console.log('STEP 10: Verify Outputs');
console.log('========================================');
console.log(` Job ID: ${finalJob.job_id}`);
console.log(` Artist: ${finalJob.artist}`);
console.log(` Title: ${finalJob.title}`);
console.log(` Theme ID: ${finalJob.theme_id || 'N/A'}`);
console.log(` YouTube URL: ${finalJob.youtube_url || 'N/A'}`);
console.log(` Dropbox URL: ${finalJob.dropbox_url || 'N/A'}`);
console.log(` Google Drive URL: ${finalJob.gdrive_url || 'N/A'}`);
// Verify required outputs
expect(finalJob.theme_id).toBeTruthy();
// Verify required fields
expect(finalJob.status).toBe('complete');
expect(finalJob.artist).toBeTruthy();
expect(finalJob.title).toBeTruthy();
// Check for download URLs
// Check for download URLs - this is the key verification
const downloadResponse = await request.get(`${API_URL}/api/jobs/${jobId}/download-urls`, {
headers: { Authorization: `Bearer ${accessToken}` },
timeout: TIMEOUTS.apiCall,
}).catch(() => null);
});
if (downloadResponse?.ok()) {
if (downloadResponse.ok()) {
const downloadUrls = await downloadResponse.json();
console.log(` Download URLs available: ${Object.keys(downloadUrls).length}`);
const urlKeys = Object.keys(downloadUrls);
console.log(` Download formats available: ${urlKeys.length}`);
// Should have multiple video formats
for (const key of urlKeys) {
console.log(` - ${key}: ${downloadUrls[key] ? 'Available' : 'Missing'}`);
}
// Verify we have at least one download URL
expect(urlKeys.length).toBeGreaterThan(0);
// Verify at least one URL is actually present
const hasUrl = urlKeys.some(key => downloadUrls[key]);
expect(hasUrl).toBe(true);
} else {
console.log(` WARNING: Could not fetch download URLs (status ${downloadResponse.status()})`);
}
// Get final job state for reference
const finalJobResponse = await request.get(`${API_URL}/api/jobs/${jobId}`, {
headers: { Authorization: `Bearer ${accessToken}` },
timeout: TIMEOUTS.apiCall,
});
if (finalJobResponse.ok()) {
const fullJob = await finalJobResponse.json();
console.log(` File URLs available: ${Object.keys(fullJob.file_urls || {}).length} categories`);
}
await page.reload();
@@ -0,0 +1,952 @@
import { test, expect, Page, BrowserContext } from '@playwright/test';
import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-testing';
/**
* E2E Happy Path Test - Real User Journey with Full UI Interactions
*
* This test validates the COMPLETE karaoke generation flow using ONLY
* browser interactions - no API shortcuts allowed.
*
* See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements.
*
* Test Flow:
* 1. Landing Page - Navigate and verify
* 2. New User Signup - Create testmail.app inbox, submit beta form, verify email
* 3. Magic Link Auth - Click magic link to authenticate
* 4. Create Job - Search for test song via UI
* 5. Audio Selection - Select audio source via UI dialog
* 6. Wait for Processing - Monitor via UI status updates
* 7. Lyrics Review - Open review UI, interact, submit via UI
* 8. Instrumental Selection - Open selection UI, interact, submit via UI
* 9. Wait for Completion - Monitor via UI
* 10. Verify Downloads - Check download links work
* 11. Verify Distribution - Check YouTube/Dropbox/GDrive indicators
* 12. Cleanup - Delete distributed content and job
*
* Environment Variables:
* - TESTMAIL_API_KEY + TESTMAIL_NAMESPACE: Required for email testing (can be skipped with E2E_TEST_TOKEN)
* - E2E_TEST_TOKEN: Pre-configured user token to skip signup flow (for iterations)
* - E2E_ADMIN_TOKEN: Required for cleanup endpoint (optional - cleanup skipped if not set)
*/
/**
* Check if we should use a pre-configured test token instead of creating a new user
*/
function hasPreConfiguredToken(): boolean {
return !!process.env.E2E_TEST_TOKEN;
}
function getPreConfiguredToken(): string {
return process.env.E2E_TEST_TOKEN || '';
}
// =============================================================================
// CONSTANTS
// =============================================================================
const PROD_URL = 'https://gen.nomadkaraoke.com';
const API_URL = 'https://api.nomadkaraoke.com';
// Test song - uses cached flacfetch results for speed
const TEST_SONG = {
artist: 'piri',
title: 'dog',
} as const;
// Timeouts
const TIMEOUTS = {
action: 30_000, // 30s for UI actions
expect: 60_000, // 60s for assertions
emailArrival: 120_000, // 2min for email to arrive
audioSearch: 120_000, // 2min for audio search
lyricsProcessing: 1500_000, // 25min for lyrics transcription (matches Cloud Tasks deadline)
videoRendering: 900_000, // 15min for video rendering
finalEncoding: 600_000, // 10min for final encoding
fullTest: 3600_000, // 60min for full test
} as const;
// =============================================================================
// HELPERS
// =============================================================================
/**
* Wait for job status to appear in the UI
*/
async function waitForJobStatus(
page: Page,
jobCard: any,
targetStatuses: string[],
timeoutMs: number
): Promise<string> {
const startTime = Date.now();
while (Date.now() - startTime < timeoutMs) {
// Check the status indicator text in the job card
const statusText = await jobCard.locator('[class*="text-"]').first().textContent() || '';
for (const target of targetStatuses) {
if (statusText.toLowerCase().includes(target.toLowerCase())) {
console.log(` Found status: ${statusText}`);
return target;
}
}
// Also check for "Action needed" indicator
const actionNeeded = await jobCard.getByText('Action needed').isVisible().catch(() => false);
if (actionNeeded) {
console.log(` Action needed - checking for target status`);
for (const target of targetStatuses) {
if (statusText.toLowerCase().includes(target.replace('awaiting_', '').replace('_', ' '))) {
return target;
}
}
}
// Wait before next check
await page.waitForTimeout(5000);
// Refresh to get latest status
const refreshBtn = page.getByRole('button', { name: /refresh/i });
if (await refreshBtn.isVisible().catch(() => false)) {
await refreshBtn.click();
await page.waitForTimeout(2000);
}
}
throw new Error(`Timeout waiting for status: ${targetStatuses.join(' or ')}`);
}
/**
* Get the auth token from localStorage
*/
async function getAuthToken(page: Page): Promise<string | null> {
return await page.evaluate(() => localStorage.getItem('karaoke_access_token'));
}
// =============================================================================
// TEST
// =============================================================================
test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
// IMPORTANT: Disable retries for this test - each retry creates a new karaoke job
// which wastes 15-20 minutes of processing time
test.describe.configure({ retries: 0 });
test('Complete flow: New user signup -> Karaoke generation -> Distribution -> Cleanup', async ({
page,
context,
}) => {
test.setTimeout(TIMEOUTS.fullTest);
// Check if we have a pre-configured token (for iterations without using testmail.app)
const usePreConfiguredToken = hasPreConfiguredToken();
// Skip if neither pre-configured token nor testmail.app is available
if (!usePreConfiguredToken && !isEmailTestingAvailable()) {
test.skip(true, 'Neither E2E_TEST_TOKEN nor TESTMAIL_API_KEY/TESTMAIL_NAMESPACE set - cannot run test');
return;
}
let accessToken: string | null = null;
let jobId: string | null = null;
let inboxId: string | null = null;
// Only create email helper if we're doing the full signup flow
const emailHelper = usePreConfiguredToken ? null : await createEmailHelper();
// Intercept API requests to disable YouTube upload for E2E tests
// This prevents test jobs from uploading to YouTube when server default is true
await page.route('**/api/audio-search/search', async (route) => {
const request = route.request();
if (request.method() === 'POST') {
const postData = request.postDataJSON();
// Explicitly disable YouTube upload for E2E test jobs
postData.enable_youtube_upload = false;
console.log(' [Request Intercept] Disabled YouTube upload for audio search');
await route.continue({
postData: JSON.stringify(postData),
});
} else {
await route.continue();
}
});
try {
// =========================================================================
// STEP 1: Landing Page
// =========================================================================
console.log('\n========================================');
console.log('STEP 1: Landing Page');
console.log('========================================');
await page.goto(PROD_URL);
await page.waitForLoadState('networkidle');
// Verify hero section
await expect(page.locator('h1')).toContainText('Karaoke Video', { timeout: TIMEOUTS.expect });
console.log(' Hero section visible');
// Verify pricing section
await expect(page.locator('#pricing')).toBeVisible({ timeout: TIMEOUTS.expect });
console.log(' Pricing section visible');
await page.screenshot({ path: 'test-results/01-landing-page.png', fullPage: true });
console.log('STEP 1 COMPLETE: Landing page loads correctly');
// =========================================================================
// STEP 2 & 3: Authentication (either via pre-configured token or beta signup)
// =========================================================================
if (usePreConfiguredToken) {
// ----- FAST PATH: Use pre-configured token -----
console.log('\n========================================');
console.log('STEP 2: Using Pre-Configured Token (skipping signup)');
console.log('========================================');
accessToken = getPreConfiguredToken();
console.log(` Using token: ${accessToken.substring(0, 8)}...`);
// Navigate to app and inject token into localStorage
await page.goto(`${PROD_URL}/app`);
await page.evaluate((token) => {
localStorage.setItem('karaoke_access_token', token);
}, accessToken);
// Reload to apply the token
await page.reload();
await page.waitForLoadState('networkidle');
await page.screenshot({ path: 'test-results/02-token-injected.png' });
console.log('STEP 2 COMPLETE: Token injected, skipping beta enrollment');
console.log('\n========================================');
console.log('STEP 3: Verify Authentication');
console.log('========================================');
// Verify we're authenticated by checking for the app UI
await expect(page.locator('body')).not.toContainText('Sign in', { timeout: TIMEOUTS.action });
console.log(' User authenticated');
// Try to find credit indicator
const creditText = await page.getByText(/credit/i).first().textContent().catch(() => 'N/A');
console.log(` Credits visible: ${creditText}`);
console.log('STEP 3 COMPLETE: User authenticated with pre-configured token');
} else {
// ----- FULL PATH: Beta signup with testmail.app -----
console.log('\n========================================');
console.log('STEP 2: New User Beta Enrollment');
console.log('========================================');
// Create test inbox
const inbox = await emailHelper!.createInbox();
inboxId = inbox.id!;
console.log(` Test email: ${inbox.emailAddress}`);
// Scroll to beta section first to ensure it's in view
const betaSection = page.locator('text=Beta Tester Program').first();
await betaSection.scrollIntoViewIfNeeded();
await page.waitForTimeout(500);
// Open beta form - click the Join Beta Program button
const betaButton = page.getByRole('button', { name: /join beta program/i });
await expect(betaButton).toBeVisible({ timeout: TIMEOUTS.action });
// Take screenshot before clicking
await page.screenshot({ path: 'test-results/02-before-beta-click.png' });
console.log(' Clicking Join Beta Program button...');
// Click and wait for the form to appear
await betaButton.click();
// Wait for beta form to appear - the button should be replaced by the form
const betaEmailInput = page.locator('#beta-email');
// Poll for form visibility with retries
let formVisible = false;
for (let attempt = 0; attempt < 5; attempt++) {
await page.waitForTimeout(1000);
formVisible = await betaEmailInput.isVisible().catch(() => false);
if (formVisible) break;
console.log(` Form not visible yet, attempt ${attempt + 1}/5`);
// Try clicking again if form didn't appear
const buttonStillVisible = await betaButton.isVisible().catch(() => false);
if (buttonStillVisible) {
console.log(' Button still visible, clicking again...');
await betaButton.click();
}
}
await page.screenshot({ path: 'test-results/02-after-beta-click.png' });
if (!formVisible) {
// Debug: log the page state
const pageContent = await page.content();
console.log(' Page HTML length:', pageContent.length);
console.log(' Looking for beta-email in HTML:', pageContent.includes('beta-email'));
throw new Error('Beta form did not appear after clicking Join Beta Program button');
}
await expect(betaEmailInput).toBeVisible({ timeout: TIMEOUTS.action });
console.log(' Beta form visible');
// Fill beta form
await betaEmailInput.fill(inbox.emailAddress!);
// Fill the promise/reason field
const promiseField = page.locator('#beta-promise, textarea');
await promiseField.first().fill('piri - dog (E2E test: testing complete karaoke generation flow)');
// Check the acceptance checkbox
const acceptCheckbox = page.locator('#beta-accept, input[type="checkbox"]');
await acceptCheckbox.first().check();
await page.screenshot({ path: 'test-results/02a-beta-form-filled.png' });
console.log(' Beta form filled');
// Submit form - look for the submit button
const submitButton = page.getByRole('button', { name: /get.*free.*credit|submit|enroll/i });
await expect(submitButton).toBeEnabled({ timeout: TIMEOUTS.action });
await submitButton.click();
// Wait for success and redirect
await expect(
page.getByText(/welcome to the beta|redirecting to the app/i)
).toBeVisible({ timeout: 15000 });
await page.waitForURL(/\/app/, { timeout: 10000 });
console.log(' Redirected to /app');
// Get the token from localStorage
accessToken = await getAuthToken(page);
if (!accessToken) {
throw new Error('No access token received after beta enrollment');
}
console.log(' Session token received');
await page.screenshot({ path: 'test-results/02b-app-after-enrollment.png' });
console.log('STEP 2 COMPLETE: Beta enrollment successful');
// =========================================================================
// STEP 3: Verify Credits (via UI)
// =========================================================================
console.log('\n========================================');
console.log('STEP 3: Verify Credits');
console.log('========================================');
// Look for credit indicator in the UI
const creditText = await page.getByText(/credit/i).first().textContent();
console.log(` Credits visible: ${creditText}`);
console.log('STEP 3 COMPLETE: User authenticated with credits');
}
// =========================================================================
// STEP 4: Create Karaoke Job (via UI)
// =========================================================================
console.log('\n========================================');
console.log('STEP 4: Create Karaoke Job');
console.log('========================================');
// Navigate to Search tab
await page.getByRole('tab', { name: /search/i }).click();
await page.waitForTimeout(1000);
// Select a theme if required
const themeSelect = page.locator('button[role="combobox"]').first();
if (await themeSelect.isVisible({ timeout: 3000 }).catch(() => false)) {
const currentTheme = await themeSelect.textContent();
if (currentTheme?.toLowerCase().includes('select')) {
await themeSelect.click();
await page.waitForTimeout(500);
await page.locator('[role="option"]').first().click();
await page.waitForTimeout(500);
console.log(' Selected first theme');
}
}
// Fill artist and title
await page.getByLabel('Artist').fill(TEST_SONG.artist);
await page.getByLabel('Title').fill(TEST_SONG.title);
console.log(` Searching for: ${TEST_SONG.artist} - ${TEST_SONG.title}`);
await page.screenshot({ path: 'test-results/04a-search-form.png' });
// Submit search
await page.getByRole('button', { name: /search.*create/i }).click();
// Wait for job card to appear
console.log(' Waiting for job card...');
const jobCard = page.locator('[class*="rounded-lg"][class*="border"]').filter({
hasText: `${TEST_SONG.artist} - ${TEST_SONG.title}`
}).first();
await expect(jobCard).toBeVisible({ timeout: TIMEOUTS.audioSearch });
console.log(' Job card visible');
// Extract job ID from the card's full text content
// The ID appears as "ID: xxxxxxxx" in the card text (e.g., "piri - dogID: af10c81f")
const cardFullText = await jobCard.textContent() || '';
const idMatch = cardFullText.match(/ID:\s*([a-f0-9]{8,})/i);
if (idMatch) {
jobId = idMatch[1];
console.log(` Job ID: ${jobId}`);
} else {
console.log(` WARNING: Could not extract job ID from card text`);
console.log(` Card text sample: ${cardFullText.substring(0, 100)}...`);
}
await page.screenshot({ path: 'test-results/04b-job-created.png' });
console.log('STEP 4 COMPLETE: Job created');
// =========================================================================
// STEP 5: Audio Selection (via UI)
// =========================================================================
console.log('\n========================================');
console.log('STEP 5: Audio Selection');
console.log('========================================');
// Wait for job to need audio selection or skip if not needed
await page.waitForTimeout(5000);
// Check if Select Audio button appears
const selectAudioBtn = jobCard.getByRole('button', { name: /select audio/i });
if (await selectAudioBtn.isVisible({ timeout: 30000 }).catch(() => false)) {
console.log(' Opening audio selection dialog...');
await selectAudioBtn.click();
await page.waitForTimeout(2000);
const dialog = page.locator('[role="dialog"]');
if (await dialog.isVisible({ timeout: 5000 })) {
await page.screenshot({ path: 'test-results/05a-audio-dialog-loading.png' });
// Wait for the loading spinner to disappear
// The AudioSearchDialog shows "Loading..." while fetching results
const loadingIndicator = dialog.getByText(/loading/i);
try {
await expect(loadingIndicator).not.toBeVisible({ timeout: 60000 });
console.log(' Audio results loaded');
} catch {
console.log(' WARNING: Loading indicator timeout');
}
await page.screenshot({ path: 'test-results/05b-audio-dialog-loaded.png' });
// Check for "No audio sources found" message
const noResultsMsg = dialog.getByText(/no audio sources found/i);
if (await noResultsMsg.isVisible({ timeout: 2000 }).catch(() => false)) {
console.log(' WARNING: No audio sources found - this may cause the test to fail');
await page.screenshot({ path: 'test-results/05b-no-audio-sources.png' });
}
// Find and click first Select button
const selectButtons = dialog.getByRole('button', { name: /^select$/i });
const count = await selectButtons.count();
console.log(` Found ${count} audio options`);
if (count > 0) {
await selectButtons.first().click();
console.log(' Audio selected via UI');
// Wait for dialog to close
await dialog.waitFor({ state: 'hidden', timeout: TIMEOUTS.audioSearch }).catch(() => {
console.log(' Dialog still open - pressing Escape');
return page.keyboard.press('Escape');
});
} else {
console.log(' ERROR: No Select buttons found - closing dialog');
await page.keyboard.press('Escape');
// This is a critical error - the job won't progress without audio selection
throw new Error('No audio sources available for selection. The job cannot proceed.');
}
}
} else {
console.log(' No audio selection needed - job auto-selected or cached');
}
await page.screenshot({ path: 'test-results/05c-after-audio.png' });
console.log('STEP 5 COMPLETE: Audio selection handled');
// =========================================================================
// STEP 6: Wait for Lyrics Processing
// =========================================================================
console.log('\n========================================');
console.log('STEP 6: Wait for Lyrics Processing');
console.log('========================================');
console.log(' Waiting for lyrics transcription (this may take 10-20 minutes)...');
// Poll the UI for status changes
let foundReviewStatus = false;
const startTime = Date.now();
let pollCount = 0;
while (Date.now() - startTime < TIMEOUTS.lyricsProcessing) {
pollCount++;
const elapsedMin = Math.floor((Date.now() - startTime) / 60000);
const elapsedSec = Math.floor(((Date.now() - startTime) % 60000) / 1000);
// Refresh the page/jobs
const refreshBtn = page.getByRole('button', { name: /refresh/i });
if (await refreshBtn.isVisible().catch(() => false)) {
await refreshBtn.click();
await page.waitForTimeout(2000);
}
// Check job card status text
const statusText = await jobCard.textContent() || '';
// Check if the "Review Lyrics" link is visible (most reliable indicator)
const reviewLink = jobCard.getByRole('link', { name: /review lyrics/i });
const reviewLinkVisible = await reviewLink.isVisible().catch(() => false);
console.log(` [${elapsedMin}m ${elapsedSec}s] Poll #${pollCount}: reviewLink=${reviewLinkVisible}, status="${statusText.substring(0, 80)}..."`);
// Primary check: Is the Review Lyrics link visible?
if (reviewLinkVisible) {
foundReviewStatus = true;
console.log(' Job ready for lyrics review (found Review Lyrics link)');
break;
}
// Secondary check: Text-based status detection
if (statusText.toLowerCase().includes('review lyrics') ||
(statusText.toLowerCase().includes('action needed') && statusText.toLowerCase().includes('review'))) {
foundReviewStatus = true;
console.log(' Job ready for lyrics review (found status text)');
break;
}
// Check for failure
if (statusText.toLowerCase().includes('failed')) {
await page.screenshot({ path: 'test-results/06-job-failed.png' });
throw new Error('Job failed during lyrics processing');
}
await page.waitForTimeout(10000); // Check every 10 seconds
}
if (!foundReviewStatus) {
const elapsedMin = Math.floor((Date.now() - startTime) / 60000);
console.log(` TIMEOUT after ${elapsedMin} minutes (${pollCount} polls)`);
await page.screenshot({ path: 'test-results/06-timeout.png' });
throw new Error(`Timeout waiting for lyrics review stage after ${elapsedMin} minutes`);
}
await page.screenshot({ path: 'test-results/06-ready-for-review.png' });
console.log('STEP 6 COMPLETE: Lyrics processing finished');
// =========================================================================
// STEP 7: Lyrics Review (via UI - REAL USER FLOW)
// The minimal path: scroll to bottom -> click "Preview Video" -> wait for video -> click "Complete Review"
// =========================================================================
console.log('\n========================================');
console.log('STEP 7: Lyrics Review (UI Interaction)');
console.log('========================================');
// Click the Review Lyrics button/link
const reviewLink = jobCard.getByRole('link', { name: /review lyrics/i });
await expect(reviewLink).toBeVisible({ timeout: TIMEOUTS.action });
// Get the review URL and open in new page
const reviewUrl = await reviewLink.getAttribute('href');
console.log(` Review URL: ${reviewUrl}`);
// Open review UI in new page
const reviewPage = await context.newPage();
await reviewPage.goto(reviewUrl!, { waitUntil: 'networkidle' });
await reviewPage.waitForTimeout(3000);
await reviewPage.screenshot({ path: 'test-results/07a-lyrics-review-opened.png', fullPage: true });
console.log(' Lyrics review UI opened');
// Wait for the page content to load
await expect(reviewPage.locator('body')).not.toBeEmpty({ timeout: TIMEOUTS.action });
// Wait for the main content to render - give MUI time to initialize
await reviewPage.waitForTimeout(2000);
// Scroll to bottom of the page to see the "Preview Video" button
console.log(' Scrolling to bottom of page...');
await reviewPage.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
await reviewPage.waitForTimeout(1000);
await reviewPage.screenshot({ path: 'test-results/07b-scrolled-to-bottom.png', fullPage: true });
// Click "Preview Video" button (at bottom of the lyrics review page)
// This button has text "Preview Video" and an OndemandVideo icon
const previewVideoBtn = reviewPage.getByRole('button', { name: /preview video/i });
await expect(previewVideoBtn).toBeVisible({ timeout: TIMEOUTS.action });
console.log(' Found "Preview Video" button');
await reviewPage.screenshot({ path: 'test-results/07c-before-preview-click.png', fullPage: true });
await previewVideoBtn.click();
console.log(' Clicked "Preview Video" button');
// Wait for the modal dialog to appear
// The modal has title "Preview Video (With Vocals)"
const previewModal = reviewPage.getByRole('dialog');
await expect(previewModal).toBeVisible({ timeout: TIMEOUTS.action });
console.log(' Preview modal opened');
await reviewPage.screenshot({ path: 'test-results/07d-preview-modal.png', fullPage: true });
// Wait for video to load in the modal
// The PreviewVideoSection component first shows "Generating preview video..."
// then renders the <video> element once the preview is ready
console.log(' Waiting for preview video generation...');
// First wait for the loading indicator to disappear (or video to appear)
// The loading state shows "Generating preview video..."
const loadingText = reviewPage.getByText(/generating preview video/i);
try {
// Wait up to 2 minutes for preview generation (it can be slow)
await expect(loadingText).not.toBeVisible({ timeout: 120000 });
console.log(' Preview generation complete');
} catch {
console.log(' WARNING: Loading indicator timeout - checking for video anyway');
}
await reviewPage.screenshot({ path: 'test-results/07e-after-generation.png', fullPage: true });
// Now check for the video element or an error message
const videoElement = reviewPage.locator('video');
const errorAlert = reviewPage.locator('[role="alert"]');
// Check if there's an error
if (await errorAlert.isVisible({ timeout: 5000 }).catch(() => false)) {
const errorText = await errorAlert.textContent();
console.log(` WARNING: Preview error: ${errorText}`);
await reviewPage.screenshot({ path: 'test-results/07e-preview-error.png', fullPage: true });
// Continue anyway - we can still click Complete Review even if preview failed
} else if (await videoElement.isVisible({ timeout: 10000 }).catch(() => false)) {
console.log(' Video element visible in modal');
// Give the video a moment to buffer/load
await reviewPage.waitForTimeout(3000);
} else {
console.log(' WARNING: No video element found, but continuing anyway');
}
await reviewPage.screenshot({ path: 'test-results/07f-video-state.png', fullPage: true });
// Click "Complete Review" button in the modal
// This button has variant="contained", color="success", and endIcon={<CloudUpload />}
const completeReviewBtn = reviewPage.getByRole('button', { name: /complete review/i });
await expect(completeReviewBtn).toBeVisible({ timeout: TIMEOUTS.action });
console.log(' Found "Complete Review" button');
await completeReviewBtn.click();
console.log(' Clicked "Complete Review" button');
// Wait for the submission to process
// The handleSubmitToServer function calls apiClient.submitCorrections() then window.close()
await reviewPage.waitForTimeout(5000);
await reviewPage.screenshot({ path: 'test-results/07g-after-complete-review.png', fullPage: true });
// The review page may close automatically after submission (window.close())
// If it's still open, close it manually
if (!reviewPage.isClosed()) {
await reviewPage.close();
console.log(' Closed review UI');
} else {
console.log(' Review UI closed automatically after submission');
}
// Refresh main app to see updated status
await page.bringToFront();
const refreshBtn = page.getByRole('button', { name: /refresh/i });
if (await refreshBtn.isVisible().catch(() => false)) {
await refreshBtn.click();
await page.waitForTimeout(3000);
}
await page.screenshot({ path: 'test-results/07h-after-review-refresh.png' });
console.log('STEP 7 COMPLETE: Lyrics review submitted via UI');
// =========================================================================
// STEP 8: Wait for Video Rendering & Instrumental Selection (via UI - REAL USER FLOW)
// The instrumental UI has: selection-option elements, submit button with id="submit-btn"
// =========================================================================
console.log('\n========================================');
console.log('STEP 8: Wait for Video Rendering & Instrumental Selection');
console.log('========================================');
// Wait for instrumental selection to become available
console.log(' Waiting for video rendering and instrumental selection...');
let foundInstrumentalStatus = false;
const renderStartTime = Date.now();
while (Date.now() - renderStartTime < TIMEOUTS.videoRendering) {
// Refresh
if (await refreshBtn.isVisible().catch(() => false)) {
await refreshBtn.click();
await page.waitForTimeout(2000);
}
// Check job card status
const statusText = await jobCard.textContent() || '';
console.log(` Status: ${statusText.substring(0, 80)}...`);
if (statusText.toLowerCase().includes('select instrumental') ||
statusText.toLowerCase().includes('awaiting_instrumental') ||
(statusText.toLowerCase().includes('action needed') && statusText.toLowerCase().includes('instrumental'))) {
foundInstrumentalStatus = true;
console.log(' Job ready for instrumental selection');
break;
}
// Check if already complete (auto-mode might have run)
if (statusText.toLowerCase().includes('complete') && !statusText.toLowerCase().includes('prep')) {
console.log(' Job already complete - skipping instrumental selection');
break;
}
// Check if encoding is in progress (past instrumental stage)
// NOTE: "rendering" status is BEFORE instrumental selection, not after!
// The flow is: rendering_video -> awaiting_instrumental_selection -> encoding -> complete
// So only skip if job is at encoding (not rendering)
if (statusText.toLowerCase().includes('encoding') && !statusText.toLowerCase().includes('awaiting')) {
console.log(' Job is encoding - instrumental selection was handled');
break;
}
if (statusText.toLowerCase().includes('failed')) {
throw new Error('Job failed during video rendering');
}
await page.waitForTimeout(10000);
}
if (foundInstrumentalStatus) {
// Click the Select Instrumental button/link
const instrumentalLink = jobCard.getByRole('link', { name: /select instrumental/i });
await expect(instrumentalLink).toBeVisible({ timeout: TIMEOUTS.action });
const instrumentalUrl = await instrumentalLink.getAttribute('href');
console.log(` Instrumental URL: ${instrumentalUrl}`);
// Open instrumental UI in new page
const instrumentalPage = await context.newPage();
// Handle both absolute and relative URLs
const fullUrl = instrumentalUrl?.startsWith('http') ? instrumentalUrl : `${PROD_URL}${instrumentalUrl}`;
// Use 'load' instead of 'networkidle' - instrumental page has ongoing audio requests
// that prevent networkidle from ever being reached
await instrumentalPage.goto(fullUrl, { waitUntil: 'load' });
await instrumentalPage.waitForTimeout(3000);
await instrumentalPage.screenshot({ path: 'test-results/08a-instrumental-opened.png', fullPage: true });
console.log(' Instrumental selection UI opened');
// Wait for the page to fully load - the instrumental UI renders selection options
await instrumentalPage.waitForSelector('.selection-option, .selection-panel', { timeout: 30000 });
// Select "Clean" instrumental option
// The options have class "selection-option" and contain labels with "Clean" or "With Backing"
const cleanOption = instrumentalPage.locator('.selection-option:has-text("Clean")').first();
if (await cleanOption.isVisible({ timeout: 5000 }).catch(() => false)) {
await cleanOption.click();
console.log(' Selected "Clean" instrumental option');
} else {
// If we can't find "Clean" specifically, the first option might already be selected
console.log(' Clean option not found - using default selection');
}
await instrumentalPage.screenshot({ path: 'test-results/08b-clean-selected.png', fullPage: true });
// Click the submit button: "✓ Confirm & Continue" (id="submit-btn")
const submitBtn = instrumentalPage.locator('#submit-btn');
await expect(submitBtn).toBeVisible({ timeout: TIMEOUTS.action });
console.log(' Found submit button');
await submitBtn.click();
console.log(' Clicked "Confirm & Continue" button');
// Wait for submission to complete - the page shows a success screen then auto-closes
await instrumentalPage.waitForTimeout(5000);
await instrumentalPage.screenshot({ path: 'test-results/08c-instrumental-submitted.png', fullPage: true });
// The page may auto-close after submission
if (!instrumentalPage.isClosed()) {
await instrumentalPage.close();
console.log(' Closed instrumental UI');
} else {
console.log(' Instrumental UI closed automatically after submission');
}
}
// Refresh main app
await page.bringToFront();
if (await refreshBtn.isVisible().catch(() => false)) {
await refreshBtn.click();
await page.waitForTimeout(3000);
}
await page.screenshot({ path: 'test-results/08d-after-instrumental.png' });
console.log('STEP 8 COMPLETE: Instrumental selection handled via UI');
// =========================================================================
// STEP 9: Wait for Final Encoding and Completion
// =========================================================================
console.log('\n========================================');
console.log('STEP 9: Wait for Completion');
console.log('========================================');
console.log(' Waiting for video rendering and encoding (this may take 15-25 minutes)...');
let isComplete = false;
const encodeStartTime = Date.now();
// After instrumental selection, job needs: video rendering (~15 min) + encoding (~10 min)
// Add extra buffer for variable processing times
const completionTimeout = TIMEOUTS.videoRendering + TIMEOUTS.finalEncoding + 600_000; // 35 minutes total
while (Date.now() - encodeStartTime < completionTimeout) {
if (await refreshBtn.isVisible().catch(() => false)) {
await refreshBtn.click();
await page.waitForTimeout(2000);
}
const statusText = await jobCard.textContent() || '';
if (statusText.toLowerCase().includes('complete') && !statusText.toLowerCase().includes('prep')) {
isComplete = true;
console.log(' Job complete!');
break;
}
if (statusText.toLowerCase().includes('failed')) {
throw new Error('Job failed during final encoding');
}
console.log(` Status: ${statusText.substring(0, 80)}...`);
await page.waitForTimeout(10000);
}
if (!isComplete) {
throw new Error('Timeout waiting for job completion');
}
await page.screenshot({ path: 'test-results/09-job-complete.png' });
console.log('STEP 9 COMPLETE: Job finished');
// =========================================================================
// STEP 10: Verify Downloads
// =========================================================================
console.log('\n========================================');
console.log('STEP 10: Verify Downloads');
console.log('========================================');
// Look for download links in the job card
const downloadLinks = jobCard.locator('a[href*="download"], a[href*="storage.googleapis.com"]');
const downloadCount = await downloadLinks.count();
console.log(` Found ${downloadCount} download links`);
if (downloadCount > 0) {
// Click first download link to verify it works
const firstDownloadUrl = await downloadLinks.first().getAttribute('href');
console.log(` First download URL: ${firstDownloadUrl?.substring(0, 80)}...`);
// Verify the download URL is accessible
// Use GET instead of HEAD since the download endpoint may not support HEAD
// The response might redirect to GCS, so accept 2xx/3xx status codes
const response = await page.request.get(firstDownloadUrl!, {
maxRedirects: 0, // Don't follow redirects, just check the initial response
});
const status = response.status();
// Accept 200 (success) or 302/307 (redirect to actual download)
expect(status >= 200 && status < 400).toBe(true);
console.log(` Download URL verified accessible (status: ${status})`);
}
await page.screenshot({ path: 'test-results/10-downloads-verified.png' });
console.log('STEP 10 COMPLETE: Downloads verified');
// =========================================================================
// STEP 11: Verify Distribution (if enabled)
// =========================================================================
console.log('\n========================================');
console.log('STEP 11: Verify Distribution');
console.log('========================================');
// Check for distribution indicators in job card or state
const cardText = await jobCard.textContent() || '';
if (cardText.toLowerCase().includes('youtube') ||
cardText.toLowerCase().includes('dropbox') ||
cardText.toLowerCase().includes('drive')) {
console.log(' Distribution indicators found in UI');
} else {
console.log(' No distribution indicators visible in UI (may be in backend state)');
}
console.log('STEP 11 COMPLETE: Distribution check done');
// =========================================================================
// STEP 12: Cleanup
// =========================================================================
console.log('\n========================================');
console.log('STEP 12: Cleanup');
console.log('========================================');
const adminToken = process.env.E2E_ADMIN_TOKEN;
if (adminToken && jobId) {
console.log(' Cleaning up distribution and job...');
try {
const cleanupResponse = await page.request.post(
`${API_URL}/api/jobs/${jobId}/cleanup-distribution`,
{
headers: {
'Authorization': `Bearer ${adminToken}`,
'Content-Type': 'application/json',
},
data: { delete_job: true },
}
);
if (cleanupResponse.ok()) {
const result = await cleanupResponse.json();
console.log(' Cleanup results:');
console.log(` YouTube: ${result.youtube?.status}`);
console.log(` Dropbox: ${result.dropbox?.status}`);
console.log(` GDrive: ${result.gdrive?.status}`);
console.log(` Job deleted: ${result.job_deleted}`);
} else {
console.log(` WARNING: Cleanup failed with status ${cleanupResponse.status()}`);
}
} catch (e) {
console.log(` WARNING: Cleanup error: ${e}`);
}
} else {
console.log(' Skipping cleanup - E2E_ADMIN_TOKEN not set or no job ID');
console.log(` Job ID for manual cleanup: ${jobId}`);
}
console.log('STEP 12 COMPLETE: Cleanup done');
// =========================================================================
// SUCCESS
// =========================================================================
console.log('\n========================================');
console.log('TEST COMPLETE: Full user journey successful!');
console.log('========================================');
console.log(`Job ID: ${jobId}`);
console.log('All UI interactions completed successfully.');
} finally {
// Always clean up the email inbox (if we created one)
if (inboxId && emailHelper) {
try {
await emailHelper.deleteInbox(inboxId);
console.log(' Cleaned up test email inbox');
} catch (e) {
console.log(` Warning: Failed to delete inbox: ${e}`);
}
}
}
});
});
@@ -0,0 +1,130 @@
import { test, expect } from '@playwright/test';
/**
* FOCUSED TEST: Instrumental Selection UI Only
*
* This test is for rapid iteration on the instrumental selection step.
* It requires an existing job that's already in "awaiting_instrumental_selection" state.
*
* Usage:
* E2E_JOB_ID=xxx E2E_INSTRUMENTAL_TOKEN=yyy npx playwright test instrumental-selection-only.spec.ts \
* --config=playwright.production.config.ts --headed
*/
const PROD_URL = 'https://gen.nomadkaraoke.com';
const API_URL = 'https://api.nomadkaraoke.com';
function getJobId(): string {
const jobId = process.env.E2E_JOB_ID;
if (!jobId) {
throw new Error('E2E_JOB_ID environment variable is required');
}
return jobId;
}
function getInstrumentalToken(): string {
const token = process.env.E2E_INSTRUMENTAL_TOKEN;
if (!token) {
throw new Error('E2E_INSTRUMENTAL_TOKEN environment variable is required');
}
return token;
}
test.describe('Instrumental Selection Only - Focused Test', () => {
test.describe.configure({ retries: 0 });
test('Complete instrumental selection via UI', async ({ page }) => {
test.setTimeout(120_000); // 2 minutes
const jobId = getJobId();
const token = getInstrumentalToken();
console.log('========================================');
console.log('FOCUSED TEST: Instrumental Selection Only');
console.log('========================================');
console.log(`Job ID: ${jobId}`);
console.log(`Instrumental Token: ${token.substring(0, 8)}...`);
console.log('');
// Build the instrumental selection URL
const baseApiUrl = encodeURIComponent(`${API_URL}/api/jobs/${jobId}`);
const instrumentalUrl = `${PROD_URL}/instrumental/?baseApiUrl=${baseApiUrl}&instrumentalToken=${token}`;
console.log(`Instrumental URL: ${instrumentalUrl}`);
console.log('');
// Navigate to the instrumental selection page
console.log('1. Opening instrumental selection UI...');
await page.goto(instrumentalUrl, { waitUntil: 'networkidle' });
await page.waitForTimeout(3000);
await page.screenshot({ path: 'test-results/instrumental-01-opened.png', fullPage: true });
console.log(' Instrumental selection UI opened');
// Wait for content to load
await expect(page.locator('body')).not.toBeEmpty({ timeout: 30_000 });
// Log page content for debugging
const bodyText = await page.locator('body').textContent();
console.log(` Page content preview: ${bodyText?.substring(0, 300)}...`);
// Check for error messages
const errorMessages = await page.locator('[class*="error"], [class*="Error"]').allTextContents();
if (errorMessages.length > 0) {
console.log(` WARNING: Error messages found: ${errorMessages.join(', ')}`);
}
// Look for selection options
console.log('2. Looking for instrumental options...');
// The instrumental selection page has radio buttons or selection cards
const selectButtons = page.locator('button:has-text("Select"), input[type="submit"], input[type="radio"], .selection-option');
const optionCount = await selectButtons.count();
console.log(` Found ${optionCount} selection elements`);
await page.screenshot({ path: 'test-results/instrumental-02-options.png', fullPage: true });
if (optionCount === 0) {
// Try looking for any clickable elements
const allButtons = await page.getByRole('button').allTextContents();
console.log(` All buttons on page: ${allButtons.join(', ')}`);
// Maybe it's a simple form with a submit button
const submitBtn = page.locator('#submit-btn, input[type="submit"], button[type="submit"]');
if (await submitBtn.isVisible().catch(() => false)) {
console.log(' Found submit button - clicking...');
await submitBtn.click();
await page.waitForTimeout(5000);
await page.screenshot({ path: 'test-results/instrumental-03-after-submit.png', fullPage: true });
console.log(' Selection submitted!');
} else {
throw new Error('No instrumental selection options found');
}
} else {
// Click the first option
console.log('3. Selecting first instrumental option...');
await selectButtons.first().click();
await page.waitForTimeout(2000);
await page.screenshot({ path: 'test-results/instrumental-03-selected.png', fullPage: true });
// Look for a submit/confirm button
const submitBtn = page.locator('#submit-btn, input[type="submit"], button[type="submit"], button:has-text("Confirm"), button:has-text("Submit")');
if (await submitBtn.isVisible().catch(() => false)) {
console.log('4. Clicking submit button...');
await submitBtn.click();
await page.waitForTimeout(5000);
console.log(' Selection submitted!');
} else {
console.log(' No submit button found - selection may have been automatic');
}
await page.screenshot({ path: 'test-results/instrumental-04-after-submit.png', fullPage: true });
}
console.log('');
console.log('========================================');
console.log('FOCUSED TEST COMPLETE');
console.log('========================================');
});
});
@@ -0,0 +1,231 @@
import { test, expect } from '@playwright/test';
/**
* FOCUSED TEST: Lyrics Review UI Only
*
* This test is for rapid iteration on the lyrics review step.
* It requires an existing job that's already in the "in_review" state.
*
* Usage:
* E2E_JOB_ID=xxx E2E_REVIEW_TOKEN=yyy npx playwright test lyrics-review-only.spec.ts \
* --config=playwright.production.config.ts
*
* To get a job in the right state:
* 1. Run the happy-path test and let it reach STEP 6 (or manually create a job)
* 2. Get the job ID and review token from the logs/UI
* 3. Run this focused test with those values
*/
const PROD_URL = 'https://gen.nomadkaraoke.com';
const API_URL = 'https://api.nomadkaraoke.com';
// Get test parameters from environment
function getJobId(): string {
const jobId = process.env.E2E_JOB_ID;
if (!jobId) {
throw new Error('E2E_JOB_ID environment variable is required');
}
return jobId;
}
function getReviewToken(): string | null {
return process.env.E2E_REVIEW_TOKEN || null;
}
function getAccessToken(): string | null {
return process.env.E2E_TEST_TOKEN || null;
}
test.describe('Lyrics Review Only - Focused Test', () => {
// Disable retries for this focused test
test.describe.configure({ retries: 0 });
test('Complete lyrics review via UI', async ({ page, context }) => {
test.setTimeout(900_000); // 15 minutes - preview generation can take up to 10 min
const jobId = getJobId();
const reviewToken = getReviewToken();
const accessToken = getAccessToken();
console.log('========================================');
console.log('FOCUSED TEST: Lyrics Review Only');
console.log('========================================');
console.log(`Job ID: ${jobId}`);
console.log(`Review Token: ${reviewToken ? reviewToken.substring(0, 8) + '...' : 'NOT SET'}`);
console.log(`Access Token: ${accessToken ? accessToken.substring(0, 8) + '...' : 'NOT SET'}`);
console.log('');
// Build the review URL
let reviewUrl: string;
if (reviewToken) {
// Direct review URL with token
reviewUrl = `${PROD_URL}/lyrics//?baseApiUrl=${encodeURIComponent(API_URL + '/api/review/' + jobId)}&reviewToken=${reviewToken}`;
} else if (accessToken) {
// Use the main app to get to the review
reviewUrl = `${PROD_URL}/lyrics//?baseApiUrl=${encodeURIComponent(API_URL + '/api/review/' + jobId)}`;
} else {
throw new Error('Either E2E_REVIEW_TOKEN or E2E_TEST_TOKEN must be set');
}
console.log(`Review URL: ${reviewUrl}`);
console.log('');
// Navigate to the review page
console.log('1. Opening lyrics review UI...');
await page.goto(reviewUrl, { waitUntil: 'networkidle' });
await page.waitForTimeout(3000);
await page.screenshot({ path: 'test-results/focused-01-review-opened.png', fullPage: true });
console.log(' Review UI opened');
// Wait for the page content to load
await expect(page.locator('body')).not.toBeEmpty({ timeout: 30_000 });
// Wait for MUI to initialize
await page.waitForTimeout(2000);
// Log page content for debugging
const bodyText = await page.locator('body').textContent();
console.log(` Page content preview: ${bodyText?.substring(0, 200)}...`);
// Look for any error messages
const errorMessages = await page.locator('[class*="error"], [class*="Error"], .MuiAlert-standardError').allTextContents();
if (errorMessages.length > 0) {
console.log(` WARNING: Error messages found: ${errorMessages.join(', ')}`);
}
// Scroll to bottom to find the "Preview Video" button
console.log('2. Scrolling to bottom of page...');
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
await page.waitForTimeout(1000);
await page.screenshot({ path: 'test-results/focused-02-scrolled.png', fullPage: true });
// List all visible buttons for debugging
const allButtons = await page.getByRole('button').allTextContents();
console.log(` Visible buttons: ${allButtons.join(', ')}`);
// Click "Preview Video" button
console.log('3. Looking for "Preview Video" button...');
const previewVideoBtn = page.getByRole('button', { name: /preview video/i });
if (await previewVideoBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
console.log(' Found "Preview Video" button - clicking...');
await page.screenshot({ path: 'test-results/focused-03-before-preview.png', fullPage: true });
await previewVideoBtn.click();
console.log(' Clicked "Preview Video" button');
} else {
// Maybe the button has different text or is not visible yet
console.log(' Preview Video button not found with that name');
console.log(' Taking screenshot for debugging...');
await page.screenshot({ path: 'test-results/focused-03-no-preview-btn.png', fullPage: true });
// Try alternative selectors
const finishBtn = page.getByRole('button', { name: /finish|complete|submit|done/i });
if (await finishBtn.isVisible().catch(() => false)) {
const btnText = await finishBtn.textContent();
console.log(` Found alternative button: "${btnText}"`);
}
throw new Error('Could not find Preview Video button');
}
// Wait for the modal dialog to appear
console.log('4. Waiting for preview modal...');
const previewModal = page.getByRole('dialog');
await expect(previewModal).toBeVisible({ timeout: 30_000 });
console.log(' Preview modal opened');
await page.screenshot({ path: 'test-results/focused-04-modal-opened.png', fullPage: true });
// List what's in the modal for debugging
const modalContent = await previewModal.textContent();
console.log(` Modal content preview: ${modalContent?.substring(0, 200)}...`);
// Wait for video preview to generate
// The PreviewVideoSection shows "Generating preview video..." while loading
console.log('5. Waiting for preview video generation...');
const loadingText = page.getByText(/generating preview video/i);
if (await loadingText.isVisible({ timeout: 5000 }).catch(() => false)) {
console.log(' Preview is generating... waiting up to 10 minutes');
try {
await expect(loadingText).not.toBeVisible({ timeout: 600_000 });
console.log(' Preview generation complete');
} catch {
console.log(' WARNING: Preview generation timed out after 10 minutes');
}
} else {
console.log(' No loading indicator found - preview may already be ready or errored');
}
await page.screenshot({ path: 'test-results/focused-05-after-generation.png', fullPage: true });
// Check for video element or error
const videoElement = page.locator('video');
const errorAlert = page.locator('[role="alert"]');
if (await errorAlert.isVisible({ timeout: 5000 }).catch(() => false)) {
const errorText = await errorAlert.textContent();
console.log(` WARNING: Preview error: ${errorText}`);
await page.screenshot({ path: 'test-results/focused-05-preview-error.png', fullPage: true });
// Continue anyway
} else if (await videoElement.isVisible({ timeout: 10000 }).catch(() => false)) {
console.log(' Video element visible');
// Check video state
const videoState = await videoElement.first().evaluate((el) => {
const video = el as HTMLVideoElement;
return {
src: video.src,
readyState: video.readyState,
networkState: video.networkState,
error: video.error?.message,
};
});
console.log(` Video state: ${JSON.stringify(videoState)}`);
await page.waitForTimeout(3000);
} else {
console.log(' WARNING: No video element found, but continuing anyway');
}
await page.screenshot({ path: 'test-results/focused-06-video-state.png', fullPage: true });
// Click "Complete Review" button in the modal
console.log('6. Looking for "Complete Review" button...');
// List all buttons in the modal
const modalButtons = await previewModal.getByRole('button').allTextContents();
console.log(` Modal buttons: ${modalButtons.join(', ')}`);
const completeReviewBtn = page.getByRole('button', { name: /complete review/i });
if (await completeReviewBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
console.log(' Found "Complete Review" button - clicking...');
await completeReviewBtn.click();
console.log(' Clicked "Complete Review" button');
} else {
// Try to find any submit-like button
const submitBtn = previewModal.getByRole('button', { name: /submit|confirm|done|save/i });
if (await submitBtn.isVisible().catch(() => false)) {
const btnText = await submitBtn.textContent();
console.log(` Found alternative submit button: "${btnText}" - clicking...`);
await submitBtn.click();
} else {
console.log(' Could not find Complete Review or similar button');
await page.screenshot({ path: 'test-results/focused-07-no-complete-btn.png', fullPage: true });
throw new Error('Could not find Complete Review button');
}
}
// Wait for the submission to process
console.log('7. Waiting for submission to complete...');
await page.waitForTimeout(5000);
await page.screenshot({ path: 'test-results/focused-08-after-submit.png', fullPage: true });
console.log('');
console.log('========================================');
console.log('FOCUSED TEST COMPLETE');
console.log('========================================');
});
});
@@ -0,0 +1,401 @@
import { test, expect } from '@playwright/test';
/**
* TEST EXISTING JOB - Test specific stages against existing jobs
*
* This test allows you to validate specific stages of the E2E flow
* against jobs that are already in the right state.
*
* Usage:
* E2E_TEST_TOKEN=xxx E2E_JOB_ID=yyy E2E_TEST_STAGE=lyrics_review \
* npx playwright test test-existing-job.spec.ts --config=playwright.production.config.ts
*
* Required:
* - E2E_TEST_TOKEN: Your user access token
* - E2E_JOB_ID: The job ID to test against
* - E2E_TEST_STAGE: The stage to test (lyrics_review, instrumental_selection, or completion)
*
* Job states required for each stage:
* - lyrics_review: Job must be in "in_review" state
* - instrumental_selection: Job must be in "awaiting_instrumental_selection" state
* - completion: Job must be in a late-stage processing state
*/
const PROD_URL = 'https://gen.nomadkaraoke.com';
function getAccessToken(): string {
const token = process.env.E2E_TEST_TOKEN;
if (!token) {
throw new Error('E2E_TEST_TOKEN environment variable is required');
}
return token;
}
function getJobId(): string {
const jobId = process.env.E2E_JOB_ID;
if (!jobId) {
throw new Error('E2E_JOB_ID environment variable is required');
}
return jobId;
}
function getTestStage(): string {
const stage = process.env.E2E_TEST_STAGE || 'lyrics_review';
const validStages = ['lyrics_review', 'instrumental_selection', 'completion'];
if (!validStages.includes(stage)) {
throw new Error(`Invalid E2E_TEST_STAGE: ${stage}. Valid options: ${validStages.join(', ')}`);
}
return stage;
}
test.describe('Test Existing Job - Focused Stage Testing', () => {
test.describe.configure({ retries: 0 });
test('Test specific stage against existing job', async ({ page, context }) => {
test.setTimeout(300_000); // 5 minutes
const accessToken = getAccessToken();
const jobId = getJobId();
const testStage = getTestStage();
console.log('========================================');
console.log('TEST EXISTING JOB');
console.log('========================================');
console.log(`Job ID: ${jobId}`);
console.log(`Test Stage: ${testStage}`);
console.log(`Access Token: ${accessToken.substring(0, 8)}...`);
console.log('');
// STEP 1: Set up authentication
console.log('1. Setting up authentication...');
await page.goto(PROD_URL, { waitUntil: 'networkidle' });
// Set the access token in localStorage
await page.evaluate((token) => {
localStorage.setItem('karaoke_access_token', token);
}, accessToken);
console.log(' Access token set in localStorage');
// STEP 2: Navigate to app page (where jobs are shown) and find the job
console.log('2. Navigating to app page...');
await page.goto(`${PROD_URL}/app`, { waitUntil: 'networkidle' });
await page.waitForTimeout(3000);
await page.screenshot({ path: 'test-results/existing-01-jobs-page.png', fullPage: true });
// Find the job card by ID
console.log(`3. Looking for job ${jobId}...`);
// The job ID appears in the job card - look for it
const jobCard = page.locator(`[data-job-id="${jobId}"], :has-text("${jobId}")`).first();
// If we can't find by ID directly, look for any job card and check
let foundJobCard = await jobCard.isVisible({ timeout: 5000 }).catch(() => false);
if (!foundJobCard) {
console.log(' Job card not found by ID, looking for job in list...');
// Get all job cards and look for ours
const allCards = page.locator('[class*="card"], [class*="Card"], [class*="job"]');
const cardCount = await allCards.count();
console.log(` Found ${cardCount} potential job cards`);
// Check if the job ID appears on the page
const pageText = await page.textContent('body');
if (pageText?.includes(jobId)) {
console.log(` Job ID ${jobId} found in page content`);
} else {
console.log(` WARNING: Job ID ${jobId} not found on page`);
await page.screenshot({ path: 'test-results/existing-02-job-not-found.png', fullPage: true });
throw new Error(`Job ${jobId} not found on jobs page`);
}
}
// Log the current job status
const statusText = await page.textContent('body') || '';
console.log(` Page content preview: ${statusText.substring(0, 300)}...`);
// STEP 3: Execute the appropriate test stage
if (testStage === 'lyrics_review') {
await testLyricsReview(page, context, jobId);
} else if (testStage === 'instrumental_selection') {
await testInstrumentalSelection(page, context, jobId);
} else if (testStage === 'completion') {
await testCompletion(page, jobId);
}
console.log('');
console.log('========================================');
console.log('TEST COMPLETE');
console.log('========================================');
});
});
async function testLyricsReview(page: any, context: any, jobId: string) {
console.log('\n========================================');
console.log('TESTING: Lyrics Review Stage');
console.log('========================================');
// Look for the Review Lyrics link
console.log('4. Looking for "Review Lyrics" link...');
const reviewLink = page.getByRole('link', { name: /review lyrics/i });
if (!(await reviewLink.isVisible({ timeout: 10000 }).catch(() => false))) {
// Maybe we need to expand the job card or the job isn't in the right state
console.log(' Review Lyrics link not visible');
// Check for Action needed indicator
const actionNeeded = page.getByText(/action needed/i);
if (await actionNeeded.isVisible().catch(() => false)) {
console.log(' Found "Action needed" indicator');
}
// List all links on the page for debugging
const allLinks = await page.getByRole('link').allTextContents();
console.log(` Available links: ${allLinks.slice(0, 20).join(', ')}`);
await page.screenshot({ path: 'test-results/existing-03-no-review-link.png', fullPage: true });
throw new Error('Job does not appear to be in the lyrics review state');
}
// Get the review URL and open in new page
const reviewUrl = await reviewLink.getAttribute('href');
console.log(` Review URL: ${reviewUrl}`);
// Open review UI in new page
console.log('5. Opening lyrics review UI...');
const reviewPage = await context.newPage();
await reviewPage.goto(reviewUrl!, { waitUntil: 'networkidle' });
await reviewPage.waitForTimeout(3000);
await reviewPage.screenshot({ path: 'test-results/existing-04-review-opened.png', fullPage: true });
console.log(' Lyrics review UI opened');
// Wait for content to load
await expect(reviewPage.locator('body')).not.toBeEmpty({ timeout: 30000 });
await reviewPage.waitForTimeout(2000);
// Log page content for debugging
const bodyText = await reviewPage.locator('body').textContent();
console.log(` Page content: ${bodyText?.substring(0, 200)}...`);
// Look for any error messages
const errorMessages = await reviewPage.locator('[class*="error"], .MuiAlert-standardError').allTextContents();
if (errorMessages.length > 0) {
console.log(` WARNING: Error messages: ${errorMessages.join(', ')}`);
}
// Scroll to bottom to find the "Preview Video" button
console.log('6. Scrolling to bottom of page...');
await reviewPage.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
await reviewPage.waitForTimeout(1000);
await reviewPage.screenshot({ path: 'test-results/existing-05-scrolled.png', fullPage: true });
// List all visible buttons
const allButtons = await reviewPage.getByRole('button').allTextContents();
console.log(` Visible buttons: ${allButtons.join(', ')}`);
// Click "Preview Video" button
console.log('7. Clicking "Preview Video" button...');
const previewVideoBtn = reviewPage.getByRole('button', { name: /preview video/i });
if (!(await previewVideoBtn.isVisible({ timeout: 5000 }).catch(() => false))) {
await reviewPage.screenshot({ path: 'test-results/existing-06-no-preview-btn.png', fullPage: true });
throw new Error('Could not find Preview Video button');
}
await previewVideoBtn.click();
console.log(' Clicked Preview Video button');
// Wait for the modal dialog to appear
console.log('8. Waiting for preview modal...');
const previewModal = reviewPage.getByRole('dialog');
await expect(previewModal).toBeVisible({ timeout: 30000 });
console.log(' Preview modal opened');
await reviewPage.screenshot({ path: 'test-results/existing-07-modal-opened.png', fullPage: true });
// Wait for video preview to generate
console.log('9. Waiting for preview video generation...');
const loadingText = reviewPage.getByText(/generating preview video/i);
if (await loadingText.isVisible({ timeout: 5000 }).catch(() => false)) {
console.log(' Preview is generating... waiting up to 2 minutes');
try {
await expect(loadingText).not.toBeVisible({ timeout: 120000 });
console.log(' Preview generation complete');
} catch {
console.log(' WARNING: Preview generation timed out');
}
} else {
console.log(' No loading indicator found - preview may already be ready');
}
await reviewPage.screenshot({ path: 'test-results/existing-08-after-generation.png', fullPage: true });
// Check for video element or error
const videoElement = reviewPage.locator('video');
const errorAlert = reviewPage.locator('[role="alert"]');
if (await errorAlert.isVisible({ timeout: 5000 }).catch(() => false)) {
const errorText = await errorAlert.textContent();
console.log(` WARNING: Preview error: ${errorText}`);
} else if (await videoElement.isVisible({ timeout: 10000 }).catch(() => false)) {
console.log(' Video element visible');
const videoState = await videoElement.first().evaluate((el) => {
const video = el as HTMLVideoElement;
return {
src: video.src?.substring(0, 50),
readyState: video.readyState,
duration: video.duration,
};
});
console.log(` Video state: ${JSON.stringify(videoState)}`);
} else {
console.log(' WARNING: No video element found');
}
// Click "Complete Review" button
console.log('10. Clicking "Complete Review" button...');
const modalButtons = await previewModal.getByRole('button').allTextContents();
console.log(` Modal buttons: ${modalButtons.join(', ')}`);
const completeReviewBtn = reviewPage.getByRole('button', { name: /complete review/i });
if (await completeReviewBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
await completeReviewBtn.click();
console.log(' Clicked Complete Review button');
} else {
await reviewPage.screenshot({ path: 'test-results/existing-09-no-complete-btn.png', fullPage: true });
throw new Error('Could not find Complete Review button');
}
// Wait for submission
console.log('11. Waiting for submission...');
await reviewPage.waitForTimeout(5000);
await reviewPage.screenshot({ path: 'test-results/existing-10-after-submit.png', fullPage: true });
console.log(' Lyrics review completed successfully!');
await reviewPage.close();
}
async function testInstrumentalSelection(page: any, context: any, jobId: string) {
console.log('\n========================================');
console.log('TESTING: Instrumental Selection Stage');
console.log('========================================');
// Look for the Select Instrumental link
console.log('4. Looking for "Select Instrumental" link...');
const selectLink = page.getByRole('link', { name: /select instrumental/i });
if (!(await selectLink.isVisible({ timeout: 10000 }).catch(() => false))) {
// List all links for debugging
const allLinks = await page.getByRole('link').allTextContents();
console.log(` Available links: ${allLinks.slice(0, 20).join(', ')}`);
await page.screenshot({ path: 'test-results/existing-03-no-select-link.png', fullPage: true });
throw new Error('Job does not appear to be in the instrumental selection state');
}
// Get the selection URL and open in new page
const selectUrl = await selectLink.getAttribute('href');
console.log(` Selection URL: ${selectUrl}`);
// Open selection UI in new page
console.log('5. Opening instrumental selection UI...');
const selectPage = await context.newPage();
await selectPage.goto(selectUrl!, { waitUntil: 'networkidle' });
await selectPage.waitForTimeout(3000);
await selectPage.screenshot({ path: 'test-results/existing-04-selection-opened.png', fullPage: true });
console.log(' Instrumental selection UI opened');
// Wait for content
await expect(selectPage.locator('body')).not.toBeEmpty({ timeout: 30000 });
// Log page content
const bodyText = await selectPage.locator('body').textContent();
console.log(` Page content: ${bodyText?.substring(0, 200)}...`);
// Find and click a Select button (the first one should be fine)
console.log('6. Looking for instrumental options...');
const selectButtons = selectPage.locator('button:has-text("Select"), input[type="submit"][value*="Select"]');
const buttonCount = await selectButtons.count();
console.log(` Found ${buttonCount} Select buttons`);
if (buttonCount === 0) {
await selectPage.screenshot({ path: 'test-results/existing-05-no-select-buttons.png', fullPage: true });
throw new Error('No instrumental selection buttons found');
}
// Click the first one
console.log('7. Clicking first instrumental option...');
await selectButtons.first().click();
console.log(' Clicked Select button');
// Wait for the selection to process
console.log('8. Waiting for selection to complete...');
await selectPage.waitForTimeout(5000);
await selectPage.screenshot({ path: 'test-results/existing-06-after-selection.png', fullPage: true });
// Check for success message or redirect
const currentUrl = selectPage.url();
console.log(` Current URL: ${currentUrl}`);
console.log(' Instrumental selection completed successfully!');
await selectPage.close();
}
async function testCompletion(page: any, jobId: string) {
console.log('\n========================================');
console.log('TESTING: Completion Stage');
console.log('========================================');
// This stage just monitors the job until completion
console.log('4. Monitoring job for completion...');
const maxWaitTime = 30 * 60 * 1000; // 30 minutes
const startTime = Date.now();
let completed = false;
while (Date.now() - startTime < maxWaitTime) {
// Refresh the page
const refreshBtn = page.getByRole('button', { name: /refresh/i });
if (await refreshBtn.isVisible().catch(() => false)) {
await refreshBtn.click();
await page.waitForTimeout(2000);
} else {
await page.reload({ waitUntil: 'networkidle' });
await page.waitForTimeout(2000);
}
// Check job status
const statusText = await page.textContent('body') || '';
console.log(` Status: ${statusText.substring(0, 100)}...`);
// Check for completion indicators
if (statusText.toLowerCase().includes('complete') &&
!statusText.toLowerCase().includes('processing')) {
completed = true;
console.log(' Job completed!');
break;
}
// Check for failure
if (statusText.toLowerCase().includes('failed')) {
throw new Error('Job failed during processing');
}
await page.waitForTimeout(30000); // Check every 30 seconds
}
if (!completed) {
throw new Error('Job did not complete within timeout');
}
await page.screenshot({ path: 'test-results/existing-completion.png', fullPage: true });
console.log(' Completion test passed!');
}
@@ -171,6 +171,71 @@ test.describe('Authentication - Beta Enrollment', () => {
await page.waitForURL(/\/app/, { timeout: 10000 });
expect(page.url()).toContain('/app');
});
test('beta enrollment shows credits after redirect to app', async ({ page }) => {
// This test verifies that after beta enrollment, the /app page
// correctly fetches and displays user data including credits.
// This catches the bug where client-side navigation didn't trigger fetchUser().
await setupApiFixtures(page, {
mocks: [
{
method: 'POST',
path: '/api/users/beta/enroll',
response: {
status: 200,
body: {
status: 'success',
message: 'Enrolled in beta',
credits_granted: 1,
session_token: 'mock-session-token-456',
},
},
},
{
method: 'GET',
path: '/api/users/me',
response: {
status: 200,
body: {
user: {
email: 'beta-tester@example.com',
credits: 1,
role: 'user',
display_name: null,
total_jobs_created: 0,
total_jobs_completed: 0,
},
has_session: true,
},
},
},
{
method: 'GET',
path: '/api/jobs',
response: { body: [] },
},
],
});
await page.goto('/');
await page.waitForLoadState('networkidle');
// Complete beta enrollment
await page.getByRole('button', { name: /join beta program/i }).click();
await page.locator('#beta-email').fill('beta-tester@example.com');
await page.locator('textarea').fill(
'I want to create a karaoke video for my favorite song!'
);
await page.locator('input[type="checkbox"]').check();
await page.getByRole('button', { name: /get my free credit/i }).click();
// Wait for redirect to /app
await page.waitForURL(/\/app/, { timeout: 10000 });
// Verify credits are displayed in the UI
// The AuthStatus component should show "1 credit" after fetchUser() completes
await expect(page.getByText(/1 credit/i)).toBeVisible({ timeout: 10000 });
});
});
test.describe('Authentication - Token Persistence', () => {
-171
View File
@@ -1,171 +0,0 @@
/**
* Tests for auto-mode state management
*/
import { renderHook, act } from "@testing-library/react"
import { useAutoMode, getAutoModeFromUrl } from "../auto-mode"
// Mock localStorage
const localStorageMock = (() => {
let store: Record<string, string> = {}
return {
getItem: jest.fn((key: string) => store[key] || null),
setItem: jest.fn((key: string, value: string) => {
store[key] = value
}),
removeItem: jest.fn((key: string) => {
delete store[key]
}),
clear: jest.fn(() => {
store = {}
}),
}
})()
Object.defineProperty(window, "localStorage", { value: localStorageMock })
describe("Auto Mode", () => {
beforeEach(() => {
localStorageMock.clear()
jest.clearAllMocks()
})
describe("useAutoMode hook", () => {
it("should initialize with enabled=false", () => {
const { result } = renderHook(() => useAutoMode())
expect(result.current.enabled).toBe(false)
})
it("should enable auto-mode", () => {
const { result } = renderHook(() => useAutoMode())
act(() => {
result.current.enable()
})
expect(result.current.enabled).toBe(true)
})
it("should disable auto-mode", () => {
const { result } = renderHook(() => useAutoMode())
act(() => {
result.current.enable()
result.current.disable()
})
expect(result.current.enabled).toBe(false)
})
it("should toggle auto-mode", () => {
const { result } = renderHook(() => useAutoMode())
act(() => {
result.current.toggle()
})
expect(result.current.enabled).toBe(true)
act(() => {
result.current.toggle()
})
expect(result.current.enabled).toBe(false)
})
it("should set enabled directly", () => {
const { result } = renderHook(() => useAutoMode())
act(() => {
result.current.setEnabled(true)
})
expect(result.current.enabled).toBe(true)
act(() => {
result.current.setEnabled(false)
})
expect(result.current.enabled).toBe(false)
})
it("should track processing jobs", () => {
const { result } = renderHook(() => useAutoMode())
expect(result.current.isProcessing("job-123-review")).toBe(false)
act(() => {
result.current.markProcessing("job-123-review")
})
expect(result.current.isProcessing("job-123-review")).toBe(true)
expect(result.current.isProcessing("job-456-review")).toBe(false)
act(() => {
result.current.unmarkProcessing("job-123-review")
})
expect(result.current.isProcessing("job-123-review")).toBe(false)
})
it("should handle multiple processing jobs", () => {
const { result } = renderHook(() => useAutoMode())
act(() => {
result.current.markProcessing("job-1-review")
result.current.markProcessing("job-2-instrumental")
})
expect(result.current.isProcessing("job-1-review")).toBe(true)
expect(result.current.isProcessing("job-2-instrumental")).toBe(true)
act(() => {
result.current.unmarkProcessing("job-1-review")
})
expect(result.current.isProcessing("job-1-review")).toBe(false)
expect(result.current.isProcessing("job-2-instrumental")).toBe(true)
})
})
describe("getAutoModeFromUrl", () => {
const originalLocation = window.location
beforeEach(() => {
// @ts-ignore - mocking location
delete window.location
// @ts-ignore - mocking location
window.location = { search: "" }
})
afterEach(() => {
window.location = originalLocation
})
it("should return false when no auto parameter", () => {
// @ts-ignore
window.location.search = ""
expect(getAutoModeFromUrl()).toBe(false)
})
it('should return true when auto=true', () => {
// @ts-ignore
window.location.search = "?auto=true"
expect(getAutoModeFromUrl()).toBe(true)
})
it('should return true when auto=1', () => {
// @ts-ignore
window.location.search = "?auto=1"
expect(getAutoModeFromUrl()).toBe(true)
})
it('should return false when auto=false', () => {
// @ts-ignore
window.location.search = "?auto=false"
expect(getAutoModeFromUrl()).toBe(false)
})
it("should handle multiple query params", () => {
// @ts-ignore
window.location.search = "?foo=bar&auto=true&baz=qux"
expect(getAutoModeFromUrl()).toBe(true)
})
})
})
+380
View File
@@ -65,6 +65,8 @@ export interface Job {
review_token?: string;
instrumental_token?: string;
audio_hash?: string;
non_interactive?: boolean;
user_email?: string;
}
export interface UploadJobResponse {
@@ -225,6 +227,7 @@ export const api = {
enable_youtube_upload?: boolean;
theme_id?: string;
color_overrides?: ColorOverrides;
non_interactive?: boolean;
}
): Promise<UploadJobResponse> {
const formData = new FormData();
@@ -250,6 +253,9 @@ export const api = {
if (options?.color_overrides) {
formData.append('color_overrides', JSON.stringify(options.color_overrides));
}
if (options?.non_interactive !== undefined) {
formData.append('non_interactive', String(options.non_interactive));
}
const response = await fetch(`${API_BASE_URL}/api/jobs/upload`, {
method: 'POST',
@@ -274,6 +280,7 @@ export const api = {
enable_youtube_upload?: boolean;
theme_id?: string;
color_overrides?: ColorOverrides;
non_interactive?: boolean;
}
): Promise<{ status: string; job_id: string; message: string }> {
const body: Record<string, any> = { url };
@@ -285,6 +292,7 @@ export const api = {
if (options?.enable_youtube_upload !== undefined) body.enable_youtube_upload = options.enable_youtube_upload;
if (options?.theme_id) body.theme_id = options.theme_id;
if (options?.color_overrides) body.color_overrides = options.color_overrides;
if (options?.non_interactive !== undefined) body.non_interactive = options.non_interactive;
const response = await fetch(`${API_BASE_URL}/api/jobs/create-from-url`, {
method: 'POST',
@@ -409,6 +417,7 @@ export const api = {
enable_youtube_upload?: boolean;
theme_id?: string;
color_overrides?: ColorOverrides;
non_interactive?: boolean;
}
): Promise<AudioSearchResponse> {
const body: Record<string, any> = { artist, title, auto_download: autoDownload };
@@ -418,6 +427,7 @@ export const api = {
if (options?.enable_youtube_upload !== undefined) body.enable_youtube_upload = options.enable_youtube_upload;
if (options?.theme_id) body.theme_id = options.theme_id;
if (options?.color_overrides) body.color_overrides = options.color_overrides;
if (options?.non_interactive !== undefined) body.non_interactive = options.non_interactive;
const response = await fetch(`${API_BASE_URL}/api/audio-search/search`, {
method: 'POST',
@@ -660,5 +670,375 @@ export interface BetaEnrollResponse {
session_token: string | null;
}
// ==========================================================================
// Admin Types
// ==========================================================================
export interface AdminStatsOverview {
total_users: number;
active_users_7d: number;
active_users_30d: number;
total_jobs: number;
jobs_last_7d: number;
jobs_last_30d: number;
jobs_by_status: {
pending: number;
processing: number;
awaiting_review: number;
awaiting_instrumental: number;
complete: number;
failed: number;
cancelled: number;
};
total_credits_issued_30d: number;
total_beta_testers: number;
}
export interface AdminUser {
email: string;
role: 'user' | 'admin';
credits: number;
display_name?: string;
total_jobs_created?: number;
total_jobs_completed?: number;
}
export interface AdminUserListResponse {
users: AdminUser[];
total: number;
offset: number;
limit: number;
has_more: boolean;
}
export interface AdminUserDetail {
email: string;
role: 'user' | 'admin';
credits: number;
display_name?: string;
is_active: boolean;
email_verified: boolean;
created_at?: string;
updated_at?: string;
last_login_at?: string;
total_jobs_created: number;
total_jobs_completed: number;
is_beta_tester: boolean;
beta_tester_status?: string;
credit_transactions: Array<{
id: string;
amount: number;
reason: string;
created_at: string;
job_id?: string;
created_by?: string;
}>;
recent_jobs: Array<{
job_id: string;
status: string;
artist?: string;
title?: string;
created_at?: string;
}>;
active_sessions_count: number;
}
export interface AdminBetaStats {
total_beta_testers: number;
active_testers: number;
pending_feedback: number;
completed_feedback: number;
total_feedback_submissions: number;
average_ratings: {
overall: number;
ease_of_use: number;
lyrics_accuracy: number;
correction_experience: number;
};
}
export interface AdminBetaFeedback {
id: string;
user_email: string;
job_id?: string;
overall_rating: number;
ease_of_use_rating?: number;
lyrics_accuracy_rating?: number;
correction_experience_rating?: number;
what_went_well?: string;
what_could_improve?: string;
additional_comments?: string;
would_recommend?: boolean;
would_use_again?: boolean;
created_at: string;
}
export interface AdminJobListParams {
status?: string;
user_email?: string;
environment?: string;
created_after?: string;
created_before?: string;
limit?: number;
}
// Audio Search Admin Types
export interface AudioSearchResultSummary {
index: number;
provider: string;
artist: string;
title: string;
is_lossless: boolean;
quality?: string;
seeders?: number;
}
export interface AudioSearchJobSummary {
job_id: string;
status: string;
user_email?: string;
audio_search_artist?: string;
audio_search_title?: string;
created_at?: string;
results_count: number;
results_summary: AudioSearchResultSummary[];
has_lossless: boolean;
providers: string[];
}
export interface AudioSearchListResponse {
jobs: AudioSearchJobSummary[];
total: number;
}
export interface ClearSearchCacheResponse {
status: string;
job_id: string;
message: string;
previous_status: string;
new_status: string;
results_cleared: number;
flacfetch_cache_cleared: boolean;
flacfetch_error?: string;
}
export interface ClearAllCacheResponse {
status: string;
message: string;
deleted_count: number;
}
export interface CacheStatsResponse {
count: number;
total_size_bytes: number;
oldest_entry?: string;
newest_entry?: string;
configured: boolean;
}
// Admin API namespace
export const adminApi = {
/**
* Get admin dashboard statistics
*/
async getStats(): Promise<AdminStatsOverview> {
const response = await fetch(`${API_BASE_URL}/api/admin/stats/overview`, {
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* List all users with pagination and search
*/
async listUsers(params?: {
limit?: number;
offset?: number;
search?: string;
sort_by?: 'created_at' | 'last_login_at' | 'credits' | 'email';
sort_order?: 'asc' | 'desc';
include_inactive?: boolean;
}): Promise<AdminUserListResponse> {
const searchParams = new URLSearchParams();
if (params?.limit) searchParams.set('limit', String(params.limit));
if (params?.offset) searchParams.set('offset', String(params.offset));
if (params?.search) searchParams.set('search', params.search);
if (params?.sort_by) searchParams.set('sort_by', params.sort_by);
if (params?.sort_order) searchParams.set('sort_order', params.sort_order);
if (params?.include_inactive) searchParams.set('include_inactive', 'true');
const url = `${API_BASE_URL}/api/users/admin/users${searchParams.toString() ? '?' + searchParams.toString() : ''}`;
const response = await fetch(url, {
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* Get detailed user information
*/
async getUserDetail(email: string): Promise<AdminUserDetail> {
const response = await fetch(`${API_BASE_URL}/api/users/admin/users/${encodeURIComponent(email)}/detail`, {
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* Add credits to a user
*/
async addCredits(email: string, amount: number, reason: string): Promise<{
status: string;
email: string;
credits_added: number;
new_balance: number;
message: string;
}> {
const response = await fetch(`${API_BASE_URL}/api/users/admin/credits`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
...getAuthHeaders()
},
body: JSON.stringify({ email, amount, reason }),
});
return handleResponse(response);
},
/**
* Set user role
*/
async setUserRole(email: string, role: 'user' | 'admin'): Promise<{ status: string; message: string }> {
const response = await fetch(`${API_BASE_URL}/api/users/admin/users/${encodeURIComponent(email)}/role?role=${role}`, {
method: 'POST',
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* Enable a user account
*/
async enableUser(email: string): Promise<{ status: string; message: string }> {
const response = await fetch(`${API_BASE_URL}/api/users/admin/users/${encodeURIComponent(email)}/enable`, {
method: 'POST',
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* Disable a user account
*/
async disableUser(email: string): Promise<{ status: string; message: string }> {
const response = await fetch(`${API_BASE_URL}/api/users/admin/users/${encodeURIComponent(email)}/disable`, {
method: 'POST',
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* List all jobs (admin view)
*/
async listAllJobs(params?: AdminJobListParams): Promise<Job[]> {
const searchParams = new URLSearchParams();
if (params?.status) searchParams.set('status', params.status);
if (params?.user_email) searchParams.set('user_email', params.user_email);
if (params?.environment) searchParams.set('environment', params.environment);
if (params?.created_after) searchParams.set('created_after', params.created_after);
if (params?.created_before) searchParams.set('created_before', params.created_before);
if (params?.limit) searchParams.set('limit', String(params.limit));
const url = `${API_BASE_URL}/api/jobs${searchParams.toString() ? '?' + searchParams.toString() : ''}`;
const response = await fetch(url, {
headers: getAuthHeaders()
});
return handleResponse<Job[]>(response);
},
/**
* Get beta program statistics
*/
async getBetaStats(): Promise<AdminBetaStats> {
const response = await fetch(`${API_BASE_URL}/api/users/admin/beta/stats`, {
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* Get beta program feedback list
*/
async getBetaFeedback(limit: number = 50): Promise<{ feedback: AdminBetaFeedback[]; total: number }> {
const response = await fetch(`${API_BASE_URL}/api/users/admin/beta/feedback?limit=${limit}`, {
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* Delete a job (admin)
*/
async deleteJob(jobId: string, deleteFiles: boolean = true): Promise<{ status: string; message: string }> {
const response = await fetch(
`${API_BASE_URL}/api/jobs/${jobId}?delete_files=${deleteFiles}`,
{ method: 'DELETE', headers: getAuthHeaders() }
);
return handleResponse(response);
},
/**
* List jobs with audio search results
*/
async listAudioSearches(params?: {
limit?: number;
status_filter?: string;
}): Promise<AudioSearchListResponse> {
const searchParams = new URLSearchParams();
if (params?.limit) searchParams.set('limit', String(params.limit));
if (params?.status_filter) searchParams.set('status_filter', params.status_filter);
const url = `${API_BASE_URL}/api/admin/audio-searches${searchParams.toString() ? '?' + searchParams.toString() : ''}`;
const response = await fetch(url, {
headers: getAuthHeaders()
});
return handleResponse(response);
},
/**
* Clear audio search cache for a job
*/
async clearAudioSearchCache(jobId: string): Promise<ClearSearchCacheResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/audio-searches/${jobId}/clear-cache`,
{ method: 'POST', headers: getAuthHeaders() }
);
return handleResponse(response);
},
/**
* Clear all flacfetch search cache
*/
async clearAllCache(): Promise<ClearAllCacheResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/cache`,
{ method: 'DELETE', headers: getAuthHeaders() }
);
return handleResponse(response);
},
/**
* Get flacfetch cache statistics
*/
async getCacheStats(): Promise<CacheStatsResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/cache/stats`,
{ headers: getAuthHeaders() }
);
return handleResponse(response);
},
};
export { ApiError };
Loaded 100 of 128 files, more files were not shown because too many files have changed in this diff. Show more