mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-21 05:18:08 +09:00
Compare commits
168
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4b01f1100f | ||
|
|
6f1bafbcec | ||
|
|
37bdb1e93a | ||
|
|
d6aaaa8544 | ||
|
|
229d23ead1 | ||
|
|
2112596133 | ||
|
|
f2d2bbacd9 | ||
|
|
d0ff5bff33 | ||
|
|
7d8b7266e2 | ||
|
|
252c05a72b | ||
|
|
b69c3a1360 | ||
|
|
989abeaf1b | ||
|
|
d3315593c9 | ||
|
|
f48fc39328 | ||
|
|
15cc62d8e3 | ||
|
|
d2c7a907af | ||
|
|
5e62efbe31 | ||
|
|
bf22d6f033 | ||
|
|
93bdacdbf2 | ||
|
|
0ed5559451 | ||
|
|
1fc735d727 | ||
|
|
8d90775946 | ||
|
|
f3dd715c81 | ||
|
|
fbdd1c4da1 | ||
|
|
050f86a280 | ||
|
|
49febf49cf | ||
|
|
3874e12ca0 | ||
|
|
148624ca36 | ||
|
|
cceb4ec1d2 | ||
|
|
c243577e73 | ||
|
|
91dc5761fc | ||
|
|
b7e61ae87b | ||
|
|
9ff7784a3c | ||
|
|
68eec8b8b9 | ||
|
|
5aa0920fae | ||
|
|
6d81ad804a | ||
|
|
81dca5c673 | ||
|
|
5c29fabd0f | ||
|
|
6951de03b3 | ||
|
|
f1d3423e9d | ||
|
|
82475a8b03 | ||
|
|
8c2be7bfe2 | ||
|
|
b7e157ec15 | ||
|
|
722ec69cda | ||
|
|
67b0601811 | ||
|
|
97a0c642f1 | ||
|
|
71dadc71a2 | ||
|
|
e81ecbbbba | ||
|
|
6897d20358 | ||
|
|
af09ee4b50 | ||
|
|
634bb3a11c | ||
|
|
79cc552226 | ||
|
|
f579f0ce71 | ||
|
|
682f5ef432 | ||
|
|
8015e2a228 | ||
|
|
55b9691fb9 | ||
|
|
6e089850a7 | ||
|
|
5f52cffc34 | ||
|
|
71ebd43771 | ||
|
|
c6bcb8cead | ||
|
|
84288b78d2 | ||
|
|
41d1eda574 | ||
|
|
95b3836fa2 | ||
|
|
8044e06b09 | ||
|
|
542052ceee | ||
|
|
6d3c348dea | ||
|
|
2328cd7dce | ||
|
|
cacdad3dc5 | ||
|
|
59e73f7c90 | ||
|
|
8782e2b455 | ||
|
|
220623c634 | ||
|
|
029e639dcf | ||
|
|
90d36a63f7 | ||
|
|
84928fed33 | ||
|
|
f144797366 | ||
|
|
a86273adf1 | ||
|
|
f0b447823b | ||
|
|
e4336c3f99 | ||
|
|
d55e8b514a | ||
|
|
e333fb2ae0 | ||
|
|
3dd2ee4a86 | ||
|
|
70b1410c0e | ||
|
|
067052448f | ||
|
|
a5bca8781a | ||
|
|
7d1b9ce960 | ||
|
|
311e0a4ad2 | ||
|
|
70977f4807 | ||
|
|
11c8d8d58e | ||
|
|
c15673b27e | ||
|
|
913efcf72a | ||
|
|
ad0a6c3475 | ||
|
|
cd15024c42 | ||
|
|
9906d4bb63 | ||
|
|
9e28f7bd99 | ||
|
|
3ca62aa555 | ||
|
|
8b808b0bf0 | ||
|
|
6985ab31b3 | ||
|
|
706e414e36 | ||
|
|
5ace8b0575 | ||
|
|
bc7d1a2308 | ||
|
|
1741470ddb | ||
|
|
ee9ea7c0c5 | ||
|
|
91b2421c37 | ||
|
|
06748776e3 | ||
|
|
91f51d6a8c | ||
|
|
f2e2617e43 | ||
|
|
198ffc2469 | ||
|
|
0a03a93948 | ||
|
|
501c301f4f | ||
|
|
867a519f6c | ||
|
|
d5db0c955d | ||
|
|
6a371a01a9 | ||
|
|
4d771c9db1 | ||
|
|
5e46cf57fb | ||
|
|
ed3e6ae039 | ||
|
|
4fe7d84789 | ||
|
|
fdd84bad36 | ||
|
|
f52964441a | ||
|
|
92affebded | ||
|
|
3221c2626b | ||
|
|
2ea21b0e51 | ||
|
|
76fa4befd8 | ||
|
|
583dd323dc | ||
|
|
ef2ecb4c55 | ||
|
|
6d28e09fce | ||
|
|
565586aca4 | ||
|
|
9147e2f21b | ||
|
|
9ad20e3a36 | ||
|
|
85a7f62b01 | ||
|
|
09f9e15ff0 | ||
|
|
dd62d30fa9 | ||
|
|
adfd3de651 | ||
|
|
871e97cae7 | ||
|
|
03e1aec641 | ||
|
|
40948c2ada | ||
|
|
2182bf7e58 | ||
|
|
ed723dd6d0 | ||
|
|
ddc4460065 | ||
|
|
b0f3d15112 | ||
|
|
213d3b4311 | ||
|
|
753499db01 | ||
|
|
1b6e682860 | ||
|
|
9baba12280 | ||
|
|
24b7b75ab2 | ||
|
|
0b59b7abfc | ||
|
|
942cbb4f66 | ||
|
|
96383a60d4 | ||
|
|
0cf14be531 | ||
|
|
95edd82555 | ||
|
|
6e02ca3056 | ||
|
|
966e8cc963 | ||
|
|
04b1c8fd3e | ||
|
|
f47b126c08 | ||
|
|
4438293b6d | ||
|
|
de4e92d9bf | ||
|
|
e7f88b60d0 | ||
|
|
fbb1cd7299 | ||
|
|
77820a44c9 | ||
|
|
e92ddd8609 | ||
|
|
a2bf2230ab | ||
|
|
75c6879b7e | ||
|
|
f51ef68179 | ||
|
|
8a5420f5f8 | ||
|
|
c9b7f1bb1a | ||
|
|
d9da1dd360 | ||
|
|
a8fa6b5f46 | ||
|
|
ebe0121938 | ||
|
|
01db9a764a |
No files matched your search
@@ -1,21 +0,0 @@
|
||||
---
|
||||
description: Git branching and PR workflow
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Branching Workflow
|
||||
|
||||
**Never commit directly to `main`.** Always work on a feature branch.
|
||||
|
||||
Create descriptive branch names like `feature/add-discord-notifications`, `fix/audio-sync-issue`, or `refactor/cleanup-api-routes`.
|
||||
|
||||
# Creating a Pull Request
|
||||
|
||||
Once all changes are complete, tests pass, and commits are pushed, create a PR with:
|
||||
|
||||
1. **Summary** - Brief overview of the changes
|
||||
2. **Changes Made** - List of specific modifications
|
||||
3. **Testing** - How changes were tested (unit tests, manual testing)
|
||||
4. **Related Issues** - Reference any GitHub issues (e.g., "Fixes #123")
|
||||
|
||||
Only merge after review and CI checks pass.
|
||||
@@ -1,14 +0,0 @@
|
||||
# Infrastructure Changes
|
||||
|
||||
All Google Cloud infrastructure (IAM, Cloud Storage, Cloud Run, Secret Manager, Artifact Registry, etc.) must be managed via Pulumi in the `infrastructure/` folder.
|
||||
|
||||
## Rules
|
||||
|
||||
- **Reading/querying** with `gcloud` CLI is fine
|
||||
- **Creating or modifying** resources must go through IaC — never use `gcloud` or the Console to make changes
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Make changes in `infrastructure/`
|
||||
2. Run `pulumi preview` and verify the output
|
||||
3. Run `pulumi up` to apply
|
||||
@@ -1,27 +0,0 @@
|
||||
---
|
||||
description: Version bumping rule for commits
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Version Management
|
||||
|
||||
When committing and pushing changes, always bump the patch version in `pyproject.toml`.
|
||||
|
||||
The version is located at `tool.poetry.version` in `pyproject.toml`. For example, if the current version is `0.70.3`, bump it to `0.70.4`.
|
||||
|
||||
This ensures every commit pushed to the repository has a unique version number for tracking and deployment purposes.
|
||||
|
||||
# Testing Requirements
|
||||
|
||||
Before committing and pushing changes, ensure all tests pass:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/ backend/tests/ -v -x --ignore=tests/integration
|
||||
```
|
||||
|
||||
Ensure all code touched is thoroughly covered by unit tests, we have a 70% coverage minimum in CI:
|
||||
```bash
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=70
|
||||
```
|
||||
|
||||
Do not commit or push code that fails tests. Fix any test failures before proceeding.
|
||||
+139
-24
@@ -25,6 +25,8 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
@@ -50,7 +52,7 @@ jobs:
|
||||
|
||||
- name: Run unit tests
|
||||
run: |
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=70
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=69
|
||||
|
||||
- name: Upload coverage reports
|
||||
uses: codecov/codecov-action@v5
|
||||
@@ -65,6 +67,8 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
@@ -97,6 +101,8 @@ jobs:
|
||||
needs: [package-unit-tests, package-integration-tests]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
@@ -162,7 +168,7 @@ jobs:
|
||||
- name: Install dependencies
|
||||
run: poetry install
|
||||
|
||||
- name: Run unit tests
|
||||
- name: Run all backend unit tests
|
||||
env:
|
||||
GOOGLE_CLOUD_PROJECT: test-project
|
||||
GCS_BUCKET_NAME: test-bucket
|
||||
@@ -170,10 +176,8 @@ jobs:
|
||||
ENVIRONMENT: test
|
||||
run: |
|
||||
poetry run pytest \
|
||||
backend/tests/test_models.py \
|
||||
backend/tests/test_job_manager.py \
|
||||
backend/tests/test_file_upload.py \
|
||||
backend/tests/test_services.py \
|
||||
backend/tests/ \
|
||||
--ignore=backend/tests/emulator \
|
||||
-v --tb=short
|
||||
|
||||
backend-emulator-tests:
|
||||
@@ -275,6 +279,8 @@ jobs:
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
@@ -364,21 +370,55 @@ jobs:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
|
||||
- name: Enable Yarn (for lyrics frontend)
|
||||
run: corepack enable
|
||||
|
||||
# ===== Build Main Frontend (Next.js) =====
|
||||
- name: Install main frontend dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Build main frontend (Next.js static export)
|
||||
working-directory: frontend
|
||||
env:
|
||||
NEXT_PUBLIC_API_URL: https://api.nomadkaraoke.com
|
||||
run: npm run build
|
||||
|
||||
# ===== Build Lyrics Review Frontend (Vite) =====
|
||||
- name: Install lyrics frontend dependencies
|
||||
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
|
||||
run: yarn install --frozen-lockfile
|
||||
|
||||
- name: Build lyrics frontend (Vite)
|
||||
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
|
||||
env:
|
||||
# Set base path for subpath deployment
|
||||
VITE_BASE_PATH: /lyrics/
|
||||
run: yarn build --base=/lyrics/
|
||||
|
||||
- name: Copy lyrics frontend to main output
|
||||
run: |
|
||||
mkdir -p frontend/out/lyrics
|
||||
cp -r lyrics_transcriber_temp/lyrics_transcriber/frontend/dist/* frontend/out/lyrics/
|
||||
|
||||
# ===== Copy Instrumental Review UI (Static HTML) =====
|
||||
- name: Copy instrumental review UI to main output
|
||||
run: |
|
||||
mkdir -p frontend/out/instrumental
|
||||
cp karaoke_gen/instrumental_review/static/index.html frontend/out/instrumental/
|
||||
|
||||
- name: Setup Pages
|
||||
uses: actions/configure-pages@v4
|
||||
|
||||
- name: Prepare frontend files
|
||||
run: |
|
||||
echo "📁 Preparing frontend files for deployment..."
|
||||
mkdir -p _site
|
||||
cp -r frontend/* _site/
|
||||
echo "✅ Frontend files copied to _site/"
|
||||
ls -la _site/
|
||||
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-pages-artifact@v3
|
||||
with:
|
||||
path: "_site"
|
||||
path: "frontend/out"
|
||||
|
||||
- name: Deploy to GitHub Pages
|
||||
id: deployment
|
||||
@@ -397,8 +437,34 @@ jobs:
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: false
|
||||
fetch-depth: 0 # Need full history for tag check
|
||||
|
||||
- name: Set up Python (for version extraction)
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
|
||||
- name: Extract version and check if deployment needed
|
||||
id: version_check
|
||||
run: |
|
||||
VERSION=$(poetry version --short)
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
echo "📋 Current version: $VERSION"
|
||||
|
||||
if git ls-remote --tags origin | grep -q "refs/tags/v$VERSION$"; then
|
||||
echo "needs_deploy=false" >> $GITHUB_OUTPUT
|
||||
echo "⏭️ Tag v$VERSION already exists - skipping backend deployment"
|
||||
echo " (Bump version in pyproject.toml to trigger a new deployment)"
|
||||
else
|
||||
echo "needs_deploy=true" >> $GITHUB_OUTPUT
|
||||
echo "✨ Tag v$VERSION does not exist - proceeding with deployment"
|
||||
fi
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: auth
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
@@ -406,17 +472,20 @@ jobs:
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Check if base image needs rebuild
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: check-base
|
||||
run: |
|
||||
echo "🔍 Checking if base image needs rebuild..."
|
||||
|
||||
CURRENT_HASH=$(sha256sum poetry.lock | cut -d' ' -f1)
|
||||
echo "Current poetry.lock hash: $CURRENT_HASH"
|
||||
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
|
||||
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
|
||||
echo "Current combined hash (poetry.lock + Dockerfile.base): $CURRENT_HASH"
|
||||
|
||||
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
|
||||
|
||||
@@ -424,14 +493,18 @@ jobs:
|
||||
EXISTING_HASH=""
|
||||
|
||||
if docker pull $BASE_IMAGE 2>/dev/null; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
if [ -z "$EXISTING_HASH" ]; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
else
|
||||
echo "⚠️ No existing base image found"
|
||||
fi
|
||||
|
||||
if [ "$CURRENT_HASH" != "$EXISTING_HASH" ]; then
|
||||
echo "🔄 Base image needs rebuild (poetry.lock changed)"
|
||||
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
|
||||
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "✅ Base image is up to date"
|
||||
@@ -439,7 +512,7 @@ jobs:
|
||||
fi
|
||||
|
||||
- name: Rebuild base image (if needed)
|
||||
if: steps.check-base.outputs.needs_rebuild == 'true'
|
||||
if: steps.version_check.outputs.needs_deploy == 'true' && steps.check-base.outputs.needs_rebuild == 'true'
|
||||
run: |
|
||||
echo "🏗️ Rebuilding base image with updated dependencies..."
|
||||
|
||||
@@ -479,9 +552,11 @@ jobs:
|
||||
done
|
||||
|
||||
- name: Trigger Cloud Build
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: |
|
||||
echo "🚀 Triggering Cloud Build for deployment..."
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Version: ${{ steps.version_check.outputs.version }}"
|
||||
|
||||
BUILD_ID=$(gcloud builds submit \
|
||||
--config=cloudbuild.yaml \
|
||||
@@ -519,21 +594,61 @@ jobs:
|
||||
sleep 30
|
||||
done
|
||||
|
||||
- name: Verify deployment
|
||||
- name: Verify deployment and version
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: |
|
||||
EXPECTED_VERSION="${{ steps.version_check.outputs.version }}"
|
||||
SERVICE_URL=$(gcloud run services describe karaoke-backend \
|
||||
--region us-central1 \
|
||||
--format 'value(status.url)')
|
||||
|
||||
echo "Service URL: $SERVICE_URL"
|
||||
sleep 10
|
||||
echo "Expected version: $EXPECTED_VERSION"
|
||||
|
||||
# Wait for new revision to be ready
|
||||
echo "Waiting for new revision to be ready..."
|
||||
sleep 15
|
||||
|
||||
# Basic health check
|
||||
curl -f "$SERVICE_URL/api/health" || exit 1
|
||||
echo "✅ Deployment verified - service is healthy"
|
||||
echo "✅ Service is healthy"
|
||||
|
||||
# Version verification with retries (Cloud Run may take time to route to new revision)
|
||||
MAX_RETRIES=6
|
||||
RETRY_DELAY=10
|
||||
for i in $(seq 1 $MAX_RETRIES); do
|
||||
echo "Checking deployed version (attempt $i/$MAX_RETRIES)..."
|
||||
DEPLOYED_VERSION=$(curl -sf "$SERVICE_URL/api/health/detailed" | jq -r '.version // empty')
|
||||
|
||||
if [ "$DEPLOYED_VERSION" = "$EXPECTED_VERSION" ]; then
|
||||
echo "✅ Version verified: $DEPLOYED_VERSION"
|
||||
break
|
||||
else
|
||||
echo "⚠️ Version mismatch: deployed=$DEPLOYED_VERSION, expected=$EXPECTED_VERSION"
|
||||
if [ $i -lt $MAX_RETRIES ]; then
|
||||
echo " Retrying in ${RETRY_DELAY}s..."
|
||||
sleep $RETRY_DELAY
|
||||
else
|
||||
echo "❌ Version verification failed after $MAX_RETRIES attempts"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
- name: Summary
|
||||
if: success()
|
||||
if: steps.version_check.outputs.needs_deploy == 'true' && success()
|
||||
run: |
|
||||
echo "🎉 Deployment successful!"
|
||||
echo "Version: ${{ steps.version_check.outputs.version }}"
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Branch: ${{ github.ref_name }}"
|
||||
echo "Run: ${{ github.run_number }}"
|
||||
|
||||
- name: Skipped deployment summary
|
||||
if: steps.version_check.outputs.needs_deploy == 'false'
|
||||
run: |
|
||||
echo "⏭️ Backend deployment skipped - version ${{ steps.version_check.outputs.version }} already deployed"
|
||||
echo ""
|
||||
echo "To deploy a new version:"
|
||||
echo " 1. Update version in pyproject.toml"
|
||||
echo " 2. Commit and push to main"
|
||||
@@ -41,6 +41,9 @@ frontend/.pnpm-debug.log*
|
||||
|
||||
# Testing
|
||||
frontend/coverage/
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/playwright/.cache/
|
||||
|
||||
# TypeScript
|
||||
frontend/*.tsbuildinfo
|
||||
@@ -73,8 +76,11 @@ dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
# Python lib directories (not frontend)
|
||||
lib/
|
||||
lib64/
|
||||
# But allow frontend lib
|
||||
!frontend/lib/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
@@ -217,3 +223,4 @@ cython_debug/
|
||||
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
flacfetch
|
||||
@@ -0,0 +1,131 @@
|
||||
# Project Guidelines for AI Assistants
|
||||
|
||||
This document contains guidelines and rules for AI assistants working on this codebase.
|
||||
|
||||
## Core Principles
|
||||
|
||||
**Always be testing.** Proactively evaluate, read, run, fix, and improve tests whenever you touch any part of the system—frontend, backend, infrastructure. Every code change should be thoroughly tested. Leave everything better tested than you found it.
|
||||
|
||||
**Always be documenting.** Continuously read, write, improve, update, and organize markdown documentation. Reference existing docs and keep them accurate as the codebase evolves.
|
||||
|
||||
**Build for maintainability.** Keep code SOLID, modular, and clean. Favor simplicity and clarity. Future developers (and AI agents) will thank you.
|
||||
|
||||
## Branching Workflow
|
||||
|
||||
**Never commit directly to `main`.** Always work on a feature branch.
|
||||
|
||||
Create descriptive branch names like `feature/add-discord-notifications`, `fix/audio-sync-issue`, or `refactor/cleanup-api-routes`.
|
||||
|
||||
### Using Git Worktrees
|
||||
|
||||
**For non-trivial changes** (new features, multi-file edits, refactors):
|
||||
- Create a git worktree in the parent folder before making any file changes:
|
||||
```bash
|
||||
git worktree add ../karaoke-gen-feat-xyz feature/xyz
|
||||
```
|
||||
- This keeps the main directory clean on `main` and enables parallel work on multiple features
|
||||
- Name the worktree folder descriptively, e.g., `../karaoke-gen-fix-audio-sync`
|
||||
|
||||
**For trivial changes** (typos, single-line fixes):
|
||||
- Creating a branch in the current directory is acceptable
|
||||
- Use `git checkout -b fix/typo-xyz` before making changes
|
||||
|
||||
**Cleanup:** When done with a worktree, remove it with:
|
||||
```bash
|
||||
git worktree remove ../karaoke-gen-feat-xyz
|
||||
```
|
||||
|
||||
### Creating a Pull Request
|
||||
|
||||
Once all changes are complete, tests pass, and commits are pushed, create a PR with:
|
||||
|
||||
1. **Summary** - Brief overview of the changes
|
||||
2. **Changes Made** - List of specific modifications
|
||||
3. **Testing** - How changes were tested (unit tests, manual testing)
|
||||
4. **Related Issues** - Reference any GitHub issues (e.g., "Fixes #123")
|
||||
|
||||
Only merge after review and CI checks pass.
|
||||
|
||||
## Infrastructure Changes
|
||||
|
||||
All Google Cloud infrastructure (IAM, Cloud Storage, Cloud Run, Secret Manager, Artifact Registry, etc.) must be managed via **Pulumi IaC** in the `infrastructure/` folder.
|
||||
|
||||
### Rules
|
||||
|
||||
- **Reading/querying** with `gcloud` CLI is fine for investigation and debugging
|
||||
- **Temporary changes** via `gcloud` CLI or Console are acceptable for urgent fixes or experimentation, but **must be cleaned up** and properly codified in the IaC afterward
|
||||
- **Permanent changes** must go through Pulumi — never leave `gcloud` or Console changes as the final state
|
||||
|
||||
### Workflow
|
||||
|
||||
1. Make changes in `infrastructure/`
|
||||
2. Run `pulumi preview` and verify the output
|
||||
3. Run `pulumi up` to apply
|
||||
|
||||
### Authentication Issues
|
||||
|
||||
If you encounter any authentication issues with `gcloud` or `pulumi` (e.g., expired tokens, permission errors), **stop and notify the user** so they can run the appropriate auth commands in their external terminal:
|
||||
|
||||
```bash
|
||||
gcloud auth login
|
||||
gcloud auth application-default login
|
||||
```
|
||||
|
||||
Do not attempt to work around auth issues or make assumptions about credentials.
|
||||
|
||||
## Version Management
|
||||
|
||||
When committing and pushing changes, always bump the patch version in `pyproject.toml`.
|
||||
|
||||
The version is located at `tool.poetry.version` in `pyproject.toml`. For example, if the current version is `0.70.3`, bump it to `0.70.4`.
|
||||
|
||||
This ensures every commit pushed to the repository has a unique version number for tracking and deployment purposes.
|
||||
|
||||
## Testing Requirements
|
||||
|
||||
**Before committing and pushing ANY changes, you MUST run the full test suite.**
|
||||
|
||||
### Backend Test Command
|
||||
|
||||
```bash
|
||||
make test 2>&1 | tail -n 500
|
||||
```
|
||||
|
||||
**Important for AI agents:**
|
||||
- Set a **10 minute timeout** when running tests (they can take a while with emulators)
|
||||
- Always pipe output through `tail -n 500` to limit output and preserve context window
|
||||
- The tail output will show test results and any failures at the end
|
||||
|
||||
This runs all backend tests in order and fails fast if any test fails:
|
||||
1. Unit tests (karaoke_gen package) with coverage check
|
||||
2. Backend unit tests
|
||||
3. E2E integration tests with emulators (auto-starts/stops emulators)
|
||||
|
||||
### Frontend Test Command
|
||||
|
||||
When working in the `frontend/` directory, run:
|
||||
|
||||
```bash
|
||||
cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
```
|
||||
|
||||
This runs:
|
||||
1. **Unit tests** (Jest) - Component and utility tests
|
||||
2. **E2E tests** (Playwright) - Browser-based integration tests
|
||||
|
||||
**Note:** E2E tests require `KARAOKE_ACCESS_TOKEN` environment variable to be set in `frontend/.env.local`.
|
||||
|
||||
Individual test commands:
|
||||
- `npm run test:unit` - Run unit tests only
|
||||
- `npm run test:e2e` - Run E2E tests only
|
||||
- `npm run test:ci` - Run unit tests with coverage
|
||||
|
||||
### Pre-commit Checklist
|
||||
|
||||
- [ ] `make test` passes (for backend changes)
|
||||
- [ ] `cd frontend && npm run test:all` passes (for frontend changes)
|
||||
- [ ] Version in `pyproject.toml` has been bumped
|
||||
- [ ] `coderabbit review --plain` has been run to get comprehensive analysis and suggestions for cleaner, more maintainable code. Apply the feedback to improve accessibility, structure, and best practices.
|
||||
|
||||
**Do not commit or push code that fails tests. Fix any test failures before proceeding.**
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# Makefile for karaoke-gen project
|
||||
# Run `make help` to see available commands
|
||||
|
||||
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "Available commands:"
|
||||
@echo " make test - Run all tests (unit + backend + emulator-based)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen)"
|
||||
@echo " make test-backend - Run backend unit tests only"
|
||||
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
|
||||
@echo " make test-all - Alias for 'make test'"
|
||||
@echo " make lint - Run linter checks"
|
||||
@echo " make emulators-start - Start GCP emulators for local development"
|
||||
@echo " make emulators-stop - Stop GCP emulators"
|
||||
@echo ""
|
||||
@echo "Before committing, run: make test"
|
||||
|
||||
# Run unit tests for karaoke_gen package
|
||||
test-unit:
|
||||
@echo "=== Running karaoke_gen unit tests ==="
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
|
||||
# Run backend unit tests
|
||||
test-backend:
|
||||
@echo "=== Running backend unit tests ==="
|
||||
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
|
||||
# Run E2E integration tests with emulators
|
||||
test-e2e:
|
||||
@echo "=== Running E2E integration tests with emulators ==="
|
||||
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
|
||||
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
|
||||
export STORAGE_EMULATOR_HOST=http://127.0.0.1:4443 && \
|
||||
export GOOGLE_CLOUD_PROJECT=test-project && \
|
||||
export GCS_BUCKET_NAME=test-bucket && \
|
||||
export ADMIN_TOKENS=test-admin-token && \
|
||||
export ENVIRONMENT=test && \
|
||||
poetry run pytest backend/tests/emulator/ -v; \
|
||||
TEST_RESULT=$$?; \
|
||||
./scripts/stop-emulators.sh; \
|
||||
exit $$TEST_RESULT
|
||||
|
||||
# Run all tests (use this before committing!)
|
||||
test: test-unit test-backend test-e2e
|
||||
@echo ""
|
||||
@echo "✅ All tests passed! Ready to commit."
|
||||
|
||||
# Alias for test
|
||||
test-all: test
|
||||
|
||||
# Lint checks
|
||||
lint:
|
||||
@echo "=== Running lint checks ==="
|
||||
poetry run ruff check karaoke_gen/ backend/
|
||||
poetry run ruff format --check karaoke_gen/ backend/
|
||||
|
||||
# Start emulators for local development
|
||||
emulators-start:
|
||||
@echo "=== Starting GCP emulators ==="
|
||||
./scripts/start-emulators.sh
|
||||
|
||||
# Stop emulators
|
||||
emulators-stop:
|
||||
@echo "=== Stopping GCP emulators ==="
|
||||
./scripts/stop-emulators.sh
|
||||
|
||||
# Clean up temporary files
|
||||
clean:
|
||||
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name ".pytest_cache" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name ".coverage" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type f -name "*.pyc" -delete 2>/dev/null || true
|
||||
rm -rf htmlcov/ .coverage coverage.xml 2>/dev/null || true
|
||||
@echo "Cleaned up temporary files"
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Karaoke Generator 🎶 🎥 🚀
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||

|
||||
|
||||
@@ -54,10 +54,44 @@ pip install karaoke-gen
|
||||
This installs both `karaoke-gen` (local) and `karaoke-gen-remote` (cloud) CLIs.
|
||||
|
||||
### Requirements
|
||||
- Python 3.10+
|
||||
- Python 3.10-3.13
|
||||
- FFmpeg
|
||||
- For local processing: CUDA-capable GPU or Apple Silicon CPU recommended
|
||||
|
||||
### Transcription Provider Setup
|
||||
|
||||
**Transcription is required** for creating karaoke videos with synchronized lyrics. The system needs word-level timing data to display lyrics in sync with the music.
|
||||
|
||||
#### Option 1: AudioShake (Recommended)
|
||||
Commercial service with high-quality transcription. Best for production use.
|
||||
|
||||
```bash
|
||||
export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
|
||||
```
|
||||
|
||||
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
|
||||
|
||||
#### Option 2: Whisper via RunPod
|
||||
Open-source alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
|
||||
```bash
|
||||
export RUNPOD_API_KEY="your_runpod_key"
|
||||
export WHISPER_RUNPOD_ID="your_whisper_endpoint_id"
|
||||
```
|
||||
|
||||
Set up a Whisper endpoint at [https://www.runpod.io/](https://www.runpod.io/)
|
||||
|
||||
#### Without Transcription (Instrumental Only)
|
||||
If you don't need synchronized lyrics, use the `--skip-lyrics` flag:
|
||||
|
||||
```bash
|
||||
karaoke-gen --skip-lyrics "Artist" "Title"
|
||||
```
|
||||
|
||||
This creates an instrumental-only karaoke video without lyrics overlay.
|
||||
|
||||
> **Note:** See `lyrics_transcriber_temp/README.md` for detailed transcription provider configuration options.
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ Local CLI (`karaoke-gen`)
|
||||
@@ -180,9 +214,11 @@ karaoke-gen-remote \
|
||||
|----------|-------------|---------|
|
||||
| `KARAOKE_GEN_URL` | Backend service URL | Required |
|
||||
| `KARAOKE_GEN_AUTH_TOKEN` | Admin auth token (for protected endpoints) | Optional |
|
||||
| `REVIEW_UI_URL` | Lyrics review UI URL | `https://lyrics.nomadkaraoke.com` |
|
||||
| `REVIEW_UI_URL` | Lyrics review UI URL | `https://gen.nomadkaraoke.com/lyrics/` |
|
||||
| `POLL_INTERVAL` | Seconds between status polls | `5` |
|
||||
|
||||
**Note:** The `REVIEW_UI_URL` defaults to the hosted lyrics review UI. For local development, set it to `http://localhost:5173` if you're running the frontend dev server.
|
||||
|
||||
### Authentication
|
||||
|
||||
The backend uses token-based authentication for admin operations (bulk delete, internal worker triggers). For basic job submission and monitoring, authentication is optional.
|
||||
@@ -473,6 +509,73 @@ Check backend health status.
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### "No suitable files found for processing"
|
||||
|
||||
This error occurs during the finalisation step when the `(With Vocals).mkv` file is missing. This file is created during lyrics transcription.
|
||||
|
||||
**Most common cause:** No transcription provider configured.
|
||||
|
||||
**Quick fix:**
|
||||
1. Check if transcription providers are configured:
|
||||
```bash
|
||||
echo $AUDIOSHAKE_API_TOKEN
|
||||
echo $RUNPOD_API_KEY
|
||||
```
|
||||
|
||||
2. If both are empty, set up a provider (see [Transcription Provider Setup](#transcription-provider-setup))
|
||||
|
||||
3. Or use `--skip-lyrics` for instrumental-only karaoke:
|
||||
```bash
|
||||
karaoke-gen --skip-lyrics "Artist" "Title"
|
||||
```
|
||||
|
||||
**Other causes:**
|
||||
- Invalid API credentials - verify your tokens are correct and active
|
||||
- API service unavailable - check service status pages
|
||||
- Network connectivity issues - ensure you can reach the API endpoints
|
||||
- Transcription timeout - try again or use a different provider
|
||||
|
||||
### Transcription Fails Silently
|
||||
|
||||
If karaoke-gen runs without errors but produces no synchronized lyrics:
|
||||
|
||||
1. **Check logs** - Run with `--log_level debug` for detailed output:
|
||||
```bash
|
||||
karaoke-gen --log_level debug "Artist" "Title"
|
||||
```
|
||||
|
||||
2. **Verify environment variables** - Ensure API tokens are exported in your shell:
|
||||
```bash
|
||||
# Check if set
|
||||
printenv | grep -E "(AUDIOSHAKE|RUNPOD|WHISPER)"
|
||||
|
||||
# Set in current session
|
||||
export AUDIOSHAKE_API_TOKEN="your_token"
|
||||
```
|
||||
|
||||
3. **Test API connectivity** - Verify you can reach the transcription service
|
||||
|
||||
### "No lyrics found from any source"
|
||||
|
||||
This warning means no reference lyrics were fetched from online sources (Genius, Spotify, Musixmatch). The transcription will still work, but auto-correction may be less accurate.
|
||||
|
||||
**To fix:**
|
||||
- Set `GENIUS_API_TOKEN` for Genius lyrics
|
||||
- Set `SPOTIFY_COOKIE_SP_DC` for Spotify lyrics
|
||||
- Set `RAPIDAPI_KEY` for Musixmatch lyrics
|
||||
- Or provide lyrics manually with `--lyrics_file /path/to/lyrics.txt`
|
||||
|
||||
### Video Quality Issues
|
||||
|
||||
If the output video has quality problems:
|
||||
- Ensure FFmpeg is properly installed: `ffmpeg -version`
|
||||
- Check available codecs: `ffmpeg -codecs`
|
||||
- For 4K output, ensure sufficient disk space (10GB+ per track)
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Development
|
||||
|
||||
### Running Tests
|
||||
|
||||
+8
-3
@@ -11,7 +11,9 @@
|
||||
# gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
|
||||
|
||||
# Use pre-built base image with all dependencies
|
||||
FROM us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest
|
||||
# Override with --build-arg BASE_IMAGE=karaoke-backend-base:local for local testing
|
||||
ARG BASE_IMAGE=us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
# Set working directory (should already exist from base)
|
||||
WORKDIR /app
|
||||
@@ -34,5 +36,8 @@ ENV PORT=8080
|
||||
# Expose port
|
||||
EXPOSE 8080
|
||||
|
||||
# Run the application
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8080"]
|
||||
# Run uvicorn directly
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
# Using --loop asyncio (instead of uvloop) to support nest_asyncio for remote flacfetch calls
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8080", "--loop", "asyncio"]
|
||||
@@ -7,12 +7,15 @@
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Install system dependencies required by karaoke-gen
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
build-essential \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Set working directory
|
||||
@@ -34,9 +37,9 @@ RUN python -m spacy download en_core_web_sm
|
||||
# Label with build info
|
||||
ARG BUILD_DATE
|
||||
ARG BUILD_ID
|
||||
ARG POETRY_LOCK_HASH
|
||||
ARG BASE_CONTENT_HASH
|
||||
LABEL org.opencontainers.image.created="${BUILD_DATE}"
|
||||
LABEL org.opencontainers.image.revision="${BUILD_ID}"
|
||||
LABEL org.opencontainers.image.title="karaoke-backend-base"
|
||||
LABEL org.opencontainers.image.description="Base image with Python dependencies for karaoke-backend"
|
||||
LABEL poetry.lock.hash="${POETRY_LOCK_HASH}"
|
||||
LABEL org.opencontainers.image.description="Base image with Python dependencies and system tools for karaoke-backend"
|
||||
LABEL base.content.hash="${BASE_CONTENT_HASH}"
|
||||
+214
-7
@@ -2,8 +2,10 @@
|
||||
FastAPI dependencies for authentication and authorization.
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional, Tuple
|
||||
from fastapi import Depends, HTTPException, Header, Query, Request
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, UTC
|
||||
from typing import Optional, Tuple, Callable
|
||||
from fastapi import Depends, HTTPException, Header, Query, Request, Path
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
from backend.services.auth_service import get_auth_service, UserType, AuthService
|
||||
@@ -12,6 +14,16 @@ from backend.services.auth_service import get_auth_service, UserType, AuthServic
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def generate_review_token() -> str:
|
||||
"""Generate a cryptographically secure review token."""
|
||||
return secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
def get_review_token_expiry(hours: int = 24) -> datetime:
|
||||
"""Get expiry time for a review token."""
|
||||
return datetime.now(UTC) + timedelta(hours=hours)
|
||||
|
||||
|
||||
# HTTP Bearer security scheme (Authorization: Bearer <token>)
|
||||
security = HTTPBearer(auto_error=False)
|
||||
|
||||
@@ -24,14 +36,21 @@ async def get_token_from_request(
|
||||
"""
|
||||
Extract access token from request.
|
||||
|
||||
Supports two methods:
|
||||
1. Authorization header: Authorization: Bearer <token>
|
||||
2. Query parameter: ?token=<token> (for download links)
|
||||
Supports three methods (in priority order):
|
||||
1. X-Admin-Token header: Used by Cloud Tasks (since OIDC overwrites Authorization)
|
||||
2. Authorization header: Authorization: Bearer <token>
|
||||
3. Query parameter: ?token=<token> (for download links)
|
||||
|
||||
Returns:
|
||||
Token string or None
|
||||
"""
|
||||
# Try Authorization header first
|
||||
# Check X-Admin-Token header first (used by Cloud Tasks)
|
||||
# Cloud Tasks OIDC token overwrites Authorization header, so we use a custom header
|
||||
admin_token = request.headers.get("X-Admin-Token")
|
||||
if admin_token:
|
||||
return admin_token
|
||||
|
||||
# Try Authorization header
|
||||
if credentials:
|
||||
return credentials.credentials
|
||||
|
||||
@@ -70,7 +89,15 @@ async def require_auth(
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
logger.warning(f"Authentication failed: {message}")
|
||||
# Log more details for debugging token issues
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
logger.warning(
|
||||
f"Authentication failed: {message}. "
|
||||
f"Token provided: {bool(token_str)}, "
|
||||
f"Token length: {len(token_str) if token_str else 0}, "
|
||||
f"Auth header present: {bool(auth_header)}, "
|
||||
f"Header prefix: {auth_header[:20] if auth_header else 'none'}..."
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {message}"
|
||||
@@ -136,3 +163,183 @@ def optional_auth(
|
||||
return None
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create a review authentication dependency.
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token)
|
||||
2. Job-specific review token (only valid for the specific job)
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
Returns:
|
||||
Dependency function that validates review access
|
||||
"""
|
||||
async def require_review_auth(
|
||||
request: Request,
|
||||
job_id: str = Path(...),
|
||||
review_token: Optional[str] = Query(None, description="Job-specific review token"),
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, alias="token", description="Full access token")
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate review access for a job.
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "review_token"
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
if full_token:
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
|
||||
if is_valid:
|
||||
logger.info(f"Review access granted via full auth ({user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
# Try review token
|
||||
if review_token:
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Validate review token matches
|
||||
if job.review_token and secrets.compare_digest(job.review_token, review_token):
|
||||
# Check expiry if set
|
||||
if job.review_token_expires_at:
|
||||
now = datetime.now(UTC)
|
||||
# Handle timezone-naive datetimes from Firestore
|
||||
expiry = job.review_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Review token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
logger.info(f"Review access granted via review_token for job {job_id}")
|
||||
return job_id, "review_token"
|
||||
else:
|
||||
logger.warning(f"Invalid review token for job {job_id}")
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid review_token for this job."
|
||||
)
|
||||
|
||||
return require_review_auth
|
||||
|
||||
|
||||
# Default instance for most review endpoints
|
||||
require_review_auth = require_review_auth_factory()
|
||||
|
||||
|
||||
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create an instrumental review authentication dependency.
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token)
|
||||
2. Job-specific instrumental token (only valid for the specific job)
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
Returns:
|
||||
Dependency function that validates instrumental review access
|
||||
"""
|
||||
async def require_instrumental_auth(
|
||||
request: Request,
|
||||
job_id: str = Path(...),
|
||||
instrumental_token: Optional[str] = Query(None, description="Job-specific instrumental review token"),
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, alias="token", description="Full access token")
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate instrumental review access for a job.
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
if full_token:
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
|
||||
if is_valid:
|
||||
logger.info(f"Instrumental access granted via full auth ({user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
# Try instrumental token
|
||||
if instrumental_token:
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Validate instrumental token matches
|
||||
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
|
||||
# Check expiry if set
|
||||
if job.instrumental_token_expires_at:
|
||||
now = datetime.now(UTC)
|
||||
# Handle timezone-naive datetimes from Firestore
|
||||
expiry = job.instrumental_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Instrumental token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Instrumental review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
|
||||
return job_id, "instrumental_token"
|
||||
else:
|
||||
logger.warning(f"Invalid instrumental token for job {job_id}")
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
|
||||
)
|
||||
|
||||
return require_instrumental_auth
|
||||
|
||||
|
||||
# Default instance for most instrumental review endpoints
|
||||
require_instrumental_auth = require_instrumental_auth_factory()
|
||||
@@ -15,9 +15,9 @@ import asyncio
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import Optional, List, Dict, Any
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.models.job import JobCreate, JobStatus
|
||||
@@ -34,8 +34,23 @@ from backend.services.audio_search_service import (
|
||||
)
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Valid style file types (from file_upload.py)
|
||||
STYLE_FILE_TYPES = {
|
||||
'style_params': {'.json'},
|
||||
'style_intro_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_karaoke_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_end_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_font': {'.ttf', '.otf'},
|
||||
'style_cdg_instrumental_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_cdg_title_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_cdg_outro_background': {'.jpg', '.jpeg', '.png'},
|
||||
}
|
||||
router = APIRouter(tags=["audio-search"])
|
||||
|
||||
# Initialize services
|
||||
@@ -48,6 +63,20 @@ worker_service = get_worker_service()
|
||||
# Pydantic models
|
||||
# ============================================================================
|
||||
|
||||
class StyleFileRequest(BaseModel):
|
||||
"""Information about a style file to be uploaded."""
|
||||
filename: str = Field(..., description="Original filename with extension")
|
||||
content_type: str = Field(..., description="MIME type of the file")
|
||||
file_type: str = Field(..., description="Type of file: 'style_params', 'style_intro_background', 'style_karaoke_background', 'style_end_background', 'style_font', 'style_cdg_instrumental_background', 'style_cdg_title_background', 'style_cdg_outro_background'")
|
||||
|
||||
|
||||
class StyleUploadUrl(BaseModel):
|
||||
"""Signed URL for uploading a style file."""
|
||||
file_type: str = Field(..., description="Type of file being uploaded")
|
||||
gcs_path: str = Field(..., description="Destination path in GCS")
|
||||
upload_url: str = Field(..., description="Signed URL to PUT the file to")
|
||||
|
||||
|
||||
class AudioSearchRequest(BaseModel):
|
||||
"""Request to search for audio by artist and title."""
|
||||
artist: str = Field(..., description="Artist name to search for")
|
||||
@@ -79,18 +108,47 @@ class AudioSearchRequest(BaseModel):
|
||||
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
|
||||
backing_vocals_models: Optional[List[str]] = Field(None, description="Models for backing vocals separation")
|
||||
other_stems_models: Optional[List[str]] = Field(None, description="Models for other stems")
|
||||
|
||||
# Style file uploads (optional)
|
||||
# Style params JSON and related assets (background images, fonts)
|
||||
style_files: Optional[List[StyleFileRequest]] = Field(None, description="Style files to upload (style_params JSON, backgrounds, fonts)")
|
||||
|
||||
|
||||
class AudioSearchResultResponse(BaseModel):
|
||||
"""A single audio search result."""
|
||||
"""A single audio search result.
|
||||
|
||||
Contains all fields needed for rich display using flacfetch's
|
||||
shared formatting functions.
|
||||
"""
|
||||
index: int
|
||||
title: str
|
||||
artist: str
|
||||
provider: str
|
||||
url: str
|
||||
duration: Optional[int] = None
|
||||
quality: Optional[str] = None
|
||||
source_id: Optional[str] = None
|
||||
provider: str # Maps to source_name in Release
|
||||
url: Optional[str] = None # Maps to download_url in Release (may be None for remote)
|
||||
duration: Optional[int] = None # Maps to duration_seconds in Release
|
||||
quality: Optional[str] = None # Stringified quality
|
||||
source_id: Optional[str] = None # Maps to info_hash in Release
|
||||
seeders: Optional[int] = None
|
||||
target_file: Optional[str] = None
|
||||
# Additional fields for rich display (from Release.to_dict())
|
||||
year: Optional[int] = None
|
||||
label: Optional[str] = None
|
||||
edition_info: Optional[str] = None
|
||||
release_type: Optional[str] = None
|
||||
channel: Optional[str] = None # For YouTube
|
||||
view_count: Optional[int] = None # For YouTube
|
||||
size_bytes: Optional[int] = None
|
||||
target_file_size: Optional[int] = None
|
||||
track_pattern: Optional[str] = None
|
||||
match_score: Optional[float] = None
|
||||
# Pre-computed display fields
|
||||
formatted_size: Optional[str] = None
|
||||
formatted_duration: Optional[str] = None
|
||||
formatted_views: Optional[str] = None
|
||||
is_lossless: Optional[bool] = None
|
||||
quality_str: Optional[str] = None
|
||||
# Full quality object for Release.from_dict()
|
||||
quality_data: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class AudioSearchResponse(BaseModel):
|
||||
@@ -102,6 +160,8 @@ class AudioSearchResponse(BaseModel):
|
||||
results_count: int = 0
|
||||
auto_download: bool = False
|
||||
server_version: str
|
||||
# Style file upload URLs (when style_files are specified in request)
|
||||
style_upload_urls: Optional[List[StyleUploadUrl]] = None
|
||||
|
||||
|
||||
class AudioSelectRequest(BaseModel):
|
||||
@@ -168,6 +228,10 @@ async def _download_and_start_processing(
|
||||
This is called either:
|
||||
- Immediately after search if auto_download=True
|
||||
- When user calls the select endpoint
|
||||
|
||||
For remote flacfetch downloads (torrent sources), the file is downloaded
|
||||
on the flacfetch VM and uploaded directly to GCS. For local downloads
|
||||
(YouTube), the file is downloaded locally and then uploaded to GCS.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
@@ -177,14 +241,17 @@ async def _download_and_start_processing(
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
if not search_results:
|
||||
raise HTTPException(status_code=400, detail="No search results available")
|
||||
|
||||
|
||||
if selection_index < 0 or selection_index >= len(search_results):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid selection index {selection_index}. Valid range: 0-{len(search_results)-1}"
|
||||
)
|
||||
|
||||
|
||||
selected = search_results[selection_index]
|
||||
|
||||
# Get remote_search_id from state_data (stored during initial search)
|
||||
remote_search_id = job.state_data.get('remote_search_id')
|
||||
|
||||
# Transition to downloading state
|
||||
job_manager.transition_to_state(
|
||||
@@ -199,33 +266,106 @@ async def _download_and_start_processing(
|
||||
)
|
||||
|
||||
try:
|
||||
# Download to temp directory
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir=temp_dir,
|
||||
)
|
||||
|
||||
# Upload to GCS
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(
|
||||
f,
|
||||
audio_gcs_path,
|
||||
content_type='audio/flac' # flacfetch typically returns FLAC
|
||||
)
|
||||
|
||||
logger.info(f"Uploaded audio to GCS: {audio_gcs_path}")
|
||||
|
||||
# Clean up temp file
|
||||
try:
|
||||
os.remove(result.filepath)
|
||||
os.rmdir(temp_dir)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to clean up temp files: {e}")
|
||||
# Determine if this is a remote torrent download
|
||||
is_torrent_source = selected.get('provider') in ['RED', 'OPS']
|
||||
is_remote_enabled = audio_search_service.is_remote_enabled()
|
||||
|
||||
# Check if we can use download_by_id (preferred - avoids re-searching)
|
||||
source_id = selected.get('source_id')
|
||||
source_name = selected.get('provider')
|
||||
target_file = selected.get('target_file')
|
||||
download_url = selected.get('url')
|
||||
|
||||
# For remote torrent downloads, have flacfetch VM upload directly to GCS
|
||||
if is_torrent_source and is_remote_enabled:
|
||||
# Generate GCS path for remote upload
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
|
||||
# Use download_by_id if we have source_id (preferred - no re-search needed)
|
||||
if source_id and source_name:
|
||||
logger.info(f"Using download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="", # Not used for remote
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination,
|
||||
)
|
||||
else:
|
||||
# Fallback to search-based download (requires re-search)
|
||||
logger.info(f"No source_id available, falling back to search-based download to: {gcs_destination}")
|
||||
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir="", # Not used for remote
|
||||
gcs_path=gcs_destination,
|
||||
remote_search_id=remote_search_id,
|
||||
)
|
||||
|
||||
# For remote downloads, filepath is already the GCS path
|
||||
if result.filepath.startswith("gs://"):
|
||||
# Extract the path portion after the bucket name
|
||||
# Format: gs://bucket/uploads/job_id/audio/filename.flac
|
||||
parts = result.filepath.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
audio_gcs_path = parts[1]
|
||||
else:
|
||||
audio_gcs_path = result.filepath
|
||||
filename = os.path.basename(result.filepath)
|
||||
else:
|
||||
# Fallback: treat as local path (shouldn't happen for remote)
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
|
||||
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
|
||||
|
||||
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
|
||||
else:
|
||||
# Local download (YouTube or fallback)
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
|
||||
# Use download_by_id if we have source_id and remote is enabled
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=temp_dir,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
)
|
||||
else:
|
||||
# Fallback to search-based download
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir=temp_dir,
|
||||
remote_search_id=remote_search_id, # Pass for potential fallback scenarios
|
||||
)
|
||||
|
||||
# Upload to GCS
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(
|
||||
f,
|
||||
audio_gcs_path,
|
||||
content_type='audio/flac' # flacfetch typically returns FLAC
|
||||
)
|
||||
|
||||
logger.info(f"Uploaded audio to GCS: {audio_gcs_path}")
|
||||
|
||||
# Clean up temp file
|
||||
try:
|
||||
os.remove(result.filepath)
|
||||
os.rmdir(temp_dir)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to clean up temp files: {e}")
|
||||
|
||||
# Update job with GCS path and transition to DOWNLOADING
|
||||
job_manager.update_job(job_id, {
|
||||
@@ -263,6 +403,7 @@ async def search_audio(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSearchRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Search for audio by artist and title, creating a new job.
|
||||
@@ -325,6 +466,7 @@ async def search_audio(
|
||||
brand_prefix=body.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
youtube_description_template=body.youtube_description, # video_worker reads this field
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
@@ -351,6 +493,66 @@ async def search_audio(
|
||||
'auto_download': body.auto_download,
|
||||
})
|
||||
|
||||
# Handle style file uploads if provided
|
||||
style_upload_urls: List[StyleUploadUrl] = []
|
||||
style_assets = {}
|
||||
|
||||
if body.style_files:
|
||||
logger.info(f"Processing {len(body.style_files)} style file uploads for job {job_id}")
|
||||
|
||||
for file_info in body.style_files:
|
||||
# Validate file type
|
||||
if file_info.file_type not in STYLE_FILE_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file type '{file_info.file_type}'. Must be one of: {', '.join(STYLE_FILE_TYPES.keys())}"
|
||||
)
|
||||
|
||||
# Validate extension
|
||||
ext = Path(file_info.filename).suffix.lower()
|
||||
allowed_exts = STYLE_FILE_TYPES[file_info.file_type]
|
||||
if ext not in allowed_exts:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid extension '{ext}' for {file_info.file_type}. Allowed: {', '.join(allowed_exts)}"
|
||||
)
|
||||
|
||||
# Generate GCS path
|
||||
if file_info.file_type == 'style_params':
|
||||
gcs_path = f"uploads/{job_id}/style/style_params.json"
|
||||
else:
|
||||
# style_intro_background -> intro_background, etc.
|
||||
asset_key = file_info.file_type.replace('style_', '')
|
||||
gcs_path = f"uploads/{job_id}/style/{asset_key}{ext}"
|
||||
|
||||
# Generate signed upload URL
|
||||
signed_url = storage_service.generate_signed_upload_url(
|
||||
gcs_path,
|
||||
content_type=file_info.content_type,
|
||||
expiration_minutes=60
|
||||
)
|
||||
|
||||
style_upload_urls.append(StyleUploadUrl(
|
||||
file_type=file_info.file_type,
|
||||
gcs_path=gcs_path,
|
||||
upload_url=signed_url
|
||||
))
|
||||
|
||||
# Track the expected asset paths
|
||||
if file_info.file_type == 'style_params':
|
||||
style_assets['style_params'] = gcs_path
|
||||
else:
|
||||
asset_key = file_info.file_type.replace('style_', '')
|
||||
style_assets[asset_key] = gcs_path
|
||||
|
||||
# Update job with style asset expectations
|
||||
style_update = {'style_assets': style_assets}
|
||||
if 'style_params' in style_assets:
|
||||
style_update['style_params_gcs_path'] = style_assets['style_params']
|
||||
job_manager.update_job(job_id, style_update)
|
||||
|
||||
logger.info(f"Generated {len(style_upload_urls)} style upload URLs for job {job_id}")
|
||||
|
||||
# Transition to searching state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -378,13 +580,17 @@ async def search_audio(
|
||||
job_manager.fail_job(job_id, f"Audio search failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Search failed: {e}")
|
||||
|
||||
# Store results in job state_data
|
||||
# Store results in job state_data, including remote_search_id if available
|
||||
results_dicts = [r.to_dict() for r in search_results]
|
||||
state_data_update = {
|
||||
'audio_search_results': results_dicts,
|
||||
'audio_search_count': len(results_dicts),
|
||||
}
|
||||
# Store remote_search_id for use during download (important for concurrent requests)
|
||||
if audio_search_service.last_remote_search_id:
|
||||
state_data_update['remote_search_id'] = audio_search_service.last_remote_search_id
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': {
|
||||
'audio_search_results': results_dicts,
|
||||
'audio_search_count': len(results_dicts),
|
||||
}
|
||||
'state_data': state_data_update
|
||||
})
|
||||
|
||||
# If auto_download, select best and start processing
|
||||
@@ -408,6 +614,7 @@ async def search_audio(
|
||||
results_count=len(search_results),
|
||||
auto_download=True,
|
||||
server_version=VERSION,
|
||||
style_upload_urls=style_upload_urls if style_upload_urls else None,
|
||||
)
|
||||
|
||||
# Interactive mode: return results for user selection
|
||||
@@ -418,20 +625,59 @@ async def search_audio(
|
||||
message=f"Found {len(search_results)} audio sources. Waiting for selection."
|
||||
)
|
||||
|
||||
# Convert to response format
|
||||
result_responses = [
|
||||
AudioSearchResultResponse(
|
||||
index=r.index,
|
||||
title=r.title,
|
||||
artist=r.artist,
|
||||
provider=r.provider,
|
||||
url=r.url,
|
||||
duration=r.duration,
|
||||
quality=r.quality,
|
||||
source_id=r.source_id,
|
||||
# Convert to response format with full Release data for rich display
|
||||
result_responses = []
|
||||
for r in search_results:
|
||||
# Get full serialized data from raw_result if available
|
||||
# raw_result can be either:
|
||||
# - A dict (from remote flacfetch API)
|
||||
# - A Release object (from local flacfetch)
|
||||
raw_dict = {}
|
||||
if r.raw_result:
|
||||
if isinstance(r.raw_result, dict):
|
||||
# Remote flacfetch API returns dicts directly
|
||||
raw_dict = r.raw_result
|
||||
else:
|
||||
# Local flacfetch returns Release objects
|
||||
try:
|
||||
raw_dict = r.raw_result.to_dict()
|
||||
except AttributeError:
|
||||
pass # Not a Release object
|
||||
|
||||
result_responses.append(
|
||||
AudioSearchResultResponse(
|
||||
index=r.index,
|
||||
title=r.title,
|
||||
artist=r.artist,
|
||||
provider=r.provider,
|
||||
url=r.url,
|
||||
duration=r.duration,
|
||||
quality=r.quality,
|
||||
source_id=r.source_id,
|
||||
seeders=raw_dict.get('seeders') or r.seeders,
|
||||
target_file=raw_dict.get('target_file') or r.target_file,
|
||||
# Additional fields for rich display
|
||||
year=raw_dict.get('year'),
|
||||
label=raw_dict.get('label'),
|
||||
edition_info=raw_dict.get('edition_info'),
|
||||
release_type=raw_dict.get('release_type'),
|
||||
channel=raw_dict.get('channel'),
|
||||
view_count=raw_dict.get('view_count'),
|
||||
size_bytes=raw_dict.get('size_bytes'),
|
||||
target_file_size=raw_dict.get('target_file_size'),
|
||||
track_pattern=raw_dict.get('track_pattern'),
|
||||
match_score=raw_dict.get('match_score'),
|
||||
# Pre-computed display fields
|
||||
formatted_size=raw_dict.get('formatted_size'),
|
||||
formatted_duration=raw_dict.get('formatted_duration'),
|
||||
formatted_views=raw_dict.get('formatted_views'),
|
||||
is_lossless=raw_dict.get('is_lossless'),
|
||||
quality_str=raw_dict.get('quality_str'),
|
||||
# Full quality object for Release.from_dict()
|
||||
# Remote API uses 'quality_data', local uses 'quality' for the dict
|
||||
quality_data=raw_dict.get('quality_data') or raw_dict.get('quality'),
|
||||
)
|
||||
)
|
||||
for r in search_results
|
||||
]
|
||||
|
||||
return AudioSearchResponse(
|
||||
status="awaiting_selection",
|
||||
@@ -441,6 +687,7 @@ async def search_audio(
|
||||
results_count=len(search_results),
|
||||
auto_download=False,
|
||||
server_version=VERSION,
|
||||
style_upload_urls=style_upload_urls if style_upload_urls else None,
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
@@ -451,7 +698,10 @@ async def search_audio(
|
||||
|
||||
|
||||
@router.get("/audio-search/{job_id}/results")
|
||||
async def get_audio_search_results(job_id: str):
|
||||
async def get_audio_search_results(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Get audio search results for a job.
|
||||
|
||||
@@ -473,6 +723,8 @@ async def get_audio_search_results(job_id: str):
|
||||
"status": "success",
|
||||
"job_id": job_id,
|
||||
"job_status": job.status,
|
||||
"artist": job.audio_search_artist or job.artist,
|
||||
"title": job.audio_search_title or job.title,
|
||||
"results": search_results,
|
||||
"results_count": len(search_results),
|
||||
}
|
||||
@@ -483,6 +735,7 @@ async def select_audio_source(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSelectRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Select an audio source and start job processing.
|
||||
@@ -504,25 +757,16 @@ async def select_audio_source(
|
||||
detail=f"Job is not awaiting audio selection (status: {job.status})"
|
||||
)
|
||||
|
||||
# Re-instantiate the audio search service (it caches raw results from search)
|
||||
# For production, we'd need to re-search or store raw results differently
|
||||
# Get search service instance
|
||||
# Note: With download_by_id, we no longer need to re-search to populate the cache.
|
||||
# The source_id stored in job.state_data['audio_search_results'] is sufficient.
|
||||
audio_search_service = get_audio_search_service()
|
||||
|
||||
# Re-run search to populate the cache
|
||||
# This is necessary because the service caches raw results in memory
|
||||
|
||||
# Validate search results exist in job state_data
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
if not search_results:
|
||||
raise HTTPException(status_code=400, detail="No search results cached for this job")
|
||||
|
||||
artist = job.audio_search_artist or job.artist
|
||||
title = job.audio_search_title or job.title
|
||||
|
||||
try:
|
||||
# Re-search to populate cache
|
||||
audio_search_service.search(artist, title)
|
||||
except Exception as e:
|
||||
logger.warning(f"Re-search failed, trying direct download: {e}")
|
||||
|
||||
selection_info = await _download_and_start_processing(
|
||||
job_id=job_id,
|
||||
selection_index=body.selection_index,
|
||||
|
||||
@@ -15,7 +15,7 @@ import json
|
||||
import logging
|
||||
import tempfile
|
||||
import os
|
||||
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Request, Body
|
||||
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Request, Body, Depends
|
||||
from pathlib import Path
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
|
||||
@@ -28,6 +28,9 @@ from backend.services.worker_service import get_worker_service
|
||||
from backend.services.credential_manager import get_credential_manager, CredentialStatus
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.services.metrics import metrics
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["jobs"])
|
||||
@@ -237,6 +240,7 @@ async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
async def upload_and_create_job(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth),
|
||||
# Required fields
|
||||
file: UploadFile = File(..., description="Audio file to process"),
|
||||
artist: str = Form(..., description="Artist name"),
|
||||
@@ -428,6 +432,9 @@ async def upload_and_create_job(
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job_id, source="upload")
|
||||
|
||||
logger.info(f"Created job {job_id} for {artist} - {title}")
|
||||
|
||||
# Upload main audio file to GCS
|
||||
@@ -770,6 +777,7 @@ def _get_gcs_path_for_file(job_id: str, file_type: str, filename: str) -> str:
|
||||
async def create_job_with_upload_urls(
|
||||
request: Request,
|
||||
body: CreateJobWithUploadUrlsRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Create a karaoke generation job and return signed URLs for direct file upload to GCS.
|
||||
@@ -882,6 +890,9 @@ async def create_job_with_upload_urls(
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job_id, source="upload")
|
||||
|
||||
logger.info(f"Created job {job_id} for {body.artist} - {body.title} (signed URL upload flow)")
|
||||
|
||||
# Generate signed upload URLs for each file
|
||||
@@ -925,6 +936,7 @@ async def mark_uploads_complete(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: UploadsCompleteRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Mark file uploads as complete and start job processing.
|
||||
@@ -1170,6 +1182,7 @@ async def create_job_from_url(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: CreateJobFromUrlRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Create a karaoke generation job from a YouTube or other online video URL.
|
||||
@@ -1265,6 +1278,9 @@ async def create_job_from_url(
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job_id, source="url")
|
||||
|
||||
logger.info(f"Created URL-based job {job_id} for URL: {body.url}")
|
||||
if artist:
|
||||
logger.info(f" Artist: {artist}")
|
||||
@@ -1344,6 +1360,7 @@ def _get_gcs_path_for_finalise_file(job_id: str, file_type: str, filename: str)
|
||||
async def create_finalise_only_job(
|
||||
request: Request,
|
||||
body: CreateFinaliseOnlyJobRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Create a finalise-only job for continuing from a local prep phase.
|
||||
@@ -1466,6 +1483,9 @@ async def create_finalise_only_job(
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job_id, source="finalise")
|
||||
|
||||
logger.info(f"Created finalise-only job {job_id} for {body.artist} - {body.title}")
|
||||
|
||||
# Generate signed upload URLs for each file
|
||||
@@ -1509,6 +1529,7 @@ async def mark_finalise_uploads_complete(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: UploadsCompleteRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Mark finalise-only file uploads as complete and start video generation.
|
||||
|
||||
@@ -1,10 +1,64 @@
|
||||
"""
|
||||
Health check routes.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
from fastapi import APIRouter
|
||||
from typing import Dict
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
from backend.version import VERSION
|
||||
from backend.services.flacfetch_client import get_flacfetch_client
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def check_transmission_status() -> Dict[str, Any]:
|
||||
"""Check if Transmission daemon is available and responsive."""
|
||||
try:
|
||||
import transmission_rpc
|
||||
host = os.environ.get("TRANSMISSION_HOST", "localhost")
|
||||
port = int(os.environ.get("TRANSMISSION_PORT", "9091"))
|
||||
|
||||
client = transmission_rpc.Client(host=host, port=port, timeout=5)
|
||||
stats = client.session_stats()
|
||||
|
||||
# Get detailed torrent info
|
||||
torrents = client.get_torrents()
|
||||
torrent_details = []
|
||||
for t in torrents:
|
||||
torrent_info = {
|
||||
"name": t.name,
|
||||
"progress": round(t.progress, 1),
|
||||
"status": str(t.status),
|
||||
"peers": t.peers_connected if hasattr(t, 'peers_connected') else 0,
|
||||
"download_speed": round(t.rate_download / 1024, 1) if hasattr(t, 'rate_download') else 0, # KB/s
|
||||
}
|
||||
# Check if stalled (downloading but no progress and no peers)
|
||||
if t.status.downloading and t.progress < 100 and torrent_info['peers'] == 0:
|
||||
torrent_info['stalled'] = True
|
||||
torrent_details.append(torrent_info)
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"host": host,
|
||||
"port": port,
|
||||
"download_dir": stats.download_dir if hasattr(stats, 'download_dir') else None,
|
||||
"active_torrent_count": stats.active_torrent_count if hasattr(stats, 'active_torrent_count') else 0,
|
||||
"torrents": torrent_details,
|
||||
}
|
||||
except ImportError as e:
|
||||
return {
|
||||
"available": False,
|
||||
"error": f"transmission_rpc not installed: {e}",
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"available": False,
|
||||
"host": os.environ.get("TRANSMISSION_HOST", "localhost"),
|
||||
"port": int(os.environ.get("TRANSMISSION_PORT", "9091")),
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
@@ -16,6 +70,67 @@ async def health_check() -> Dict[str, str]:
|
||||
}
|
||||
|
||||
|
||||
async def check_flacfetch_service_status() -> Dict[str, Any]:
|
||||
"""Check if remote flacfetch service is available and healthy."""
|
||||
client = get_flacfetch_client()
|
||||
|
||||
if not client:
|
||||
return {
|
||||
"configured": False,
|
||||
"message": "Remote flacfetch service not configured (FLACFETCH_API_URL not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await client.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"available": True,
|
||||
"status": health.get("status"),
|
||||
"version": health.get("version"),
|
||||
"transmission": health.get("transmission", {}),
|
||||
"disk": health.get("disk", {}),
|
||||
"providers": health.get("providers", {}),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
Detailed health check including dependencies.
|
||||
|
||||
Use this to debug issues with Transmission, flacfetch service, etc.
|
||||
"""
|
||||
transmission_status = check_transmission_status()
|
||||
flacfetch_status = await check_flacfetch_service_status()
|
||||
|
||||
# Log for debugging
|
||||
if not transmission_status.get("available"):
|
||||
logger.warning(f"Local Transmission not available: {transmission_status.get('error')}")
|
||||
else:
|
||||
logger.info(f"Local Transmission available at {transmission_status.get('host')}:{transmission_status.get('port')}")
|
||||
|
||||
if flacfetch_status.get("configured") and not flacfetch_status.get("available"):
|
||||
logger.warning(f"Remote flacfetch service not available: {flacfetch_status.get('error')}")
|
||||
elif flacfetch_status.get("available"):
|
||||
logger.info(f"Remote flacfetch service healthy: {flacfetch_status.get('status')}")
|
||||
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"dependencies": {
|
||||
"transmission_local": transmission_status,
|
||||
"flacfetch_remote": flacfetch_status,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/readiness")
|
||||
async def readiness_check() -> Dict[str, str]:
|
||||
"""Readiness check endpoint for Cloud Run."""
|
||||
|
||||
@@ -6,11 +6,17 @@ They are protected by admin authentication.
|
||||
|
||||
With Cloud Tasks integration, these endpoints may be called multiple times
|
||||
(retry on failure). Idempotency checks prevent duplicate processing.
|
||||
|
||||
Observability:
|
||||
- Extracts trace context from incoming requests (propagated via Cloud Tasks)
|
||||
- Creates worker spans linked to the original request trace
|
||||
- All logs include job_id for easy filtering in Cloud Logging
|
||||
"""
|
||||
import logging
|
||||
import asyncio
|
||||
import time
|
||||
from typing import Tuple, Optional
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.workers.audio_worker import process_audio_separation
|
||||
@@ -21,6 +27,12 @@ from backend.workers.render_video_worker import process_render_video
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.tracing import (
|
||||
extract_trace_context,
|
||||
start_span_with_context,
|
||||
add_span_attribute,
|
||||
add_span_event,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -57,7 +69,7 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"Job {job_id} not found for {worker_name} worker")
|
||||
logger.warning(f"[job:{job_id}] Job not found for {worker_name} worker")
|
||||
return WorkerResponse(
|
||||
status="not_found",
|
||||
job_id=job_id,
|
||||
@@ -70,7 +82,7 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
|
||||
stage = worker_progress.get('stage')
|
||||
|
||||
if stage == 'running':
|
||||
logger.info(f"{worker_name.capitalize()} worker already running for job {job_id}, skipping")
|
||||
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already running, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_running",
|
||||
job_id=job_id,
|
||||
@@ -78,7 +90,7 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
|
||||
)
|
||||
|
||||
if stage == 'complete':
|
||||
logger.info(f"{worker_name.capitalize()} worker already complete for job {job_id}, skipping")
|
||||
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already complete, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_complete",
|
||||
job_id=job_id,
|
||||
@@ -93,6 +105,7 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
|
||||
@router.post("/workers/audio", response_model=WorkerResponse)
|
||||
async def trigger_audio_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
@@ -107,11 +120,18 @@ async def trigger_audio_worker(
|
||||
The worker runs in the background and updates job state as it progresses.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
logger.info(f"Triggering audio worker for job {job_id}")
|
||||
|
||||
# Extract trace context from incoming request (propagated via Cloud Tasks)
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=audio")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "audio")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "audio")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
@@ -119,6 +139,7 @@ async def trigger_audio_worker(
|
||||
# while the worker continues processing
|
||||
background_tasks.add_task(process_audio_separation, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
@@ -129,6 +150,7 @@ async def trigger_audio_worker(
|
||||
@router.post("/workers/lyrics", response_model=WorkerResponse)
|
||||
async def trigger_lyrics_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
@@ -143,16 +165,24 @@ async def trigger_lyrics_worker(
|
||||
The worker runs in the background and updates job state as it progresses.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
logger.info(f"Triggering lyrics worker for job {job_id}")
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=lyrics")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "lyrics")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "lyrics")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(process_lyrics_transcription, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
@@ -163,6 +193,7 @@ async def trigger_lyrics_worker(
|
||||
@router.post("/workers/screens", response_model=WorkerResponse)
|
||||
async def trigger_screens_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
@@ -174,16 +205,24 @@ async def trigger_screens_worker(
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
logger.info(f"Triggering screens worker for job {job_id}")
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=screens")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "screens")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "screens")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(generate_screens, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
@@ -194,6 +233,7 @@ async def trigger_screens_worker(
|
||||
@router.post("/workers/video", response_model=WorkerResponse)
|
||||
async def trigger_video_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
@@ -206,16 +246,24 @@ async def trigger_video_worker(
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
logger.info(f"Triggering video worker for job {job_id}")
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=video")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "video")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "video")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(generate_video, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
@@ -226,6 +274,7 @@ async def trigger_video_worker(
|
||||
@router.post("/workers/render-video", response_model=WorkerResponse)
|
||||
async def trigger_render_video_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
@@ -242,16 +291,24 @@ async def trigger_render_video_worker(
|
||||
Next state: AWAITING_INSTRUMENTAL_SELECTION
|
||||
"""
|
||||
job_id = request.job_id
|
||||
logger.info(f"Triggering render-video worker for job {job_id}")
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=render-video")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "render-video")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "render")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(process_render_video, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
|
||||
+83
-19
@@ -26,8 +26,9 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.config import get_settings
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.api.dependencies import require_admin, require_auth, require_instrumental_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.metrics import metrics
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -55,7 +56,8 @@ async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
@router.post("", response_model=JobResponse)
|
||||
async def create_job(
|
||||
request: URLSubmissionRequest,
|
||||
background_tasks: BackgroundTasks
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> JobResponse:
|
||||
"""
|
||||
Create a new karaoke generation job from a URL.
|
||||
@@ -81,6 +83,9 @@ async def create_job(
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job.job_id, source="url")
|
||||
|
||||
# Trigger both workers in parallel using asyncio.gather
|
||||
# (FastAPI's BackgroundTasks runs async tasks sequentially)
|
||||
background_tasks.add_task(_trigger_workers_parallel, job.job_id)
|
||||
@@ -102,7 +107,10 @@ async def create_job(
|
||||
|
||||
|
||||
@router.get("/{job_id}", response_model=Job)
|
||||
async def get_job(job_id: str) -> Job:
|
||||
async def get_job(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> Job:
|
||||
"""Get job status and details."""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
@@ -122,7 +130,8 @@ async def list_jobs(
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[str] = None,
|
||||
created_before: Optional[str] = None,
|
||||
limit: int = 100
|
||||
limit: int = 100,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filters.
|
||||
@@ -174,7 +183,11 @@ async def list_jobs(
|
||||
|
||||
|
||||
@router.delete("/{job_id}")
|
||||
async def delete_job(job_id: str, delete_files: bool = True) -> dict:
|
||||
async def delete_job(
|
||||
job_id: str,
|
||||
delete_files: bool = True,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""Delete a job and optionally its output files."""
|
||||
try:
|
||||
job = job_manager.get_job(job_id)
|
||||
@@ -301,7 +314,10 @@ async def bulk_delete_jobs(
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/{job_id}/review-data")
|
||||
async def get_review_data(job_id: str) -> Dict[str, Any]:
|
||||
async def get_review_data(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get data needed for lyrics review interface.
|
||||
|
||||
@@ -342,7 +358,11 @@ async def get_review_data(job_id: str) -> Dict[str, Any]:
|
||||
|
||||
|
||||
@router.post("/{job_id}/start-review")
|
||||
async def start_review(job_id: str, request: StartReviewRequest) -> dict:
|
||||
async def start_review(
|
||||
job_id: str,
|
||||
request: StartReviewRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Mark job as IN_REVIEW (user opened review interface).
|
||||
|
||||
@@ -364,7 +384,8 @@ async def start_review(job_id: str, request: StartReviewRequest) -> dict:
|
||||
async def submit_corrections(
|
||||
job_id: str,
|
||||
submission: CorrectionsSubmission,
|
||||
background_tasks: BackgroundTasks
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Save corrected lyrics during human review.
|
||||
@@ -422,7 +443,8 @@ async def submit_corrections(
|
||||
@router.post("/{job_id}/complete-review")
|
||||
async def complete_review(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Complete the human review and trigger video rendering.
|
||||
@@ -470,13 +492,18 @@ async def complete_review(
|
||||
|
||||
|
||||
@router.get("/{job_id}/instrumental-options")
|
||||
async def get_instrumental_options(job_id: str) -> Dict[str, Any]:
|
||||
async def get_instrumental_options(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get instrumental audio options for user selection.
|
||||
|
||||
Returns signed URLs for both options:
|
||||
1. Clean instrumental (no backing vocals)
|
||||
2. Instrumental with backing vocals
|
||||
|
||||
Accepts either full auth token or job-specific instrumental_token.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
@@ -525,7 +552,10 @@ async def get_instrumental_options(job_id: str) -> Dict[str, Any]:
|
||||
|
||||
|
||||
@router.get("/{job_id}/instrumental-analysis")
|
||||
async def get_instrumental_analysis(job_id: str) -> Dict[str, Any]:
|
||||
async def get_instrumental_analysis(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get audio analysis data for instrumental selection.
|
||||
|
||||
@@ -536,6 +566,8 @@ async def get_instrumental_analysis(job_id: str) -> Dict[str, Any]:
|
||||
|
||||
This endpoint enables intelligent instrumental selection by providing
|
||||
detailed analysis of the backing vocals track.
|
||||
|
||||
Accepts either full auth token or job-specific instrumental_token.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
@@ -599,7 +631,11 @@ async def get_instrumental_analysis(job_id: str) -> Dict[str, Any]:
|
||||
|
||||
|
||||
@router.get("/{job_id}/audio-stream/{stem_type}")
|
||||
async def stream_audio(job_id: str, stem_type: str):
|
||||
async def stream_audio(
|
||||
job_id: str,
|
||||
stem_type: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
|
||||
):
|
||||
"""
|
||||
Stream an audio file for playback in the browser.
|
||||
|
||||
@@ -611,6 +647,8 @@ async def stream_audio(job_id: str, stem_type: str):
|
||||
|
||||
Returns audio as a streaming response with proper headers for
|
||||
browser audio playback.
|
||||
|
||||
Accepts either full auth token or job-specific instrumental_token.
|
||||
"""
|
||||
from fastapi.responses import StreamingResponse
|
||||
import tempfile
|
||||
@@ -687,6 +725,7 @@ async def stream_audio(job_id: str, stem_type: str):
|
||||
async def create_custom_instrumental(
|
||||
job_id: str,
|
||||
request: CreateCustomInstrumentalRequest,
|
||||
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Create a custom instrumental by muting regions of backing vocals.
|
||||
@@ -698,6 +737,8 @@ async def create_custom_instrumental(
|
||||
|
||||
After calling this endpoint, the user can select "custom" in the
|
||||
select-instrumental endpoint.
|
||||
|
||||
Accepts either full auth token or job-specific instrumental_token.
|
||||
"""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
@@ -776,7 +817,11 @@ async def create_custom_instrumental(
|
||||
|
||||
|
||||
@router.get("/{job_id}/waveform-data")
|
||||
async def get_waveform_data(job_id: str, num_points: int = 500) -> Dict[str, Any]:
|
||||
async def get_waveform_data(
|
||||
job_id: str,
|
||||
num_points: int = 500,
|
||||
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get waveform amplitude data for client-side rendering.
|
||||
|
||||
@@ -793,6 +838,8 @@ async def get_waveform_data(job_id: str, num_points: int = 500) -> Dict[str, Any
|
||||
"duration_seconds": 180.5,
|
||||
"num_points": 500
|
||||
}
|
||||
|
||||
Accepts either full auth token or job-specific instrumental_token.
|
||||
"""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
@@ -833,13 +880,16 @@ async def get_waveform_data(job_id: str, num_points: int = 500) -> Dict[str, Any
|
||||
async def select_instrumental(
|
||||
job_id: str,
|
||||
selection: InstrumentalSelection,
|
||||
background_tasks: BackgroundTasks
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Submit instrumental selection.
|
||||
|
||||
This is the SECOND critical human-in-the-loop interaction point.
|
||||
After selection, the job proceeds to video generation.
|
||||
|
||||
Accepts either full auth token or job-specific instrumental_token.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
@@ -881,7 +931,10 @@ async def select_instrumental(
|
||||
|
||||
|
||||
@router.get("/{job_id}/download-urls")
|
||||
async def get_download_urls(job_id: str) -> dict:
|
||||
async def get_download_urls(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Get download URLs for all job output files.
|
||||
|
||||
@@ -923,7 +976,12 @@ async def get_download_urls(job_id: str) -> dict:
|
||||
|
||||
|
||||
@router.get("/{job_id}/download/{category}/{file_key}")
|
||||
async def download_file(job_id: str, category: str, file_key: str):
|
||||
async def download_file(
|
||||
job_id: str,
|
||||
category: str,
|
||||
file_key: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Stream download a specific file from a completed job.
|
||||
|
||||
@@ -1010,7 +1068,11 @@ async def download_file(job_id: str, category: str, file_key: str):
|
||||
|
||||
|
||||
@router.post("/{job_id}/cancel")
|
||||
async def cancel_job(job_id: str, request: CancelJobRequest) -> dict:
|
||||
async def cancel_job(
|
||||
job_id: str,
|
||||
request: CancelJobRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Cancel a job.
|
||||
|
||||
@@ -1031,7 +1093,8 @@ async def cancel_job(job_id: str, request: CancelJobRequest) -> dict:
|
||||
@router.post("/{job_id}/retry")
|
||||
async def retry_job(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Retry a failed job from the last successful checkpoint.
|
||||
@@ -1185,7 +1248,8 @@ async def retry_job(
|
||||
async def get_worker_logs(
|
||||
job_id: str,
|
||||
since_index: int = 0,
|
||||
worker: Optional[str] = None
|
||||
worker: Optional[str] = None,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get worker logs for debugging.
|
||||
|
||||
@@ -16,9 +16,9 @@ import json
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Set
|
||||
from typing import Dict, Any, Set, Tuple
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from fastapi import APIRouter, HTTPException, Request, Depends
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
from starlette.background import BackgroundTask
|
||||
|
||||
@@ -27,6 +27,8 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.job_logging import job_log_context, JobLogger
|
||||
from backend.services.tracing import create_span, add_span_attribute, add_span_event
|
||||
from backend.api.dependencies import require_auth, require_review_auth
|
||||
from backend.services.auth_service import UserType
|
||||
|
||||
# LyricsTranscriber imports for preview generation
|
||||
from lyrics_transcriber.types import CorrectionResult
|
||||
@@ -63,7 +65,10 @@ async def ping(job_id: str):
|
||||
|
||||
|
||||
@router.get("/{job_id}/correction-data")
|
||||
async def get_correction_data(job_id: str):
|
||||
async def get_correction_data(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Get correction data for the review interface.
|
||||
|
||||
@@ -129,14 +134,21 @@ async def get_correction_data(job_id: str):
|
||||
|
||||
|
||||
@router.get("/{job_id}/audio/{audio_hash}")
|
||||
async def get_audio_with_hash(job_id: str, audio_hash: str):
|
||||
async def get_audio_with_hash(
|
||||
job_id: str,
|
||||
audio_hash: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the audio file for playback (with hash parameter)."""
|
||||
return await _stream_audio(job_id)
|
||||
|
||||
|
||||
@router.get("/{job_id}/audio/")
|
||||
@router.get("/{job_id}/audio")
|
||||
async def get_audio_no_hash(job_id: str):
|
||||
async def get_audio_no_hash(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the audio file for playback (without hash parameter)."""
|
||||
return await _stream_audio(job_id)
|
||||
|
||||
@@ -189,7 +201,11 @@ async def _stream_audio(job_id: str):
|
||||
|
||||
|
||||
@router.post("/{job_id}/complete")
|
||||
async def complete_review(job_id: str, updated_data: Dict[str, Any]):
|
||||
async def complete_review(
|
||||
job_id: str,
|
||||
updated_data: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Complete the review and save corrected lyrics.
|
||||
|
||||
@@ -244,7 +260,11 @@ async def complete_review(job_id: str, updated_data: Dict[str, Any]):
|
||||
|
||||
|
||||
@router.post("/{job_id}/handlers")
|
||||
async def update_handlers(job_id: str, enabled_handlers: list):
|
||||
async def update_handlers(
|
||||
job_id: str,
|
||||
enabled_handlers: list,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Update enabled correction handlers (optional feature).
|
||||
|
||||
@@ -256,7 +276,11 @@ async def update_handlers(job_id: str, enabled_handlers: list):
|
||||
|
||||
|
||||
@router.post("/{job_id}/add-lyrics")
|
||||
async def add_lyrics(job_id: str, data: Dict[str, str]):
|
||||
async def add_lyrics(
|
||||
job_id: str,
|
||||
data: Dict[str, str],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Add custom lyrics source and rerun correction.
|
||||
|
||||
@@ -354,7 +378,11 @@ async def add_lyrics(job_id: str, data: Dict[str, str]):
|
||||
|
||||
|
||||
@router.post("/{job_id}/preview-video")
|
||||
async def generate_preview_video(job_id: str, updated_data: Dict[str, Any]):
|
||||
async def generate_preview_video(
|
||||
job_id: str,
|
||||
updated_data: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Generate a preview video with the current corrections.
|
||||
|
||||
@@ -469,7 +497,11 @@ async def generate_preview_video(job_id: str, updated_data: Dict[str, Any]):
|
||||
|
||||
|
||||
@router.get("/{job_id}/preview-video/{preview_hash}")
|
||||
async def get_preview_video(job_id: str, preview_hash: str):
|
||||
async def get_preview_video(
|
||||
job_id: str,
|
||||
preview_hash: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the generated preview video."""
|
||||
storage = StorageService()
|
||||
|
||||
@@ -510,7 +542,11 @@ async def get_preview_video(job_id: str, preview_hash: str):
|
||||
|
||||
|
||||
@router.post("/{job_id}/v1/annotations")
|
||||
async def submit_annotation(job_id: str, annotation: Dict[str, Any]):
|
||||
async def submit_annotation(
|
||||
job_id: str,
|
||||
annotation: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Submit a correction annotation for ML training data.
|
||||
|
||||
@@ -522,7 +558,10 @@ async def submit_annotation(job_id: str, annotation: Dict[str, Any]):
|
||||
|
||||
|
||||
@router.get("/{job_id}/v1/annotations/stats")
|
||||
async def get_annotation_stats(job_id: str):
|
||||
async def get_annotation_stats(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Get annotation statistics."""
|
||||
return {
|
||||
"total_annotations": 0,
|
||||
|
||||
@@ -56,6 +56,12 @@ class Settings(BaseSettings):
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
# Flacfetch remote service (for torrent downloads)
|
||||
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
|
||||
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
|
||||
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
|
||||
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
|
||||
|
||||
# Default distribution settings (can be overridden per-request)
|
||||
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
|
||||
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
|
||||
|
||||
+4
-5
@@ -9,16 +9,15 @@ from fastapi.middleware.cors import CORSMiddleware
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
|
||||
|
||||
from backend.version import VERSION
|
||||
|
||||
|
||||
# Configure logging with trace ID support
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, settings.log_level.upper()),
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
# Configure structured logging (JSON in Cloud Run, human-readable locally)
|
||||
# This must happen before any logging calls
|
||||
setup_structured_logging()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Initialize OpenTelemetry tracing (must happen before app creation)
|
||||
|
||||
@@ -251,6 +251,12 @@ class Job(BaseModel):
|
||||
finalise_only: bool = False # Skip prep, run only finalisation (requires uploaded prep outputs)
|
||||
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
|
||||
|
||||
# Review authentication (Batch 7)
|
||||
review_token: Optional[str] = None # Job-scoped token for lyrics review UI access (generated when entering AWAITING_REVIEW)
|
||||
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
|
||||
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
|
||||
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
|
||||
|
||||
# Processing state
|
||||
track_output_dir: Optional[str] = None # Local output directory (temp)
|
||||
audio_hash: Optional[str] = None # Hash for deduplication
|
||||
|
||||
@@ -1,79 +1,69 @@
|
||||
"""
|
||||
Audio search service for finding and downloading audio files.
|
||||
|
||||
This service integrates with flacfetch to search for audio from various sources
|
||||
(YouTube, music trackers, etc.) and download the selected audio file.
|
||||
This service integrates with karaoke_gen.audio_fetcher (which wraps flacfetch)
|
||||
to search for audio from various sources (YouTube, music trackers, etc.)
|
||||
and download the selected audio file.
|
||||
|
||||
This is a thin wrapper that adds backend-specific functionality:
|
||||
- Caching raw results for API-based selection flow
|
||||
- Firestore-compatible serialization
|
||||
- Singleton pattern for service lifecycle
|
||||
- Remote flacfetch service integration for torrent downloads
|
||||
|
||||
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
|
||||
HTTP API for search and download operations. This is required for torrent
|
||||
downloads since Cloud Run doesn't support BitTorrent peer connections.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from dataclasses import dataclass, asdict
|
||||
from typing import List, Optional, Dict, Any
|
||||
from typing import List, Optional
|
||||
|
||||
import nest_asyncio
|
||||
|
||||
# Import shared classes from karaoke_gen.audio_fetcher - single source of truth
|
||||
# Note: Import directly from audio_fetcher module, not karaoke_gen.__init__
|
||||
# to avoid pulling in heavier dependencies like lyrics_transcriber
|
||||
from karaoke_gen.audio_fetcher import (
|
||||
FlacFetcher,
|
||||
AudioSearchResult,
|
||||
AudioFetchResult,
|
||||
AudioFetcherError,
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
|
||||
from .flacfetch_client import get_flacfetch_client, FlacfetchServiceError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioSearchResult:
|
||||
"""
|
||||
Represents a single search result for audio.
|
||||
|
||||
This is a serializable representation of a search result that can be
|
||||
stored in Firestore and sent to clients.
|
||||
"""
|
||||
title: str
|
||||
artist: str
|
||||
provider: str
|
||||
url: str
|
||||
duration: Optional[int] = None # Duration in seconds
|
||||
quality: Optional[str] = None # e.g., "FLAC", "320kbps", etc.
|
||||
source_id: Optional[str] = None # Unique ID from the source
|
||||
index: int = 0 # Index in the results list (for selection)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""Convert to dict for JSON serialization."""
|
||||
return asdict(self)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "AudioSearchResult":
|
||||
"""Create from dict."""
|
||||
return cls(**data)
|
||||
# Re-export exception under backend's naming convention for compatibility
|
||||
AudioSearchError = AudioFetcherError
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioDownloadResult:
|
||||
"""Result of downloading an audio file."""
|
||||
filepath: str
|
||||
artist: str
|
||||
title: str
|
||||
provider: str
|
||||
duration: Optional[int] = None
|
||||
quality: Optional[str] = None
|
||||
|
||||
|
||||
class AudioSearchError(Exception):
|
||||
"""Base exception for audio search errors."""
|
||||
pass
|
||||
|
||||
|
||||
class NoResultsError(AudioSearchError):
|
||||
"""Raised when no search results are found."""
|
||||
pass
|
||||
|
||||
|
||||
class DownloadError(AudioSearchError):
|
||||
"""Raised when download fails."""
|
||||
pass
|
||||
# Also alias AudioFetchResult as AudioDownloadResult for backwards compatibility
|
||||
AudioDownloadResult = AudioFetchResult
|
||||
|
||||
|
||||
class AudioSearchService:
|
||||
"""
|
||||
Service for searching and downloading audio files using flacfetch.
|
||||
Service for searching and downloading audio files.
|
||||
|
||||
This service is used by the backend to:
|
||||
1. Search for audio by artist/title
|
||||
2. Return results for user selection
|
||||
3. Download the selected audio to a local file
|
||||
This is a thin wrapper around karaoke_gen.audio_fetcher.FlacFetcher that adds
|
||||
backend-specific functionality:
|
||||
- Caches raw results for API-based selection (client picks by index)
|
||||
- Provides Firestore-compatible serialization
|
||||
- Manages service lifecycle as singleton
|
||||
- Remote flacfetch service integration for torrent downloads
|
||||
|
||||
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
|
||||
HTTP API for search and download operations. This enables torrent downloads
|
||||
which are not possible in Cloud Run due to network restrictions.
|
||||
|
||||
The actual flacfetch integration is in karaoke_gen.audio_fetcher.FlacFetcher,
|
||||
which is shared between local CLI and cloud backend.
|
||||
"""
|
||||
|
||||
# Sentinel value to indicate "use environment variable"
|
||||
@@ -81,72 +71,64 @@ class AudioSearchService:
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
redacted_api_key: Optional[str] = _USE_ENV,
|
||||
red_api_key: Optional[str] = _USE_ENV,
|
||||
red_api_url: Optional[str] = _USE_ENV,
|
||||
ops_api_key: Optional[str] = _USE_ENV,
|
||||
ops_api_url: Optional[str] = _USE_ENV,
|
||||
):
|
||||
"""
|
||||
Initialize the audio search service.
|
||||
|
||||
Args:
|
||||
redacted_api_key: API key for Redacted tracker (optional, uses env if not provided)
|
||||
red_api_key: API key for RED tracker (optional, uses env if not provided)
|
||||
red_api_url: Base URL for RED tracker API (optional, uses env if not provided)
|
||||
ops_api_key: API key for OPS tracker (optional, uses env if not provided)
|
||||
ops_api_url: Base URL for OPS tracker API (optional, uses env if not provided)
|
||||
"""
|
||||
# Use environment variables if not explicitly provided
|
||||
if redacted_api_key is self._USE_ENV:
|
||||
self._redacted_api_key = os.environ.get("REDACTED_API_KEY")
|
||||
else:
|
||||
self._redacted_api_key = redacted_api_key
|
||||
if red_api_key is self._USE_ENV:
|
||||
red_api_key = os.environ.get("RED_API_KEY")
|
||||
if red_api_url is self._USE_ENV:
|
||||
red_api_url = os.environ.get("RED_API_URL")
|
||||
|
||||
if ops_api_key is self._USE_ENV:
|
||||
self._ops_api_key = os.environ.get("OPS_API_KEY")
|
||||
else:
|
||||
self._ops_api_key = ops_api_key
|
||||
|
||||
self._manager = None
|
||||
self._raw_results = {} # Cache raw results for download
|
||||
|
||||
def _get_manager(self):
|
||||
"""Lazily initialize and return the FetchManager."""
|
||||
if self._manager is None:
|
||||
try:
|
||||
from flacfetch.core.manager import FetchManager
|
||||
from flacfetch.providers.youtube import YouTubeProvider
|
||||
except ImportError as e:
|
||||
logger.error(f"flacfetch not installed: {e}")
|
||||
raise AudioSearchError(
|
||||
"flacfetch library is not installed. "
|
||||
"Install with: pip install flacfetch"
|
||||
) from e
|
||||
|
||||
self._manager = FetchManager()
|
||||
|
||||
# Add providers based on available API keys
|
||||
if self._redacted_api_key:
|
||||
try:
|
||||
from flacfetch.providers.redacted import RedactedProvider
|
||||
self._manager.add_provider(RedactedProvider(api_key=self._redacted_api_key))
|
||||
logger.debug("Added Redacted provider")
|
||||
except ImportError:
|
||||
logger.warning("Redacted provider not available")
|
||||
|
||||
if self._ops_api_key:
|
||||
try:
|
||||
from flacfetch.providers.ops import OPSProvider
|
||||
self._manager.add_provider(OPSProvider(api_key=self._ops_api_key))
|
||||
logger.debug("Added OPS provider")
|
||||
except ImportError:
|
||||
logger.warning("OPS provider not available")
|
||||
|
||||
# Always add YouTube as a fallback provider
|
||||
self._manager.add_provider(YouTubeProvider())
|
||||
logger.debug("Added YouTube provider")
|
||||
ops_api_key = os.environ.get("OPS_API_KEY")
|
||||
if ops_api_url is self._USE_ENV:
|
||||
ops_api_url = os.environ.get("OPS_API_URL")
|
||||
|
||||
return self._manager
|
||||
# Check for remote flacfetch client
|
||||
self._remote_client = get_flacfetch_client()
|
||||
|
||||
# Log which mode we're using
|
||||
if self._remote_client:
|
||||
logger.info("AudioSearchService using REMOTE flacfetch service (torrent downloads enabled)")
|
||||
else:
|
||||
logger.info("AudioSearchService using LOCAL flacfetch (YouTube only in Cloud Run)")
|
||||
|
||||
# Delegate to shared FlacFetcher implementation (for local mode or fallback)
|
||||
self._fetcher = FlacFetcher(
|
||||
red_api_key=red_api_key,
|
||||
red_api_url=red_api_url,
|
||||
ops_api_key=ops_api_key,
|
||||
ops_api_url=ops_api_url,
|
||||
)
|
||||
|
||||
# Cache search results for API-based selection flow
|
||||
# Key: index, Value: AudioSearchResult (with raw_result)
|
||||
self._cached_results: List[AudioSearchResult] = []
|
||||
|
||||
# Cache for remote search results (search_id -> results)
|
||||
self._remote_search_id: Optional[str] = None
|
||||
|
||||
def search(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Search for audio matching the given artist and title.
|
||||
|
||||
Results are cached internally for later download via download().
|
||||
|
||||
If a remote flacfetch service is configured (FLACFETCH_API_URL), uses that
|
||||
for better torrent download support. Otherwise falls back to local flacfetch.
|
||||
|
||||
Args:
|
||||
artist: The artist name to search for
|
||||
title: The track title to search for
|
||||
@@ -156,49 +138,94 @@ class AudioSearchService:
|
||||
|
||||
Raises:
|
||||
NoResultsError: If no results are found
|
||||
AudioSearchError: For other errors
|
||||
AudioSearchError: For other errors (e.g., flacfetch not installed)
|
||||
"""
|
||||
try:
|
||||
from flacfetch.core.models import TrackQuery
|
||||
except ImportError as e:
|
||||
raise AudioSearchError("flacfetch not installed") from e
|
||||
|
||||
manager = self._get_manager()
|
||||
query = TrackQuery(artist=artist, title=title)
|
||||
|
||||
logger.info(f"Searching for audio: {artist} - {title}")
|
||||
|
||||
try:
|
||||
results = manager.search(query)
|
||||
# Try remote flacfetch service first if configured
|
||||
if self._remote_client:
|
||||
return self._search_remote(artist, title)
|
||||
|
||||
# Fallback to local flacfetch
|
||||
results = self._fetcher.search(artist, title)
|
||||
|
||||
# Cache results for later download
|
||||
self._cached_results = results
|
||||
self._remote_search_id = None
|
||||
|
||||
logger.info(f"Found {len(results)} results for: {artist} - {title}")
|
||||
return results
|
||||
|
||||
except (NoResultsError, AudioFetcherError):
|
||||
# Re-raise these as-is
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Search failed: {e}")
|
||||
raise AudioSearchError(f"Search failed: {e}") from e
|
||||
|
||||
def _search_remote(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Search using the remote flacfetch service.
|
||||
|
||||
if not results:
|
||||
raise NoResultsError(f"No results found for: {artist} - {title}")
|
||||
|
||||
# Convert to our AudioSearchResult format and cache raw results
|
||||
search_results = []
|
||||
self._raw_results.clear()
|
||||
|
||||
for i, result in enumerate(results):
|
||||
search_result = AudioSearchResult(
|
||||
title=getattr(result, "title", title),
|
||||
artist=getattr(result, "artist", artist),
|
||||
provider=getattr(result, "provider", "Unknown"),
|
||||
url=getattr(result, "url", ""),
|
||||
duration=getattr(result, "duration", None),
|
||||
quality=getattr(result, "quality", None),
|
||||
source_id=getattr(result, "id", str(i)),
|
||||
index=i,
|
||||
)
|
||||
search_results.append(search_result)
|
||||
Runs async code in a sync context for compatibility with existing API.
|
||||
"""
|
||||
try:
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
# Apply to the current loop if one exists, otherwise it's a no-op
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(loop)
|
||||
except RuntimeError:
|
||||
pass # No running loop, nest_asyncio not needed
|
||||
|
||||
# Cache raw result for later download
|
||||
self._raw_results[i] = result
|
||||
|
||||
logger.info(f"Found {len(search_results)} results for: {artist} - {title}")
|
||||
return search_results
|
||||
# Run async search in sync context
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
response = loop.run_until_complete(
|
||||
self._remote_client.search(artist, title)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Check for empty results
|
||||
if not response.get("results"):
|
||||
raise NoResultsError(f"No results found for: {artist} - {title}")
|
||||
|
||||
# Store search_id for download
|
||||
self._remote_search_id = response.get("search_id")
|
||||
|
||||
# Convert remote results to AudioSearchResult objects
|
||||
results = []
|
||||
for item in response.get("results", []):
|
||||
result = AudioSearchResult(
|
||||
title=item.get("title", "Unknown"),
|
||||
artist=item.get("artist", "Unknown"),
|
||||
url=item.get("download_url", ""), # May be empty for remote
|
||||
provider=item.get("provider", "Unknown"),
|
||||
duration=item.get("duration_seconds"),
|
||||
quality=item.get("quality", ""),
|
||||
source_id=item.get("info_hash"),
|
||||
index=item.get("index", 0),
|
||||
seeders=item.get("seeders"),
|
||||
target_file=item.get("target_file"),
|
||||
# Store full remote data in raw_result for rich display
|
||||
raw_result=item,
|
||||
)
|
||||
results.append(result)
|
||||
|
||||
# Cache results
|
||||
self._cached_results = results
|
||||
|
||||
logger.info(f"Found {len(results)} results from remote flacfetch for: {artist} - {title}")
|
||||
return results
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
logger.warning(f"Remote flacfetch search failed, falling back to local: {e}")
|
||||
# Fallback to local search
|
||||
self._remote_search_id = None
|
||||
results = self._fetcher.search(artist, title)
|
||||
self._cached_results = results
|
||||
return results
|
||||
|
||||
def select_best(self, results: List[AudioSearchResult]) -> int:
|
||||
"""
|
||||
@@ -212,90 +239,310 @@ class AudioSearchService:
|
||||
Returns:
|
||||
Index of the best result
|
||||
"""
|
||||
if not results:
|
||||
return 0
|
||||
|
||||
manager = self._get_manager()
|
||||
|
||||
# Get raw results for the indices we have
|
||||
raw_results = [self._raw_results.get(r.index) for r in results if r.index in self._raw_results]
|
||||
|
||||
if raw_results:
|
||||
try:
|
||||
best = manager.select_best(raw_results)
|
||||
# Find index of best result
|
||||
for i, raw in enumerate(raw_results):
|
||||
if raw == best:
|
||||
return results[i].index
|
||||
except Exception as e:
|
||||
logger.warning(f"select_best failed, using first result: {e}")
|
||||
|
||||
# Fallback: return first result
|
||||
return results[0].index if results else 0
|
||||
return self._fetcher.select_best(results)
|
||||
|
||||
def download(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
remote_search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download audio from a search result.
|
||||
|
||||
Download audio from a cached search result.
|
||||
|
||||
This method uses the cached results from the last search() call.
|
||||
The API flow is:
|
||||
1. Client calls search() -> gets list of results
|
||||
2. Client picks an index
|
||||
3. Client calls download(index) -> gets downloaded file
|
||||
|
||||
If a remote flacfetch service is configured and the search was performed
|
||||
remotely (for torrent sources), uses the remote service for download.
|
||||
|
||||
Args:
|
||||
result_index: Index of the result to download (from search results)
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
|
||||
gcs_path: Optional GCS path for remote uploads (e.g., "uploads/job123/audio/")
|
||||
remote_search_id: Optional search_id for remote downloads. If provided,
|
||||
uses this instead of the cached _remote_search_id. This is important
|
||||
for concurrent requests where the service-level cache may be stale.
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
|
||||
Raises:
|
||||
DownloadError: If download fails or no cached result for index
|
||||
"""
|
||||
if result_index < 0 or result_index >= len(self._cached_results):
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. "
|
||||
"Run search() first."
|
||||
)
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
logger.info(f"Downloading: {result.artist} - {result.title} from {result.provider}")
|
||||
|
||||
# Use provided search_id or fall back to service-level cache
|
||||
effective_search_id = remote_search_id or self._remote_search_id
|
||||
|
||||
# Check if this was a remote search (torrent sources need remote download)
|
||||
if effective_search_id and self._remote_client:
|
||||
# Check if this is a torrent source that needs remote handling
|
||||
is_torrent = result.provider in ["RED", "OPS"]
|
||||
|
||||
if is_torrent:
|
||||
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
|
||||
|
||||
# Delegate to shared FlacFetcher (local download)
|
||||
fetch_result = self._fetcher.download(result, output_dir, output_filename)
|
||||
|
||||
logger.info(f"Downloaded to: {fetch_result.filepath}")
|
||||
|
||||
return fetch_result
|
||||
|
||||
def download_by_id(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download audio directly by source ID (no prior search required).
|
||||
|
||||
This is the preferred method when you have stored the source_id from a
|
||||
previous search and want to download later without re-searching. This
|
||||
avoids unnecessary API calls to private trackers.
|
||||
|
||||
Args:
|
||||
source_name: Provider name (RED, OPS, YouTube, Spotify)
|
||||
source_id: Source-specific ID (torrent ID, video ID, track ID)
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
target_file: For torrents, specific file to extract from the torrent
|
||||
download_url: For YouTube/Spotify, direct URL (optional)
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
Raises:
|
||||
DownloadError: If download fails
|
||||
"""
|
||||
if result_index not in self._raw_results:
|
||||
raise DownloadError(f"No cached result for index {result_index}. Run search() first.")
|
||||
|
||||
raw_result = self._raw_results[result_index]
|
||||
manager = self._get_manager()
|
||||
|
||||
# Ensure output directory exists
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
# Get metadata from raw result
|
||||
artist = getattr(raw_result, "artist", "Unknown")
|
||||
title = getattr(raw_result, "title", "Unknown")
|
||||
provider = getattr(raw_result, "provider", "Unknown")
|
||||
|
||||
# Generate filename if not provided
|
||||
if output_filename is None:
|
||||
output_filename = f"{artist} - {title}"
|
||||
|
||||
logger.info(f"Downloading: {artist} - {title} from {provider}")
|
||||
logger.info(f"Download by ID: {source_name} ID={source_id}")
|
||||
|
||||
# For torrent sources, must use remote client
|
||||
if source_name in ["RED", "OPS"]:
|
||||
if not self._remote_client:
|
||||
raise DownloadError(
|
||||
f"Cannot download from {source_name} without remote flacfetch service"
|
||||
)
|
||||
return self._download_by_id_remote(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=output_dir,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
|
||||
# For YouTube/Spotify, can use local flacfetch via FetchManager.download_by_id
|
||||
# But we currently only have remote support, so use remote if available
|
||||
if self._remote_client:
|
||||
return self._download_by_id_remote(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=output_dir,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
|
||||
# Local download for YouTube/Spotify (fallback)
|
||||
# Use the local fetcher's download_by_id if available
|
||||
raise DownloadError(
|
||||
f"Local download_by_id not yet implemented for {source_name}. "
|
||||
"Configure FLACFETCH_API_URL for remote downloads."
|
||||
)
|
||||
|
||||
def _download_by_id_remote(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str, # Not used for remote downloads (files go to GCS or remote server)
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download by ID using the remote flacfetch service.
|
||||
|
||||
Note: output_dir is accepted for API compatibility but ignored for remote
|
||||
downloads. Remote downloads either go to GCS (if gcs_path is set) or to
|
||||
the remote server's download directory.
|
||||
"""
|
||||
# output_dir is intentionally unused - remote downloads don't use local paths
|
||||
_ = output_dir
|
||||
try:
|
||||
# Enable nested event loops
|
||||
try:
|
||||
running_loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(running_loop)
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
# Start download
|
||||
download_id = loop.run_until_complete(
|
||||
self._remote_client.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(f"Remote download by ID started: {download_id}")
|
||||
|
||||
# Wait for completion
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = loop.run_until_complete(
|
||||
self._remote_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=600,
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Determine file path
|
||||
filepath = final_status.get("gcs_path") or final_status.get("output_path")
|
||||
|
||||
if not filepath:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download by ID complete: {filepath}")
|
||||
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist="", # Not available without search
|
||||
title="", # Not available without search
|
||||
provider=source_name,
|
||||
quality="", # Not available without search
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
raise DownloadError(f"Remote download by ID failed: {e}") from e
|
||||
except Exception as e:
|
||||
logger.error(f"Remote download by ID error: {e}")
|
||||
raise DownloadError(f"Remote download by ID failed: {e}") from e
|
||||
|
||||
def _download_remote(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download using the remote flacfetch service.
|
||||
|
||||
The remote service downloads via torrent and optionally uploads to GCS.
|
||||
|
||||
Args:
|
||||
result_index: Index of the result to download
|
||||
output_dir: Directory to save downloaded file
|
||||
output_filename: Optional filename
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
search_id: Remote search ID to use (overrides self._remote_search_id)
|
||||
"""
|
||||
effective_search_id = search_id or self._remote_search_id
|
||||
if not effective_search_id:
|
||||
raise DownloadError("No remote search ID - run search() first")
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
try:
|
||||
filepath = manager.download(
|
||||
raw_result,
|
||||
output_path=output_dir,
|
||||
output_filename=output_filename,
|
||||
)
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
try:
|
||||
running_loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(running_loop)
|
||||
except RuntimeError:
|
||||
pass # No running loop, nest_asyncio not needed
|
||||
|
||||
if filepath is None:
|
||||
raise DownloadError(f"Download returned no file path for: {artist} - {title}")
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
# Start download
|
||||
download_id = loop.run_until_complete(
|
||||
self._remote_client.download(
|
||||
search_id=effective_search_id,
|
||||
result_index=result_index,
|
||||
output_filename=output_filename,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(f"Remote download started: {download_id}")
|
||||
|
||||
# Wait for completion with progress logging
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = loop.run_until_complete(
|
||||
self._remote_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=600, # 10 minute timeout
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
logger.info(f"Downloaded to: {filepath}")
|
||||
# Determine file path
|
||||
filepath = final_status.get("gcs_path") or final_status.get("output_path")
|
||||
|
||||
if not filepath:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download complete: {filepath}")
|
||||
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist=artist,
|
||||
title=title,
|
||||
provider=provider,
|
||||
duration=getattr(raw_result, "duration", None),
|
||||
quality=getattr(raw_result, "quality", None),
|
||||
artist=result.artist,
|
||||
title=result.title,
|
||||
provider=result.provider,
|
||||
quality=result.quality,
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
raise DownloadError(f"Remote download failed: {e}") from e
|
||||
except Exception as e:
|
||||
raise DownloadError(f"Failed to download {artist} - {title}: {e}") from e
|
||||
logger.error(f"Remote download error: {e}")
|
||||
raise DownloadError(f"Remote download failed: {e}") from e
|
||||
|
||||
def search_and_download_auto(
|
||||
self,
|
||||
@@ -303,6 +550,7 @@ class AudioSearchService:
|
||||
title: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Search for audio and automatically download the best result.
|
||||
@@ -315,6 +563,7 @@ class AudioSearchService:
|
||||
title: The track title to search for
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
@@ -328,10 +577,96 @@ class AudioSearchService:
|
||||
|
||||
# Select best
|
||||
best_index = self.select_best(results)
|
||||
logger.info(f"Auto-selected result {best_index}: {results[best_index].artist} - {results[best_index].title} ({results[best_index].provider})")
|
||||
best_result = results[best_index]
|
||||
logger.info(
|
||||
f"Auto-selected result {best_index}: "
|
||||
f"{best_result.artist} - {best_result.title} ({best_result.provider})"
|
||||
)
|
||||
|
||||
# Download
|
||||
return self.download(best_index, output_dir, output_filename)
|
||||
return self.download(best_index, output_dir, output_filename, gcs_path)
|
||||
|
||||
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Async version of search for use in async routes.
|
||||
|
||||
Note: Currently wraps sync search in executor. Future optimization
|
||||
could make this fully async.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(None, self.search, artist, title)
|
||||
|
||||
async def download_async(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
remote_search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Async version of download for use in async routes.
|
||||
|
||||
Note: Currently wraps sync download in executor. Future optimization
|
||||
could make this fully async.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download(result_index, output_dir, output_filename, gcs_path, remote_search_id)
|
||||
)
|
||||
|
||||
async def download_by_id_async(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Async version of download_by_id for use in async routes.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download_by_id(
|
||||
source_name, source_id, output_dir, output_filename,
|
||||
target_file, download_url, gcs_path
|
||||
)
|
||||
)
|
||||
|
||||
def is_remote_enabled(self) -> bool:
|
||||
"""Check if remote flacfetch service is configured."""
|
||||
return self._remote_client is not None
|
||||
|
||||
@property
|
||||
def last_remote_search_id(self) -> Optional[str]:
|
||||
"""Get the search_id from the last remote search.
|
||||
|
||||
This should be stored in job state_data after search and passed
|
||||
back to download() to ensure correct remote download handling.
|
||||
"""
|
||||
return self._remote_search_id
|
||||
|
||||
async def check_remote_health(self) -> Optional[dict]:
|
||||
"""
|
||||
Check health of remote flacfetch service.
|
||||
|
||||
Returns:
|
||||
Health status dict if remote service is configured and healthy,
|
||||
None if not configured or unhealthy.
|
||||
"""
|
||||
if not self._remote_client:
|
||||
return None
|
||||
|
||||
try:
|
||||
return await self._remote_client.health_check()
|
||||
except Exception as e:
|
||||
logger.warning(f"Remote flacfetch health check failed: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# Singleton instance
|
||||
@@ -344,5 +679,3 @@ def get_audio_search_service() -> AudioSearchService:
|
||||
if _audio_search_service is None:
|
||||
_audio_search_service = AudioSearchService()
|
||||
return _audio_search_service
|
||||
|
||||
|
||||
@@ -68,6 +68,16 @@ class AuthService:
|
||||
logger.info("Admin token validated")
|
||||
return True, UserType.ADMIN, -1, "Admin access granted"
|
||||
|
||||
# Debug: Log token comparison info (only first/last few chars for security)
|
||||
if self.admin_tokens:
|
||||
expected_prefix = self.admin_tokens[0][:8] if len(self.admin_tokens[0]) >= 8 else self.admin_tokens[0]
|
||||
provided_prefix = token[:8] if len(token) >= 8 else token
|
||||
logger.debug(
|
||||
f"Token mismatch: expected prefix '{expected_prefix}...', "
|
||||
f"got '{provided_prefix}...', "
|
||||
f"expected len={len(self.admin_tokens[0])}, got len={len(token)}"
|
||||
)
|
||||
|
||||
# Check stored tokens in Firestore
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
|
||||
@@ -0,0 +1,473 @@
|
||||
"""
|
||||
Client for the remote flacfetch HTTP API service.
|
||||
|
||||
This client communicates with a dedicated flacfetch VM that handles:
|
||||
- BitTorrent downloads from private trackers (RED, OPS)
|
||||
- YouTube downloads
|
||||
- GCS uploads of downloaded files
|
||||
|
||||
The flacfetch service provides:
|
||||
- Full peer connectivity for torrents (not possible in Cloud Run)
|
||||
- Indefinite seeding of completed torrents
|
||||
- Automatic disk cleanup
|
||||
|
||||
Usage:
|
||||
client = get_flacfetch_client()
|
||||
if client:
|
||||
# Search for audio
|
||||
search_result = await client.search("ABBA", "Waterloo")
|
||||
|
||||
# Download the best result
|
||||
download_id = await client.download(
|
||||
search_id=search_result["search_id"],
|
||||
result_index=0,
|
||||
gcs_path="uploads/job123/audio/",
|
||||
)
|
||||
|
||||
# Wait for completion
|
||||
result = await client.wait_for_download(download_id)
|
||||
print(f"Downloaded to: {result['gcs_path']}")
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FlacfetchServiceError(Exception):
|
||||
"""Error communicating with or returned from flacfetch service."""
|
||||
pass
|
||||
|
||||
|
||||
class FlacfetchClient:
|
||||
"""
|
||||
Client for remote flacfetch HTTP API.
|
||||
|
||||
All endpoints require authentication via X-API-Key header.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str,
|
||||
api_key: str,
|
||||
timeout: int = 60,
|
||||
):
|
||||
"""
|
||||
Initialize flacfetch client.
|
||||
|
||||
Args:
|
||||
base_url: Base URL of flacfetch service (e.g., http://10.0.0.5:8080)
|
||||
api_key: API key for authentication
|
||||
timeout: Default request timeout in seconds
|
||||
"""
|
||||
self.base_url = base_url.rstrip('/')
|
||||
self.api_key = api_key
|
||||
self.timeout = timeout
|
||||
logger.info(f"FlacfetchClient initialized with base_url={self.base_url}")
|
||||
|
||||
def _headers(self) -> Dict[str, str]:
|
||||
"""Get request headers with authentication."""
|
||||
return {
|
||||
"X-API-Key": self.api_key,
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check if flacfetch service is healthy.
|
||||
|
||||
Returns:
|
||||
Health status dict with transmission, disk, and provider info
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: If service is unhealthy or unreachable
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/health",
|
||||
headers=self._headers(),
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
if data.get("status") not in ["healthy", "degraded"]:
|
||||
raise FlacfetchServiceError(f"Service unhealthy: {data}")
|
||||
|
||||
return data
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
|
||||
|
||||
async def search(
|
||||
self,
|
||||
artist: str,
|
||||
title: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Search for audio matching artist and title.
|
||||
|
||||
Args:
|
||||
artist: Artist name
|
||||
title: Track title
|
||||
|
||||
Returns:
|
||||
Search response dict with search_id and results list
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On search failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/search",
|
||||
headers=self._headers(),
|
||||
json={"artist": artist, "title": title},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
if resp.status_code == 404:
|
||||
# No results found - return empty results
|
||||
return {
|
||||
"search_id": None,
|
||||
"artist": artist,
|
||||
"title": title,
|
||||
"results": [],
|
||||
"results_count": 0,
|
||||
}
|
||||
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Search request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def download(
|
||||
self,
|
||||
search_id: str,
|
||||
result_index: int,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Start downloading an audio file.
|
||||
|
||||
Args:
|
||||
search_id: Search ID from previous search
|
||||
result_index: Index of result to download
|
||||
output_filename: Optional custom filename (without extension)
|
||||
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
|
||||
|
||||
Returns:
|
||||
Download ID for tracking progress
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"search_id": search_id,
|
||||
"result_index": result_index,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Start downloading directly by source ID (no prior search required).
|
||||
|
||||
This is useful when you have stored the source_id from a previous search
|
||||
and want to download later without re-searching.
|
||||
|
||||
Args:
|
||||
source_name: Provider name (RED, OPS, YouTube, Spotify)
|
||||
source_id: Source-specific ID (torrent ID, video ID, track ID)
|
||||
output_filename: Optional custom filename (without extension)
|
||||
target_file: For torrents, specific file to extract
|
||||
download_url: For YouTube/Spotify, direct URL (optional)
|
||||
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
|
||||
|
||||
Returns:
|
||||
Download ID for tracking progress
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"source_name": source_name,
|
||||
"source_id": source_id,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if target_file:
|
||||
payload["target_file"] = target_file
|
||||
if download_url:
|
||||
payload["download_url"] = download_url
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download-by-id",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of a download.
|
||||
|
||||
Args:
|
||||
download_id: Download ID from start_download
|
||||
|
||||
Returns:
|
||||
Status dict with progress, speed, path info
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On status check failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/download/{download_id}/status",
|
||||
headers=self._headers(),
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Status request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 404:
|
||||
raise FlacfetchServiceError(f"Download not found: {download_id}")
|
||||
raise FlacfetchServiceError(f"Status check failed: {e.response.status_code}")
|
||||
|
||||
async def wait_for_download(
|
||||
self,
|
||||
download_id: str,
|
||||
timeout: int = 600,
|
||||
poll_interval: float = 2.0,
|
||||
progress_callback: Optional[callable] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Wait for a download to complete.
|
||||
|
||||
Args:
|
||||
download_id: Download ID to wait for
|
||||
timeout: Maximum wait time in seconds
|
||||
poll_interval: Time between status checks
|
||||
progress_callback: Optional callback(status_dict) for progress updates
|
||||
|
||||
Returns:
|
||||
Final status dict with output_path or gcs_path
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download failure or timeout
|
||||
"""
|
||||
elapsed = 0.0
|
||||
|
||||
while elapsed < timeout:
|
||||
status = await self.get_download_status(download_id)
|
||||
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(status)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback error: {e}")
|
||||
|
||||
download_status = status.get("status")
|
||||
|
||||
if download_status in ["complete", "seeding"]:
|
||||
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
|
||||
return status
|
||||
elif download_status == "failed":
|
||||
error = status.get("error", "Unknown error")
|
||||
raise FlacfetchServiceError(f"Download failed: {error}")
|
||||
elif download_status == "cancelled":
|
||||
raise FlacfetchServiceError("Download was cancelled")
|
||||
|
||||
# Log progress
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
peers = status.get("peers", 0)
|
||||
logger.debug(
|
||||
f"Download {download_id}: {progress:.1f}% "
|
||||
f"({speed:.1f} KB/s, {peers} peers)"
|
||||
)
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
elapsed += poll_interval
|
||||
|
||||
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
|
||||
|
||||
async def list_torrents(self) -> Dict[str, Any]:
|
||||
"""
|
||||
List all torrents in Transmission.
|
||||
|
||||
Returns:
|
||||
Torrent list response with torrent info and totals
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/torrents",
|
||||
headers=self._headers(),
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
|
||||
|
||||
async def delete_torrent(
|
||||
self,
|
||||
torrent_id: int,
|
||||
delete_data: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Delete a torrent from Transmission.
|
||||
|
||||
Args:
|
||||
torrent_id: Transmission torrent ID
|
||||
delete_data: Also delete downloaded files
|
||||
|
||||
Returns:
|
||||
Delete response
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.delete(
|
||||
f"{self.base_url}/torrents/{torrent_id}",
|
||||
headers=self._headers(),
|
||||
params={"delete_data": str(delete_data).lower()},
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
|
||||
|
||||
async def cleanup_torrents(
|
||||
self,
|
||||
strategy: str = "oldest",
|
||||
target_free_gb: float = 10.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Trigger disk cleanup.
|
||||
|
||||
Args:
|
||||
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
|
||||
target_free_gb: Target free space to achieve
|
||||
|
||||
Returns:
|
||||
Cleanup response with removed count and freed space
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/torrents/cleanup",
|
||||
headers=self._headers(),
|
||||
json={
|
||||
"strategy": strategy,
|
||||
"target_free_gb": target_free_gb,
|
||||
},
|
||||
timeout=60,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_client: Optional[FlacfetchClient] = None
|
||||
|
||||
|
||||
def get_flacfetch_client() -> Optional[FlacfetchClient]:
|
||||
"""
|
||||
Get the flacfetch client if configured.
|
||||
|
||||
Returns:
|
||||
FlacfetchClient instance if FLACFETCH_API_URL and FLACFETCH_API_KEY are set,
|
||||
None otherwise (indicating local-only mode).
|
||||
"""
|
||||
global _client
|
||||
|
||||
if _client is None:
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
|
||||
if settings.flacfetch_api_url and settings.flacfetch_api_key:
|
||||
_client = FlacfetchClient(
|
||||
base_url=settings.flacfetch_api_url,
|
||||
api_key=settings.flacfetch_api_key,
|
||||
)
|
||||
logger.info("FlacfetchClient initialized for remote service")
|
||||
else:
|
||||
logger.debug("FlacfetchClient not configured (missing URL or API key)")
|
||||
|
||||
return _client
|
||||
|
||||
|
||||
def reset_flacfetch_client():
|
||||
"""Reset the singleton client (for testing)."""
|
||||
global _client
|
||||
_client = None
|
||||
|
||||
@@ -53,8 +53,14 @@ class JobManager:
|
||||
enable_txt=job_create.enable_txt,
|
||||
enable_youtube_upload=job_create.enable_youtube_upload,
|
||||
youtube_description=job_create.youtube_description,
|
||||
youtube_description_template=job_create.youtube_description_template, # video_worker reads this
|
||||
webhook_url=job_create.webhook_url,
|
||||
user_email=job_create.user_email,
|
||||
# Distribution settings
|
||||
brand_prefix=job_create.brand_prefix,
|
||||
discord_webhook_url=job_create.discord_webhook_url,
|
||||
dropbox_path=job_create.dropbox_path,
|
||||
gdrive_folder_id=job_create.gdrive_folder_id,
|
||||
# Two-phase workflow (Batch 6)
|
||||
prep_only=job_create.prep_only,
|
||||
finalise_only=job_create.finalise_only,
|
||||
@@ -280,6 +286,24 @@ class JobManager:
|
||||
if progress is not None:
|
||||
updates['progress'] = progress
|
||||
|
||||
# Generate review token when entering AWAITING_REVIEW state
|
||||
if new_status == JobStatus.AWAITING_REVIEW:
|
||||
from backend.api.dependencies import generate_review_token, get_review_token_expiry
|
||||
review_token = generate_review_token()
|
||||
review_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
|
||||
updates['review_token'] = review_token
|
||||
updates['review_token_expires_at'] = review_token_expires
|
||||
logger.info(f"Generated review token for job {job_id}, expires in 48 hours")
|
||||
|
||||
# Generate instrumental token when entering AWAITING_INSTRUMENTAL_SELECTION state
|
||||
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
|
||||
from backend.api.dependencies import generate_review_token, get_review_token_expiry
|
||||
instrumental_token = generate_review_token() # Reuse same token generator
|
||||
instrumental_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
|
||||
updates['instrumental_token'] = instrumental_token
|
||||
updates['instrumental_token_expires_at'] = instrumental_token_expires
|
||||
logger.info(f"Generated instrumental token for job {job_id}, expires in 48 hours")
|
||||
|
||||
# If we have state_data_updates, merge them with existing state_data
|
||||
merged_state_data = None
|
||||
if state_data_updates:
|
||||
@@ -304,6 +328,20 @@ class JobManager:
|
||||
message=message
|
||||
)
|
||||
|
||||
# Apply review token update separately if generated
|
||||
if new_status == JobStatus.AWAITING_REVIEW and 'review_token' in updates:
|
||||
self.firestore.update_job(job_id, {
|
||||
'review_token': updates['review_token'],
|
||||
'review_token_expires_at': updates['review_token_expires_at']
|
||||
})
|
||||
|
||||
# Apply instrumental token update separately if generated
|
||||
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION and 'instrumental_token' in updates:
|
||||
self.firestore.update_job(job_id, {
|
||||
'instrumental_token': updates['instrumental_token'],
|
||||
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
|
||||
})
|
||||
|
||||
logger.info(f"Job {job_id} transitioned to {new_status}")
|
||||
return True
|
||||
|
||||
|
||||
@@ -0,0 +1,413 @@
|
||||
"""
|
||||
Custom metrics for Cloud Monitoring.
|
||||
|
||||
This module provides application-level metrics for tracking job processing,
|
||||
worker performance, and external API usage. Metrics can be viewed in:
|
||||
1. Cloud Logging (via structured log entries)
|
||||
2. Cloud Monitoring (via log-based metrics or OpenTelemetry)
|
||||
|
||||
The metrics service uses a pragmatic approach:
|
||||
- Always emits metrics as structured log entries (works immediately)
|
||||
- Uses the same JSON format as Cloud Logging
|
||||
- Can be enhanced with OpenTelemetry metrics exporters when available
|
||||
|
||||
Usage:
|
||||
from backend.services.metrics import metrics
|
||||
|
||||
# Record a job completion
|
||||
metrics.record_job_completed("abc123", source="upload")
|
||||
|
||||
# Record worker duration
|
||||
metrics.record_worker_duration("audio", 45.2, success=True)
|
||||
|
||||
# Record external API call
|
||||
with metrics.time_external_api("modal"):
|
||||
response = await modal_client.separate_audio(...)
|
||||
"""
|
||||
import logging
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from backend.services.tracing import get_current_trace_id, get_current_span_id
|
||||
|
||||
|
||||
logger = logging.getLogger("metrics")
|
||||
|
||||
|
||||
@dataclass
|
||||
class MetricLabels:
|
||||
"""Common metric labels."""
|
||||
job_id: Optional[str] = None
|
||||
worker: Optional[str] = None
|
||||
status: Optional[str] = None
|
||||
source: Optional[str] = None
|
||||
api: Optional[str] = None
|
||||
operation: Optional[str] = None
|
||||
bucket: Optional[str] = None
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""Convert to dict, excluding None values."""
|
||||
return {k: v for k, v in self.__dict__.items() if v is not None}
|
||||
|
||||
|
||||
class MetricsService:
|
||||
"""
|
||||
Application metrics service.
|
||||
|
||||
Emits metrics as structured log entries that can be:
|
||||
1. Queried directly in Cloud Logging
|
||||
2. Converted to Cloud Monitoring metrics via log-based metrics
|
||||
3. Exported via OpenTelemetry (when configured)
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize the metrics service."""
|
||||
self._logger = logging.getLogger("metrics")
|
||||
# Ensure metrics logger outputs at INFO level
|
||||
self._logger.setLevel(logging.INFO)
|
||||
|
||||
def _emit_metric(self, metric_name: str, metric_type: str, value: float, labels: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Emit a metric as a structured log entry.
|
||||
|
||||
The log format is designed to be easily parsed by Cloud Logging
|
||||
and converted to log-based metrics.
|
||||
|
||||
Args:
|
||||
metric_name: Name of the metric (e.g., "jobs_total")
|
||||
metric_type: Type of metric (counter, histogram, gauge)
|
||||
value: Metric value
|
||||
labels: Metric labels/dimensions
|
||||
"""
|
||||
# Add trace context if available
|
||||
trace_id = get_current_trace_id()
|
||||
span_id = get_current_span_id()
|
||||
|
||||
# Build metric entry
|
||||
metric_entry = {
|
||||
"metric_name": metric_name,
|
||||
"metric_type": metric_type,
|
||||
"metric_value": value,
|
||||
**labels,
|
||||
}
|
||||
|
||||
if trace_id:
|
||||
metric_entry["trace_id"] = trace_id
|
||||
if span_id:
|
||||
metric_entry["span_id"] = span_id
|
||||
|
||||
# Emit as structured log entry
|
||||
# Use INFO level so metrics always show up
|
||||
self._logger.info(
|
||||
f"METRIC {metric_name}={value}",
|
||||
extra=metric_entry
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# Job Metrics
|
||||
# =========================================
|
||||
|
||||
def record_job_created(self, job_id: str, source: str = "unknown") -> None:
|
||||
"""
|
||||
Record a new job creation.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
source: Job source (upload, url, search)
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="jobs_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"job_id": job_id, "status": "created", "source": source}
|
||||
)
|
||||
|
||||
def record_job_completed(self, job_id: str, source: str = "unknown") -> None:
|
||||
"""
|
||||
Record a job completion.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
source: Job source (upload, url, search)
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="jobs_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"job_id": job_id, "status": "completed", "source": source}
|
||||
)
|
||||
|
||||
def record_job_failed(self, job_id: str, source: str = "unknown", error: Optional[str] = None) -> None:
|
||||
"""
|
||||
Record a job failure.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
source: Job source (upload, url, search)
|
||||
error: Optional error message
|
||||
"""
|
||||
labels = {"job_id": job_id, "status": "failed", "source": source}
|
||||
if error:
|
||||
labels["error"] = error[:200] # Truncate long errors
|
||||
self._emit_metric(
|
||||
metric_name="jobs_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
def record_job_duration(self, job_id: str, duration_seconds: float, source: str = "unknown") -> None:
|
||||
"""
|
||||
Record total job processing duration.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
duration_seconds: Total processing time in seconds
|
||||
source: Job source (upload, url, search)
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="job_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels={"job_id": job_id, "source": source}
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# Worker Metrics
|
||||
# =========================================
|
||||
|
||||
def record_worker_started(self, worker: str, job_id: str) -> None:
|
||||
"""
|
||||
Record a worker invocation start.
|
||||
|
||||
Args:
|
||||
worker: Worker name (audio, lyrics, screens, video, render_video)
|
||||
job_id: Job ID
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="worker_invocations_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"worker": worker, "job_id": job_id, "status": "started"}
|
||||
)
|
||||
|
||||
def record_worker_duration(self, worker: str, duration_seconds: float, success: bool, job_id: Optional[str] = None) -> None:
|
||||
"""
|
||||
Record worker processing duration.
|
||||
|
||||
Args:
|
||||
worker: Worker name
|
||||
duration_seconds: Processing time in seconds
|
||||
success: Whether worker completed successfully
|
||||
job_id: Optional job ID
|
||||
"""
|
||||
labels = {
|
||||
"worker": worker,
|
||||
"success": str(success).lower(),
|
||||
}
|
||||
if job_id:
|
||||
labels["job_id"] = job_id
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="job_stage_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
# Also emit a counter for success/failure tracking
|
||||
self._emit_metric(
|
||||
metric_name="worker_invocations_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"worker": worker, "success": str(success).lower(), "job_id": job_id or "unknown"}
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# GCS Metrics
|
||||
# =========================================
|
||||
|
||||
def record_gcs_operation(
|
||||
self,
|
||||
operation: str,
|
||||
bucket: str,
|
||||
success: bool,
|
||||
size_bytes: Optional[int] = None,
|
||||
duration_seconds: Optional[float] = None,
|
||||
job_id: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Record a GCS operation.
|
||||
|
||||
Args:
|
||||
operation: Operation type (upload, download, delete)
|
||||
bucket: GCS bucket name
|
||||
success: Whether operation succeeded
|
||||
size_bytes: Optional file size in bytes
|
||||
duration_seconds: Optional operation duration
|
||||
job_id: Optional job ID
|
||||
"""
|
||||
labels = {
|
||||
"operation": operation,
|
||||
"bucket": bucket,
|
||||
"success": str(success).lower(),
|
||||
}
|
||||
if job_id:
|
||||
labels["job_id"] = job_id
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="gcs_operations_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
if size_bytes is not None:
|
||||
self._emit_metric(
|
||||
metric_name="gcs_operation_bytes",
|
||||
metric_type="histogram",
|
||||
value=size_bytes,
|
||||
labels={**labels, "operation": operation}
|
||||
)
|
||||
|
||||
if duration_seconds is not None:
|
||||
self._emit_metric(
|
||||
metric_name="gcs_operation_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels={**labels, "operation": operation}
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# External API Metrics
|
||||
# =========================================
|
||||
|
||||
def record_external_api_call(
|
||||
self,
|
||||
api: str,
|
||||
success: bool,
|
||||
duration_seconds: float,
|
||||
job_id: Optional[str] = None,
|
||||
error: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Record an external API call.
|
||||
|
||||
Args:
|
||||
api: API name (modal, audioshake, genius, spotify)
|
||||
success: Whether call succeeded
|
||||
duration_seconds: API call duration
|
||||
job_id: Optional job ID
|
||||
error: Optional error message
|
||||
"""
|
||||
labels = {
|
||||
"api": api,
|
||||
"success": str(success).lower(),
|
||||
}
|
||||
if job_id:
|
||||
labels["job_id"] = job_id
|
||||
if error:
|
||||
labels["error"] = error[:100] # Truncate
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="external_api_calls_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="external_api_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels={"api": api, "success": str(success).lower()}
|
||||
)
|
||||
|
||||
@contextmanager
|
||||
def time_external_api(self, api: str, job_id: Optional[str] = None):
|
||||
"""
|
||||
Context manager to time an external API call.
|
||||
|
||||
Usage:
|
||||
with metrics.time_external_api("modal", job_id) as timer:
|
||||
response = await client.call_api()
|
||||
timer.set_success(True)
|
||||
|
||||
Args:
|
||||
api: API name
|
||||
job_id: Optional job ID
|
||||
|
||||
Yields:
|
||||
Timer object with set_success() method
|
||||
"""
|
||||
timer = _ApiTimer()
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
yield timer
|
||||
except Exception as e:
|
||||
timer.set_success(False)
|
||||
timer.error = str(e)
|
||||
raise
|
||||
finally:
|
||||
duration = time.time() - start_time
|
||||
self.record_external_api_call(
|
||||
api=api,
|
||||
success=timer.success,
|
||||
duration_seconds=duration,
|
||||
job_id=job_id,
|
||||
error=timer.error,
|
||||
)
|
||||
|
||||
@contextmanager
|
||||
def time_worker(self, worker: str, job_id: str):
|
||||
"""
|
||||
Context manager to time a worker execution.
|
||||
|
||||
Usage:
|
||||
with metrics.time_worker("audio", job_id) as timer:
|
||||
await process_audio()
|
||||
timer.set_success(True)
|
||||
|
||||
Args:
|
||||
worker: Worker name
|
||||
job_id: Job ID
|
||||
|
||||
Yields:
|
||||
Timer object with set_success() method
|
||||
"""
|
||||
timer = _ApiTimer()
|
||||
self.record_worker_started(worker, job_id)
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
yield timer
|
||||
except Exception as e:
|
||||
timer.set_success(False)
|
||||
raise
|
||||
finally:
|
||||
duration = time.time() - start_time
|
||||
self.record_worker_duration(
|
||||
worker=worker,
|
||||
duration_seconds=duration,
|
||||
success=timer.success,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
|
||||
class _ApiTimer:
|
||||
"""Helper class for tracking API call success state."""
|
||||
|
||||
def __init__(self):
|
||||
self.success = True # Assume success unless set otherwise
|
||||
self.error: Optional[str] = None
|
||||
|
||||
def set_success(self, success: bool) -> None:
|
||||
self.success = success
|
||||
|
||||
|
||||
# Global metrics instance
|
||||
metrics = MetricsService()
|
||||
|
||||
@@ -0,0 +1,246 @@
|
||||
"""
|
||||
Structured logging with trace correlation for Cloud Logging.
|
||||
|
||||
This module provides JSON-formatted logging that integrates with Google Cloud Logging
|
||||
and correlates logs with OpenTelemetry traces in Cloud Trace.
|
||||
|
||||
When running in Cloud Run:
|
||||
- Logs are output as JSON for Cloud Logging to parse
|
||||
- Each log entry includes trace ID and span ID for correlation
|
||||
- Custom fields (job_id, worker) are preserved in the log structure
|
||||
|
||||
Usage:
|
||||
# In main.py, before any logging:
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
setup_structured_logging()
|
||||
|
||||
# Then use standard logging:
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.info("Processing started", extra={"job_id": "abc123"})
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
# Cloud Logging severity mapping (compatible with GCP)
|
||||
# https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity
|
||||
SEVERITY_MAP = {
|
||||
"DEBUG": "DEBUG",
|
||||
"INFO": "INFO",
|
||||
"WARNING": "WARNING",
|
||||
"ERROR": "ERROR",
|
||||
"CRITICAL": "CRITICAL",
|
||||
}
|
||||
|
||||
|
||||
class StructuredFormatter(logging.Formatter):
|
||||
"""
|
||||
JSON formatter with trace correlation for Google Cloud Logging.
|
||||
|
||||
Produces log entries in the format expected by Cloud Logging, including:
|
||||
- logging.googleapis.com/trace: Links to Cloud Trace
|
||||
- logging.googleapis.com/spanId: Current span ID
|
||||
- severity: Cloud Logging severity level
|
||||
- Custom fields passed via 'extra' dict
|
||||
"""
|
||||
|
||||
def __init__(self, project_id: Optional[str] = None):
|
||||
"""
|
||||
Initialize the structured formatter.
|
||||
|
||||
Args:
|
||||
project_id: GCP project ID for trace URLs (auto-detected if not provided)
|
||||
"""
|
||||
super().__init__()
|
||||
self.project_id = project_id or os.environ.get("GOOGLE_CLOUD_PROJECT") or os.environ.get("GCP_PROJECT")
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
"""
|
||||
Format a log record as JSON.
|
||||
|
||||
Args:
|
||||
record: The log record to format
|
||||
|
||||
Returns:
|
||||
JSON string for Cloud Logging
|
||||
"""
|
||||
from backend.services.tracing import get_current_trace_id, get_current_span_id
|
||||
|
||||
# Build base log entry
|
||||
log_entry: Dict[str, Any] = {
|
||||
"timestamp": datetime.utcfromtimestamp(record.created).isoformat() + "Z",
|
||||
"severity": SEVERITY_MAP.get(record.levelname, record.levelname),
|
||||
"message": record.getMessage(),
|
||||
"logger": record.name,
|
||||
}
|
||||
|
||||
# Add trace correlation if available
|
||||
trace_id = get_current_trace_id()
|
||||
span_id = get_current_span_id()
|
||||
|
||||
if trace_id and self.project_id:
|
||||
# Format for Cloud Logging trace correlation
|
||||
log_entry["logging.googleapis.com/trace"] = f"projects/{self.project_id}/traces/{trace_id}"
|
||||
|
||||
if span_id:
|
||||
log_entry["logging.googleapis.com/spanId"] = span_id
|
||||
|
||||
# Add custom fields from 'extra' dict
|
||||
# Common fields we want to extract from log records
|
||||
custom_fields = ["job_id", "worker", "operation", "duration", "status", "error"]
|
||||
for field in custom_fields:
|
||||
value = getattr(record, field, None)
|
||||
if value is not None:
|
||||
log_entry[field] = value
|
||||
|
||||
# Add source location for debugging
|
||||
if record.pathname and record.lineno:
|
||||
log_entry["logging.googleapis.com/sourceLocation"] = {
|
||||
"file": record.pathname,
|
||||
"line": record.lineno,
|
||||
"function": record.funcName,
|
||||
}
|
||||
|
||||
# Add exception info if present
|
||||
if record.exc_info:
|
||||
log_entry["exception"] = self.formatException(record.exc_info)
|
||||
|
||||
# Remove None values for cleaner output
|
||||
return json.dumps({k: v for k, v in log_entry.items() if v is not None})
|
||||
|
||||
|
||||
class HumanReadableFormatter(logging.Formatter):
|
||||
"""
|
||||
Human-readable formatter for local development.
|
||||
|
||||
Includes trace context when available but outputs in traditional format.
|
||||
"""
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
"""Format a log record for human readability."""
|
||||
from backend.services.tracing import get_current_trace_id
|
||||
|
||||
# Build base message
|
||||
timestamp = datetime.utcfromtimestamp(record.created).strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
# Add job_id prefix if available
|
||||
job_id = getattr(record, "job_id", None)
|
||||
job_prefix = f"[job:{job_id[:8]}] " if job_id else ""
|
||||
|
||||
# Add trace ID suffix if available
|
||||
trace_id = get_current_trace_id()
|
||||
trace_suffix = f" [trace:{trace_id[:8]}]" if trace_id else ""
|
||||
|
||||
message = f"{timestamp} - {record.name} - {record.levelname} - {job_prefix}{record.getMessage()}{trace_suffix}"
|
||||
|
||||
# Add exception info if present
|
||||
if record.exc_info:
|
||||
message += "\n" + self.formatException(record.exc_info)
|
||||
|
||||
return message
|
||||
|
||||
|
||||
def is_cloud_run() -> bool:
|
||||
"""Check if we're running in Cloud Run."""
|
||||
return os.environ.get("K_SERVICE") is not None
|
||||
|
||||
|
||||
def setup_structured_logging(force_json: bool = False, log_level: Optional[str] = None) -> None:
|
||||
"""
|
||||
Configure structured logging for the application.
|
||||
|
||||
In Cloud Run: Uses JSON format with trace correlation for Cloud Logging
|
||||
In development: Uses human-readable format with optional trace context
|
||||
|
||||
Args:
|
||||
force_json: Force JSON output even in development
|
||||
log_level: Override log level (default from settings or INFO)
|
||||
"""
|
||||
from backend.config import settings
|
||||
|
||||
# Determine log level
|
||||
level = log_level or getattr(settings, "log_level", "INFO")
|
||||
level = getattr(logging, level.upper(), logging.INFO)
|
||||
|
||||
# Choose formatter based on environment
|
||||
if is_cloud_run() or force_json:
|
||||
formatter = StructuredFormatter()
|
||||
else:
|
||||
formatter = HumanReadableFormatter()
|
||||
|
||||
# Create handler
|
||||
handler = logging.StreamHandler(sys.stdout)
|
||||
handler.setFormatter(formatter)
|
||||
handler.setLevel(level)
|
||||
|
||||
# Configure root logger
|
||||
root_logger = logging.getLogger()
|
||||
root_logger.setLevel(level)
|
||||
|
||||
# Remove existing handlers and add our handler
|
||||
root_logger.handlers = [handler]
|
||||
|
||||
# Also configure uvicorn loggers to use our format
|
||||
for logger_name in ["uvicorn", "uvicorn.access", "uvicorn.error"]:
|
||||
logger = logging.getLogger(logger_name)
|
||||
logger.handlers = [handler]
|
||||
logger.propagate = False
|
||||
|
||||
|
||||
class JobLogAdapter(logging.LoggerAdapter):
|
||||
"""
|
||||
Logger adapter that automatically adds job context to log records.
|
||||
|
||||
Usage:
|
||||
logger = logging.getLogger(__name__)
|
||||
job_logger = JobLogAdapter(logger, job_id="abc123", worker="audio")
|
||||
job_logger.info("Processing started") # Automatically includes job_id and worker
|
||||
"""
|
||||
|
||||
def __init__(self, logger: logging.Logger, job_id: str, worker: Optional[str] = None, **extra):
|
||||
"""
|
||||
Initialize the adapter.
|
||||
|
||||
Args:
|
||||
logger: Base logger instance
|
||||
job_id: Job ID to include in all log records
|
||||
worker: Optional worker name
|
||||
**extra: Additional fields to include in all log records
|
||||
"""
|
||||
super().__init__(logger, {"job_id": job_id, "worker": worker, **extra})
|
||||
|
||||
def process(self, msg: str, kwargs: Dict[str, Any]) -> tuple:
|
||||
"""Process a logging call to add extra context."""
|
||||
# Merge extra dict from adapter with any extra passed to the log call
|
||||
extra = kwargs.get("extra", {})
|
||||
extra.update(self.extra)
|
||||
kwargs["extra"] = extra
|
||||
return msg, kwargs
|
||||
|
||||
|
||||
def get_job_logger(job_id: str, worker: Optional[str] = None, name: Optional[str] = None) -> JobLogAdapter:
|
||||
"""
|
||||
Get a logger adapter configured for a specific job.
|
||||
|
||||
This is a convenience function for creating job-specific loggers.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to include in all log records
|
||||
worker: Optional worker name
|
||||
name: Logger name (defaults to "job")
|
||||
|
||||
Returns:
|
||||
JobLogAdapter configured for the job
|
||||
|
||||
Usage:
|
||||
logger = get_job_logger("abc123", "audio")
|
||||
logger.info("Starting audio separation")
|
||||
logger.error("Failed to process", extra={"error": str(e)})
|
||||
"""
|
||||
base_logger = logging.getLogger(name or "job")
|
||||
return JobLogAdapter(base_logger, job_id=job_id, worker=worker)
|
||||
|
||||
@@ -19,6 +19,25 @@ Usage:
|
||||
with tracer.start_as_current_span("my-operation") as span:
|
||||
span.set_attribute("job_id", job_id)
|
||||
# ... do work ...
|
||||
|
||||
# For job-specific spans with automatic job_id attribute
|
||||
from backend.services.tracing import job_span
|
||||
|
||||
with job_span("audio-worker", job_id) as span:
|
||||
# job_id is automatically set as an attribute
|
||||
span.set_attribute("stage", "separation")
|
||||
# ... do work ...
|
||||
|
||||
# For propagating trace context through Cloud Tasks
|
||||
from backend.services.tracing import inject_trace_context, extract_trace_context
|
||||
|
||||
# In worker_service.py (when creating task):
|
||||
headers = inject_trace_context({})
|
||||
|
||||
# In internal.py (when receiving task):
|
||||
context = extract_trace_context(request.headers)
|
||||
with tracer.start_as_current_span("worker", context=context):
|
||||
...
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
@@ -380,3 +399,145 @@ def get_current_span_id() -> Optional[str]:
|
||||
if ctx.is_valid:
|
||||
return format(ctx.span_id, '016x')
|
||||
return None
|
||||
|
||||
|
||||
# =====================================================
|
||||
# Job-Aware Tracing Utilities
|
||||
# =====================================================
|
||||
|
||||
@contextmanager
|
||||
def job_span(
|
||||
name: str,
|
||||
job_id: str,
|
||||
attributes: Optional[Dict[str, Any]] = None,
|
||||
kind: trace.SpanKind = trace.SpanKind.INTERNAL,
|
||||
):
|
||||
"""
|
||||
Create a traced span with job_id automatically set as an attribute.
|
||||
|
||||
This is the preferred way to create spans in job processing workers.
|
||||
The job_id attribute enables filtering spans by job in Cloud Trace.
|
||||
|
||||
Usage:
|
||||
with job_span("audio-separation", job_id) as span:
|
||||
span.set_attribute("stage", "clean_instrumental")
|
||||
# ... do work ...
|
||||
|
||||
Args:
|
||||
name: Span name (appears in Cloud Trace)
|
||||
job_id: Job ID to attach to the span
|
||||
attributes: Additional span attributes
|
||||
kind: Span kind (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER)
|
||||
|
||||
Yields:
|
||||
The active span
|
||||
"""
|
||||
all_attributes = {
|
||||
"job_id": job_id,
|
||||
"service.operation": name,
|
||||
}
|
||||
if attributes:
|
||||
all_attributes.update(attributes)
|
||||
|
||||
with create_span(name, all_attributes, kind) as span:
|
||||
yield span
|
||||
|
||||
|
||||
def inject_trace_context(headers: Dict[str, str]) -> Dict[str, str]:
|
||||
"""
|
||||
Inject current trace context into HTTP headers.
|
||||
|
||||
Use this when creating Cloud Tasks or making HTTP calls to propagate
|
||||
the trace context so child spans link to the parent trace.
|
||||
|
||||
Usage:
|
||||
headers = inject_trace_context({
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": "Bearer token",
|
||||
})
|
||||
# Use headers in HTTP request or Cloud Task
|
||||
|
||||
Args:
|
||||
headers: Existing headers dict to add trace context to
|
||||
|
||||
Returns:
|
||||
Headers dict with traceparent/tracestate headers added
|
||||
"""
|
||||
propagator = TraceContextTextMapPropagator()
|
||||
carrier = dict(headers) # Make a copy
|
||||
propagator.inject(carrier)
|
||||
return carrier
|
||||
|
||||
|
||||
def extract_trace_context(headers: Dict[str, str]) -> Optional[trace.Context]:
|
||||
"""
|
||||
Extract trace context from HTTP headers.
|
||||
|
||||
Use this when receiving Cloud Tasks or HTTP requests to continue
|
||||
the trace from the parent span.
|
||||
|
||||
Usage:
|
||||
context = extract_trace_context(request.headers)
|
||||
with tracer.start_as_current_span("worker", context=context):
|
||||
# ... processing ...
|
||||
|
||||
Args:
|
||||
headers: HTTP headers containing traceparent/tracestate
|
||||
|
||||
Returns:
|
||||
Context object for use with start_as_current_span, or None
|
||||
"""
|
||||
propagator = TraceContextTextMapPropagator()
|
||||
return propagator.extract(headers)
|
||||
|
||||
|
||||
def start_span_with_context(
|
||||
name: str,
|
||||
context: Optional[trace.Context],
|
||||
job_id: Optional[str] = None,
|
||||
attributes: Optional[Dict[str, Any]] = None,
|
||||
):
|
||||
"""
|
||||
Start a span as child of the given context (or current context if None).
|
||||
|
||||
This is useful for starting worker spans that should link to the
|
||||
trace from the original API request.
|
||||
|
||||
Usage:
|
||||
context = extract_trace_context(request.headers)
|
||||
with start_span_with_context("audio-worker", context, job_id) as span:
|
||||
# ... processing ...
|
||||
|
||||
Args:
|
||||
name: Span name
|
||||
context: Parent context (from extract_trace_context) or None
|
||||
job_id: Optional job ID to set as attribute
|
||||
attributes: Additional span attributes
|
||||
|
||||
Returns:
|
||||
Context manager yielding the span
|
||||
"""
|
||||
tracer = get_tracer()
|
||||
|
||||
all_attributes = attributes or {}
|
||||
if job_id:
|
||||
all_attributes["job_id"] = job_id
|
||||
all_attributes["service.operation"] = name
|
||||
|
||||
@contextmanager
|
||||
def _span():
|
||||
kwargs = {"kind": trace.SpanKind.INTERNAL}
|
||||
if context is not None:
|
||||
kwargs["context"] = context
|
||||
|
||||
with tracer.start_as_current_span(name, **kwargs) as span:
|
||||
for key, value in all_attributes.items():
|
||||
span.set_attribute(key, value)
|
||||
try:
|
||||
yield span
|
||||
except Exception as e:
|
||||
span.set_status(Status(StatusCode.ERROR, str(e)))
|
||||
span.record_exception(e)
|
||||
raise
|
||||
|
||||
return _span()
|
||||
@@ -10,6 +10,10 @@ SOLID Principles:
|
||||
- Single Responsibility: Only handles worker coordination
|
||||
- Dependency Inversion: Depends on HTTP abstraction, not implementation
|
||||
- Open/Closed: Can add new workers without modifying existing code
|
||||
|
||||
Observability:
|
||||
- Propagates trace context through Cloud Tasks for distributed tracing
|
||||
- All worker invocations create spans linked to original request trace
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
@@ -18,6 +22,7 @@ from typing import Optional
|
||||
import httpx
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.services.tracing import inject_trace_context
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -180,6 +185,9 @@ class WorkerService:
|
||||
- Deduplication (via task name if needed)
|
||||
- OIDC authentication for Cloud Run
|
||||
|
||||
Observability:
|
||||
- Injects trace context headers so worker spans link to parent trace
|
||||
|
||||
Args:
|
||||
worker_type: Worker type
|
||||
job_id: Job ID to process
|
||||
@@ -205,14 +213,33 @@ class WorkerService:
|
||||
# Build queue path
|
||||
parent = self.tasks_client.queue_path(project, location, queue_name)
|
||||
|
||||
# Build base headers
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
# Add admin auth via custom header
|
||||
# NOTE: We use X-Admin-Token instead of Authorization because Cloud Tasks
|
||||
# OIDC token overwrites the Authorization header when oidc_token is specified.
|
||||
# The OIDC token handles Cloud Run authentication (allows Cloud Tasks to invoke
|
||||
# the service), while X-Admin-Token handles application-level authentication.
|
||||
if self._admin_token:
|
||||
headers["X-Admin-Token"] = self._admin_token
|
||||
logger.debug(
|
||||
f"[job:{job_id}] Using admin token for Cloud Task auth via X-Admin-Token, "
|
||||
f"token prefix: {self._admin_token[:8]}..., len={len(self._admin_token)}"
|
||||
)
|
||||
|
||||
# Inject trace context for distributed tracing
|
||||
# This allows worker spans to link back to the original request trace
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": tasks_v2.HttpMethod.POST,
|
||||
"url": f"{self._base_url}/api/internal/workers/{worker_type}",
|
||||
"headers": {
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
"headers": headers,
|
||||
"body": json.dumps({"job_id": job_id}).encode(),
|
||||
# Use OIDC token for Cloud Run authentication
|
||||
# This allows Cloud Tasks to invoke the Cloud Run service
|
||||
@@ -222,23 +249,18 @@ class WorkerService:
|
||||
},
|
||||
}
|
||||
|
||||
# Add admin auth header if available
|
||||
# This is needed for the internal endpoint's require_admin dependency
|
||||
if self._admin_token:
|
||||
task["http_request"]["headers"]["Authorization"] = f"Bearer {self._admin_token}"
|
||||
|
||||
# Create task
|
||||
# Note: We don't set a task name, allowing Cloud Tasks to generate one
|
||||
# This prevents duplicate task errors on retries
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"Created Cloud Task for {worker_type} worker, job {job_id}: {response.name}"
|
||||
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name}"
|
||||
)
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to enqueue Cloud Task for {worker_type}/{job_id}: {e}",
|
||||
f"[job:{job_id}] Failed to enqueue Cloud Task for {worker_type}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
@@ -255,6 +277,9 @@ class WorkerService:
|
||||
This is the original implementation - direct HTTP POST to internal endpoint.
|
||||
The endpoint adds the worker function to FastAPI BackgroundTasks.
|
||||
|
||||
Observability:
|
||||
- Injects trace context headers so worker spans link to parent trace
|
||||
|
||||
Args:
|
||||
worker_type: Worker type
|
||||
job_id: Job ID to process
|
||||
@@ -269,6 +294,9 @@ class WorkerService:
|
||||
if self._admin_token:
|
||||
headers["Authorization"] = f"Bearer {self._admin_token}"
|
||||
|
||||
# Inject trace context for distributed tracing
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout_seconds) as client:
|
||||
url = f"{self._base_url}/api/internal/workers/{worker_type}"
|
||||
|
||||
@@ -279,22 +307,22 @@ class WorkerService:
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(f"Successfully triggered {worker_type} worker for job {job_id}")
|
||||
logger.info(f"[job:{job_id}] Successfully triggered {worker_type} worker")
|
||||
return True
|
||||
else:
|
||||
logger.error(
|
||||
f"Failed to trigger {worker_type} worker for job {job_id}: "
|
||||
f"[job:{job_id}] Failed to trigger {worker_type} worker: "
|
||||
f"HTTP {response.status_code} - {response.text}"
|
||||
)
|
||||
return False
|
||||
|
||||
except httpx.TimeoutException:
|
||||
logger.error(f"Timeout triggering {worker_type} worker for job {job_id}")
|
||||
logger.error(f"[job:{job_id}] Timeout triggering {worker_type} worker")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error triggering {worker_type} worker for job {job_id}: {e}",
|
||||
f"[job:{job_id}] Error triggering {worker_type} worker: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
+104
-4
@@ -12,13 +12,19 @@ from unittest.mock import Mock, MagicMock, AsyncMock, patch
|
||||
from datetime import datetime, UTC
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
# Mock google.auth.default for unit tests if not using emulator
|
||||
# This prevents DefaultCredentialsError during imports
|
||||
# Mock google.auth.default AND firestore for unit tests if not using emulator
|
||||
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
|
||||
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# Only mock the default() function, not the entire module
|
||||
# This allows other google.auth imports to work normally
|
||||
# Set up test environment variables BEFORE importing any backend modules
|
||||
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
|
||||
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
|
||||
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
|
||||
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
|
||||
os.environ.setdefault('ENVIRONMENT', 'test')
|
||||
|
||||
# Mock google.auth.default - prevents credential errors
|
||||
try:
|
||||
import google.auth
|
||||
mock_credentials = MagicMock()
|
||||
@@ -30,10 +36,104 @@ if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
except ImportError:
|
||||
# If google.auth not installed, that's ok for some tests
|
||||
pass
|
||||
|
||||
# Mock google.cloud.firestore.Client - prevents Firestore initialization
|
||||
# This is CRITICAL: when AuthService is instantiated, it creates FirestoreService
|
||||
# which tries to create a real Firestore client. The client returns MagicMocks
|
||||
# which fail when used as enum values (e.g., UserType).
|
||||
try:
|
||||
import google.cloud.firestore as firestore_module
|
||||
|
||||
# Create a mock client that returns proper values for get_token
|
||||
mock_firestore_client = MagicMock()
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_snapshot = MagicMock()
|
||||
|
||||
# For auth token lookups, return None (token not found in Firestore)
|
||||
# This forces validation to use admin tokens from environment
|
||||
mock_snapshot.exists = False
|
||||
mock_snapshot.to_dict.return_value = None
|
||||
mock_doc.get.return_value = mock_snapshot
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_firestore_client.collection.return_value = mock_collection
|
||||
|
||||
# Replace the Client class
|
||||
original_client = firestore_module.Client
|
||||
firestore_module.Client = MagicMock(return_value=mock_firestore_client)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# Mock google.cloud.storage.Client - prevents GCS initialization
|
||||
try:
|
||||
import google.cloud.storage as storage_module
|
||||
mock_storage_client = MagicMock()
|
||||
mock_bucket = MagicMock()
|
||||
mock_storage_client.bucket.return_value = mock_bucket
|
||||
storage_module.Client = MagicMock(return_value=mock_storage_client)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
from backend.models.job import Job, JobStatus, JobCreate
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def mock_auth_dependency(request):
|
||||
"""Mock the require_auth dependency for all tests using FastAPI's dependency override system."""
|
||||
# Skip for emulator tests and integration tests which use real auth
|
||||
test_path = str(request.fspath)
|
||||
if 'emulator' in test_path or 'integration' in test_path:
|
||||
yield
|
||||
return
|
||||
|
||||
# Skip if FIRESTORE_EMULATOR_HOST is set (running in emulator environment)
|
||||
import os
|
||||
if os.environ.get('FIRESTORE_EMULATOR_HOST'):
|
||||
yield
|
||||
return
|
||||
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.api.dependencies import require_auth, require_admin, require_review_auth, require_instrumental_auth
|
||||
from backend.main import app
|
||||
|
||||
# Create mock auth functions
|
||||
async def mock_require_auth():
|
||||
"""Mock require_auth to always return valid admin credentials."""
|
||||
return ("test-admin-token", UserType.ADMIN, 999)
|
||||
|
||||
async def mock_require_admin():
|
||||
"""Mock require_admin to always return valid admin credentials."""
|
||||
return ("test-admin-token", UserType.ADMIN, 999)
|
||||
|
||||
async def mock_require_review_auth(job_id: str = "test123"):
|
||||
"""Mock require_review_auth to always return valid review access."""
|
||||
return (job_id, "full")
|
||||
|
||||
async def mock_require_instrumental_auth(job_id: str = "test123"):
|
||||
"""Mock require_instrumental_auth to always return valid instrumental review access."""
|
||||
return (job_id, "full")
|
||||
|
||||
# Use FastAPI's dependency override system
|
||||
app.dependency_overrides[require_auth] = mock_require_auth
|
||||
app.dependency_overrides[require_admin] = mock_require_admin
|
||||
app.dependency_overrides[require_review_auth] = mock_require_review_auth
|
||||
app.dependency_overrides[require_instrumental_auth] = mock_require_instrumental_auth
|
||||
|
||||
yield
|
||||
|
||||
# Clean up after test
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers():
|
||||
"""Get authentication headers for test requests."""
|
||||
return {
|
||||
"Authorization": "Bearer test-admin-token",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_firestore():
|
||||
"""Mock Firestore client for unit tests."""
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -27,15 +27,15 @@ pytestmark = pytest.mark.skipif(
|
||||
class TestEmulatorBasics:
|
||||
"""Basic emulator connectivity tests."""
|
||||
|
||||
def test_health_endpoint(self, client):
|
||||
def test_health_endpoint(self, client, auth_headers):
|
||||
"""Test health endpoint works."""
|
||||
response = client.get("/api/health")
|
||||
response = client.get("/api/health", )
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "healthy"
|
||||
|
||||
def test_root_endpoint(self, client):
|
||||
def test_root_endpoint(self, client, auth_headers):
|
||||
"""Test root endpoint works."""
|
||||
response = client.get("/")
|
||||
response = client.get("/", )
|
||||
assert response.status_code == 200
|
||||
assert response.json()["service"] == "karaoke-gen-backend"
|
||||
|
||||
@@ -281,13 +281,13 @@ class TestReviewEndpoints:
|
||||
time.sleep(0.2)
|
||||
|
||||
# Should fail - job not in AWAITING_REVIEW status
|
||||
response = client.get(f"/api/review/{job_id}/correction-data")
|
||||
response = client.get(f"/api/review/{job_id}/correction-data", headers=auth_headers)
|
||||
assert response.status_code == 400
|
||||
assert "not ready for review" in response.json()["detail"].lower()
|
||||
|
||||
def test_review_audio_no_job(self, client):
|
||||
def test_review_audio_no_job(self, client, auth_headers):
|
||||
"""Test audio endpoint returns 404 for nonexistent job."""
|
||||
response = client.get("/api/review/nonexistent-job/audio/")
|
||||
response = client.get("/api/review/nonexistent-job/audio/", headers=auth_headers)
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_review_preview_video_stub(self, client, auth_headers):
|
||||
@@ -303,6 +303,7 @@ class TestReviewEndpoints:
|
||||
# Preview video should return error since job not ready
|
||||
response = client.post(
|
||||
f"/api/review/{job_id}/preview-video",
|
||||
headers=auth_headers,
|
||||
json={"corrections": [], "corrected_segments": []}
|
||||
)
|
||||
# Should return 400 (job not in AWAITING_REVIEW state)
|
||||
@@ -320,6 +321,7 @@ class TestReviewEndpoints:
|
||||
# Annotations endpoint should accept data (even if it just logs it)
|
||||
response = client.post(
|
||||
f"/api/review/{job_id}/v1/annotations",
|
||||
headers=auth_headers,
|
||||
json={"type": "test", "data": "test annotation"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
@@ -28,20 +28,20 @@ class TestHealthRoutes:
|
||||
from backend.main import app
|
||||
return TestClient(app)
|
||||
|
||||
def test_health_endpoint_returns_200(self, client):
|
||||
def test_health_endpoint_returns_200(self, client, auth_headers):
|
||||
"""Test /api/health returns 200 OK."""
|
||||
response = client.get("/api/health")
|
||||
response = client.get("/api/health", )
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_health_endpoint_returns_healthy_status(self, client):
|
||||
def test_health_endpoint_returns_healthy_status(self, client, auth_headers):
|
||||
"""Test health endpoint returns healthy status."""
|
||||
response = client.get("/api/health")
|
||||
response = client.get("/api/health", )
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
|
||||
def test_root_endpoint_returns_200(self, client):
|
||||
def test_root_endpoint_returns_200(self, client, auth_headers):
|
||||
"""Test root endpoint returns 200."""
|
||||
response = client.get("/")
|
||||
response = client.get("/", )
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,294 @@
|
||||
"""
|
||||
Tests for audio_analysis_service.py - Audio analysis for backing vocals.
|
||||
|
||||
These tests mock the storage service and shared karaoke_gen modules to verify:
|
||||
- GCS file download/upload operations
|
||||
- Audio analysis delegation to shared AudioAnalyzer
|
||||
- Waveform generation delegation to shared WaveformGenerator
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch, ANY
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
|
||||
class TestAudioAnalysisServiceInit:
|
||||
"""Test AudioAnalysisService initialization."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_init_creates_dependencies(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test initialization creates storage service and analyzers."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
mock_storage_class.assert_called_once()
|
||||
mock_analyzer_class.assert_called_once_with(
|
||||
silence_threshold_db=-40.0,
|
||||
min_segment_duration_ms=100,
|
||||
)
|
||||
mock_waveform_class.assert_called_once()
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_init_with_custom_params(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test initialization with custom threshold and duration."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
service = AudioAnalysisService(
|
||||
silence_threshold_db=-50.0,
|
||||
min_segment_duration_ms=200,
|
||||
)
|
||||
|
||||
mock_analyzer_class.assert_called_once_with(
|
||||
silence_threshold_db=-50.0,
|
||||
min_segment_duration_ms=200,
|
||||
)
|
||||
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_init_with_provided_storage_service(
|
||||
self, mock_waveform_class, mock_analyzer_class
|
||||
):
|
||||
"""Test initialization with externally provided storage service."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
service = AudioAnalysisService(storage_service=mock_storage)
|
||||
|
||||
assert service.storage_service is mock_storage
|
||||
|
||||
|
||||
class TestAnalyzeBackingVocals:
|
||||
"""Test analyze_backing_vocals method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_analyze_downloads_and_analyzes(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test analyze_backing_vocals downloads file and runs analysis."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
# Setup mocks
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.has_audible_content = True
|
||||
mock_result.segment_count = 5
|
||||
mock_result.recommended_selection = Mock(value="with_backing")
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
# Call method
|
||||
result = service.analyze_backing_vocals(
|
||||
gcs_audio_path="uploads/job123/backing_vocals.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify storage download was called
|
||||
mock_storage.download_file.assert_called_once()
|
||||
call_args = mock_storage.download_file.call_args
|
||||
assert call_args[0][0] == "uploads/job123/backing_vocals.flac"
|
||||
|
||||
# Verify analyzer was called
|
||||
mock_analyzer.analyze.assert_called_once()
|
||||
|
||||
# Verify result
|
||||
assert result is mock_result
|
||||
assert result.has_audible_content is True
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_analyze_cleans_up_temp_files(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test that temp files are cleaned up after analysis."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.has_audible_content = False
|
||||
mock_result.segment_count = 0
|
||||
mock_result.recommended_selection = Mock(value="clean")
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
# Call method
|
||||
service.analyze_backing_vocals(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Temp directory should be cleaned up (no assertion needed,
|
||||
# TemporaryDirectory context manager handles this)
|
||||
# Just verify the method completes without error
|
||||
|
||||
|
||||
class TestAnalyzeAndGenerateWaveform:
|
||||
"""Test analyze_and_generate_waveform method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_analyze_and_generate_waveform(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test analyze_and_generate_waveform does both operations."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.has_audible_content = True
|
||||
mock_result.audible_segments = [Mock(), Mock()]
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer.silence_threshold_db = -40.0
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
mock_waveform = Mock()
|
||||
mock_waveform_class.return_value = mock_waveform
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
# Call method
|
||||
result, waveform_path = service.analyze_and_generate_waveform(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
gcs_waveform_destination="uploads/job123/waveform.png",
|
||||
)
|
||||
|
||||
# Verify analysis was performed
|
||||
mock_analyzer.analyze.assert_called_once()
|
||||
|
||||
# Verify waveform was generated
|
||||
mock_waveform.generate.assert_called_once()
|
||||
call_kwargs = mock_waveform.generate.call_args.kwargs
|
||||
assert call_kwargs["segments"] == mock_result.audible_segments
|
||||
assert call_kwargs["show_time_axis"] is True
|
||||
|
||||
# Verify waveform was uploaded
|
||||
assert mock_storage.upload_file.call_count == 1
|
||||
|
||||
# Verify return values
|
||||
assert result is mock_result
|
||||
assert waveform_path == "uploads/job123/waveform.png"
|
||||
|
||||
|
||||
class TestGetWaveformData:
|
||||
"""Test get_waveform_data method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_get_waveform_data(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test get_waveform_data returns amplitude data."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_waveform = Mock()
|
||||
mock_waveform.generate_data_only.return_value = (
|
||||
[0.1, 0.3, 0.5, 0.3, 0.1], # amplitudes
|
||||
180.5, # duration
|
||||
)
|
||||
mock_waveform_class.return_value = mock_waveform
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
amplitudes, duration = service.get_waveform_data(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
num_points=500,
|
||||
)
|
||||
|
||||
# Verify storage download
|
||||
mock_storage.download_file.assert_called_once()
|
||||
|
||||
# Verify waveform generator called
|
||||
mock_waveform.generate_data_only.assert_called_once()
|
||||
call_kwargs = mock_waveform.generate_data_only.call_args.kwargs
|
||||
assert call_kwargs["num_points"] == 500
|
||||
|
||||
# Verify return values
|
||||
assert amplitudes == [0.1, 0.3, 0.5, 0.3, 0.1]
|
||||
assert duration == 180.5
|
||||
|
||||
|
||||
class TestGenerateWaveformWithMutes:
|
||||
"""Test generate_waveform_with_mutes method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_generate_waveform_with_mutes(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test generate_waveform_with_mutes highlights mute regions."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.audible_segments = [Mock(), Mock()]
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
mock_waveform = Mock()
|
||||
mock_waveform_class.return_value = mock_waveform
|
||||
|
||||
# Create mock mute regions
|
||||
mock_mute1 = Mock()
|
||||
mock_mute2 = Mock()
|
||||
mute_regions = [mock_mute1, mock_mute2]
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
result_path = service.generate_waveform_with_mutes(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
gcs_waveform_destination="uploads/job123/waveform_muted.png",
|
||||
mute_regions=mute_regions,
|
||||
)
|
||||
|
||||
# Verify waveform generator was called with mute_regions
|
||||
mock_waveform.generate.assert_called_once()
|
||||
call_kwargs = mock_waveform.generate.call_args.kwargs
|
||||
assert call_kwargs["mute_regions"] == mute_regions
|
||||
assert call_kwargs["segments"] == mock_result.audible_segments
|
||||
|
||||
# Verify upload
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
# Verify return
|
||||
assert result_path == "uploads/job123/waveform_muted.png"
|
||||
|
||||
@@ -0,0 +1,386 @@
|
||||
"""
|
||||
Tests for audio_editing_service.py - Custom instrumental creation.
|
||||
|
||||
These tests mock the storage service and shared karaoke_gen modules to verify:
|
||||
- GCS file download/upload operations
|
||||
- Audio editing delegation to shared AudioEditor
|
||||
- Mute region validation
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch, ANY
|
||||
|
||||
|
||||
class TestAudioEditingServiceInit:
|
||||
"""Test AudioEditingService initialization."""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_init_creates_dependencies(self, mock_editor_class, mock_storage_class):
|
||||
"""Test initialization creates storage service and editor."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mock_storage_class.assert_called_once()
|
||||
mock_editor_class.assert_called_once_with(output_format="flac")
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_init_with_custom_format(self, mock_editor_class, mock_storage_class):
|
||||
"""Test initialization with custom output format."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService(output_format="wav")
|
||||
|
||||
mock_editor_class.assert_called_once_with(output_format="wav")
|
||||
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_init_with_provided_storage_service(self, mock_editor_class):
|
||||
"""Test initialization with externally provided storage service."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
service = AudioEditingService(storage_service=mock_storage)
|
||||
|
||||
assert service.storage_service is mock_storage
|
||||
|
||||
|
||||
class TestCreateCustomInstrumental:
|
||||
"""Test create_custom_instrumental method."""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_custom_instrumental_success(
|
||||
self, mock_editor_class, mock_storage_class
|
||||
):
|
||||
"""Test creating a custom instrumental successfully."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.total_muted_duration_seconds = 15.5
|
||||
mock_result.mute_regions_applied = [Mock(), Mock(), Mock()]
|
||||
mock_result.output_path = None # Will be set by service
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.create_custom_instrumental.return_value = mock_result
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Create mock mute regions
|
||||
mock_mute1 = Mock()
|
||||
mock_mute2 = Mock()
|
||||
mute_regions = [mock_mute1, mock_mute2]
|
||||
|
||||
result = service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="uploads/job123/custom.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify downloads
|
||||
assert mock_storage.download_file.call_count == 2
|
||||
|
||||
# Verify editor was called
|
||||
mock_editor.create_custom_instrumental.assert_called_once()
|
||||
call_kwargs = mock_editor.create_custom_instrumental.call_args.kwargs
|
||||
assert call_kwargs["mute_regions"] == mute_regions
|
||||
|
||||
# Verify upload
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
# Verify result path was updated
|
||||
assert result.output_path == "uploads/job123/custom.flac"
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_custom_instrumental_downloads_both_files(
|
||||
self, mock_editor_class, mock_storage_class
|
||||
):
|
||||
"""Test that both input files are downloaded."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.total_muted_duration_seconds = 0
|
||||
mock_result.mute_regions_applied = []
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.create_custom_instrumental.return_value = mock_result
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=[],
|
||||
gcs_output_path="uploads/job123/output.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify both files were downloaded
|
||||
download_calls = mock_storage.download_file.call_args_list
|
||||
assert len(download_calls) == 2
|
||||
|
||||
# First call should be clean instrumental
|
||||
assert download_calls[0][0][0] == "uploads/job123/clean.flac"
|
||||
# Second call should be backing vocals
|
||||
assert download_calls[1][0][0] == "uploads/job123/backing.flac"
|
||||
|
||||
|
||||
class TestCreatePreview:
|
||||
"""Test create_preview method."""
|
||||
|
||||
@patch("pydub.AudioSegment")
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_preview_success(
|
||||
self, mock_editor_class, mock_storage_class, mock_audio_segment
|
||||
):
|
||||
"""Test creating a preview successfully."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_preview = Mock()
|
||||
mock_preview.export = Mock()
|
||||
mock_preview.__getitem__ = Mock(return_value=mock_preview)
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.preview_with_mutes.return_value = mock_preview
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
result = service.create_preview(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=[Mock(), Mock()],
|
||||
gcs_preview_path="uploads/job123/preview.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify editor was called
|
||||
mock_editor.preview_with_mutes.assert_called_once()
|
||||
|
||||
# Verify result
|
||||
assert result == "uploads/job123/preview.flac"
|
||||
|
||||
@patch("pydub.AudioSegment")
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_preview_with_duration_limit(
|
||||
self, mock_editor_class, mock_storage_class, mock_audio_segment
|
||||
):
|
||||
"""Test creating a preview with duration limit."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_preview = Mock()
|
||||
mock_truncated = Mock()
|
||||
mock_preview.__getitem__ = Mock(return_value=mock_truncated)
|
||||
mock_truncated.export = Mock()
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.preview_with_mutes.return_value = mock_preview
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
service.create_preview(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=[],
|
||||
gcs_preview_path="uploads/job123/preview.flac",
|
||||
job_id="job123",
|
||||
preview_duration_seconds=30.0,
|
||||
)
|
||||
|
||||
# Verify preview was truncated (30 seconds = 30000ms)
|
||||
mock_preview.__getitem__.assert_called_once_with(slice(None, 30000))
|
||||
|
||||
|
||||
class TestMuteBackingVocalsOnly:
|
||||
"""Test mute_backing_vocals_only method."""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_mute_backing_vocals_only(self, mock_editor_class, mock_storage_class):
|
||||
"""Test muting backing vocals without combining."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [Mock(), Mock()]
|
||||
|
||||
result = service.mute_backing_vocals_only(
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="uploads/job123/muted_backing.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify only one file was downloaded
|
||||
mock_storage.download_file.assert_called_once()
|
||||
assert mock_storage.download_file.call_args[0][0] == "uploads/job123/backing.flac"
|
||||
|
||||
# Verify editor method was called
|
||||
mock_editor.apply_mute_to_single_track.assert_called_once()
|
||||
call_kwargs = mock_editor.apply_mute_to_single_track.call_args.kwargs
|
||||
assert call_kwargs["mute_regions"] == mute_regions
|
||||
|
||||
# Verify upload
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
# Verify result
|
||||
assert result == "uploads/job123/muted_backing.flac"
|
||||
|
||||
|
||||
class TestValidateMuteRegions:
|
||||
"""Test validate_mute_regions method.
|
||||
|
||||
Note: MuteRegion uses pydantic validation that rejects invalid values at
|
||||
creation time. These tests use Mock objects to test the validation logic
|
||||
in the service, simulating edge cases that would normally be rejected.
|
||||
"""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_valid_mute_regions(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation passes for valid mute regions."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=10.0, end_seconds=15.0),
|
||||
MuteRegion(start_seconds=30.0, end_seconds=45.0),
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
assert errors == []
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_negative_start_seconds(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation catches negative start time.
|
||||
|
||||
Uses Mock since MuteRegion pydantic model rejects negative values.
|
||||
"""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Use mock to simulate invalid region (pydantic would reject this)
|
||||
mock_region = Mock()
|
||||
mock_region.start_seconds = -5.0
|
||||
mock_region.end_seconds = 10.0
|
||||
|
||||
errors = service.validate_mute_regions([mock_region], total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "cannot be negative" in errors[0]
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_end_before_start(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation catches end time before start time."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Use mock to simulate invalid region
|
||||
mock_region = Mock()
|
||||
mock_region.start_seconds = 30.0
|
||||
mock_region.end_seconds = 20.0
|
||||
|
||||
errors = service.validate_mute_regions([mock_region], total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "must be after" in errors[0]
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_start_exceeds_duration(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation catches start time exceeding audio duration."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=200.0, end_seconds=210.0),
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "exceeds audio duration" in errors[0]
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_end_exceeds_duration_is_warning_not_error(
|
||||
self, mock_editor_class, mock_storage_class
|
||||
):
|
||||
"""Test that end time exceeding duration is logged as warning, not error."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=170.0, end_seconds=200.0), # End exceeds 180s
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
# Should not be an error (will be clamped)
|
||||
assert len(errors) == 0
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_multiple_errors(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation reports multiple errors."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Use mocks for invalid regions
|
||||
mock_region1 = Mock()
|
||||
mock_region1.start_seconds = -5.0
|
||||
mock_region1.end_seconds = 10.0
|
||||
|
||||
mock_region2 = Mock()
|
||||
mock_region2.start_seconds = 30.0
|
||||
mock_region2.end_seconds = 20.0 # End before start
|
||||
|
||||
mock_region3 = Mock()
|
||||
mock_region3.start_seconds = 200.0 # Exceeds duration
|
||||
mock_region3.end_seconds = 210.0
|
||||
|
||||
mute_regions = [mock_region1, mock_region2, mock_region3]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 3
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -3,13 +3,28 @@ Unit tests for audio analysis and editing services.
|
||||
|
||||
These tests verify the backend service wrappers that use the shared
|
||||
karaoke_gen.instrumental_review module with GCS integration.
|
||||
|
||||
NOTE: These tests require ffmpeg to be installed. They will be skipped
|
||||
if ffmpeg is not available (e.g., in CI environments without ffmpeg).
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, ANY
|
||||
|
||||
# Check if ffmpeg is available
|
||||
def _ffmpeg_available():
|
||||
"""Check if ffmpeg is available on the system."""
|
||||
return shutil.which('ffmpeg') is not None
|
||||
|
||||
# Skip all tests in this module if ffmpeg is not available
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not _ffmpeg_available(),
|
||||
reason="ffmpeg not available - required for audio processing tests"
|
||||
)
|
||||
|
||||
from pydub import AudioSegment
|
||||
from pydub.generators import Sine
|
||||
|
||||
|
||||
@@ -0,0 +1,408 @@
|
||||
"""
|
||||
Tests for dropbox_service.py - Dropbox file operations.
|
||||
|
||||
These tests mock the Dropbox SDK and Secret Manager to verify:
|
||||
- Credential loading from Secret Manager
|
||||
- Folder listing and brand code calculation
|
||||
- File and folder uploads
|
||||
- Shared link creation
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
|
||||
|
||||
class TestDropboxServiceInit:
|
||||
"""Test DropboxService initialization."""
|
||||
|
||||
def test_init_creates_service(self):
|
||||
"""Test initialization creates service with no client."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
assert service._client is None
|
||||
assert service._is_configured is False
|
||||
|
||||
|
||||
class TestLoadCredentials:
|
||||
"""Test _load_credentials method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_load_credentials_success(self, mock_sm_client_class):
|
||||
"""Test successful credential loading from Secret Manager."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"access_token": "access-token-123",
|
||||
"refresh_token": "refresh-token-456",
|
||||
"app_key": "app-key",
|
||||
"app_secret": "app-secret",
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is not None
|
||||
assert creds["access_token"] == "access-token-123"
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_load_credentials_failure(self, mock_sm_client_class):
|
||||
"""Test handling when Secret Manager fails."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_sm_client.access_secret_version.side_effect = Exception("Access denied")
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is None
|
||||
|
||||
|
||||
class TestIsConfigured:
|
||||
"""Test is_configured property."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_is_configured_true(self, mock_sm_client_class):
|
||||
"""Test is_configured returns True when credentials available."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"access_token": "token"
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
assert service.is_configured is True
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_is_configured_false_no_token(self, mock_sm_client_class):
|
||||
"""Test is_configured returns False when no access_token."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"refresh_token": "refresh" # Missing access_token
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
assert service.is_configured is False
|
||||
|
||||
|
||||
class TestDropboxClient:
|
||||
"""Test client property."""
|
||||
|
||||
@patch("dropbox.Dropbox")
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_client_creates_dropbox_instance(self, mock_sm_client_class, mock_dropbox_class):
|
||||
"""Test client property creates Dropbox SDK client."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"access_token": "token",
|
||||
"refresh_token": "refresh",
|
||||
"app_key": "key",
|
||||
"app_secret": "secret",
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox_class.return_value = mock_dropbox
|
||||
|
||||
service = DropboxService()
|
||||
client = service.client
|
||||
|
||||
mock_dropbox_class.assert_called_once_with(
|
||||
oauth2_access_token="token",
|
||||
oauth2_refresh_token="refresh",
|
||||
app_key="key",
|
||||
app_secret="secret",
|
||||
)
|
||||
assert client == mock_dropbox
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_client_raises_on_missing_credentials(self, mock_sm_client_class):
|
||||
"""Test client raises RuntimeError when credentials missing."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_sm_client.access_secret_version.side_effect = Exception("Not found")
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
_ = service.client
|
||||
|
||||
assert "not configured" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestListFolders:
|
||||
"""Test list_folders method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_list_folders(self, mock_sm_client_class):
|
||||
"""Test listing folders at a path."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
from dropbox.files import FolderMetadata
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create mock folder entries
|
||||
mock_folder1 = Mock(spec=FolderMetadata)
|
||||
mock_folder1.name = "NOMAD-0001"
|
||||
mock_folder2 = Mock(spec=FolderMetadata)
|
||||
mock_folder2.name = "NOMAD-0002"
|
||||
mock_file = Mock() # Not a FolderMetadata
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.entries = [mock_folder1, mock_folder2, mock_file]
|
||||
mock_result.has_more = False
|
||||
|
||||
service = DropboxService()
|
||||
# Directly set the client to avoid needing to mock the whole init chain
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.files_list_folder.return_value = mock_result
|
||||
service._client = mock_dropbox
|
||||
|
||||
folders = service.list_folders("/Karaoke/Tracks")
|
||||
|
||||
mock_dropbox.files_list_folder.assert_called_once_with("/Karaoke/Tracks")
|
||||
assert folders == ["NOMAD-0001", "NOMAD-0002"]
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_list_folders_adds_leading_slash(self, mock_sm_client_class):
|
||||
"""Test that path without leading slash gets one added."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.entries = []
|
||||
mock_result.has_more = False
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.files_list_folder.return_value = mock_result
|
||||
service._client = mock_dropbox
|
||||
|
||||
service.list_folders("path/without/slash")
|
||||
|
||||
mock_dropbox.files_list_folder.assert_called_once_with("/path/without/slash")
|
||||
|
||||
|
||||
class TestGetNextBrandCode:
|
||||
"""Test get_next_brand_code method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_get_next_brand_code(self, mock_sm_client_class):
|
||||
"""Test calculating next brand code from existing folders."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
# Mock list_folders to return existing codes
|
||||
with patch.object(service, "list_folders") as mock_list:
|
||||
mock_list.return_value = [
|
||||
"NOMAD-1161",
|
||||
"NOMAD-1162",
|
||||
"NOMAD-1163",
|
||||
"Other Folder",
|
||||
"NOMAD-0001",
|
||||
]
|
||||
|
||||
next_code = service.get_next_brand_code("/path", "NOMAD")
|
||||
|
||||
assert next_code == "NOMAD-1164"
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_get_next_brand_code_empty_folder(self, mock_sm_client_class):
|
||||
"""Test brand code calculation with no existing codes."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with patch.object(service, "list_folders") as mock_list:
|
||||
mock_list.return_value = []
|
||||
|
||||
next_code = service.get_next_brand_code("/path", "BRAND")
|
||||
|
||||
assert next_code == "BRAND-0001"
|
||||
|
||||
|
||||
class TestUploadFile:
|
||||
"""Test upload_file method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_small_file(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a small file directly."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test file
|
||||
test_file = tmp_path / "test.txt"
|
||||
test_file.write_text("Small file content")
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
service._client = mock_dropbox
|
||||
|
||||
service.upload_file(str(test_file), "/Uploads/test.txt")
|
||||
|
||||
mock_dropbox.files_upload.assert_called_once()
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_file_adds_leading_slash(self, mock_sm_client_class, tmp_path):
|
||||
"""Test upload adds leading slash to remote path."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
test_file = tmp_path / "test.txt"
|
||||
test_file.write_text("content")
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
service._client = mock_dropbox
|
||||
|
||||
service.upload_file(str(test_file), "uploads/test.txt")
|
||||
|
||||
# Check that the path has leading slash
|
||||
call_args = mock_dropbox.files_upload.call_args
|
||||
assert call_args[0][1] == "/uploads/test.txt"
|
||||
|
||||
|
||||
class TestUploadFolder:
|
||||
"""Test upload_folder method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a folder with multiple files."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test folder with files
|
||||
(tmp_path / "file1.txt").write_text("content1")
|
||||
(tmp_path / "file2.txt").write_text("content2")
|
||||
(tmp_path / "subdir").mkdir() # Should be skipped
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with patch.object(service, "upload_file") as mock_upload:
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Should upload 2 files (not subdir)
|
||||
assert mock_upload.call_count == 2
|
||||
|
||||
|
||||
class TestCreateSharedLink:
|
||||
"""Test create_shared_link method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_create_shared_link_new(self, mock_sm_client_class):
|
||||
"""Test creating a new shared link."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
mock_link = Mock()
|
||||
mock_link.url = "https://dropbox.com/s/abc123/file.mp4"
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.sharing_create_shared_link_with_settings.return_value = mock_link
|
||||
service._client = mock_dropbox
|
||||
|
||||
url = service.create_shared_link("/Uploads/file.mp4")
|
||||
|
||||
assert url == "https://dropbox.com/s/abc123/file.mp4"
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_create_shared_link_existing(self, mock_sm_client_class):
|
||||
"""Test getting existing shared link when one already exists."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
from dropbox.exceptions import ApiError
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Simulate "link already exists" error
|
||||
mock_error = Mock()
|
||||
mock_error.is_shared_link_already_exists.return_value = True
|
||||
|
||||
mock_existing_link = Mock()
|
||||
mock_existing_link.url = "https://dropbox.com/s/existing/file.mp4"
|
||||
|
||||
mock_links_result = Mock()
|
||||
mock_links_result.links = [mock_existing_link]
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.sharing_create_shared_link_with_settings.side_effect = \
|
||||
ApiError("req_id", mock_error, "message", "headers")
|
||||
mock_dropbox.sharing_list_shared_links.return_value = mock_links_result
|
||||
service._client = mock_dropbox
|
||||
|
||||
url = service.create_shared_link("/Uploads/file.mp4")
|
||||
|
||||
assert url == "https://dropbox.com/s/existing/file.mp4"
|
||||
|
||||
@@ -96,15 +96,15 @@ def auth_headers():
|
||||
class TestEmulatorBasics:
|
||||
"""Basic emulator connectivity tests."""
|
||||
|
||||
def test_health_endpoint(self, client):
|
||||
def test_health_endpoint(self, client, auth_headers):
|
||||
"""Test health endpoint works."""
|
||||
response = client.get("/api/health")
|
||||
response = client.get("/api/health", )
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "healthy"
|
||||
|
||||
def test_root_endpoint(self, client):
|
||||
def test_root_endpoint(self, client, auth_headers):
|
||||
"""Test root endpoint works."""
|
||||
response = client.get("/")
|
||||
response = client.get("/", )
|
||||
assert response.status_code == 200
|
||||
assert response.json()["service"] == "karaoke-gen-backend"
|
||||
|
||||
|
||||
@@ -0,0 +1,632 @@
|
||||
"""
|
||||
Tests for the flacfetch remote HTTP API client.
|
||||
|
||||
This module tests FlacfetchClient which handles communication with the
|
||||
dedicated flacfetch VM for torrent/audio downloads.
|
||||
|
||||
These tests use mocked HTTP responses to verify:
|
||||
- Request formatting (headers, payloads)
|
||||
- Response parsing
|
||||
- Error handling
|
||||
- State management (singleton pattern)
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, Mock, patch, MagicMock
|
||||
import asyncio
|
||||
|
||||
import httpx
|
||||
|
||||
from backend.services.flacfetch_client import (
|
||||
FlacfetchClient,
|
||||
FlacfetchServiceError,
|
||||
get_flacfetch_client,
|
||||
reset_flacfetch_client,
|
||||
)
|
||||
|
||||
|
||||
class TestFlacfetchClientInit:
|
||||
"""Test FlacfetchClient initialization."""
|
||||
|
||||
def test_init_with_required_params(self):
|
||||
"""Test client initializes with base_url and api_key."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key-123",
|
||||
)
|
||||
|
||||
assert client.base_url == "http://localhost:8080"
|
||||
assert client.api_key == "test-key-123"
|
||||
assert client.timeout == 60 # Default timeout
|
||||
|
||||
def test_init_strips_trailing_slash(self):
|
||||
"""Test base_url trailing slash is stripped."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080/",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
assert client.base_url == "http://localhost:8080"
|
||||
|
||||
def test_init_with_custom_timeout(self):
|
||||
"""Test custom timeout is set."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
timeout=120,
|
||||
)
|
||||
|
||||
assert client.timeout == 120
|
||||
|
||||
def test_headers_include_api_key(self):
|
||||
"""Test _headers() includes X-API-Key."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="secret-key-456",
|
||||
)
|
||||
|
||||
headers = client._headers()
|
||||
|
||||
assert headers["X-API-Key"] == "secret-key-456"
|
||||
assert headers["Content-Type"] == "application/json"
|
||||
|
||||
|
||||
class TestFlacfetchClientHealthCheck:
|
||||
"""Test health_check() method."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_check_success(self):
|
||||
"""Test successful health check."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {
|
||||
"status": "healthy",
|
||||
"transmission": {"available": True, "version": "4.0.5"},
|
||||
"disk": {"free_gb": 15.5},
|
||||
}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
result = await client.health_check()
|
||||
|
||||
assert result["status"] == "healthy"
|
||||
assert result["transmission"]["available"] is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_check_degraded_status(self):
|
||||
"""Test health check with degraded status still succeeds."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {"status": "degraded"}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
result = await client.health_check()
|
||||
|
||||
assert result["status"] == "degraded"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_check_unhealthy_raises_error(self):
|
||||
"""Test health check with unhealthy status raises error."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {"status": "unhealthy", "error": "Transmission down"}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
await client.health_check()
|
||||
|
||||
assert "unhealthy" in str(exc_info.value)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_check_network_error(self):
|
||||
"""Test health check handles network errors."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(side_effect=httpx.RequestError("Connection refused"))
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
await client.health_check()
|
||||
|
||||
assert "Cannot reach" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestFlacfetchClientSearch:
|
||||
"""Test search() method."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_success(self):
|
||||
"""Test successful search returns results."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.json.return_value = {
|
||||
"search_id": "search_abc123",
|
||||
"results": [
|
||||
{
|
||||
"title": "Waterloo",
|
||||
"artist": "ABBA",
|
||||
"provider": "RED",
|
||||
"quality": "FLAC 16bit CD",
|
||||
"seeders": 50,
|
||||
"index": 0,
|
||||
}
|
||||
],
|
||||
"results_count": 1,
|
||||
}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
result = await client.search("ABBA", "Waterloo")
|
||||
|
||||
assert result["search_id"] == "search_abc123"
|
||||
assert len(result["results"]) == 1
|
||||
assert result["results"][0]["title"] == "Waterloo"
|
||||
|
||||
# Verify request was made correctly
|
||||
mock_client.post.assert_called_once()
|
||||
call_args = mock_client.post.call_args
|
||||
assert call_args.kwargs["json"] == {"artist": "ABBA", "title": "Waterloo"}
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_no_results_returns_empty(self):
|
||||
"""Test search with no results returns empty list."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 404
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
result = await client.search("Unknown", "Artist")
|
||||
|
||||
assert result["results"] == []
|
||||
assert result["search_id"] is None
|
||||
|
||||
|
||||
class TestFlacfetchClientDownload:
|
||||
"""Test download() method."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_starts_successfully(self):
|
||||
"""Test download() returns download_id."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {
|
||||
"download_id": "dl_xyz789",
|
||||
"status": "queued",
|
||||
}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
download_id = await client.download(
|
||||
search_id="search_abc123",
|
||||
result_index=0,
|
||||
)
|
||||
|
||||
assert download_id == "dl_xyz789"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_with_gcs_path(self):
|
||||
"""Test download() includes GCS upload parameters."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {"download_id": "dl_xyz789"}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
await client.download(
|
||||
search_id="search_abc123",
|
||||
result_index=0,
|
||||
gcs_path="uploads/job123/audio/",
|
||||
)
|
||||
|
||||
# Verify GCS params were included
|
||||
call_args = mock_client.post.call_args
|
||||
payload = call_args.kwargs["json"]
|
||||
assert payload["upload_to_gcs"] is True
|
||||
assert payload["gcs_path"] == "uploads/job123/audio/"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_with_custom_filename(self):
|
||||
"""Test download() with custom output filename."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {"download_id": "dl_xyz789"}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.post = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
await client.download(
|
||||
search_id="search_abc123",
|
||||
result_index=0,
|
||||
output_filename="ABBA - Waterloo",
|
||||
)
|
||||
|
||||
call_args = mock_client.post.call_args
|
||||
payload = call_args.kwargs["json"]
|
||||
assert payload["output_filename"] == "ABBA - Waterloo"
|
||||
|
||||
|
||||
class TestFlacfetchClientGetDownloadStatus:
|
||||
"""Test get_download_status() method."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_download_status_success(self):
|
||||
"""Test getting download status."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {
|
||||
"download_id": "dl_xyz789",
|
||||
"status": "downloading",
|
||||
"progress": 45.5,
|
||||
"download_speed_kbps": 1250.0,
|
||||
"peers": 5,
|
||||
}
|
||||
mock_response.raise_for_status = Mock()
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
status = await client.get_download_status("dl_xyz789")
|
||||
|
||||
assert status["status"] == "downloading"
|
||||
assert status["progress"] == 45.5
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_download_status_not_found(self):
|
||||
"""Test getting status for non-existent download."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 404
|
||||
mock_response.raise_for_status = Mock(
|
||||
side_effect=httpx.HTTPStatusError("Not found", request=Mock(), response=mock_response)
|
||||
)
|
||||
|
||||
with patch("httpx.AsyncClient") as mock_client_class:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
await client.get_download_status("nonexistent")
|
||||
|
||||
assert "not found" in str(exc_info.value).lower()
|
||||
|
||||
|
||||
class TestFlacfetchClientWaitForDownload:
|
||||
"""Test wait_for_download() method."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_completes(self):
|
||||
"""Test waiting for download to complete."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
# Mock get_download_status to return complete status
|
||||
async def mock_get_status(download_id):
|
||||
return {
|
||||
"download_id": download_id,
|
||||
"status": "complete",
|
||||
"gcs_path": "gs://bucket/uploads/test.flac",
|
||||
}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
result = await client.wait_for_download("dl_xyz789", timeout=10)
|
||||
|
||||
assert result["status"] == "complete"
|
||||
assert result["gcs_path"] == "gs://bucket/uploads/test.flac"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_seeding_status_completes(self):
|
||||
"""Test that 'seeding' status is treated as complete."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
async def mock_get_status(download_id):
|
||||
return {
|
||||
"download_id": download_id,
|
||||
"status": "seeding",
|
||||
"output_path": "/downloads/test.flac",
|
||||
}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
result = await client.wait_for_download("dl_xyz789", timeout=10)
|
||||
|
||||
assert result["status"] == "seeding"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_failed(self):
|
||||
"""Test wait raises error when download fails."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
async def mock_get_status(download_id):
|
||||
return {
|
||||
"download_id": download_id,
|
||||
"status": "failed",
|
||||
"error": "Torrent download failed: no peers",
|
||||
}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
await client.wait_for_download("dl_xyz789", timeout=10)
|
||||
|
||||
assert "failed" in str(exc_info.value).lower()
|
||||
assert "no peers" in str(exc_info.value)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_cancelled(self):
|
||||
"""Test wait raises error when download is cancelled."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
async def mock_get_status(download_id):
|
||||
return {
|
||||
"download_id": download_id,
|
||||
"status": "cancelled",
|
||||
}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
await client.wait_for_download("dl_xyz789", timeout=10)
|
||||
|
||||
assert "cancelled" in str(exc_info.value).lower()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_timeout(self):
|
||||
"""Test wait raises error on timeout."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
call_count = 0
|
||||
async def mock_get_status(download_id):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
return {
|
||||
"download_id": download_id,
|
||||
"status": "downloading",
|
||||
"progress": 10.0 * call_count,
|
||||
}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
with pytest.raises(FlacfetchServiceError) as exc_info:
|
||||
# Very short timeout with short poll interval
|
||||
await client.wait_for_download("dl_xyz789", timeout=0.2, poll_interval=0.1)
|
||||
|
||||
assert "timed out" in str(exc_info.value).lower()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wait_for_download_calls_progress_callback(self):
|
||||
"""Test progress callback is called during wait."""
|
||||
client = FlacfetchClient(
|
||||
base_url="http://localhost:8080",
|
||||
api_key="test-key",
|
||||
)
|
||||
|
||||
call_count = 0
|
||||
async def mock_get_status(download_id):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count >= 3:
|
||||
return {"status": "complete", "gcs_path": "gs://test"}
|
||||
return {"status": "downloading", "progress": 33.0 * call_count}
|
||||
|
||||
client.get_download_status = mock_get_status
|
||||
|
||||
progress_updates = []
|
||||
def progress_callback(status):
|
||||
progress_updates.append(status)
|
||||
|
||||
await client.wait_for_download(
|
||||
"dl_xyz789",
|
||||
timeout=10,
|
||||
poll_interval=0.01,
|
||||
progress_callback=progress_callback,
|
||||
)
|
||||
|
||||
# Should have received progress updates
|
||||
assert len(progress_updates) >= 2
|
||||
|
||||
|
||||
class TestGetFlacfetchClient:
|
||||
"""Test get_flacfetch_client() singleton factory."""
|
||||
|
||||
def test_returns_none_when_not_configured(self):
|
||||
"""Test returns None when FLACFETCH_API_URL not set."""
|
||||
reset_flacfetch_client()
|
||||
|
||||
with patch("backend.config.get_settings") as mock_settings:
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url=None,
|
||||
flacfetch_api_key=None,
|
||||
)
|
||||
|
||||
client = get_flacfetch_client()
|
||||
|
||||
assert client is None
|
||||
|
||||
def test_returns_none_when_key_missing(self):
|
||||
"""Test returns None when API key not set."""
|
||||
reset_flacfetch_client()
|
||||
|
||||
with patch("backend.config.get_settings") as mock_settings:
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url="http://localhost:8080",
|
||||
flacfetch_api_key=None,
|
||||
)
|
||||
|
||||
client = get_flacfetch_client()
|
||||
|
||||
assert client is None
|
||||
|
||||
def test_returns_client_when_configured(self):
|
||||
"""Test returns FlacfetchClient when both URL and key are set."""
|
||||
reset_flacfetch_client()
|
||||
|
||||
with patch("backend.config.get_settings") as mock_settings:
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url="http://10.0.0.5:8080",
|
||||
flacfetch_api_key="secret-key",
|
||||
)
|
||||
|
||||
client = get_flacfetch_client()
|
||||
|
||||
assert client is not None
|
||||
assert isinstance(client, FlacfetchClient)
|
||||
assert client.base_url == "http://10.0.0.5:8080"
|
||||
assert client.api_key == "secret-key"
|
||||
|
||||
def test_returns_same_instance(self):
|
||||
"""Test singleton returns same instance."""
|
||||
reset_flacfetch_client()
|
||||
|
||||
with patch("backend.config.get_settings") as mock_settings:
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url="http://localhost:8080",
|
||||
flacfetch_api_key="test-key",
|
||||
)
|
||||
|
||||
client1 = get_flacfetch_client()
|
||||
client2 = get_flacfetch_client()
|
||||
|
||||
assert client1 is client2
|
||||
|
||||
def test_reset_clears_singleton(self):
|
||||
"""Test reset_flacfetch_client() clears the singleton."""
|
||||
reset_flacfetch_client()
|
||||
|
||||
with patch("backend.config.get_settings") as mock_settings:
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url="http://localhost:8080",
|
||||
flacfetch_api_key="test-key",
|
||||
)
|
||||
|
||||
client1 = get_flacfetch_client()
|
||||
reset_flacfetch_client()
|
||||
|
||||
# Change the settings
|
||||
mock_settings.return_value = Mock(
|
||||
flacfetch_api_url="http://other:8080",
|
||||
flacfetch_api_key="other-key",
|
||||
)
|
||||
|
||||
client2 = get_flacfetch_client()
|
||||
|
||||
assert client1 is not client2
|
||||
assert client2.base_url == "http://other:8080"
|
||||
|
||||
@@ -0,0 +1,524 @@
|
||||
"""
|
||||
Tests for gdrive_service.py - Google Drive file operations.
|
||||
|
||||
These tests mock the Google API client and Secret Manager to verify:
|
||||
- Credential loading from Secret Manager
|
||||
- Folder creation and lookup
|
||||
- File uploads with proper MIME types
|
||||
- Public share folder structure uploads
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
|
||||
|
||||
class TestGoogleDriveServiceInit:
|
||||
"""Test GoogleDriveService initialization."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_init_creates_service(self, mock_get_settings):
|
||||
"""Test initialization creates service with settings."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
assert service.settings == mock_settings
|
||||
assert service._service is None
|
||||
assert service._credentials_data is None
|
||||
assert service._loaded is False
|
||||
|
||||
|
||||
class TestLoadCredentials:
|
||||
"""Test _load_credentials method."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_load_credentials_success(self, mock_get_settings):
|
||||
"""Test successful credential loading from Secret Manager."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"token": "access-token",
|
||||
"refresh_token": "refresh-token",
|
||||
"client_id": "client-id",
|
||||
"client_secret": "client-secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is not None
|
||||
assert creds["refresh_token"] == "refresh-token"
|
||||
mock_settings.get_secret.assert_called_once_with("gdrive-oauth-credentials")
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_load_credentials_fallback_to_youtube(self, mock_get_settings):
|
||||
"""Test fallback to YouTube credentials when Drive creds not found."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
# First call returns None (Drive creds not found)
|
||||
# Second call returns YouTube creds
|
||||
mock_settings.get_secret.side_effect = [
|
||||
None,
|
||||
json.dumps({
|
||||
"refresh_token": "youtube-token",
|
||||
"client_id": "youtube-id",
|
||||
"client_secret": "youtube-secret",
|
||||
})
|
||||
]
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is not None
|
||||
assert creds["refresh_token"] == "youtube-token"
|
||||
assert mock_settings.get_secret.call_count == 2
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_load_credentials_not_found(self, mock_get_settings):
|
||||
"""Test handling when no credentials found."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is None
|
||||
assert service._loaded is True
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_load_credentials_missing_required_fields(self, mock_get_settings):
|
||||
"""Test handling when credentials missing required fields."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"token": "access-token",
|
||||
# Missing: refresh_token, client_id, client_secret
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is None
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_load_credentials_cached(self, mock_get_settings):
|
||||
"""Test credentials are cached after first load."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
creds1 = service._load_credentials()
|
||||
creds2 = service._load_credentials()
|
||||
|
||||
assert creds1 == creds2
|
||||
assert mock_settings.get_secret.call_count == 1
|
||||
|
||||
|
||||
class TestIsConfigured:
|
||||
"""Test is_configured property."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_is_configured_true(self, mock_get_settings):
|
||||
"""Test is_configured returns True when credentials exist."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
assert service.is_configured is True
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_is_configured_false(self, mock_get_settings):
|
||||
"""Test is_configured returns False when no credentials."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
assert service.is_configured is False
|
||||
|
||||
|
||||
class TestDriveService:
|
||||
"""Test service property."""
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_service_creates_drive_client(
|
||||
self, mock_get_settings, mock_creds_class, mock_build
|
||||
):
|
||||
"""Test service property creates Google Drive API client."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"token": "access-token",
|
||||
"refresh_token": "refresh-token",
|
||||
"client_id": "client-id",
|
||||
"client_secret": "client-secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
mock_creds = Mock()
|
||||
mock_creds_class.return_value = mock_creds
|
||||
|
||||
mock_drive_service = Mock()
|
||||
mock_build.return_value = mock_drive_service
|
||||
|
||||
service = GoogleDriveService()
|
||||
result = service.service
|
||||
|
||||
mock_build.assert_called_once_with("drive", "v3", credentials=mock_creds)
|
||||
assert result == mock_drive_service
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_service_raises_on_missing_credentials(self, mock_get_settings):
|
||||
"""Test service raises RuntimeError when credentials missing."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
_ = service.service
|
||||
|
||||
assert "not configured" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestGetOrCreateFolder:
|
||||
"""Test get_or_create_folder method."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_get_existing_folder(self, mock_get_settings):
|
||||
"""Test finding an existing folder."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
# Mock the Drive API
|
||||
mock_files_api = Mock()
|
||||
mock_list_result = Mock()
|
||||
mock_list_result.execute.return_value = {
|
||||
"files": [{"id": "folder-id-123", "name": "MP4"}]
|
||||
}
|
||||
mock_files_api.list.return_value = mock_list_result
|
||||
|
||||
mock_drive = Mock()
|
||||
mock_drive.files.return_value = mock_files_api
|
||||
service._service = mock_drive
|
||||
service._loaded = True
|
||||
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
|
||||
|
||||
folder_id = service.get_or_create_folder("parent-123", "MP4")
|
||||
|
||||
assert folder_id == "folder-id-123"
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_create_new_folder(self, mock_get_settings):
|
||||
"""Test creating a new folder when it doesn't exist."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
# Mock the Drive API
|
||||
mock_files_api = Mock()
|
||||
|
||||
# list returns empty (folder doesn't exist)
|
||||
mock_list_result = Mock()
|
||||
mock_list_result.execute.return_value = {"files": []}
|
||||
mock_files_api.list.return_value = mock_list_result
|
||||
|
||||
# create returns new folder
|
||||
mock_create_result = Mock()
|
||||
mock_create_result.execute.return_value = {"id": "new-folder-id"}
|
||||
mock_files_api.create.return_value = mock_create_result
|
||||
|
||||
mock_drive = Mock()
|
||||
mock_drive.files.return_value = mock_files_api
|
||||
service._service = mock_drive
|
||||
service._loaded = True
|
||||
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
|
||||
|
||||
folder_id = service.get_or_create_folder("parent-123", "NewFolder")
|
||||
|
||||
assert folder_id == "new-folder-id"
|
||||
mock_files_api.create.assert_called_once()
|
||||
|
||||
|
||||
class TestUploadFile:
|
||||
"""Test upload_file method."""
|
||||
|
||||
@patch("googleapiclient.http.MediaFileUpload")
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_upload_file_success(self, mock_get_settings, mock_media, tmp_path):
|
||||
"""Test uploading a file to Drive."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
# Create test file
|
||||
test_file = tmp_path / "test.mp4"
|
||||
test_file.write_bytes(b"video content")
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
mock_files_api = Mock()
|
||||
|
||||
# list returns empty (no existing file)
|
||||
mock_list_result = Mock()
|
||||
mock_list_result.execute.return_value = {"files": []}
|
||||
mock_files_api.list.return_value = mock_list_result
|
||||
|
||||
# create returns new file
|
||||
mock_create_result = Mock()
|
||||
mock_create_result.execute.return_value = {"id": "file-id-123"}
|
||||
mock_files_api.create.return_value = mock_create_result
|
||||
|
||||
mock_drive = Mock()
|
||||
mock_drive.files.return_value = mock_files_api
|
||||
service._service = mock_drive
|
||||
service._loaded = True
|
||||
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
|
||||
|
||||
file_id = service.upload_file(str(test_file), "parent-123", "video.mp4")
|
||||
|
||||
assert file_id == "file-id-123"
|
||||
mock_media.assert_called_once()
|
||||
# Verify MIME type was set correctly
|
||||
call_kwargs = mock_media.call_args.kwargs
|
||||
assert call_kwargs["mimetype"] == "video/mp4"
|
||||
|
||||
@patch("googleapiclient.http.MediaFileUpload")
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_upload_file_replaces_existing(self, mock_get_settings, mock_media, tmp_path):
|
||||
"""Test upload deletes existing file before upload."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
test_file = tmp_path / "test.mp4"
|
||||
test_file.write_bytes(b"content")
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
mock_files_api = Mock()
|
||||
|
||||
# list returns existing file
|
||||
mock_list_result = Mock()
|
||||
mock_list_result.execute.return_value = {
|
||||
"files": [{"id": "existing-file-id"}]
|
||||
}
|
||||
mock_files_api.list.return_value = mock_list_result
|
||||
|
||||
mock_delete_result = Mock()
|
||||
mock_delete_result.execute.return_value = {}
|
||||
mock_files_api.delete.return_value = mock_delete_result
|
||||
|
||||
mock_create_result = Mock()
|
||||
mock_create_result.execute.return_value = {"id": "new-file-id"}
|
||||
mock_files_api.create.return_value = mock_create_result
|
||||
|
||||
mock_drive = Mock()
|
||||
mock_drive.files.return_value = mock_files_api
|
||||
service._service = mock_drive
|
||||
service._loaded = True
|
||||
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
|
||||
|
||||
file_id = service.upload_file(str(test_file), "parent-123", "video.mp4")
|
||||
|
||||
# Should have deleted existing file
|
||||
mock_files_api.delete.assert_called_once_with(fileId="existing-file-id")
|
||||
assert file_id == "new-file-id"
|
||||
|
||||
|
||||
class TestUploadToPublicShare:
|
||||
"""Test upload_to_public_share method."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_upload_to_public_share(self, mock_get_settings, tmp_path):
|
||||
"""Test uploading files to public share folder structure."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
# Create test files
|
||||
mp4_file = tmp_path / "output.mp4"
|
||||
mp4_file.write_bytes(b"4k video")
|
||||
mp4_720_file = tmp_path / "output_720p.mp4"
|
||||
mp4_720_file.write_bytes(b"720p video")
|
||||
cdg_file = tmp_path / "output.zip"
|
||||
cdg_file.write_bytes(b"cdg package")
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
# Mock methods
|
||||
with patch.object(service, "get_or_create_folder") as mock_get_folder:
|
||||
with patch.object(service, "upload_file") as mock_upload:
|
||||
mock_get_folder.side_effect = [
|
||||
"mp4-folder-id",
|
||||
"mp4-720-folder-id",
|
||||
"cdg-folder-id",
|
||||
]
|
||||
mock_upload.side_effect = [
|
||||
"mp4-file-id",
|
||||
"720p-file-id",
|
||||
"cdg-file-id",
|
||||
]
|
||||
|
||||
output_files = {
|
||||
"final_karaoke_lossy_mp4": str(mp4_file),
|
||||
"final_karaoke_lossy_720p_mp4": str(mp4_720_file),
|
||||
"final_karaoke_cdg_zip": str(cdg_file),
|
||||
}
|
||||
|
||||
result = service.upload_to_public_share(
|
||||
root_folder_id="root-123",
|
||||
brand_code="NOMAD-1163",
|
||||
base_name="Artist - Title",
|
||||
output_files=output_files,
|
||||
)
|
||||
|
||||
# Should have created/found 3 folders
|
||||
assert mock_get_folder.call_count == 3
|
||||
|
||||
# Should have uploaded 3 files
|
||||
assert mock_upload.call_count == 3
|
||||
|
||||
# Check result
|
||||
assert result["mp4"] == "mp4-file-id"
|
||||
assert result["mp4_720p"] == "720p-file-id"
|
||||
assert result["cdg"] == "cdg-file-id"
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_upload_to_public_share_skips_missing_files(
|
||||
self, mock_get_settings, tmp_path
|
||||
):
|
||||
"""Test upload skips files that don't exist."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
# Only create one file
|
||||
mp4_file = tmp_path / "output.mp4"
|
||||
mp4_file.write_bytes(b"video")
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
with patch.object(service, "get_or_create_folder") as mock_get_folder:
|
||||
with patch.object(service, "upload_file") as mock_upload:
|
||||
mock_get_folder.return_value = "folder-id"
|
||||
mock_upload.return_value = "file-id"
|
||||
|
||||
output_files = {
|
||||
"final_karaoke_lossy_mp4": str(mp4_file),
|
||||
"final_karaoke_lossy_720p_mp4": "/nonexistent/file.mp4",
|
||||
"final_karaoke_cdg_zip": None,
|
||||
}
|
||||
|
||||
result = service.upload_to_public_share(
|
||||
root_folder_id="root-123",
|
||||
brand_code="CODE",
|
||||
base_name="Name",
|
||||
output_files=output_files,
|
||||
)
|
||||
|
||||
# Should only upload the one file that exists
|
||||
assert mock_upload.call_count == 1
|
||||
assert len(result) == 1
|
||||
|
||||
|
||||
class TestGetGdriveService:
|
||||
"""Test get_gdrive_service singleton."""
|
||||
|
||||
@patch("backend.services.gdrive_service.get_settings")
|
||||
def test_get_gdrive_service_singleton(self, mock_get_settings):
|
||||
"""Test get_gdrive_service returns singleton instance."""
|
||||
from backend.services.gdrive_service import get_gdrive_service
|
||||
import backend.services.gdrive_service as gdrive_module
|
||||
|
||||
# Reset singleton
|
||||
gdrive_module._gdrive_service = None
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service1 = get_gdrive_service()
|
||||
service2 = get_gdrive_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
@@ -11,7 +11,6 @@ Tests for:
|
||||
import pytest
|
||||
from datetime import datetime
|
||||
from unittest.mock import MagicMock, patch, AsyncMock
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
@@ -117,16 +116,14 @@ def silent_job():
|
||||
class TestGetInstrumentalAnalysis:
|
||||
"""Tests for GET /api/jobs/{job_id}/instrumental-analysis."""
|
||||
|
||||
def test_get_analysis_returns_data(self, mock_job_manager, mock_storage_service, sample_job):
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_returns_data(self, mock_job_manager, mock_storage_service, sample_job):
|
||||
"""GET /instrumental-analysis should return analysis data."""
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
import asyncio
|
||||
|
||||
result = asyncio.get_event_loop().run_until_complete(
|
||||
get_instrumental_analysis("test-job-123")
|
||||
)
|
||||
result = await get_instrumental_analysis("test-job-123")
|
||||
|
||||
assert result["job_id"] == "test-job-123"
|
||||
assert result["artist"] == "Test Artist"
|
||||
@@ -134,23 +131,22 @@ class TestGetInstrumentalAnalysis:
|
||||
assert result["analysis"]["has_audible_content"] is True
|
||||
assert len(result["analysis"]["audible_segments"]) == 2
|
||||
|
||||
def test_get_analysis_includes_audio_urls(self, mock_job_manager, mock_storage_service, sample_job):
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_includes_audio_urls(self, mock_job_manager, mock_storage_service, sample_job):
|
||||
"""GET /instrumental-analysis should include audio URLs."""
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
import asyncio
|
||||
|
||||
result = asyncio.get_event_loop().run_until_complete(
|
||||
get_instrumental_analysis("test-job-123")
|
||||
)
|
||||
result = await get_instrumental_analysis("test-job-123")
|
||||
|
||||
assert "audio_urls" in result
|
||||
assert "clean_instrumental" in result["audio_urls"]
|
||||
assert "backing_vocals" in result["audio_urls"]
|
||||
assert "with_backing" in result["audio_urls"]
|
||||
|
||||
def test_get_analysis_includes_waveform_url(self, mock_job_manager, sample_job):
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_includes_waveform_url(self, mock_job_manager, sample_job):
|
||||
"""GET /instrumental-analysis should include waveform URL."""
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
@@ -161,59 +157,50 @@ class TestGetInstrumentalAnalysis:
|
||||
mock_storage_cls.return_value = mock_storage
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
import asyncio
|
||||
|
||||
result = asyncio.get_event_loop().run_until_complete(
|
||||
get_instrumental_analysis("test-job-123")
|
||||
)
|
||||
result = await get_instrumental_analysis("test-job-123")
|
||||
|
||||
assert "waveform_url" in result
|
||||
# Waveform URL should be generated from the file_urls
|
||||
assert result["waveform_url"] == "https://storage.googleapis.com/signed-url"
|
||||
|
||||
def test_get_analysis_silent_audio(self, mock_job_manager, mock_storage_service, silent_job):
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_silent_audio(self, mock_job_manager, mock_storage_service, silent_job):
|
||||
"""GET /instrumental-analysis should correctly report silent audio."""
|
||||
mock_job_manager.get_job.return_value = silent_job
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
import asyncio
|
||||
|
||||
result = asyncio.get_event_loop().run_until_complete(
|
||||
get_instrumental_analysis("test-job-silent")
|
||||
)
|
||||
result = await get_instrumental_analysis("test-job-silent")
|
||||
|
||||
assert result["analysis"]["has_audible_content"] is False
|
||||
assert result["analysis"]["recommended_selection"] == "clean"
|
||||
assert len(result["analysis"]["audible_segments"]) == 0
|
||||
|
||||
def test_get_analysis_job_not_found(self, mock_job_manager):
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_job_not_found(self, mock_job_manager):
|
||||
"""GET /instrumental-analysis should return 404 for non-existent job."""
|
||||
mock_job_manager.get_job.return_value = None
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
from fastapi import HTTPException
|
||||
import asyncio
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
asyncio.get_event_loop().run_until_complete(
|
||||
get_instrumental_analysis("non-existent")
|
||||
)
|
||||
await get_instrumental_analysis("non-existent")
|
||||
|
||||
assert exc_info.value.status_code == 404
|
||||
|
||||
def test_get_analysis_wrong_status(self, mock_job_manager, sample_job):
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_analysis_wrong_status(self, mock_job_manager, sample_job):
|
||||
"""GET /instrumental-analysis should return 400 for wrong job status."""
|
||||
sample_job.status = JobStatus.PENDING
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import get_instrumental_analysis
|
||||
from fastapi import HTTPException
|
||||
import asyncio
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
asyncio.get_event_loop().run_until_complete(
|
||||
get_instrumental_analysis("test-job-123")
|
||||
)
|
||||
await get_instrumental_analysis("test-job-123")
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
|
||||
@@ -226,39 +213,34 @@ class TestAudioStream:
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import stream_audio
|
||||
import asyncio
|
||||
|
||||
# This will try to stream from GCS which we can't test easily
|
||||
# Instead, verify the function exists and has correct signature
|
||||
assert callable(stream_audio)
|
||||
|
||||
def test_stream_invalid_stem_type(self, mock_job_manager, sample_job):
|
||||
@pytest.mark.asyncio
|
||||
async def test_stream_invalid_stem_type(self, mock_job_manager, sample_job):
|
||||
"""Should return 400 for invalid stem type."""
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import stream_audio
|
||||
from fastapi import HTTPException
|
||||
import asyncio
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
asyncio.get_event_loop().run_until_complete(
|
||||
stream_audio("test-job-123", "invalid_stem")
|
||||
)
|
||||
await stream_audio("test-job-123", "invalid_stem")
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
|
||||
def test_stream_job_not_found(self, mock_job_manager):
|
||||
@pytest.mark.asyncio
|
||||
async def test_stream_job_not_found(self, mock_job_manager):
|
||||
"""Should return 404 for non-existent job."""
|
||||
mock_job_manager.get_job.return_value = None
|
||||
|
||||
from backend.api.routes.jobs import stream_audio
|
||||
from fastapi import HTTPException
|
||||
import asyncio
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
asyncio.get_event_loop().run_until_complete(
|
||||
stream_audio("non-existent", "backing_vocals")
|
||||
)
|
||||
await stream_audio("non-existent", "backing_vocals")
|
||||
|
||||
assert exc_info.value.status_code == 404
|
||||
|
||||
@@ -293,27 +275,26 @@ class TestCreateCustomInstrumental:
|
||||
|
||||
assert "At least one mute region is required" in str(exc_info.value)
|
||||
|
||||
def test_create_custom_job_not_found(self, mock_job_manager):
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_custom_job_not_found(self, mock_job_manager):
|
||||
"""Should return 404 for non-existent job."""
|
||||
mock_job_manager.get_job.return_value = None
|
||||
|
||||
from backend.api.routes.jobs import create_custom_instrumental
|
||||
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
|
||||
from fastapi import HTTPException
|
||||
import asyncio
|
||||
|
||||
request = CreateCustomInstrumentalRequest(
|
||||
mute_regions=[MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)]
|
||||
)
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
asyncio.get_event_loop().run_until_complete(
|
||||
create_custom_instrumental("non-existent", request)
|
||||
)
|
||||
await create_custom_instrumental("non-existent", request)
|
||||
|
||||
assert exc_info.value.status_code == 404
|
||||
|
||||
def test_create_custom_wrong_status(self, mock_job_manager, sample_job):
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_custom_wrong_status(self, mock_job_manager, sample_job):
|
||||
"""Should return 400 for wrong job status."""
|
||||
sample_job.status = JobStatus.PENDING
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
@@ -321,16 +302,13 @@ class TestCreateCustomInstrumental:
|
||||
from backend.api.routes.jobs import create_custom_instrumental
|
||||
from backend.models.requests import CreateCustomInstrumentalRequest, MuteRegionRequest
|
||||
from fastapi import HTTPException
|
||||
import asyncio
|
||||
|
||||
request = CreateCustomInstrumentalRequest(
|
||||
mute_regions=[MuteRegionRequest(start_seconds=10.0, end_seconds=20.0)]
|
||||
)
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
asyncio.get_event_loop().run_until_complete(
|
||||
create_custom_instrumental("test-job-123", request)
|
||||
)
|
||||
await create_custom_instrumental("test-job-123", request)
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
|
||||
@@ -338,34 +316,30 @@ class TestCreateCustomInstrumental:
|
||||
class TestGetWaveformData:
|
||||
"""Tests for GET /api/jobs/{job_id}/waveform-data."""
|
||||
|
||||
def test_get_waveform_data_job_not_found(self, mock_job_manager):
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_waveform_data_job_not_found(self, mock_job_manager):
|
||||
"""Should return 404 for non-existent job."""
|
||||
mock_job_manager.get_job.return_value = None
|
||||
|
||||
from backend.api.routes.jobs import get_waveform_data
|
||||
from fastapi import HTTPException
|
||||
import asyncio
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
asyncio.get_event_loop().run_until_complete(
|
||||
get_waveform_data("non-existent")
|
||||
)
|
||||
await get_waveform_data("non-existent")
|
||||
|
||||
assert exc_info.value.status_code == 404
|
||||
|
||||
def test_get_waveform_data_wrong_status(self, mock_job_manager, sample_job):
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_waveform_data_wrong_status(self, mock_job_manager, sample_job):
|
||||
"""Should return 400 for wrong job status."""
|
||||
sample_job.status = JobStatus.PENDING
|
||||
mock_job_manager.get_job.return_value = sample_job
|
||||
|
||||
from backend.api.routes.jobs import get_waveform_data
|
||||
from fastapi import HTTPException
|
||||
import asyncio
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
asyncio.get_event_loop().run_until_complete(
|
||||
get_waveform_data("test-job-123")
|
||||
)
|
||||
await get_waveform_data("test-job-123")
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
|
||||
|
||||
@@ -98,7 +98,7 @@ class TestAudioWorkerEndpoint:
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_audio_worker_requires_auth(self, client):
|
||||
def test_audio_worker_requires_auth(self, client, auth_headers):
|
||||
"""Test audio worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
@@ -128,7 +128,7 @@ class TestLyricsWorkerEndpoint:
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_lyrics_worker_requires_auth(self, client):
|
||||
def test_lyrics_worker_requires_auth(self, client, auth_headers):
|
||||
"""Test lyrics worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/lyrics",
|
||||
@@ -158,7 +158,7 @@ class TestScreensWorkerEndpoint:
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_screens_worker_requires_auth(self, client):
|
||||
def test_screens_worker_requires_auth(self, client, auth_headers):
|
||||
"""Test screens worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/screens",
|
||||
@@ -188,7 +188,7 @@ class TestVideoWorkerEndpoint:
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_video_worker_requires_auth(self, client):
|
||||
def test_video_worker_requires_auth(self, client, auth_headers):
|
||||
"""Test video worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/video",
|
||||
@@ -218,7 +218,7 @@ class TestRenderVideoWorkerEndpoint:
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_render_video_worker_requires_auth(self, client):
|
||||
def test_render_video_worker_requires_auth(self, client, auth_headers):
|
||||
"""Test render-video worker requires authentication."""
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
|
||||
@@ -102,6 +102,37 @@ class TestJobCreation:
|
||||
|
||||
assert job.status == JobStatus.PENDING
|
||||
assert job.progress == 0
|
||||
|
||||
def test_create_job_with_distribution_settings(self, job_manager, mock_firestore_service):
|
||||
"""Test that distribution settings are passed from JobCreate to Job.
|
||||
|
||||
This was a bug where brand_prefix, dropbox_path, gdrive_folder_id, and
|
||||
discord_webhook_url were NOT being passed to the Job constructor.
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
brand_prefix="NOMAD",
|
||||
discord_webhook_url="https://discord.com/webhook/test",
|
||||
dropbox_path="/Karaoke/Tracks-Organized",
|
||||
gdrive_folder_id="1abc123xyz",
|
||||
enable_youtube_upload=True,
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
# Verify distribution settings are passed through
|
||||
assert job.brand_prefix == "NOMAD"
|
||||
assert job.discord_webhook_url == "https://discord.com/webhook/test"
|
||||
assert job.dropbox_path == "/Karaoke/Tracks-Organized"
|
||||
assert job.gdrive_folder_id == "1abc123xyz"
|
||||
assert job.enable_youtube_upload is True
|
||||
|
||||
# Verify Firestore was called with job containing these fields
|
||||
mock_firestore_service.create_job.assert_called_once()
|
||||
created_job = mock_firestore_service.create_job.call_args[0][0]
|
||||
assert created_job.brand_prefix == "NOMAD"
|
||||
assert created_job.dropbox_path == "/Karaoke/Tracks-Organized"
|
||||
|
||||
|
||||
class TestJobRetrieval:
|
||||
|
||||
@@ -68,20 +68,20 @@ def client(mock_job_manager, mock_worker_service):
|
||||
class TestGetJob:
|
||||
"""Tests for GET /api/jobs/{job_id}."""
|
||||
|
||||
def test_get_job_returns_200(self, client, mock_job_manager, mock_job):
|
||||
def test_get_job_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test getting an existing job returns 200."""
|
||||
response = client.get("/api/jobs/test123")
|
||||
response = client.get("/api/jobs/test123", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_get_job_returns_job_data(self, client, mock_job_manager, mock_job):
|
||||
def test_get_job_returns_job_data(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test response contains job data."""
|
||||
response = client.get("/api/jobs/test123")
|
||||
response = client.get("/api/jobs/test123", headers=auth_headers)
|
||||
data = response.json()
|
||||
assert data["job_id"] == "test123"
|
||||
assert data["status"] == "pending"
|
||||
assert data["artist"] == "Test Artist"
|
||||
|
||||
def test_get_nonexistent_job_returns_404(self, mock_worker_service):
|
||||
def test_get_nonexistent_job_returns_404(self, mock_worker_service, auth_headers):
|
||||
"""Test getting non-existent job returns 404."""
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = None
|
||||
@@ -95,58 +95,60 @@ class TestGetJob:
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
client = TestClient(app)
|
||||
response = client.get("/api/jobs/nonexistent")
|
||||
response = client.get("/api/jobs/nonexistent", headers=auth_headers)
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
class TestListJobs:
|
||||
"""Tests for GET /api/jobs."""
|
||||
|
||||
def test_list_jobs_returns_200(self, client):
|
||||
def test_list_jobs_returns_200(self, client, auth_headers):
|
||||
"""Test listing jobs returns 200."""
|
||||
response = client.get("/api/jobs")
|
||||
response = client.get("/api/jobs", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_list_jobs_returns_array(self, client, mock_job_manager):
|
||||
def test_list_jobs_returns_array(self, client, mock_job_manager, auth_headers):
|
||||
"""Test response is an array of jobs."""
|
||||
response = client.get("/api/jobs")
|
||||
response = client.get("/api/jobs", headers=auth_headers)
|
||||
data = response.json()
|
||||
assert isinstance(data, list)
|
||||
assert len(data) == 1
|
||||
assert data[0]["job_id"] == "test123"
|
||||
|
||||
def test_list_jobs_with_status_filter(self, client, mock_job_manager):
|
||||
def test_list_jobs_with_status_filter(self, client, mock_job_manager, auth_headers):
|
||||
"""Test listing jobs with status filter."""
|
||||
response = client.get("/api/jobs?status=pending")
|
||||
response = client.get("/api/jobs?status=pending", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_list_jobs_with_limit(self, client, mock_job_manager):
|
||||
def test_list_jobs_with_limit(self, client, mock_job_manager, auth_headers):
|
||||
"""Test listing jobs with limit."""
|
||||
response = client.get("/api/jobs?limit=10")
|
||||
response = client.get("/api/jobs?limit=10", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
class TestCreateJob:
|
||||
"""Tests for POST /api/jobs."""
|
||||
|
||||
def test_create_job_with_url_returns_200(self, client, mock_job_manager):
|
||||
def test_create_job_with_url_returns_200(self, client, mock_job_manager, auth_headers):
|
||||
"""Test creating job with URL returns 200."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
json={"url": "https://youtube.com/watch?v=test123"}
|
||||
json={"url": "https://youtube.com/watch?v=test123"},
|
||||
headers=auth_headers
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_create_job_returns_job_id(self, client, mock_job_manager):
|
||||
def test_create_job_returns_job_id(self, client, mock_job_manager, auth_headers):
|
||||
"""Test create response contains job_id."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
json={"url": "https://youtube.com/watch?v=test"}
|
||||
json={"url": "https://youtube.com/watch?v=test"},
|
||||
headers=auth_headers
|
||||
)
|
||||
data = response.json()
|
||||
assert "job_id" in data
|
||||
|
||||
def test_create_job_with_artist_title(self, client, mock_job_manager):
|
||||
def test_create_job_with_artist_title(self, client, mock_job_manager, auth_headers):
|
||||
"""Test creating job with artist and title."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
@@ -154,7 +156,8 @@ class TestCreateJob:
|
||||
"url": "https://youtube.com/watch?v=test",
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
},
|
||||
headers=auth_headers
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
@@ -162,12 +165,12 @@ class TestCreateJob:
|
||||
class TestDeleteJob:
|
||||
"""Tests for DELETE /api/jobs/{job_id}."""
|
||||
|
||||
def test_delete_job_returns_200(self, client, mock_job_manager):
|
||||
def test_delete_job_returns_200(self, client, mock_job_manager, auth_headers):
|
||||
"""Test deleting job returns 200."""
|
||||
response = client.delete("/api/jobs/test123")
|
||||
response = client.delete("/api/jobs/test123", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_delete_nonexistent_job(self, mock_worker_service):
|
||||
def test_delete_nonexistent_job(self, mock_worker_service, auth_headers):
|
||||
"""Test deleting non-existent job."""
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.delete_job.return_value = False
|
||||
@@ -181,7 +184,7 @@ class TestDeleteJob:
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
client = TestClient(app)
|
||||
response = client.delete("/api/jobs/nonexistent")
|
||||
response = client.delete("/api/jobs/nonexistent", headers=auth_headers)
|
||||
# Either 200 or 404 depending on implementation
|
||||
assert response.status_code in [200, 404]
|
||||
|
||||
@@ -189,12 +192,13 @@ class TestDeleteJob:
|
||||
class TestCancelJob:
|
||||
"""Tests for POST /api/jobs/{job_id}/cancel."""
|
||||
|
||||
def test_cancel_job_returns_200(self, client, mock_job_manager, mock_job):
|
||||
def test_cancel_job_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test cancelling job returns 200."""
|
||||
mock_job_manager.cancel_job.return_value = mock_job
|
||||
response = client.post(
|
||||
"/api/jobs/test123/cancel",
|
||||
json={}
|
||||
json={},
|
||||
headers=auth_headers
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
@@ -202,7 +206,7 @@ class TestCancelJob:
|
||||
class TestSubmitCorrections:
|
||||
"""Tests for POST /api/jobs/{job_id}/corrections."""
|
||||
|
||||
def test_submit_corrections_returns_200(self, client, mock_job_manager, mock_job):
|
||||
def test_submit_corrections_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test submitting corrections returns 200."""
|
||||
# Job needs to be in AWAITING_REVIEW or IN_REVIEW status
|
||||
review_job = Job(
|
||||
@@ -222,7 +226,8 @@ class TestSubmitCorrections:
|
||||
"lines": [],
|
||||
"metadata": {"source": "test"}
|
||||
}
|
||||
}
|
||||
},
|
||||
headers=auth_headers
|
||||
)
|
||||
# Should be 200 or validation error
|
||||
assert response.status_code in [200, 400, 422]
|
||||
@@ -231,11 +236,12 @@ class TestSubmitCorrections:
|
||||
class TestSelectInstrumental:
|
||||
"""Tests for POST /api/jobs/{job_id}/select-instrumental."""
|
||||
|
||||
def test_select_instrumental_requires_selection_field(self, client, mock_job_manager, mock_job):
|
||||
def test_select_instrumental_requires_selection_field(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test instrumental selection requires selection field."""
|
||||
response = client.post(
|
||||
"/api/jobs/test123/select-instrumental",
|
||||
json={} # Missing selection field
|
||||
json={}, # Missing selection field
|
||||
headers=auth_headers
|
||||
)
|
||||
# Missing field should cause validation error
|
||||
assert response.status_code == 422
|
||||
@@ -244,9 +250,9 @@ class TestSelectInstrumental:
|
||||
class TestStartReview:
|
||||
"""Tests for POST /api/jobs/{job_id}/start-review."""
|
||||
|
||||
def test_start_review_endpoint_exists(self, client, mock_job_manager, mock_job):
|
||||
def test_start_review_endpoint_exists(self, client, mock_job_manager, mock_job, auth_headers):
|
||||
"""Test start review endpoint exists."""
|
||||
response = client.post("/api/jobs/test123/start-review")
|
||||
response = client.post("/api/jobs/test123/start-review", headers=auth_headers)
|
||||
# Should not be 404 or 405
|
||||
assert response.status_code not in [404, 405]
|
||||
|
||||
@@ -0,0 +1,448 @@
|
||||
"""
|
||||
Tests for storage_service.py - Google Cloud Storage operations.
|
||||
|
||||
These tests mock the GCS client to verify:
|
||||
- Upload operations (file, fileobj, JSON)
|
||||
- Download operations (file, JSON)
|
||||
- Signed URL generation
|
||||
- File operations (delete, list, exists)
|
||||
"""
|
||||
import json
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
from io import BytesIO
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
|
||||
class TestStorageServiceInit:
|
||||
"""Test StorageService initialization."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_init_creates_client_and_bucket(self, mock_settings, mock_client_class):
|
||||
"""Test initialization creates GCS client and gets bucket."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_client = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
|
||||
mock_client_class.assert_called_once_with(project="test-project")
|
||||
mock_client.bucket.assert_called_once_with("test-bucket")
|
||||
assert service.client == mock_client
|
||||
assert service.bucket == mock_bucket
|
||||
|
||||
|
||||
class TestStorageServiceUpload:
|
||||
"""Test upload operations."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_upload_file(self, mock_settings, mock_client_class):
|
||||
"""Test uploading a file to GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
result = service.upload_file("/local/path/file.flac", "uploads/job123/file.flac")
|
||||
|
||||
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
|
||||
mock_blob.upload_from_filename.assert_called_once_with("/local/path/file.flac")
|
||||
assert result == "uploads/job123/file.flac"
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_upload_file_raises_on_error(self, mock_settings, mock_client_class):
|
||||
"""Test upload_file raises exception on GCS error."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.upload_from_filename.side_effect = Exception("GCS error")
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
service.upload_file("/local/path/file.flac", "dest/file.flac")
|
||||
|
||||
assert "GCS error" in str(exc_info.value)
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_upload_fileobj(self, mock_settings, mock_client_class):
|
||||
"""Test uploading a file object to GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
file_obj = BytesIO(b"test content")
|
||||
|
||||
result = service.upload_fileobj(file_obj, "uploads/test.txt", content_type="text/plain")
|
||||
|
||||
mock_bucket.blob.assert_called_once_with("uploads/test.txt")
|
||||
assert mock_blob.content_type == "text/plain"
|
||||
mock_blob.upload_from_file.assert_called_once_with(file_obj, rewind=True)
|
||||
assert result == "uploads/test.txt"
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_upload_json(self, mock_settings, mock_client_class):
|
||||
"""Test uploading JSON data to GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
data = {"key": "value", "nested": {"data": True}}
|
||||
|
||||
result = service.upload_json("data/config.json", data)
|
||||
|
||||
mock_bucket.blob.assert_called_once_with("data/config.json")
|
||||
assert mock_blob.content_type == "application/json"
|
||||
# Check the uploaded content
|
||||
call_args = mock_blob.upload_from_string.call_args
|
||||
uploaded_content = call_args[0][0]
|
||||
assert json.loads(uploaded_content) == data
|
||||
assert result == "data/config.json"
|
||||
|
||||
|
||||
class TestStorageServiceDownload:
|
||||
"""Test download operations."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_download_file(self, mock_settings, mock_client_class):
|
||||
"""Test downloading a file from GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
result = service.download_file("uploads/job123/file.flac", "/local/path/file.flac")
|
||||
|
||||
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
|
||||
mock_blob.download_to_filename.assert_called_once_with("/local/path/file.flac")
|
||||
assert result == "/local/path/file.flac"
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_download_file_raises_on_error(self, mock_settings, mock_client_class):
|
||||
"""Test download_file raises exception on GCS error."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.download_to_filename.side_effect = Exception("File not found")
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
service.download_file("missing/file.flac", "/local/path")
|
||||
|
||||
assert "File not found" in str(exc_info.value)
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_download_json(self, mock_settings, mock_client_class):
|
||||
"""Test downloading and parsing JSON from GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.download_as_text.return_value = '{"key": "value", "count": 42}'
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
result = service.download_json("data/config.json")
|
||||
|
||||
assert result == {"key": "value", "count": 42}
|
||||
|
||||
|
||||
class TestStorageServiceDelete:
|
||||
"""Test delete operations."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_delete_file(self, mock_settings, mock_client_class):
|
||||
"""Test deleting a single file from GCS."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
service.delete_file("uploads/job123/file.flac")
|
||||
|
||||
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
|
||||
mock_blob.delete.assert_called_once()
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_delete_folder(self, mock_settings, mock_client_class):
|
||||
"""Test deleting all files with a prefix (folder)."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
# Create mock blobs
|
||||
mock_blob1 = Mock()
|
||||
mock_blob1.name = "uploads/job123/file1.flac"
|
||||
mock_blob2 = Mock()
|
||||
mock_blob2.name = "uploads/job123/file2.flac"
|
||||
mock_blob3 = Mock()
|
||||
mock_blob3.name = "uploads/job123/subdir/file3.flac"
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2, mock_blob3]
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
count = service.delete_folder("uploads/job123/")
|
||||
|
||||
mock_bucket.list_blobs.assert_called_once_with(prefix="uploads/job123/")
|
||||
mock_blob1.delete.assert_called_once()
|
||||
mock_blob2.delete.assert_called_once()
|
||||
mock_blob3.delete.assert_called_once()
|
||||
assert count == 3
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_delete_folder_handles_errors_gracefully(self, mock_settings, mock_client_class):
|
||||
"""Test delete_folder continues even if some deletes fail."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob1 = Mock()
|
||||
mock_blob1.name = "file1.flac"
|
||||
mock_blob2 = Mock()
|
||||
mock_blob2.name = "file2.flac"
|
||||
mock_blob2.delete.side_effect = Exception("Permission denied")
|
||||
mock_blob3 = Mock()
|
||||
mock_blob3.name = "file3.flac"
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2, mock_blob3]
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
count = service.delete_folder("uploads/")
|
||||
|
||||
# Should have deleted 2 files (blob2 failed)
|
||||
assert count == 2
|
||||
|
||||
|
||||
class TestStorageServiceFileOperations:
|
||||
"""Test file listing and existence checks."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_list_files(self, mock_settings, mock_client_class):
|
||||
"""Test listing files with a prefix."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob1 = Mock()
|
||||
mock_blob1.name = "uploads/job123/audio.flac"
|
||||
mock_blob2 = Mock()
|
||||
mock_blob2.name = "uploads/job123/lyrics.txt"
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2]
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
files = service.list_files("uploads/job123/")
|
||||
|
||||
assert files == ["uploads/job123/audio.flac", "uploads/job123/lyrics.txt"]
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_file_exists_returns_true(self, mock_settings, mock_client_class):
|
||||
"""Test file_exists returns True for existing file."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.exists.return_value = True
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
result = service.file_exists("uploads/job123/file.flac")
|
||||
|
||||
assert result is True
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_file_exists_returns_false(self, mock_settings, mock_client_class):
|
||||
"""Test file_exists returns False for missing file."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.exists.return_value = False
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
service = StorageService()
|
||||
result = service.file_exists("missing/file.flac")
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestStorageServiceSignedUrls:
|
||||
"""Test signed URL generation."""
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_generate_signed_url(self, mock_settings, mock_client_class):
|
||||
"""Test generating a signed download URL."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
# Mock google.auth.default to return credentials without service_account_email
|
||||
with patch("google.auth.default") as mock_auth_default:
|
||||
mock_credentials = Mock(spec=[]) # No service_account_email attr
|
||||
mock_auth_default.return_value = (mock_credentials, "test-project")
|
||||
|
||||
service = StorageService()
|
||||
url = service.generate_signed_url("uploads/file.flac", expiration_minutes=30)
|
||||
|
||||
assert url == "https://storage.googleapis.com/signed-url"
|
||||
mock_blob.generate_signed_url.assert_called_once()
|
||||
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
|
||||
assert call_kwargs["method"] == "GET"
|
||||
assert call_kwargs["version"] == "v4"
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_generate_signed_upload_url(self, mock_settings, mock_client_class):
|
||||
"""Test generating a signed upload URL."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-upload-url"
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
with patch("google.auth.default") as mock_auth_default:
|
||||
mock_credentials = Mock(spec=[])
|
||||
mock_auth_default.return_value = (mock_credentials, "test-project")
|
||||
|
||||
service = StorageService()
|
||||
url = service.generate_signed_upload_url(
|
||||
"uploads/file.flac",
|
||||
content_type="audio/flac",
|
||||
expiration_minutes=15
|
||||
)
|
||||
|
||||
assert url == "https://storage.googleapis.com/signed-upload-url"
|
||||
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
|
||||
assert call_kwargs["method"] == "PUT"
|
||||
assert call_kwargs["headers"] == {"Content-Type": "audio/flac"}
|
||||
|
||||
@patch("backend.services.storage_service.storage.Client")
|
||||
@patch("backend.services.storage_service.settings")
|
||||
def test_signed_url_with_service_account(self, mock_settings, mock_client_class):
|
||||
"""Test signed URL generation with service account credentials."""
|
||||
mock_settings.google_cloud_project = "test-project"
|
||||
mock_settings.gcs_bucket_name = "test-bucket"
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
mock_client = Mock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
mock_client_class.return_value = mock_client
|
||||
|
||||
with patch("google.auth.default") as mock_auth_default:
|
||||
with patch("google.auth.transport.requests.Request") as mock_request:
|
||||
# Mock credentials with service_account_email
|
||||
mock_credentials = Mock()
|
||||
mock_credentials.service_account_email = "sa@project.iam.gserviceaccount.com"
|
||||
mock_credentials.token = "access-token-123"
|
||||
mock_auth_default.return_value = (mock_credentials, "test-project")
|
||||
|
||||
service = StorageService()
|
||||
url = service.generate_signed_url("uploads/file.flac")
|
||||
|
||||
# Should have refreshed credentials and used IAM signing
|
||||
mock_credentials.refresh.assert_called_once()
|
||||
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
|
||||
assert call_kwargs["service_account_email"] == "sa@project.iam.gserviceaccount.com"
|
||||
assert call_kwargs["access_token"] == "access-token-123"
|
||||
|
||||
@@ -16,12 +16,20 @@ from unittest.mock import Mock, MagicMock, patch, AsyncMock
|
||||
from io import BytesIO
|
||||
|
||||
# Test the remote CLI's style parsing
|
||||
# Skip this class if flacfetch is not properly installed (remote_cli depends on it)
|
||||
try:
|
||||
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, Config
|
||||
_remote_cli_available = True
|
||||
except ImportError:
|
||||
_remote_cli_available = False
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _remote_cli_available, reason="flacfetch not properly installed")
|
||||
class TestRemoteCLIStyleParsing:
|
||||
"""Test that remote CLI correctly parses style_params.json."""
|
||||
|
||||
def test_parse_style_params_extracts_file_paths(self):
|
||||
"""Test that _parse_style_params extracts all file references."""
|
||||
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, Config
|
||||
|
||||
# Create a mock style_params.json
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
@@ -79,7 +87,6 @@ class TestRemoteCLIStyleParsing:
|
||||
|
||||
def test_parse_style_params_handles_missing_files(self):
|
||||
"""Test that _parse_style_params ignores non-existent files."""
|
||||
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, Config
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
style_params = {
|
||||
|
||||
@@ -10,6 +10,14 @@ from io import BytesIO
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def upload_auth_headers():
|
||||
"""Auth headers for file upload tests (no Content-Type, let multipart work)."""
|
||||
return {
|
||||
"Authorization": "Bearer test-admin-token"
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job():
|
||||
"""Create a standard mock job."""
|
||||
@@ -65,37 +73,41 @@ def client(mock_services):
|
||||
class TestFileUploadEndpoint:
|
||||
"""Tests for POST /api/jobs/upload."""
|
||||
|
||||
def test_upload_flac_returns_200(self, client, mock_services):
|
||||
def test_upload_flac_returns_200(self, client, mock_services, upload_auth_headers):
|
||||
"""Test uploading FLAC file returns 200."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.flac", BytesIO(b"fake audio"), "audio/flac")},
|
||||
data={"artist": "Test Artist", "title": "Test Song"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_upload_mp3_returns_200(self, client, mock_services):
|
||||
def test_upload_mp3_returns_200(self, client, mock_services, upload_auth_headers):
|
||||
"""Test uploading MP3 file returns 200."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.mp3", BytesIO(b"fake audio"), "audio/mpeg")},
|
||||
data={"artist": "Test Artist", "title": "Test Song"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_upload_wav_returns_200(self, client, mock_services):
|
||||
def test_upload_wav_returns_200(self, client, mock_services, upload_auth_headers):
|
||||
"""Test uploading WAV file returns 200."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.wav", BytesIO(b"fake audio"), "audio/wav")},
|
||||
data={"artist": "Test Artist", "title": "Test Song"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_upload_returns_job_id(self, client, mock_services):
|
||||
def test_upload_returns_job_id(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload response contains job_id."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.flac", BytesIO(b"fake audio"), "audio/flac")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
@@ -103,10 +115,11 @@ class TestFileUploadEndpoint:
|
||||
assert "job_id" in data
|
||||
assert data["status"] == "success"
|
||||
|
||||
def test_upload_returns_filename(self, client, mock_services):
|
||||
def test_upload_returns_filename(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload response contains original filename."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("my_song.flac", BytesIO(b"fake audio"), "audio/flac")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
@@ -114,84 +127,93 @@ class TestFileUploadEndpoint:
|
||||
assert "filename" in data
|
||||
assert data["filename"] == "my_song.flac"
|
||||
|
||||
def test_upload_rejects_txt_file(self, client, mock_services):
|
||||
def test_upload_rejects_txt_file(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload rejects text files."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.txt", BytesIO(b"not audio"), "text/plain")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_upload_rejects_pdf_file(self, client, mock_services):
|
||||
def test_upload_rejects_pdf_file(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload rejects PDF files."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("doc.pdf", BytesIO(b"pdf content"), "application/pdf")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_upload_rejects_exe_file(self, client, mock_services):
|
||||
def test_upload_rejects_exe_file(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload rejects executable files."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("app.exe", BytesIO(b"exe content"), "application/octet-stream")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_upload_requires_artist(self, client, mock_services):
|
||||
def test_upload_requires_artist(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload requires artist field."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.flac", BytesIO(b"audio"), "audio/flac")},
|
||||
data={"title": "Song"} # Missing artist
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_upload_requires_title(self, client, mock_services):
|
||||
def test_upload_requires_title(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload requires title field."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.flac", BytesIO(b"audio"), "audio/flac")},
|
||||
data={"artist": "Artist"} # Missing title
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_upload_requires_file(self, client, mock_services):
|
||||
def test_upload_requires_file(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload requires file."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
data={"artist": "Artist", "title": "Song"}
|
||||
# Missing file
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_upload_triggers_workers(self, client, mock_services):
|
||||
def test_upload_triggers_workers(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload triggers audio and lyrics workers."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.flac", BytesIO(b"audio"), "audio/flac")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
# Workers should be triggered in background
|
||||
|
||||
def test_upload_creates_job(self, client, mock_services):
|
||||
def test_upload_creates_job(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload creates job in job manager."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.flac", BytesIO(b"audio"), "audio/flac")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
mock_services['job_manager'].create_job.assert_called()
|
||||
|
||||
def test_upload_stores_file_to_gcs(self, client, mock_services):
|
||||
def test_upload_stores_file_to_gcs(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload stores file to GCS."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.flac", BytesIO(b"audio"), "audio/flac")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
@@ -202,28 +224,31 @@ class TestFileUploadEndpoint:
|
||||
class TestUploadValidation:
|
||||
"""Tests for upload validation logic."""
|
||||
|
||||
def test_upload_accepts_m4a(self, client, mock_services):
|
||||
def test_upload_accepts_m4a(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload accepts M4A files."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.m4a", BytesIO(b"audio"), "audio/mp4")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_upload_accepts_ogg(self, client, mock_services):
|
||||
def test_upload_accepts_ogg(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload accepts OGG files."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.ogg", BytesIO(b"audio"), "audio/ogg")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_upload_accepts_aac(self, client, mock_services):
|
||||
def test_upload_accepts_aac(self, client, mock_services, upload_auth_headers):
|
||||
"""Test upload accepts AAC files."""
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers=upload_auth_headers,
|
||||
files={"file": ("test.aac", BytesIO(b"audio"), "audio/aac")},
|
||||
data={"artist": "Test", "title": "Song"}
|
||||
)
|
||||
|
||||
@@ -546,6 +546,36 @@ class TestRenderVideoWorkerConfiguration:
|
||||
assert subtitle_offset == -250
|
||||
|
||||
|
||||
class TestRenderVideoWorkerCountdownPadding:
|
||||
"""Tests for render_video_worker countdown detection and audio padding.
|
||||
|
||||
This ensures that when corrections with countdown timestamps are loaded,
|
||||
the audio is padded to match. This prevents video desynchronization.
|
||||
"""
|
||||
|
||||
def test_countdown_processor_import(self):
|
||||
"""Test that render_video_worker imports CountdownProcessor."""
|
||||
from backend.workers.render_video_worker import CountdownProcessor
|
||||
assert CountdownProcessor is not None
|
||||
|
||||
def test_countdown_processor_has_process_method(self):
|
||||
"""Test that CountdownProcessor has the process method (main API)."""
|
||||
from lyrics_transcriber.output.countdown_processor import CountdownProcessor
|
||||
|
||||
countdown_processor = CountdownProcessor(cache_dir="/tmp")
|
||||
assert hasattr(countdown_processor, 'process')
|
||||
assert callable(countdown_processor.process)
|
||||
|
||||
def test_countdown_processor_constants(self):
|
||||
"""Test that CountdownProcessor has expected constants."""
|
||||
from lyrics_transcriber.output.countdown_processor import CountdownProcessor
|
||||
|
||||
# Verify the constants exist (these control countdown behavior)
|
||||
assert hasattr(CountdownProcessor, 'COUNTDOWN_TEXT')
|
||||
assert hasattr(CountdownProcessor, 'COUNTDOWN_PADDING_SECONDS')
|
||||
assert hasattr(CountdownProcessor, 'COUNTDOWN_THRESHOLD_SECONDS')
|
||||
|
||||
|
||||
class TestDownloadHelpers:
|
||||
"""Tests for download helper functions in workers."""
|
||||
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
"""
|
||||
Tests for youtube_service.py - YouTube credential management.
|
||||
|
||||
These tests mock Secret Manager to verify:
|
||||
- Credential loading and parsing
|
||||
- Validation of required fields
|
||||
- Error handling for missing/invalid credentials
|
||||
"""
|
||||
import json
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
|
||||
class TestYouTubeServiceInit:
|
||||
"""Test YouTubeService initialization."""
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_init_creates_service(self, mock_get_settings):
|
||||
"""Test initialization creates service with settings."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
|
||||
assert service.settings == mock_settings
|
||||
assert service._credentials is None
|
||||
assert service._loaded is False
|
||||
|
||||
|
||||
class TestLoadCredentials:
|
||||
"""Test load_credentials method."""
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_load_credentials_success(self, mock_get_settings):
|
||||
"""Test successful credential loading from Secret Manager."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"token": "access-token-123",
|
||||
"refresh_token": "refresh-token-456",
|
||||
"token_uri": "https://oauth2.googleapis.com/token",
|
||||
"client_id": "client-id.apps.googleusercontent.com",
|
||||
"client_secret": "client-secret-789",
|
||||
"scopes": ["https://www.googleapis.com/auth/youtube.upload"],
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
result = service.load_credentials()
|
||||
|
||||
assert result is True
|
||||
assert service._credentials is not None
|
||||
assert service._credentials["refresh_token"] == "refresh-token-456"
|
||||
mock_settings.get_secret.assert_called_once_with("youtube-oauth-credentials")
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_load_credentials_not_found(self, mock_get_settings):
|
||||
"""Test handling when credentials are not in Secret Manager."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
result = service.load_credentials()
|
||||
|
||||
assert result is False
|
||||
assert service._credentials is None
|
||||
assert service._loaded is True
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_load_credentials_missing_required_fields(self, mock_get_settings):
|
||||
"""Test handling when credentials are missing required fields."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"token": "access-token",
|
||||
# Missing: refresh_token, token_uri, client_id, client_secret
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
result = service.load_credentials()
|
||||
|
||||
assert result is False
|
||||
assert service._credentials is None
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_load_credentials_invalid_json(self, mock_get_settings):
|
||||
"""Test handling of invalid JSON in credentials."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = "not valid json {"
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
result = service.load_credentials()
|
||||
|
||||
assert result is False
|
||||
assert service._credentials is None
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_load_credentials_exception(self, mock_get_settings):
|
||||
"""Test handling of exceptions during credential loading."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.side_effect = Exception("Secret Manager error")
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
result = service.load_credentials()
|
||||
|
||||
assert result is False
|
||||
assert service._loaded is True
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_load_credentials_cached(self, mock_get_settings):
|
||||
"""Test that credentials are cached after first load."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"token_uri": "uri",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
|
||||
# First call
|
||||
result1 = service.load_credentials()
|
||||
# Second call
|
||||
result2 = service.load_credentials()
|
||||
|
||||
assert result1 is True
|
||||
assert result2 is True
|
||||
# Secret Manager should only be called once
|
||||
assert mock_settings.get_secret.call_count == 1
|
||||
|
||||
|
||||
class TestGetCredentialsDict:
|
||||
"""Test get_credentials_dict method."""
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_get_credentials_dict_loads_if_needed(self, mock_get_settings):
|
||||
"""Test get_credentials_dict loads credentials if not already loaded."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"token_uri": "uri",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
|
||||
# Don't explicitly call load_credentials
|
||||
result = service.get_credentials_dict()
|
||||
|
||||
assert result is not None
|
||||
assert result["refresh_token"] == "token"
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_get_credentials_dict_returns_none_if_not_configured(
|
||||
self, mock_get_settings
|
||||
):
|
||||
"""Test get_credentials_dict returns None when not configured."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
result = service.get_credentials_dict()
|
||||
|
||||
assert result is None
|
||||
|
||||
|
||||
class TestIsConfigured:
|
||||
"""Test is_configured property."""
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_is_configured_true(self, mock_get_settings):
|
||||
"""Test is_configured returns True when credentials exist."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = json.dumps({
|
||||
"refresh_token": "token",
|
||||
"token_uri": "uri",
|
||||
"client_id": "id",
|
||||
"client_secret": "secret",
|
||||
})
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
|
||||
assert service.is_configured is True
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_is_configured_false(self, mock_get_settings):
|
||||
"""Test is_configured returns False when credentials missing."""
|
||||
from backend.services.youtube_service import YouTubeService
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service = YouTubeService()
|
||||
|
||||
assert service.is_configured is False
|
||||
|
||||
|
||||
class TestGetYouTubeService:
|
||||
"""Test get_youtube_service singleton."""
|
||||
|
||||
@patch("backend.services.youtube_service.get_settings")
|
||||
def test_get_youtube_service_singleton(self, mock_get_settings):
|
||||
"""Test get_youtube_service returns singleton instance."""
|
||||
from backend.services.youtube_service import get_youtube_service
|
||||
import backend.services.youtube_service as youtube_module
|
||||
|
||||
# Reset singleton
|
||||
youtube_module._youtube_service = None
|
||||
|
||||
mock_settings = Mock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
service1 = get_youtube_service()
|
||||
service2 = get_youtube_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
+124
-87
@@ -7,11 +7,17 @@ Handles the audio processing track of parallel processing:
|
||||
3. Post-processing: Combine instrumentals, normalize
|
||||
|
||||
Re-uses karaoke_gen.audio_processor.AudioProcessor for remote GPU separation.
|
||||
|
||||
Observability:
|
||||
- All operations wrapped in tracing spans for Cloud Trace visibility
|
||||
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
|
||||
- Worker start/end timing logged with WORKER_START/WORKER_END markers
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
from typing import Optional, Dict, Any
|
||||
from pathlib import Path
|
||||
|
||||
@@ -20,6 +26,8 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.config import get_settings
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
from backend.services.tracing import job_span, add_span_event, add_span_attribute
|
||||
from backend.services.metrics import metrics
|
||||
|
||||
# Import from karaoke_gen package
|
||||
from karaoke_gen.audio_processor import AudioProcessor
|
||||
@@ -215,6 +223,7 @@ async def process_audio_separation(job_id: str) -> bool:
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
start_time = time.time()
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
settings = get_settings()
|
||||
@@ -222,7 +231,8 @@ async def process_audio_separation(job_id: str) -> bool:
|
||||
# Create job logger for remote debugging FIRST
|
||||
job_log = create_job_logger(job_id, "audio")
|
||||
|
||||
# Log immediately to verify logging is working
|
||||
# Log with structured markers for easy Cloud Logging queries
|
||||
logger.info(f"[job:{job_id}] WORKER_START worker=audio")
|
||||
job_log.info("=== AUDIO WORKER STARTED ===")
|
||||
job_log.info(f"Job ID: {job_id}")
|
||||
|
||||
@@ -232,7 +242,7 @@ async def process_audio_separation(job_id: str) -> bool:
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
logger.error(f"[job:{job_id}] Job not found in Firestore")
|
||||
job_log.error(f"Job {job_id} not found in Firestore!")
|
||||
return False
|
||||
|
||||
@@ -241,91 +251,114 @@ async def process_audio_separation(job_id: str) -> bool:
|
||||
job_log.info(f"Created temp directory: {temp_dir}")
|
||||
|
||||
try:
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (karaoke_gen.audio_processor) are
|
||||
# only captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting audio separation for {job.artist} - {job.title}")
|
||||
logger.info(f"Starting audio separation for job {job_id}")
|
||||
|
||||
# Ensure AUDIO_SEPARATOR_API_URL is set
|
||||
api_url = os.environ.get("AUDIO_SEPARATOR_API_URL")
|
||||
if not api_url:
|
||||
raise Exception("AUDIO_SEPARATOR_API_URL environment variable not set. "
|
||||
"Cannot perform audio separation without remote API access.")
|
||||
job_log.info(f"Audio separator API: {api_url}")
|
||||
|
||||
# Download audio file from GCS or URL
|
||||
job_log.info("Downloading audio file...")
|
||||
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager_instance=job_manager)
|
||||
if not audio_path:
|
||||
raise Exception("Failed to download audio file")
|
||||
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
# The status is managed at a higher level - workers just track their progress
|
||||
job_manager.update_state_data(job_id, 'audio_progress', {
|
||||
'stage': 'separating_stage1',
|
||||
'progress': 10,
|
||||
'message': 'Starting audio separation (Stage 1: Clean instrumental)'
|
||||
})
|
||||
|
||||
# Create AudioProcessor instance (reuses karaoke_gen code)
|
||||
# Use model configuration from job if provided, otherwise use defaults
|
||||
job_log.info("Creating AudioProcessor instance...")
|
||||
if job.clean_instrumental_model:
|
||||
job_log.info(f" Using clean instrumental model: {job.clean_instrumental_model}")
|
||||
if job.backing_vocals_models:
|
||||
job_log.info(f" Using backing vocals models: {job.backing_vocals_models}")
|
||||
if job.other_stems_models:
|
||||
job_log.info(f" Using other stems models: {job.other_stems_models}")
|
||||
|
||||
audio_processor = create_audio_processor(
|
||||
temp_dir,
|
||||
clean_instrumental_model=job.clean_instrumental_model,
|
||||
backing_vocals_models=job.backing_vocals_models,
|
||||
other_stems_models=job.other_stems_models
|
||||
)
|
||||
|
||||
# Format artist-title for file naming (matches CLI behavior)
|
||||
artist_title = f"{job.artist} - {job.title}"
|
||||
|
||||
# Run audio separation (calls Modal API internally)
|
||||
# This returns a dict with paths to all separated stems
|
||||
job_log.info("Starting audio separation (this may take 5-10 minutes)...")
|
||||
job_log.info(" Stage 1: Clean instrumental separation (MDX models)")
|
||||
job_log.info(" Stage 2: Backing vocals separation (Demucs model)")
|
||||
logger.info(f"Job {job_id}: Calling audio_processor.process_audio_separation()")
|
||||
separation_result = audio_processor.process_audio_separation(
|
||||
audio_file=audio_path,
|
||||
artist_title=artist_title,
|
||||
track_output_dir=temp_dir
|
||||
)
|
||||
|
||||
job_log.info("Audio separation complete!")
|
||||
job_log.info(f" Generated {len(separation_result)} stem files")
|
||||
logger.info(f"Job {job_id}: Audio separation complete, organizing results")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
job_manager.update_state_data(job_id, 'audio_progress', {
|
||||
'stage': 'audio_complete',
|
||||
'progress': 45,
|
||||
'message': 'Audio separation complete, uploading stems'
|
||||
})
|
||||
|
||||
# Upload all stems to GCS
|
||||
await upload_separation_results(job_id, separation_result, storage, job_manager)
|
||||
|
||||
logger.info(f"Job {job_id}: All stems uploaded successfully")
|
||||
|
||||
# Mark audio processing complete
|
||||
# This will check if lyrics are also complete and transition to next stage if so
|
||||
job_manager.mark_audio_complete(job_id)
|
||||
|
||||
return True
|
||||
# Wrap entire worker in a tracing span
|
||||
with job_span("audio-worker", job_id, {"artist": job.artist, "title": job.title}) as root_span:
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (karaoke_gen.audio_processor) are
|
||||
# only captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting audio separation for {job.artist} - {job.title}")
|
||||
logger.info(f"[job:{job_id}] Starting audio separation for {job.artist} - {job.title}")
|
||||
|
||||
# Ensure AUDIO_SEPARATOR_API_URL is set
|
||||
api_url = os.environ.get("AUDIO_SEPARATOR_API_URL")
|
||||
if not api_url:
|
||||
raise Exception("AUDIO_SEPARATOR_API_URL environment variable not set. "
|
||||
"Cannot perform audio separation without remote API access.")
|
||||
job_log.info(f"Audio separator API: {api_url}")
|
||||
add_span_attribute("audio_separator_api", api_url)
|
||||
|
||||
# Download audio file from GCS or URL
|
||||
with job_span("download-audio", job_id) as download_span:
|
||||
job_log.info("Downloading audio file...")
|
||||
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager_instance=job_manager)
|
||||
if not audio_path:
|
||||
raise Exception("Failed to download audio file")
|
||||
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
|
||||
download_span.set_attribute("audio_file", os.path.basename(audio_path))
|
||||
download_span.set_attribute("source", "url" if job.url else "gcs")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
# The status is managed at a higher level - workers just track their progress
|
||||
job_manager.update_state_data(job_id, 'audio_progress', {
|
||||
'stage': 'separating_stage1',
|
||||
'progress': 10,
|
||||
'message': 'Starting audio separation (Stage 1: Clean instrumental)'
|
||||
})
|
||||
|
||||
# Create AudioProcessor instance (reuses karaoke_gen code)
|
||||
# Use model configuration from job if provided, otherwise use defaults
|
||||
job_log.info("Creating AudioProcessor instance...")
|
||||
if job.clean_instrumental_model:
|
||||
job_log.info(f" Using clean instrumental model: {job.clean_instrumental_model}")
|
||||
add_span_attribute("clean_model", job.clean_instrumental_model)
|
||||
if job.backing_vocals_models:
|
||||
job_log.info(f" Using backing vocals models: {job.backing_vocals_models}")
|
||||
if job.other_stems_models:
|
||||
job_log.info(f" Using other stems models: {job.other_stems_models}")
|
||||
|
||||
audio_processor = create_audio_processor(
|
||||
temp_dir,
|
||||
clean_instrumental_model=job.clean_instrumental_model,
|
||||
backing_vocals_models=job.backing_vocals_models,
|
||||
other_stems_models=job.other_stems_models
|
||||
)
|
||||
|
||||
# Format artist-title for file naming (matches CLI behavior)
|
||||
artist_title = f"{job.artist} - {job.title}"
|
||||
|
||||
# Run audio separation (calls Modal API internally)
|
||||
# This returns a dict with paths to all separated stems
|
||||
with job_span("modal-separation", job_id) as sep_span:
|
||||
sep_start = time.time()
|
||||
job_log.info("Starting audio separation (this may take 5-10 minutes)...")
|
||||
job_log.info(" Stage 1: Clean instrumental separation (MDX models)")
|
||||
job_log.info(" Stage 2: Backing vocals separation (Demucs model)")
|
||||
add_span_event("separation_started")
|
||||
logger.info(f"[job:{job_id}] Calling Modal API for audio separation")
|
||||
|
||||
with metrics.time_external_api("modal", job_id):
|
||||
separation_result = audio_processor.process_audio_separation(
|
||||
audio_file=audio_path,
|
||||
artist_title=artist_title,
|
||||
track_output_dir=temp_dir
|
||||
)
|
||||
|
||||
sep_duration = time.time() - sep_start
|
||||
sep_span.set_attribute("duration_seconds", sep_duration)
|
||||
sep_span.set_attribute("stem_count", len(separation_result))
|
||||
add_span_event("separation_completed", {"duration_seconds": sep_duration})
|
||||
|
||||
job_log.info("Audio separation complete!")
|
||||
job_log.info(f" Generated {len(separation_result)} stem files")
|
||||
logger.info(f"[job:{job_id}] Audio separation complete, organizing results")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
job_manager.update_state_data(job_id, 'audio_progress', {
|
||||
'stage': 'audio_complete',
|
||||
'progress': 45,
|
||||
'message': 'Audio separation complete, uploading stems'
|
||||
})
|
||||
|
||||
# Upload all stems to GCS
|
||||
with job_span("upload-stems", job_id) as upload_span:
|
||||
await upload_separation_results(job_id, separation_result, storage, job_manager)
|
||||
upload_span.set_attribute("stem_count", len(separation_result))
|
||||
|
||||
logger.info(f"[job:{job_id}] All stems uploaded successfully")
|
||||
|
||||
# Mark audio processing complete
|
||||
# This will check if lyrics are also complete and transition to next stage if so
|
||||
job_manager.mark_audio_complete(job_id)
|
||||
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
logger.info(f"[job:{job_id}] WORKER_END worker=audio status=success duration={duration:.1f}s")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Audio separation failed: {e}", exc_info=True)
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[job:{job_id}] WORKER_END worker=audio status=error duration={duration:.1f}s error={e}")
|
||||
job_manager.mark_job_failed(
|
||||
job_id=job_id,
|
||||
error_message=f"Audio separation failed: {str(e)}",
|
||||
@@ -344,7 +377,7 @@ async def process_audio_separation(job_id: str) -> bool:
|
||||
# Cleanup temporary directory
|
||||
if os.path.exists(temp_dir):
|
||||
shutil.rmtree(temp_dir)
|
||||
logger.debug(f"Cleaned up temp directory: {temp_dir}")
|
||||
logger.debug(f"[job:{job_id}] Cleaned up temp directory: {temp_dir}")
|
||||
|
||||
|
||||
async def download_audio(
|
||||
@@ -372,10 +405,14 @@ async def download_audio(
|
||||
Path to downloaded audio file, or None if failed
|
||||
"""
|
||||
try:
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
# Case 1: File was uploaded to GCS
|
||||
if job.input_media_gcs_path:
|
||||
logger.info(f"Job {job_id}: Downloading uploaded file from GCS: {job.input_media_gcs_path}")
|
||||
local_path = os.path.join(temp_dir, job.filename or "input.flac")
|
||||
# Sanitize filename to handle Unicode chars that cause HTTP header encoding issues
|
||||
safe_filename = sanitize_filename(job.filename) if job.filename else "input.flac"
|
||||
local_path = os.path.join(temp_dir, safe_filename)
|
||||
storage.download_file(job.input_media_gcs_path, local_path)
|
||||
logger.info(f"Job {job_id}: Downloaded uploaded file to {local_path}")
|
||||
return local_path
|
||||
|
||||
+207
-162
@@ -12,12 +12,18 @@ Handles the lyrics processing track of parallel processing:
|
||||
Re-uses:
|
||||
- karaoke_gen.lyrics_processor.LyricsProcessor for lyrics fetching and orchestration
|
||||
- lyrics_transcriber library (submodule) for transcription and correction
|
||||
|
||||
Observability:
|
||||
- All operations wrapped in tracing spans for Cloud Trace visibility
|
||||
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
|
||||
- Worker start/end timing logged with WORKER_START/WORKER_END markers
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
import json
|
||||
from typing import Optional, Dict, Any
|
||||
from pathlib import Path
|
||||
@@ -28,6 +34,8 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
from backend.workers.style_helper import load_style_config
|
||||
from backend.services.tracing import job_span, add_span_event, add_span_attribute
|
||||
from backend.services.metrics import metrics
|
||||
|
||||
# Import from karaoke_gen package
|
||||
from karaoke_gen.lyrics_processor import LyricsProcessor
|
||||
@@ -108,13 +116,15 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
start_time = time.time()
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
# Create job logger for remote debugging FIRST
|
||||
job_log = create_job_logger(job_id, "lyrics")
|
||||
|
||||
# Log immediately to verify logging is working
|
||||
# Log with structured markers for easy Cloud Logging queries
|
||||
logger.info(f"[job:{job_id}] WORKER_START worker=lyrics")
|
||||
job_log.info("=== LYRICS WORKER STARTED ===")
|
||||
job_log.info(f"Job ID: {job_id}")
|
||||
|
||||
@@ -127,166 +137,188 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
temp_dir = None
|
||||
|
||||
try:
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (lyrics_transcriber) are only captured
|
||||
# by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
job_log.error(f"Job {job_id} not found in Firestore!")
|
||||
return False
|
||||
|
||||
# Create temporary working directory
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_lyrics_{job_id}_")
|
||||
job_log.info(f"Created temp directory: {temp_dir}")
|
||||
job_log.info(f"Starting lyrics transcription for {job.artist} - {job.title}")
|
||||
job_log.info(f"Log capture enabled for: {', '.join(LYRICS_WORKER_LOGGERS)}")
|
||||
logger.info(f"Starting lyrics transcription for job {job_id}")
|
||||
|
||||
# Log environment configuration
|
||||
job_log.info("Checking API configuration...")
|
||||
if os.environ.get("GENIUS_API_TOKEN"):
|
||||
job_log.info(" Genius API: configured")
|
||||
else:
|
||||
job_log.warning(" Genius API: NOT configured")
|
||||
if os.environ.get("SPOTIFY_COOKIE_SP_DC"):
|
||||
job_log.info(" Spotify: configured")
|
||||
else:
|
||||
job_log.warning(" Spotify: NOT configured")
|
||||
if os.environ.get("RAPIDAPI_KEY"):
|
||||
job_log.info(" Musixmatch (RapidAPI): configured")
|
||||
else:
|
||||
job_log.warning(" Musixmatch (RapidAPI): NOT configured")
|
||||
|
||||
# Ensure required environment variables are set
|
||||
if not os.environ.get("AUDIOSHAKE_API_TOKEN"):
|
||||
job_log.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
|
||||
logger.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
|
||||
|
||||
# Download audio file from GCS (waits for audio worker if URL job)
|
||||
job_log.info("Downloading audio file from GCS...")
|
||||
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager)
|
||||
if not audio_path:
|
||||
raise Exception("Failed to download audio file")
|
||||
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
# The status is managed at a higher level - workers just track their progress
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
'stage': 'transcribing',
|
||||
'progress': 10,
|
||||
'message': 'Starting lyrics transcription via AudioShake'
|
||||
})
|
||||
|
||||
# Load style configuration (downloads assets from GCS if available)
|
||||
# This is needed for max_line_length and other style settings in LyricsTranscriber
|
||||
job_log.info("Loading style configuration for lyrics processing...")
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
style_params_json_path = style_config.get_style_params_path() if style_config.has_custom_styles() else None
|
||||
|
||||
if style_params_json_path:
|
||||
job_log.info(f"Using custom style params: {style_params_json_path}")
|
||||
# Log the contents of the style params for debugging
|
||||
try:
|
||||
with open(style_params_json_path, 'r') as f:
|
||||
style_content = json.load(f)
|
||||
job_log.info(f"Style params sections: {list(style_content.keys())}")
|
||||
if 'karaoke' in style_content:
|
||||
karaoke_style = style_content['karaoke']
|
||||
job_log.info(f" karaoke.background_image: {karaoke_style.get('background_image', 'NOT SET')}")
|
||||
job_log.info(f" karaoke.font_path: {karaoke_style.get('font_path', 'NOT SET')}")
|
||||
except Exception as e:
|
||||
job_log.warning(f"Could not read style params for logging: {e}")
|
||||
else:
|
||||
job_log.info("No custom style params found, using defaults")
|
||||
|
||||
# Download user-provided lyrics file if available
|
||||
lyrics_file_path = None
|
||||
if hasattr(job, 'lyrics_file_gcs_path') and job.lyrics_file_gcs_path:
|
||||
lyrics_file_path = os.path.join(temp_dir, "user_lyrics.txt")
|
||||
job_log.info(f"Downloading user-provided lyrics file: {job.lyrics_file_gcs_path}")
|
||||
storage.download_file(job.lyrics_file_gcs_path, lyrics_file_path)
|
||||
job_log.info(f"User lyrics file downloaded to: {lyrics_file_path}")
|
||||
|
||||
# Get lyrics configuration from job
|
||||
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
|
||||
if subtitle_offset != 0:
|
||||
job_log.info(f"Subtitle offset: {subtitle_offset}ms")
|
||||
|
||||
# Create LyricsProcessor instance (reuses karaoke_gen code)
|
||||
job_log.info("Creating LyricsProcessor instance...")
|
||||
job_log.info(f" style_params_json: {style_params_json_path}")
|
||||
job_log.info(f" lyrics_file: {lyrics_file_path}")
|
||||
job_log.info(f" subtitle_offset_ms: {subtitle_offset}")
|
||||
lyrics_processor = create_lyrics_processor(
|
||||
style_params_json=style_params_json_path,
|
||||
lyrics_file=lyrics_file_path,
|
||||
subtitle_offset_ms=subtitle_offset
|
||||
)
|
||||
|
||||
# Run transcription + correction
|
||||
# This will:
|
||||
# 1. Fetch reference lyrics from Genius/Spotify/Musixmatch/LRCLib
|
||||
# 2. Transcribe audio via AudioShake API
|
||||
# 3. Run automatic correction with lyrics_transcriber
|
||||
# 4. Generate corrections JSON (no interactive review)
|
||||
# 5. Skip video generation (render_video=False)
|
||||
|
||||
# Use lyrics_artist/lyrics_title overrides if provided, else fall back to job artist/title
|
||||
lyrics_search_artist = getattr(job, 'lyrics_artist', None) or job.artist
|
||||
lyrics_search_title = getattr(job, 'lyrics_title', None) or job.title
|
||||
|
||||
job_log.info("Starting LyricsTranscriber processing...")
|
||||
job_log.info(f" Artist: {job.artist}")
|
||||
job_log.info(f" Title: {job.title}")
|
||||
if lyrics_search_artist != job.artist:
|
||||
job_log.info(f" Lyrics search artist override: {lyrics_search_artist}")
|
||||
if lyrics_search_title != job.title:
|
||||
job_log.info(f" Lyrics search title override: {lyrics_search_title}")
|
||||
logger.info(f"Job {job_id}: Calling lyrics_processor.transcribe_lyrics()")
|
||||
|
||||
# Run transcription in thread pool to avoid blocking the event loop
|
||||
# (transcribe_lyrics is synchronous and takes 1-2 minutes)
|
||||
# Note: We pass the original artist/title for file naming, but the LyricsProcessor
|
||||
# will use lyrics_artist/lyrics_title for the search (set via constructor args)
|
||||
result = await asyncio.to_thread(
|
||||
lyrics_processor.transcribe_lyrics,
|
||||
input_audio_wav=audio_path,
|
||||
artist=job.artist, # Original artist for file naming
|
||||
title=job.title, # Original title for file naming
|
||||
track_output_dir=temp_dir,
|
||||
lyrics_artist=lyrics_search_artist, # Override for lyrics search
|
||||
lyrics_title=lyrics_search_title # Override for lyrics search
|
||||
)
|
||||
|
||||
job_log.info("Transcription processing complete")
|
||||
logger.info(f"Job {job_id}: Transcription complete, uploading results")
|
||||
|
||||
# Upload lyrics results to GCS
|
||||
job_log.info("Uploading lyrics results to GCS...")
|
||||
await upload_lyrics_results(job_id, temp_dir, result, storage, job_manager, job_log)
|
||||
|
||||
job_log.info("All lyrics data uploaded successfully")
|
||||
logger.info(f"Job {job_id}: All lyrics data uploaded successfully")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
'stage': 'lyrics_complete',
|
||||
'progress': 45,
|
||||
'message': 'Lyrics transcription complete'
|
||||
})
|
||||
|
||||
# Mark lyrics processing complete
|
||||
# This will check if audio is also complete and transition to next stage if so
|
||||
job_log.info("Lyrics worker complete, checking if audio is also done...")
|
||||
job_manager.mark_lyrics_complete(job_id)
|
||||
|
||||
return True
|
||||
# Wrap entire worker in a tracing span
|
||||
with job_span("lyrics-worker", job_id) as root_span:
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (lyrics_transcriber) are only captured
|
||||
# by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"[job:{job_id}] Job not found in Firestore")
|
||||
job_log.error(f"Job {job_id} not found in Firestore!")
|
||||
return False
|
||||
|
||||
add_span_attribute("artist", job.artist)
|
||||
add_span_attribute("title", job.title)
|
||||
|
||||
# Create temporary working directory
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_lyrics_{job_id}_")
|
||||
job_log.info(f"Created temp directory: {temp_dir}")
|
||||
job_log.info(f"Starting lyrics transcription for {job.artist} - {job.title}")
|
||||
job_log.info(f"Log capture enabled for: {', '.join(LYRICS_WORKER_LOGGERS)}")
|
||||
logger.info(f"[job:{job_id}] Starting lyrics transcription for {job.artist} - {job.title}")
|
||||
|
||||
# Log environment configuration
|
||||
with job_span("check-api-config", job_id):
|
||||
job_log.info("Checking API configuration...")
|
||||
apis_configured = []
|
||||
if os.environ.get("GENIUS_API_TOKEN"):
|
||||
job_log.info(" Genius API: configured")
|
||||
apis_configured.append("genius")
|
||||
else:
|
||||
job_log.warning(" Genius API: NOT configured")
|
||||
if os.environ.get("SPOTIFY_COOKIE_SP_DC"):
|
||||
job_log.info(" Spotify: configured")
|
||||
apis_configured.append("spotify")
|
||||
else:
|
||||
job_log.warning(" Spotify: NOT configured")
|
||||
if os.environ.get("RAPIDAPI_KEY"):
|
||||
job_log.info(" Musixmatch (RapidAPI): configured")
|
||||
apis_configured.append("musixmatch")
|
||||
else:
|
||||
job_log.warning(" Musixmatch (RapidAPI): NOT configured")
|
||||
add_span_attribute("lyrics_apis", ",".join(apis_configured))
|
||||
|
||||
# Ensure required environment variables are set
|
||||
if not os.environ.get("AUDIOSHAKE_API_TOKEN"):
|
||||
job_log.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
|
||||
logger.warning(f"[job:{job_id}] AUDIOSHAKE_API_TOKEN not set - transcription may fail")
|
||||
|
||||
# Download audio file from GCS (waits for audio worker if URL job)
|
||||
with job_span("download-audio", job_id) as download_span:
|
||||
job_log.info("Downloading audio file from GCS...")
|
||||
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager)
|
||||
if not audio_path:
|
||||
raise Exception("Failed to download audio file")
|
||||
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
|
||||
download_span.set_attribute("audio_file", os.path.basename(audio_path))
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
# The status is managed at a higher level - workers just track their progress
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
'stage': 'transcribing',
|
||||
'progress': 10,
|
||||
'message': 'Starting lyrics transcription via AudioShake'
|
||||
})
|
||||
|
||||
# Load style configuration (downloads assets from GCS if available)
|
||||
# This is needed for max_line_length and other style settings in LyricsTranscriber
|
||||
with job_span("load-style-config", job_id):
|
||||
job_log.info("Loading style configuration for lyrics processing...")
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
style_params_json_path = style_config.get_style_params_path() if style_config.has_custom_styles() else None
|
||||
|
||||
if style_params_json_path:
|
||||
job_log.info(f"Using custom style params: {style_params_json_path}")
|
||||
# Log the contents of the style params for debugging
|
||||
try:
|
||||
with open(style_params_json_path, 'r') as f:
|
||||
style_content = json.load(f)
|
||||
job_log.info(f"Style params sections: {list(style_content.keys())}")
|
||||
if 'karaoke' in style_content:
|
||||
karaoke_style = style_content['karaoke']
|
||||
job_log.info(f" karaoke.background_image: {karaoke_style.get('background_image', 'NOT SET')}")
|
||||
job_log.info(f" karaoke.font_path: {karaoke_style.get('font_path', 'NOT SET')}")
|
||||
except Exception as e:
|
||||
job_log.warning(f"Could not read style params for logging: {e}")
|
||||
else:
|
||||
job_log.info("No custom style params found, using defaults")
|
||||
|
||||
# Download user-provided lyrics file if available
|
||||
lyrics_file_path = None
|
||||
if hasattr(job, 'lyrics_file_gcs_path') and job.lyrics_file_gcs_path:
|
||||
with job_span("download-user-lyrics", job_id):
|
||||
lyrics_file_path = os.path.join(temp_dir, "user_lyrics.txt")
|
||||
job_log.info(f"Downloading user-provided lyrics file: {job.lyrics_file_gcs_path}")
|
||||
storage.download_file(job.lyrics_file_gcs_path, lyrics_file_path)
|
||||
job_log.info(f"User lyrics file downloaded to: {lyrics_file_path}")
|
||||
|
||||
# Get lyrics configuration from job
|
||||
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
|
||||
if subtitle_offset != 0:
|
||||
job_log.info(f"Subtitle offset: {subtitle_offset}ms")
|
||||
add_span_attribute("subtitle_offset_ms", subtitle_offset)
|
||||
|
||||
# Create LyricsProcessor instance (reuses karaoke_gen code)
|
||||
job_log.info("Creating LyricsProcessor instance...")
|
||||
job_log.info(f" style_params_json: {style_params_json_path}")
|
||||
job_log.info(f" lyrics_file: {lyrics_file_path}")
|
||||
job_log.info(f" subtitle_offset_ms: {subtitle_offset}")
|
||||
lyrics_processor = create_lyrics_processor(
|
||||
style_params_json=style_params_json_path,
|
||||
lyrics_file=lyrics_file_path,
|
||||
subtitle_offset_ms=subtitle_offset
|
||||
)
|
||||
|
||||
# Use lyrics_artist/lyrics_title overrides if provided, else fall back to job artist/title
|
||||
lyrics_search_artist = getattr(job, 'lyrics_artist', None) or job.artist
|
||||
lyrics_search_title = getattr(job, 'lyrics_title', None) or job.title
|
||||
|
||||
job_log.info("Starting LyricsTranscriber processing...")
|
||||
job_log.info(f" Artist: {job.artist}")
|
||||
job_log.info(f" Title: {job.title}")
|
||||
if lyrics_search_artist != job.artist:
|
||||
job_log.info(f" Lyrics search artist override: {lyrics_search_artist}")
|
||||
if lyrics_search_title != job.title:
|
||||
job_log.info(f" Lyrics search title override: {lyrics_search_title}")
|
||||
logger.info(f"[job:{job_id}] Calling lyrics_processor.transcribe_lyrics()")
|
||||
|
||||
# Run transcription + correction
|
||||
with job_span("audioshake-transcription", job_id) as trans_span:
|
||||
trans_start = time.time()
|
||||
add_span_event("transcription_started")
|
||||
|
||||
# Run transcription in thread pool to avoid blocking the event loop
|
||||
# (transcribe_lyrics is synchronous and takes 1-2 minutes)
|
||||
# Note: transcribe_lyrics internally calls AudioShake and lyrics APIs
|
||||
with metrics.time_external_api("audioshake", job_id):
|
||||
result = await asyncio.to_thread(
|
||||
lyrics_processor.transcribe_lyrics,
|
||||
input_audio_wav=audio_path,
|
||||
artist=job.artist, # Original artist for file naming
|
||||
title=job.title, # Original title for file naming
|
||||
track_output_dir=temp_dir,
|
||||
lyrics_artist=lyrics_search_artist, # Override for lyrics search
|
||||
lyrics_title=lyrics_search_title # Override for lyrics search
|
||||
)
|
||||
|
||||
trans_duration = time.time() - trans_start
|
||||
trans_span.set_attribute("duration_seconds", trans_duration)
|
||||
add_span_event("transcription_completed", {"duration_seconds": trans_duration})
|
||||
|
||||
job_log.info("Transcription processing complete")
|
||||
logger.info(f"[job:{job_id}] Transcription complete, uploading results")
|
||||
|
||||
# Upload lyrics results to GCS
|
||||
with job_span("upload-lyrics-results", job_id):
|
||||
job_log.info("Uploading lyrics results to GCS...")
|
||||
await upload_lyrics_results(job_id, temp_dir, result, storage, job_manager, job_log)
|
||||
|
||||
job_log.info("All lyrics data uploaded successfully")
|
||||
logger.info(f"[job:{job_id}] All lyrics data uploaded successfully")
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
'stage': 'lyrics_complete',
|
||||
'progress': 45,
|
||||
'message': 'Lyrics transcription complete'
|
||||
})
|
||||
|
||||
# Mark lyrics processing complete
|
||||
# This will check if audio is also complete and transition to next stage if so
|
||||
job_log.info("Lyrics worker complete, checking if audio is also done...")
|
||||
job_manager.mark_lyrics_complete(job_id)
|
||||
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
logger.info(f"[job:{job_id}] WORKER_END worker=lyrics status=success duration={duration:.1f}s")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
job_log.error(f"Lyrics transcription failed: {str(e)}", exc_info=True)
|
||||
logger.error(f"Job {job_id}: Lyrics transcription failed: {e}", exc_info=True)
|
||||
logger.error(f"[job:{job_id}] WORKER_END worker=lyrics status=error duration={duration:.1f}s error={e}")
|
||||
job_manager.mark_job_failed(
|
||||
job_id=job_id,
|
||||
error_message=f"Lyrics transcription failed: {str(e)}",
|
||||
@@ -305,7 +337,7 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
# Cleanup temporary directory (only if it was created)
|
||||
if temp_dir and os.path.exists(temp_dir):
|
||||
shutil.rmtree(temp_dir)
|
||||
logger.debug(f"Cleaned up temp directory: {temp_dir}")
|
||||
logger.debug(f"[job:{job_id}] Cleaned up temp directory: {temp_dir}")
|
||||
|
||||
|
||||
async def download_audio(
|
||||
@@ -447,11 +479,24 @@ async def upload_lyrics_results(
|
||||
corrections_data = json.load(f)
|
||||
|
||||
# Store metadata in state_data
|
||||
job_manager.update_state_data(job_id, 'lyrics_metadata', {
|
||||
# Include countdown padding info for instrumental synchronization
|
||||
lyrics_metadata = {
|
||||
'segment_count': len(corrections_data.get('corrected_segments', [])),
|
||||
'has_corrections': True,
|
||||
'ready_for_review': True
|
||||
})
|
||||
'ready_for_review': True,
|
||||
}
|
||||
|
||||
# Add countdown padding info from transcription result
|
||||
# This is needed by video_worker to pad instrumentals for sync
|
||||
if transcription_result.get("countdown_padding_added"):
|
||||
lyrics_metadata['has_countdown_padding'] = True
|
||||
lyrics_metadata['countdown_padding_seconds'] = transcription_result.get("countdown_padding_seconds", 3.0)
|
||||
logger.info(f"Job {job_id}: Countdown padding detected: {lyrics_metadata['countdown_padding_seconds']}s")
|
||||
else:
|
||||
lyrics_metadata['has_countdown_padding'] = False
|
||||
lyrics_metadata['countdown_padding_seconds'] = 0.0
|
||||
|
||||
job_manager.update_state_data(job_id, 'lyrics_metadata', lyrics_metadata)
|
||||
except Exception as e:
|
||||
logger.warning(f"Job {job_id}: Could not parse corrections JSON: {e}")
|
||||
else:
|
||||
|
||||
@@ -14,10 +14,16 @@ This worker:
|
||||
Key insight: We use OutputGenerator from lyrics_transcriber library
|
||||
WITHOUT using its blocking ReviewServer. This allows async operation
|
||||
in Cloud Run.
|
||||
|
||||
Observability:
|
||||
- All operations wrapped in tracing spans for Cloud Trace visibility
|
||||
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
|
||||
- Worker start/end timing logged with WORKER_START/WORKER_END markers
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import time
|
||||
import json
|
||||
from typing import Optional, Dict, Any
|
||||
from dataclasses import dataclass
|
||||
@@ -27,9 +33,11 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.config import get_settings
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
from backend.services.tracing import job_span, add_span_event, add_span_attribute
|
||||
|
||||
# Import from lyrics_transcriber (submodule)
|
||||
from lyrics_transcriber.output.generator import OutputGenerator
|
||||
from lyrics_transcriber.output.countdown_processor import CountdownProcessor
|
||||
from lyrics_transcriber.types import CorrectionResult
|
||||
from lyrics_transcriber.core.config import OutputConfig
|
||||
|
||||
@@ -62,12 +70,16 @@ async def process_render_video(job_id: str) -> bool:
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
start_time = time.time()
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
settings = get_settings()
|
||||
|
||||
# Create job logger for remote debugging FIRST
|
||||
job_log = create_job_logger(job_id, "render_video")
|
||||
|
||||
# Log with structured markers for easy Cloud Logging queries
|
||||
logger.info(f"[job:{job_id}] WORKER_START worker=render-video")
|
||||
job_log.info("=== RENDER VIDEO WORKER STARTED ===")
|
||||
job_log.info(f"Job ID: {job_id}")
|
||||
|
||||
@@ -77,225 +89,267 @@ async def process_render_video(job_id: str) -> bool:
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
logger.error(f"[job:{job_id}] Job not found in Firestore")
|
||||
job_log.error(f"Job {job_id} not found in Firestore!")
|
||||
return False
|
||||
|
||||
job_log.info(f"Starting video render for {job.artist} - {job.title}")
|
||||
logger.info(f"Job {job_id}: Starting video render (post-review)")
|
||||
logger.info(f"[job:{job_id}] Starting video render (post-review)")
|
||||
|
||||
try:
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (lyrics_transcriber.output) are only
|
||||
# captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
# Transition to RENDERING_VIDEO
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.RENDERING_VIDEO,
|
||||
progress=75,
|
||||
message="Rendering karaoke video with corrected lyrics"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
job_log.info(f"Created temp directory: {temp_dir}")
|
||||
# 1. Download corrected corrections data
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
# Wrap entire worker in a tracing span
|
||||
with job_span("render-video-worker", job_id, {"artist": job.artist, "title": job.title}) as root_span:
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (lyrics_transcriber.output) are only
|
||||
# captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
# Transition to RENDERING_VIDEO
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.RENDERING_VIDEO,
|
||||
progress=75,
|
||||
message="Rendering karaoke video with corrected lyrics"
|
||||
)
|
||||
|
||||
# Try updated corrections first (from human review), fall back to original
|
||||
corrections_gcs_updated = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
corrections_gcs_original = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
|
||||
# Get GCS path from file_urls
|
||||
corrections_url = job.file_urls.get('lyrics', {}).get('corrections_updated')
|
||||
if not corrections_url:
|
||||
corrections_url = job.file_urls.get('lyrics', {}).get('corrections')
|
||||
|
||||
if not corrections_url:
|
||||
# Try direct GCS paths
|
||||
if storage.file_exists(corrections_gcs_updated):
|
||||
corrections_gcs = corrections_gcs_updated
|
||||
elif storage.file_exists(corrections_gcs_original):
|
||||
corrections_gcs = corrections_gcs_original
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
job_log.info(f"Created temp directory: {temp_dir}")
|
||||
|
||||
# Try updated corrections first (from human review), fall back to original
|
||||
corrections_gcs_updated = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
corrections_gcs_original = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
|
||||
# Get GCS path from file_urls
|
||||
corrections_url = job.file_urls.get('lyrics', {}).get('corrections_updated')
|
||||
if not corrections_url:
|
||||
corrections_url = job.file_urls.get('lyrics', {}).get('corrections')
|
||||
|
||||
if not corrections_url:
|
||||
# Try direct GCS paths
|
||||
if storage.file_exists(corrections_gcs_updated):
|
||||
corrections_gcs = corrections_gcs_updated
|
||||
elif storage.file_exists(corrections_gcs_original):
|
||||
corrections_gcs = corrections_gcs_original
|
||||
else:
|
||||
raise FileNotFoundError(f"No corrections file found for job {job_id}")
|
||||
else:
|
||||
raise FileNotFoundError(f"No corrections file found for job {job_id}")
|
||||
else:
|
||||
# Extract GCS path from URL
|
||||
corrections_gcs = _extract_gcs_path(corrections_url)
|
||||
|
||||
# 2. Load the ORIGINAL corrections (has full structure)
|
||||
original_corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
original_corrections_path = os.path.join(temp_dir, "corrections_original.json")
|
||||
|
||||
job_log.info(f"Downloading original corrections from {original_corrections_gcs}")
|
||||
logger.info(f"Job {job_id}: Downloading original corrections from {original_corrections_gcs}")
|
||||
storage.download_file(original_corrections_gcs, original_corrections_path)
|
||||
|
||||
with open(original_corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
# 3. Check if there are updated corrections (from review UI)
|
||||
# The frontend sends only partial data: {corrections, corrected_segments}
|
||||
updated_corrections_gcs = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
|
||||
if storage.file_exists(updated_corrections_gcs):
|
||||
job_log.info("Found updated corrections from review, merging")
|
||||
logger.info(f"Job {job_id}: Found updated corrections, merging")
|
||||
updated_path = os.path.join(temp_dir, "corrections_updated.json")
|
||||
storage.download_file(updated_corrections_gcs, updated_path)
|
||||
# Extract GCS path from URL
|
||||
corrections_gcs = _extract_gcs_path(corrections_url)
|
||||
|
||||
with open(updated_path, 'r', encoding='utf-8') as f:
|
||||
updated_data = json.load(f)
|
||||
# 2. Load the ORIGINAL corrections (has full structure)
|
||||
original_corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
original_corrections_path = os.path.join(temp_dir, "corrections_original.json")
|
||||
|
||||
# Merge: update the original with the user's corrections
|
||||
if 'corrections' in updated_data:
|
||||
original_data['corrections'] = updated_data['corrections']
|
||||
if 'corrected_segments' in updated_data:
|
||||
original_data['corrected_segments'] = updated_data['corrected_segments']
|
||||
job_log.info(f"Downloading original corrections from {original_corrections_gcs}")
|
||||
logger.info(f"Job {job_id}: Downloading original corrections from {original_corrections_gcs}")
|
||||
storage.download_file(original_corrections_gcs, original_corrections_path)
|
||||
|
||||
job_log.info("Merged user corrections into original data")
|
||||
logger.info(f"Job {job_id}: Merged user corrections into original data")
|
||||
|
||||
# 4. Convert to CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
job_log.info(f"Loaded CorrectionResult with {len(correction_result.corrected_segments)} segments")
|
||||
logger.info(f"Job {job_id}: Loaded CorrectionResult with {len(correction_result.corrected_segments)} segments")
|
||||
|
||||
# 3. Download audio file
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
audio_gcs_path = job.input_media_gcs_path
|
||||
|
||||
if not audio_gcs_path:
|
||||
raise FileNotFoundError(f"No input audio path for job {job_id}")
|
||||
|
||||
job_log.info(f"Downloading audio from {audio_gcs_path}")
|
||||
logger.info(f"Job {job_id}: Downloading audio from {audio_gcs_path}")
|
||||
storage.download_file(audio_gcs_path, audio_path)
|
||||
job_log.info(f"Audio downloaded: {os.path.getsize(audio_path)} bytes")
|
||||
|
||||
# 4. Get or create styles using the unified style loader
|
||||
job_log.info("Loading style configuration...")
|
||||
job_log.info(f" job.style_params_gcs_path: {job.style_params_gcs_path}")
|
||||
job_log.info(f" job.style_assets: {list(job.style_assets.keys()) if job.style_assets else 'None'}")
|
||||
|
||||
styles_path, style_data = load_styles_from_gcs(
|
||||
style_params_gcs_path=job.style_params_gcs_path,
|
||||
style_assets=job.style_assets,
|
||||
temp_dir=temp_dir,
|
||||
download_func=storage.download_file,
|
||||
logger=job_log,
|
||||
)
|
||||
|
||||
# 5. Configure OutputGenerator
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
job_log.info(f"Using styles from: {styles_path}")
|
||||
|
||||
# Get subtitle offset from job (user-specified timing adjustment)
|
||||
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
|
||||
if subtitle_offset != 0:
|
||||
job_log.info(f"Applying subtitle offset: {subtitle_offset}ms")
|
||||
|
||||
config = OutputConfig(
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
output_styles_json=styles_path,
|
||||
render_video=True,
|
||||
generate_cdg=False, # CDG optional, generated separately
|
||||
generate_plain_text=True,
|
||||
generate_lrc=True,
|
||||
video_resolution="4k",
|
||||
subtitle_offset_ms=subtitle_offset
|
||||
)
|
||||
|
||||
job_log.info(f"OutputConfig: output_styles_json={config.output_styles_json}, render_video={config.render_video}")
|
||||
|
||||
output_generator = OutputGenerator(config, logger)
|
||||
|
||||
# 6. Generate outputs (video, LRC, ASS, etc.)
|
||||
output_prefix = f"{job.artist or 'Unknown'} - {job.title or 'Unknown'}"
|
||||
job_log.info(f"Generating outputs with prefix '{output_prefix}'")
|
||||
logger.info(f"Job {job_id}: Generating outputs with prefix '{output_prefix}'")
|
||||
|
||||
outputs = output_generator.generate_outputs(
|
||||
transcription_corrected=correction_result,
|
||||
lyrics_results={}, # Reference lyrics already in correction_result
|
||||
output_prefix=output_prefix,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title
|
||||
)
|
||||
|
||||
# 7. Upload video to GCS
|
||||
if outputs.video and os.path.exists(outputs.video):
|
||||
video_size = os.path.getsize(outputs.video)
|
||||
job_log.info(f"Video generated: {video_size} bytes")
|
||||
video_gcs_path = f"jobs/{job_id}/videos/with_vocals.mkv"
|
||||
video_url = storage.upload_file(outputs.video, video_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'videos', 'with_vocals', video_url)
|
||||
job_log.info(f"Uploaded with_vocals.mkv to GCS")
|
||||
logger.info(f"Job {job_id}: Uploaded with_vocals.mkv ({video_size} bytes)")
|
||||
else:
|
||||
job_log.error("Video generation failed - no output file produced!")
|
||||
raise Exception("Video generation failed - no output file produced")
|
||||
|
||||
# 8. Upload LRC file
|
||||
if outputs.lrc and os.path.exists(outputs.lrc):
|
||||
lrc_gcs_path = f"jobs/{job_id}/lyrics/karaoke.lrc"
|
||||
lrc_url = storage.upload_file(outputs.lrc, lrc_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'lrc', lrc_url)
|
||||
job_log.info("Uploaded karaoke.lrc")
|
||||
logger.info(f"Job {job_id}: Uploaded karaoke.lrc")
|
||||
|
||||
# 9. Upload ASS subtitle file
|
||||
if outputs.ass and os.path.exists(outputs.ass):
|
||||
ass_gcs_path = f"jobs/{job_id}/lyrics/karaoke.ass"
|
||||
ass_url = storage.upload_file(outputs.ass, ass_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'ass', ass_url)
|
||||
job_log.info("Uploaded karaoke.ass")
|
||||
logger.info(f"Job {job_id}: Uploaded karaoke.ass")
|
||||
|
||||
# 10. Upload corrected text files
|
||||
if outputs.corrected_txt and os.path.exists(outputs.corrected_txt):
|
||||
txt_gcs_path = f"jobs/{job_id}/lyrics/corrected.txt"
|
||||
txt_url = storage.upload_file(outputs.corrected_txt, txt_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'corrected_txt', txt_url)
|
||||
job_log.info("Uploaded corrected.txt")
|
||||
logger.info(f"Job {job_id}: Uploaded corrected.txt")
|
||||
|
||||
# 11. Analyze backing vocals for intelligent instrumental selection
|
||||
job_log.info("Analyzing backing vocals for instrumental selection...")
|
||||
await _analyze_backing_vocals(job_id, job_manager, storage, job_log)
|
||||
|
||||
# 12. Transition based on prep_only flag
|
||||
if getattr(job, 'prep_only', False):
|
||||
# Prep-only mode: stop here and mark as prep complete
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.PREP_COMPLETE,
|
||||
progress=100,
|
||||
message="Prep phase complete - download outputs to continue locally"
|
||||
with open(original_corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
# 3. Check if there are updated corrections (from review UI)
|
||||
# The frontend sends only partial data: {corrections, corrected_segments}
|
||||
updated_corrections_gcs = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
|
||||
if storage.file_exists(updated_corrections_gcs):
|
||||
job_log.info("Found updated corrections from review, merging")
|
||||
logger.info(f"Job {job_id}: Found updated corrections, merging")
|
||||
updated_path = os.path.join(temp_dir, "corrections_updated.json")
|
||||
storage.download_file(updated_corrections_gcs, updated_path)
|
||||
|
||||
with open(updated_path, 'r', encoding='utf-8') as f:
|
||||
updated_data = json.load(f)
|
||||
|
||||
# Merge: update the original with the user's corrections
|
||||
if 'corrections' in updated_data:
|
||||
original_data['corrections'] = updated_data['corrections']
|
||||
if 'corrected_segments' in updated_data:
|
||||
original_data['corrected_segments'] = updated_data['corrected_segments']
|
||||
|
||||
job_log.info("Merged user corrections into original data")
|
||||
logger.info(f"Job {job_id}: Merged user corrections into original data")
|
||||
|
||||
# 4. Convert to CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
job_log.info(f"Loaded CorrectionResult with {len(correction_result.corrected_segments)} segments")
|
||||
logger.info(f"Job {job_id}: Loaded CorrectionResult with {len(correction_result.corrected_segments)} segments")
|
||||
|
||||
# 5. Download audio file
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
audio_gcs_path = job.input_media_gcs_path
|
||||
|
||||
if not audio_gcs_path:
|
||||
raise FileNotFoundError(f"No input audio path for job {job_id}")
|
||||
|
||||
job_log.info(f"Downloading audio from {audio_gcs_path}")
|
||||
logger.info(f"Job {job_id}: Downloading audio from {audio_gcs_path}")
|
||||
storage.download_file(audio_gcs_path, audio_path)
|
||||
job_log.info(f"Audio downloaded: {os.path.getsize(audio_path)} bytes")
|
||||
|
||||
# Process countdown intro if needed (for songs that start too quickly)
|
||||
# This adds "3... 2... 1..." segment and pads audio with 3s silence
|
||||
# Note: Countdown is deferred to this stage (not during lyrics transcription)
|
||||
# so the review UI can show the original timing without the 3s shift
|
||||
countdown_processor = CountdownProcessor(cache_dir=temp_dir, logger=logger)
|
||||
correction_result, audio_path, padding_added, padding_seconds = countdown_processor.process(
|
||||
correction_result=correction_result,
|
||||
audio_filepath=audio_path,
|
||||
)
|
||||
job_log.info("=== RENDER VIDEO WORKER COMPLETE (PREP ONLY) ===")
|
||||
logger.info(f"Job {job_id}: Prep-only video render complete")
|
||||
else:
|
||||
# Normal mode: proceed to instrumental selection
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
|
||||
progress=80,
|
||||
message="Video rendered - select your instrumental"
|
||||
if padding_added:
|
||||
job_log.info(f"Countdown added: {padding_seconds}s padding applied to audio and timestamps shifted")
|
||||
else:
|
||||
job_log.info("No countdown needed - song starts after 3 seconds")
|
||||
|
||||
# 6. Get or create styles using the unified style loader
|
||||
job_log.info("Loading style configuration...")
|
||||
job_log.info(f" job.style_params_gcs_path: {job.style_params_gcs_path}")
|
||||
job_log.info(f" job.style_assets: {list(job.style_assets.keys()) if job.style_assets else 'None'}")
|
||||
|
||||
styles_path, style_data = load_styles_from_gcs(
|
||||
style_params_gcs_path=job.style_params_gcs_path,
|
||||
style_assets=job.style_assets,
|
||||
temp_dir=temp_dir,
|
||||
download_func=storage.download_file,
|
||||
logger=job_log,
|
||||
)
|
||||
job_log.info("=== RENDER VIDEO WORKER COMPLETE ===")
|
||||
logger.info(f"Job {job_id}: Video render complete, awaiting instrumental selection")
|
||||
return True
|
||||
|
||||
# 7. Configure OutputGenerator
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
job_log.info(f"Using styles from: {styles_path}")
|
||||
|
||||
# Get subtitle offset from job (user-specified timing adjustment)
|
||||
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
|
||||
if subtitle_offset != 0:
|
||||
job_log.info(f"Applying subtitle offset: {subtitle_offset}ms")
|
||||
|
||||
config = OutputConfig(
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
output_styles_json=styles_path,
|
||||
render_video=True,
|
||||
generate_cdg=False, # CDG optional, generated separately
|
||||
generate_plain_text=True,
|
||||
generate_lrc=True,
|
||||
video_resolution="4k",
|
||||
subtitle_offset_ms=subtitle_offset
|
||||
)
|
||||
|
||||
job_log.info(f"OutputConfig: output_styles_json={config.output_styles_json}, render_video={config.render_video}")
|
||||
|
||||
output_generator = OutputGenerator(config, logger)
|
||||
|
||||
# 8. Generate outputs (video, LRC, ASS, etc.)
|
||||
output_prefix = f"{job.artist or 'Unknown'} - {job.title or 'Unknown'}"
|
||||
job_log.info(f"Generating outputs with prefix '{output_prefix}'")
|
||||
logger.info(f"Job {job_id}: Generating outputs with prefix '{output_prefix}'")
|
||||
|
||||
outputs = output_generator.generate_outputs(
|
||||
transcription_corrected=correction_result,
|
||||
lyrics_results={}, # Reference lyrics already in correction_result
|
||||
output_prefix=output_prefix,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title
|
||||
)
|
||||
|
||||
# 9. Upload video to GCS
|
||||
if outputs.video and os.path.exists(outputs.video):
|
||||
video_size = os.path.getsize(outputs.video)
|
||||
job_log.info(f"Video generated: {video_size} bytes")
|
||||
video_gcs_path = f"jobs/{job_id}/videos/with_vocals.mkv"
|
||||
video_url = storage.upload_file(outputs.video, video_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'videos', 'with_vocals', video_url)
|
||||
job_log.info(f"Uploaded with_vocals.mkv to GCS")
|
||||
logger.info(f"Job {job_id}: Uploaded with_vocals.mkv ({video_size} bytes)")
|
||||
else:
|
||||
job_log.error("Video generation failed - no output file produced!")
|
||||
raise Exception("Video generation failed - no output file produced")
|
||||
|
||||
# 10. Upload LRC file
|
||||
if outputs.lrc and os.path.exists(outputs.lrc):
|
||||
lrc_gcs_path = f"jobs/{job_id}/lyrics/karaoke.lrc"
|
||||
lrc_url = storage.upload_file(outputs.lrc, lrc_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'lrc', lrc_url)
|
||||
job_log.info("Uploaded karaoke.lrc")
|
||||
logger.info(f"Job {job_id}: Uploaded karaoke.lrc")
|
||||
|
||||
# 11. Upload ASS subtitle file
|
||||
if outputs.ass and os.path.exists(outputs.ass):
|
||||
ass_gcs_path = f"jobs/{job_id}/lyrics/karaoke.ass"
|
||||
ass_url = storage.upload_file(outputs.ass, ass_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'ass', ass_url)
|
||||
job_log.info("Uploaded karaoke.ass")
|
||||
logger.info(f"Job {job_id}: Uploaded karaoke.ass")
|
||||
|
||||
# 12. Upload corrected text files
|
||||
if outputs.corrected_txt and os.path.exists(outputs.corrected_txt):
|
||||
txt_gcs_path = f"jobs/{job_id}/lyrics/corrected.txt"
|
||||
txt_url = storage.upload_file(outputs.corrected_txt, txt_gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'corrected_txt', txt_url)
|
||||
job_log.info("Uploaded corrected.txt")
|
||||
logger.info(f"Job {job_id}: Uploaded corrected.txt")
|
||||
|
||||
# 13. Analyze backing vocals for intelligent instrumental selection
|
||||
job_log.info("Analyzing backing vocals for instrumental selection...")
|
||||
await _analyze_backing_vocals(job_id, job_manager, storage, job_log)
|
||||
|
||||
# 14. Transition based on prep_only flag and existing instrumental
|
||||
if getattr(job, 'prep_only', False):
|
||||
# Prep-only mode: stop here and mark as prep complete
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.PREP_COMPLETE,
|
||||
progress=100,
|
||||
message="Prep phase complete - download outputs to continue locally"
|
||||
)
|
||||
job_log.info("=== RENDER VIDEO WORKER COMPLETE (PREP ONLY) ===")
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
root_span.set_attribute("prep_only", True)
|
||||
logger.info(f"[job:{job_id}] WORKER_END worker=render-video status=success duration={duration:.1f}s prep_only=true")
|
||||
elif getattr(job, 'existing_instrumental_gcs_path', None):
|
||||
# Existing instrumental provided - skip selection, auto-use it
|
||||
# Store selection as 'custom' to indicate user-provided instrumental
|
||||
job_manager.update_state_data(job_id, 'instrumental_selection', 'custom')
|
||||
job_log.info("Existing instrumental provided - skipping selection, using user-provided file")
|
||||
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.INSTRUMENTAL_SELECTED,
|
||||
progress=82,
|
||||
message="Using user-provided instrumental"
|
||||
)
|
||||
job_log.info("=== RENDER VIDEO WORKER COMPLETE (EXISTING INSTRUMENTAL) ===")
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
root_span.set_attribute("existing_instrumental", True)
|
||||
logger.info(f"[job:{job_id}] WORKER_END worker=render-video status=success duration={duration:.1f}s existing_instrumental=true")
|
||||
|
||||
# Trigger video worker directly since no user selection needed
|
||||
from backend.workers.video_worker import run_video_worker
|
||||
job_log.info("Triggering video worker for final encoding...")
|
||||
await run_video_worker(job_id, job_manager, storage)
|
||||
else:
|
||||
# Normal mode: proceed to instrumental selection
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
|
||||
progress=80,
|
||||
message="Video rendered - select your instrumental"
|
||||
)
|
||||
job_log.info("=== RENDER VIDEO WORKER COMPLETE ===")
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
logger.info(f"[job:{job_id}] WORKER_END worker=render-video status=success duration={duration:.1f}s")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
job_log.error(f"Video render failed: {e}")
|
||||
logger.error(f"Job {job_id}: Video render failed: {e}", exc_info=True)
|
||||
logger.error(f"[job:{job_id}] WORKER_END worker=render-video status=error duration={duration:.1f}s error={e}")
|
||||
job_manager.fail_job(job_id, f"Video render failed: {str(e)}")
|
||||
return False
|
||||
|
||||
|
||||
+112
-102
@@ -16,17 +16,16 @@ processing complete (via mark_audio_complete/mark_lyrics_complete coordination).
|
||||
|
||||
Integrates with karaoke_gen.video_generator.VideoGenerator.
|
||||
|
||||
SOLID Principles:
|
||||
- Single Responsibility: Only generates title/end screens
|
||||
- Open/Closed: Extensible for different screen styles
|
||||
- Liskov Substitution: Can swap different video generators
|
||||
- Interface Segregation: Focused interface for screen generation
|
||||
- Dependency Inversion: Depends on abstractions (VideoGenerator interface)
|
||||
Observability:
|
||||
- All operations wrapped in tracing spans for Cloud Trace visibility
|
||||
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
|
||||
- Worker start/end timing logged with WORKER_START/WORKER_END markers
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
from typing import Optional, Dict, Any
|
||||
from pathlib import Path
|
||||
|
||||
@@ -36,6 +35,7 @@ from backend.services.storage_service import StorageService
|
||||
from backend.config import get_settings
|
||||
from backend.workers.style_helper import load_style_config, StyleConfig
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
from backend.services.tracing import job_span, add_span_event, add_span_attribute
|
||||
|
||||
# Import from karaoke_gen package
|
||||
from karaoke_gen.video_generator import VideoGenerator
|
||||
@@ -64,6 +64,7 @@ async def generate_screens(job_id: str) -> bool:
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
start_time = time.time()
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
settings = get_settings()
|
||||
@@ -71,118 +72,127 @@ async def generate_screens(job_id: str) -> bool:
|
||||
# Create job logger for remote debugging
|
||||
job_log = create_job_logger(job_id, "screens")
|
||||
|
||||
# Log with structured markers for easy Cloud Logging queries
|
||||
logger.info(f"[job:{job_id}] WORKER_START worker=screens")
|
||||
|
||||
# Set up log capture for VideoGenerator and style_helper
|
||||
log_handler = setup_job_logging(job_id, "screens", *SCREENS_WORKER_LOGGERS)
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
logger.error(f"[job:{job_id}] Job not found")
|
||||
return False
|
||||
|
||||
# Validate both audio and lyrics are complete
|
||||
if not _validate_prerequisites(job):
|
||||
logger.error(f"Job {job_id}: Prerequisites not met for screen generation")
|
||||
logger.error(f"[job:{job_id}] Prerequisites not met for screen generation")
|
||||
return False
|
||||
|
||||
# Create temporary working directory
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_screens_{job_id}_")
|
||||
|
||||
try:
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (karaoke_gen.video_generator) are only
|
||||
# captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting screen generation for {job.artist} - {job.title}")
|
||||
logger.info(f"Starting screen generation for job {job_id}")
|
||||
|
||||
# Transition to GENERATING_SCREENS state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.GENERATING_SCREENS,
|
||||
progress=50,
|
||||
message="Generating title and end screens"
|
||||
)
|
||||
|
||||
# Log style assets info
|
||||
style_assets = getattr(job, 'style_assets', {}) or {}
|
||||
job_log.info(f"Style assets from job: {list(style_assets.keys()) if style_assets else 'None'}")
|
||||
if style_assets:
|
||||
for key, path in style_assets.items():
|
||||
job_log.info(f" {key}: {path}")
|
||||
|
||||
# Load style configuration (downloads assets from GCS if available)
|
||||
job_log.info("Loading style configuration from GCS...")
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
if style_config.has_custom_styles():
|
||||
job_log.info("Using CUSTOM style configuration")
|
||||
logger.info(f"Job {job_id}: Using custom style configuration")
|
||||
else:
|
||||
job_log.warning("Using DEFAULT style configuration (no custom styles found)")
|
||||
logger.info(f"Job {job_id}: Using default style configuration")
|
||||
|
||||
# Initialize video generator
|
||||
video_generator = _create_video_generator(temp_dir)
|
||||
|
||||
# Generate title screen with style config
|
||||
job_log.info("Generating title screen...")
|
||||
title_screen_path = await _generate_title_screen(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
video_generator=video_generator,
|
||||
style_config=style_config,
|
||||
temp_dir=temp_dir,
|
||||
job_log=job_log
|
||||
)
|
||||
|
||||
if not title_screen_path:
|
||||
raise Exception("Title screen generation failed")
|
||||
job_log.info(f"Title screen generated: {title_screen_path}")
|
||||
|
||||
# Generate end screen with style config
|
||||
job_log.info("Generating end screen...")
|
||||
end_screen_path = await _generate_end_screen(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
video_generator=video_generator,
|
||||
style_config=style_config,
|
||||
temp_dir=temp_dir,
|
||||
job_log=job_log
|
||||
)
|
||||
|
||||
if not end_screen_path:
|
||||
raise Exception("End screen generation failed")
|
||||
job_log.info(f"End screen generated: {end_screen_path}")
|
||||
|
||||
# Upload screens to GCS
|
||||
await _upload_screens(
|
||||
job_id=job_id,
|
||||
job_manager=job_manager,
|
||||
storage=storage,
|
||||
title_screen_path=title_screen_path,
|
||||
end_screen_path=end_screen_path
|
||||
)
|
||||
|
||||
# Apply countdown padding if needed
|
||||
# (This is handled automatically by checking lyrics metadata)
|
||||
await _apply_countdown_padding_if_needed(job_id, job_manager, job)
|
||||
|
||||
# Transition to AWAITING_REVIEW
|
||||
# Human must review lyrics before video can be rendered
|
||||
logger.info(f"Job {job_id}: Screens generated, awaiting lyrics review")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_REVIEW,
|
||||
progress=55,
|
||||
message="Ready for lyrics review. Please review and correct lyrics."
|
||||
)
|
||||
|
||||
# TODO: Send notification to user that selection is needed
|
||||
# await notify_user(job_id, "instrumental_selection_ready")
|
||||
|
||||
return True
|
||||
# Wrap entire worker in a tracing span
|
||||
with job_span("screens-worker", job_id, {"artist": job.artist, "title": job.title}) as root_span:
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting screen generation for {job.artist} - {job.title}")
|
||||
logger.info(f"[job:{job_id}] Starting screen generation for {job.artist} - {job.title}")
|
||||
|
||||
# Transition to GENERATING_SCREENS state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.GENERATING_SCREENS,
|
||||
progress=50,
|
||||
message="Generating title and end screens"
|
||||
)
|
||||
|
||||
# Log style assets info
|
||||
style_assets = getattr(job, 'style_assets', {}) or {}
|
||||
job_log.info(f"Style assets from job: {list(style_assets.keys()) if style_assets else 'None'}")
|
||||
if style_assets:
|
||||
for key, path in style_assets.items():
|
||||
job_log.info(f" {key}: {path}")
|
||||
|
||||
# Load style configuration (downloads assets from GCS if available)
|
||||
with job_span("load-style-config", job_id):
|
||||
job_log.info("Loading style configuration from GCS...")
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
if style_config.has_custom_styles():
|
||||
job_log.info("Using CUSTOM style configuration")
|
||||
logger.info(f"[job:{job_id}] Using custom style configuration")
|
||||
add_span_attribute("style_type", "custom")
|
||||
else:
|
||||
job_log.warning("Using DEFAULT style configuration (no custom styles found)")
|
||||
logger.info(f"[job:{job_id}] Using default style configuration")
|
||||
add_span_attribute("style_type", "default")
|
||||
|
||||
# Initialize video generator
|
||||
video_generator = _create_video_generator(temp_dir)
|
||||
|
||||
# Generate title screen with style config
|
||||
with job_span("generate-title-screen", job_id):
|
||||
job_log.info("Generating title screen...")
|
||||
title_screen_path = await _generate_title_screen(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
video_generator=video_generator,
|
||||
style_config=style_config,
|
||||
temp_dir=temp_dir,
|
||||
job_log=job_log
|
||||
)
|
||||
|
||||
if not title_screen_path:
|
||||
raise Exception("Title screen generation failed")
|
||||
job_log.info(f"Title screen generated: {title_screen_path}")
|
||||
|
||||
# Generate end screen with style config
|
||||
with job_span("generate-end-screen", job_id):
|
||||
job_log.info("Generating end screen...")
|
||||
end_screen_path = await _generate_end_screen(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
video_generator=video_generator,
|
||||
style_config=style_config,
|
||||
temp_dir=temp_dir,
|
||||
job_log=job_log
|
||||
)
|
||||
|
||||
if not end_screen_path:
|
||||
raise Exception("End screen generation failed")
|
||||
job_log.info(f"End screen generated: {end_screen_path}")
|
||||
|
||||
# Upload screens to GCS
|
||||
with job_span("upload-screens", job_id):
|
||||
await _upload_screens(
|
||||
job_id=job_id,
|
||||
job_manager=job_manager,
|
||||
storage=storage,
|
||||
title_screen_path=title_screen_path,
|
||||
end_screen_path=end_screen_path
|
||||
)
|
||||
|
||||
# Apply countdown padding if needed
|
||||
await _apply_countdown_padding_if_needed(job_id, job_manager, job)
|
||||
|
||||
# Transition to AWAITING_REVIEW
|
||||
# Human must review lyrics before video can be rendered
|
||||
logger.info(f"[job:{job_id}] Screens generated, awaiting lyrics review")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_REVIEW,
|
||||
progress=55,
|
||||
message="Ready for lyrics review. Please review and correct lyrics."
|
||||
)
|
||||
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
logger.info(f"[job:{job_id}] WORKER_END worker=screens status=success duration={duration:.1f}s")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Screen generation failed: {e}", exc_info=True)
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[job:{job_id}] WORKER_END worker=screens status=error duration={duration:.1f}s error={e}")
|
||||
job_manager.mark_job_failed(
|
||||
job_id=job_id,
|
||||
error_message=f"Screen generation failed: {str(e)}",
|
||||
@@ -201,7 +211,7 @@ async def generate_screens(job_id: str) -> bool:
|
||||
# Cleanup temporary directory
|
||||
if os.path.exists(temp_dir):
|
||||
shutil.rmtree(temp_dir)
|
||||
logger.debug(f"Cleaned up temp directory: {temp_dir}")
|
||||
logger.debug(f"[job:{job_id}] Cleaned up temp directory: {temp_dir}")
|
||||
|
||||
|
||||
def _validate_prerequisites(job) -> bool:
|
||||
|
||||
@@ -15,11 +15,17 @@ The worker's job is simply to:
|
||||
4. Upload results back to GCS
|
||||
|
||||
This reuses 100% of the existing karaoke_gen functionality.
|
||||
|
||||
Observability:
|
||||
- All operations wrapped in tracing spans for Cloud Trace visibility
|
||||
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
|
||||
- Worker start/end timing logged with WORKER_START/WORKER_END markers
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
import json
|
||||
from typing import Optional, Dict, Any
|
||||
from pathlib import Path
|
||||
@@ -32,6 +38,7 @@ from backend.services.youtube_service import get_youtube_service
|
||||
from backend.config import get_settings
|
||||
from backend.workers.style_helper import load_style_config
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
from backend.services.tracing import job_span, add_span_event, add_span_attribute
|
||||
|
||||
# Import from karaoke_gen package - reuse existing implementation
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
@@ -61,6 +68,7 @@ async def generate_video(job_id: str) -> bool:
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
start_time = time.time()
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
settings = get_settings()
|
||||
@@ -68,17 +76,20 @@ async def generate_video(job_id: str) -> bool:
|
||||
# Create job logger for remote debugging
|
||||
job_log = create_job_logger(job_id, "video")
|
||||
|
||||
# Log with structured markers for easy Cloud Logging queries
|
||||
logger.info(f"[job:{job_id}] WORKER_START worker=video")
|
||||
|
||||
# Set up log capture for KaraokeFinalise
|
||||
log_handler = setup_job_logging(job_id, "video", *VIDEO_WORKER_LOGGERS)
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
logger.error(f"[job:{job_id}] Job not found")
|
||||
return False
|
||||
|
||||
# Validate prerequisites
|
||||
if not _validate_prerequisites(job):
|
||||
logger.error(f"Job {job_id}: Prerequisites not met for video generation")
|
||||
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
|
||||
return False
|
||||
|
||||
# Create temporary working directory
|
||||
@@ -105,12 +116,14 @@ async def generate_video(job_id: str) -> bool:
|
||||
job_log.warning("YouTube credentials not available - upload will be skipped")
|
||||
|
||||
try:
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (karaoke_gen.karaoke_finalise) are only
|
||||
# captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting video finalization for {job.artist} - {job.title}")
|
||||
logger.info(f"Starting video generation for job {job_id}")
|
||||
# Wrap entire worker in a tracing span
|
||||
with job_span("video-worker", job_id, {"artist": job.artist, "title": job.title}) as root_span:
|
||||
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
|
||||
# This ensures logs from third-party libraries (karaoke_gen.karaoke_finalise) are only
|
||||
# captured by this job's handler, not handlers from other concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting video finalization for {job.artist} - {job.title}")
|
||||
logger.info(f"[job:{job_id}] Starting video generation for {job.artist} - {job.title}")
|
||||
|
||||
# Transition to GENERATING_VIDEO state
|
||||
job_manager.transition_to_state(
|
||||
@@ -122,9 +135,10 @@ async def generate_video(job_id: str) -> bool:
|
||||
|
||||
# Download and set up files in the format KaraokeFinalise expects
|
||||
base_name = f"{job.artist} - {job.title}"
|
||||
job_log.info("Downloading files from GCS (this may take a few minutes for large files)...")
|
||||
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
|
||||
job_log.info("All files downloaded successfully")
|
||||
with job_span("download-files", job_id):
|
||||
job_log.info("Downloading files from GCS (this may take a few minutes for large files)...")
|
||||
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
|
||||
job_log.info("All files downloaded successfully")
|
||||
|
||||
# Load style config for CDG styles
|
||||
job_log.info("Loading CDG style configuration...")
|
||||
@@ -162,6 +176,14 @@ async def generate_video(job_id: str) -> bool:
|
||||
message="Encoding videos (15-20 min)"
|
||||
)
|
||||
|
||||
# Get countdown padding info from lyrics metadata
|
||||
# This ensures instrumental is padded to match vocals if countdown was added
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
job_log.info(f"Countdown padding detected: {countdown_padding_seconds}s - instrumental will be padded if needed")
|
||||
|
||||
# Log finalization parameters
|
||||
job_log.info("Creating KaraokeFinalise with parameters:")
|
||||
job_log.info(f" enable_cdg: {getattr(job, 'enable_cdg', False)}")
|
||||
@@ -169,6 +191,7 @@ async def generate_video(job_id: str) -> bool:
|
||||
job_log.info(f" brand_prefix: {getattr(job, 'brand_prefix', None)}")
|
||||
job_log.info(f" discord_webhook: {'configured' if getattr(job, 'discord_webhook_url', None) else 'not configured'}")
|
||||
job_log.info(f" instrumental source: {instrumental_source}")
|
||||
job_log.info(f" countdown_padding_seconds: {countdown_padding_seconds}")
|
||||
if existing_instrumental_path:
|
||||
job_log.info(f" using user-provided instrumental (selection was: {instrumental_selection})")
|
||||
|
||||
@@ -209,6 +232,8 @@ async def generate_video(job_id: str) -> bool:
|
||||
non_interactive=True,
|
||||
server_side_mode=True,
|
||||
selected_instrumental_file=instrumental_file,
|
||||
# Audio synchronization - ensure instrumental matches vocal padding
|
||||
countdown_padding_seconds=countdown_padding_seconds,
|
||||
)
|
||||
|
||||
# Call process() - this does ALL the work:
|
||||
@@ -216,25 +241,34 @@ async def generate_video(job_id: str) -> bool:
|
||||
# - Generates CDG/TXT packages if enabled
|
||||
# - Posts Discord notification if configured
|
||||
# - Handles brand code generation
|
||||
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
|
||||
logger.info(f"Job {job_id}: Starting KaraokeFinalise.process()")
|
||||
result = finalise.process()
|
||||
with job_span("karaoke-finalise", job_id) as finalise_span:
|
||||
finalise_start = time.time()
|
||||
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
|
||||
logger.info(f"[job:{job_id}] Starting KaraokeFinalise.process()")
|
||||
add_span_event("finalise_started")
|
||||
|
||||
result = finalise.process()
|
||||
|
||||
finalise_duration = time.time() - finalise_start
|
||||
finalise_span.set_attribute("duration_seconds", finalise_duration)
|
||||
add_span_event("finalise_completed", {"duration_seconds": finalise_duration})
|
||||
|
||||
job_log.info("KaraokeFinalise.process() complete!")
|
||||
if result.get('brand_code'):
|
||||
job_log.info(f"Brand code: {result.get('brand_code')}")
|
||||
logger.info(f"Job {job_id}: KaraokeFinalise.process() complete")
|
||||
logger.info(f"Job {job_id}: Brand code: {result.get('brand_code')}")
|
||||
logger.info(f"[job:{job_id}] KaraokeFinalise.process() complete in {finalise_duration:.1f}s")
|
||||
logger.info(f"[job:{job_id}] Brand code: {result.get('brand_code')}")
|
||||
|
||||
# Native API distribution uploads (used by remote CLI instead of rclone)
|
||||
await _handle_native_distribution(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
job_log=job_log,
|
||||
job_manager=job_manager,
|
||||
temp_dir=temp_dir,
|
||||
result=result,
|
||||
)
|
||||
with job_span("distribution", job_id):
|
||||
await _handle_native_distribution(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
job_log=job_log,
|
||||
job_manager=job_manager,
|
||||
temp_dir=temp_dir,
|
||||
result=result,
|
||||
)
|
||||
|
||||
# Upload generated files to GCS
|
||||
job_manager.transition_to_state(
|
||||
@@ -244,10 +278,11 @@ async def generate_video(job_id: str) -> bool:
|
||||
message="Uploading final files"
|
||||
)
|
||||
|
||||
await _upload_results(job_id, job_manager, storage, temp_dir, result)
|
||||
with job_span("upload-results", job_id):
|
||||
await _upload_results(job_id, job_manager, storage, temp_dir, result)
|
||||
|
||||
# Mark job as complete
|
||||
logger.info(f"Job {job_id}: Video generation complete")
|
||||
logger.info(f"[job:{job_id}] Video generation complete")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.COMPLETE,
|
||||
@@ -264,10 +299,15 @@ async def generate_video(job_id: str) -> bool:
|
||||
}
|
||||
})
|
||||
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
root_span.set_attribute("brand_code", result.get('brand_code', ''))
|
||||
logger.info(f"[job:{job_id}] WORKER_END worker=video status=success duration={duration:.1f}s")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Video generation failed: {e}", exc_info=True)
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[job:{job_id}] WORKER_END worker=video status=error duration={duration:.1f}s error={e}")
|
||||
job_manager.mark_job_failed(
|
||||
job_id=job_id,
|
||||
error_message=f"Video generation failed: {str(e)}",
|
||||
@@ -438,7 +478,8 @@ def _validate_prerequisites(job) -> bool:
|
||||
logger.error(f"Job {job.job_id}: No instrumental selected")
|
||||
return False
|
||||
|
||||
if instrumental_selection not in ['clean', 'with_backing']:
|
||||
# 'custom' is valid when user provided existing_instrumental
|
||||
if instrumental_selection not in ['clean', 'with_backing', 'custom']:
|
||||
logger.error(f"Job {job.job_id}: Invalid instrumental selection: {instrumental_selection}")
|
||||
return False
|
||||
|
||||
@@ -454,7 +495,14 @@ def _validate_prerequisites(job) -> bool:
|
||||
logger.error(f"Job {job.job_id}: Missing lyrics video")
|
||||
return False
|
||||
|
||||
# Check instrumental exists
|
||||
# Check instrumental exists - for 'custom', check existing_instrumental_gcs_path instead of stems
|
||||
if instrumental_selection == 'custom':
|
||||
existing_instrumental_path = getattr(job, 'existing_instrumental_gcs_path', None)
|
||||
if not existing_instrumental_path:
|
||||
logger.error(f"Job {job.job_id}: Custom instrumental selected but no existing_instrumental_gcs_path")
|
||||
return False
|
||||
return True # Other stem checks not needed for custom instrumental
|
||||
|
||||
stems = job.file_urls.get('stems', {})
|
||||
instrumental_key = 'instrumental_clean' if instrumental_selection == 'clean' else 'instrumental_with_backing'
|
||||
if not stems.get(instrumental_key):
|
||||
|
||||
@@ -1,22 +1,23 @@
|
||||
# Cloud Build configuration for base image
|
||||
#
|
||||
# This builds the base image with all Python dependencies pre-installed.
|
||||
# Run this manually when poetry.lock changes:
|
||||
# Run this manually when poetry.lock OR Dockerfile.base changes:
|
||||
# gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
|
||||
#
|
||||
# The base image is used by the main Dockerfile for fast app builds.
|
||||
# The poetry.lock hash is stored as a label for change detection.
|
||||
# A combined hash of poetry.lock + Dockerfile.base is stored as a label for change detection.
|
||||
|
||||
steps:
|
||||
# Calculate poetry.lock hash for change detection
|
||||
# Calculate combined hash for change detection (poetry.lock + Dockerfile.base)
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
entrypoint: 'bash'
|
||||
args:
|
||||
- '-c'
|
||||
- |
|
||||
POETRY_LOCK_HASH=$$(sha256sum poetry.lock | cut -d' ' -f1)
|
||||
echo "Poetry.lock hash: $$POETRY_LOCK_HASH"
|
||||
echo "$$POETRY_LOCK_HASH" > /workspace/POETRY_LOCK_HASH
|
||||
# Hash both files to detect changes in either
|
||||
CONTENT_HASH=$$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
|
||||
echo "Combined content hash (poetry.lock + Dockerfile.base): $$CONTENT_HASH"
|
||||
echo "$$CONTENT_HASH" > /workspace/CONTENT_HASH
|
||||
|
||||
# Build base image with all dependencies
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
@@ -24,11 +25,11 @@ steps:
|
||||
args:
|
||||
- '-c'
|
||||
- |
|
||||
POETRY_LOCK_HASH=$$(cat /workspace/POETRY_LOCK_HASH)
|
||||
CONTENT_HASH=$$(cat /workspace/CONTENT_HASH)
|
||||
docker build \
|
||||
--build-arg BUILD_DATE=$BUILD_ID \
|
||||
--build-arg BUILD_ID=$BUILD_ID \
|
||||
--build-arg POETRY_LOCK_HASH=$$POETRY_LOCK_HASH \
|
||||
--build-arg BASE_CONTENT_HASH=$$CONTENT_HASH \
|
||||
-t us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend-base:latest \
|
||||
-t us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend-base:$BUILD_ID \
|
||||
-f backend/Dockerfile.base \
|
||||
|
||||
+1
-1
@@ -76,7 +76,7 @@ steps:
|
||||
--max-instances 50 \
|
||||
--min-instances 0 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest"
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
|
||||
|
||||
images:
|
||||
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
|
||||
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
# Codecov Configuration
|
||||
# Docs: https://docs.codecov.com/docs/codecovyml-reference
|
||||
|
||||
coverage:
|
||||
# Coverage value precision (2 decimal places)
|
||||
precision: 2
|
||||
round: down
|
||||
# Green range starts at 70% (our target)
|
||||
range: "70...100"
|
||||
|
||||
status:
|
||||
# Project-level coverage status check
|
||||
project:
|
||||
default:
|
||||
target: 70% # Pass if coverage >= 70%
|
||||
threshold: 2% # Allow 2% drop without failing
|
||||
if_ci_failed: error
|
||||
|
||||
# Patch coverage (new/changed lines only)
|
||||
patch:
|
||||
default:
|
||||
target: 70% # New code should also meet 70%
|
||||
threshold: 5% # More lenient for patches
|
||||
only_pulls: true # Only run on PRs, not on main branch pushes
|
||||
|
||||
# Disable PR comments entirely since we're satisfied with 70%+ coverage
|
||||
# Status checks will still appear on the PR
|
||||
comment: false
|
||||
|
||||
# Alternative: Uncomment below to only comment when coverage changes
|
||||
# comment:
|
||||
# layout: "diff, flags, files"
|
||||
# behavior: default
|
||||
# require_changes: true # Only comment if coverage actually changes
|
||||
# require_base: true # Require base report to exist
|
||||
|
||||
@@ -249,7 +249,7 @@ This section provides a comprehensive comparison of all CLI parameters between `
|
||||
| # | Parameter | Local | Remote | Description |
|
||||
|---|-----------|-------|--------|-------------|
|
||||
| R1 | `--service-url` | N/A | 🔹 | **Backend service URL.** URL of the karaoke-gen cloud backend. Can also be set via `KARAOKE_GEN_URL` environment variable. Required for remote mode. |
|
||||
| R2 | `--review-ui-url` | N/A | 🔹 | **Lyrics review UI URL.** URL of the hosted lyrics review web application. Default: `https://lyrics.nomadkaraoke.com`. |
|
||||
| R2 | `--review-ui-url` | N/A | 🔹 | **Lyrics review UI URL.** URL of the hosted lyrics review web application. Default: `https://gen.nomadkaraoke.com/lyrics/`. |
|
||||
| R3 | `--poll-interval` | N/A | 🔹 | **Status poll frequency.** Seconds between job status checks while monitoring. Default: 5 seconds. |
|
||||
| R4 | `--environment` | N/A | 🔹 | **Job environment tag.** Tag jobs with environment label (test/production/development) for filtering and cleanup. Sent as `X-Environment` header. |
|
||||
| R5 | `--client-id` | N/A | 🔹 | **Client identifier tag.** Tag jobs with client/user identifier for filtering. Sent as `X-Client-ID` header. |
|
||||
@@ -681,6 +681,10 @@ The karaoke-gen system supports multiple interfaces to the same core functionali
|
||||
| `dropbox-oauth-credentials` | ✅ | Dropbox API OAuth tokens |
|
||||
| `gdrive-service-account` | ✅ | Google Drive service account |
|
||||
| `discord-webhook-url` | ✅ | Default Discord webhook |
|
||||
| `red-api-key` | ⏳ | RED tracker API key (flacfetch audio search) |
|
||||
| `red-api-url` | ⏳ | RED tracker base URL (flacfetch audio search) |
|
||||
| `ops-api-key` | ⏳ | OPS tracker API key (flacfetch audio search) |
|
||||
| `ops-api-url` | ⏳ | OPS tracker base URL (flacfetch audio search) |
|
||||
|
||||
### Default Distribution Settings
|
||||
|
||||
|
||||
@@ -0,0 +1,608 @@
|
||||
# Observability & Tracing Plan for Karaoke Generator
|
||||
|
||||
**Created:** 2025-12-22
|
||||
**Status:** Planning
|
||||
**Priority:** High
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The karaoke-gen backend has **partial observability infrastructure** but **no rich tracing implemented**. While the code includes OpenTelemetry setup and tracing utilities, actual job processing is minimally instrumented. This means debugging job failures requires manually querying Cloud Logging with grep-style searches rather than clicking through a trace waterfall.
|
||||
|
||||
### Current State: 🟡 Partial
|
||||
|
||||
| Capability | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| **Logs exported to Cloud Logging** | ✅ Working | Standard Python logs visible |
|
||||
| **OpenTelemetry SDK installed** | ✅ Working | Dependencies in pyproject.toml |
|
||||
| **Cloud Trace exporter configured** | 🟡 Broken | 403 permission errors (fixed in infra today) |
|
||||
| **FastAPI auto-instrumentation** | ✅ Working | HTTP requests create spans |
|
||||
| **HTTPX instrumentation** | ✅ Working | Outgoing HTTP calls traced |
|
||||
| **Job ID correlation in logs** | ❌ Missing | Can't filter logs by job ID |
|
||||
| **Worker tracing spans** | ❌ Missing | No spans in audio/lyrics workers |
|
||||
| **Custom metrics** | ❌ Missing | No Cloud Monitoring metrics |
|
||||
| **Dashboard** | ❌ Missing | No pre-built observability view |
|
||||
| **Alerting** | ❌ Missing | No alerts for job failures |
|
||||
|
||||
### What You Can Do Today
|
||||
|
||||
```bash
|
||||
# Query Cloud Logging for a specific job (manual grep)
|
||||
gcloud logging read 'resource.type="cloud_run_revision" AND textPayload:"JOB_ID"' \
|
||||
--project=nomadkaraoke --limit=100
|
||||
|
||||
# View traces (once permission is fixed)
|
||||
# Go to: https://console.cloud.google.com/traces/list?project=nomadkaraoke
|
||||
```
|
||||
|
||||
### What You Should Be Able To Do
|
||||
|
||||
1. Go to a dashboard, enter job ID → see entire job lifecycle
|
||||
2. Click trace → see waterfall of audio worker, lyrics worker, Cloud Tasks
|
||||
3. See metrics: jobs/hour, success rate, p95 latency by stage
|
||||
4. Get alerted when job failure rate exceeds threshold
|
||||
|
||||
---
|
||||
|
||||
## Current Implementation Analysis
|
||||
|
||||
### 1. Tracing Infrastructure (Exists, Partially Working)
|
||||
|
||||
**File: `backend/services/tracing.py`**
|
||||
|
||||
```python
|
||||
# Good: SDK is set up, exporter is configured
|
||||
setup_tracing(service_name="karaoke-backend", service_version=VERSION)
|
||||
|
||||
# Good: Utility functions exist
|
||||
@traced("my-function") # Decorator for automatic spans
|
||||
with create_span("operation", {"job_id": job_id}): # Context manager
|
||||
add_span_attribute("key", "value") # Add to current span
|
||||
```
|
||||
|
||||
**Problem:** These utilities are barely used! Only `backend/api/routes/review.py` actually creates spans for business logic. The critical workers (`audio_worker.py`, `lyrics_worker.py`, etc.) have zero tracing.
|
||||
|
||||
### 2. Logging Infrastructure (Exists, Not Correlated)
|
||||
|
||||
**File: `backend/workers/worker_logging.py`**
|
||||
|
||||
```python
|
||||
# Good: Logs go to Firestore for CLI streaming
|
||||
class JobLogHandler(logging.Handler):
|
||||
"""Forwards logs to Firestore job document"""
|
||||
|
||||
# Good: Context vars prevent log mixing in concurrent jobs
|
||||
with job_logging_context(job_id):
|
||||
# logs isolated per job
|
||||
```
|
||||
|
||||
**Problem:** Logs go to Firestore for CLI display, but Cloud Logging has no job_id label. You can't filter Cloud Logging by job ID.
|
||||
|
||||
### 3. Permission Issues (Fixed Today)
|
||||
|
||||
The Cloud Trace exporter was failing with 403 errors because the Cloud Run service account lacked `roles/cloudtrace.agent`. This was just fixed in `infrastructure/__main__.py`:
|
||||
|
||||
```python
|
||||
# NEW: Added today
|
||||
cloud_trace_iam = gcp.projects.IAMMember(
|
||||
"karaoke-backend-cloudtrace-agent",
|
||||
role="roles/cloudtrace.agent",
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis: What's Missing
|
||||
|
||||
### Gap 1: No Trace Context Propagation to Workers
|
||||
|
||||
When Cloud Tasks triggers a worker (audio, lyrics), a new trace is started. The parent trace from the initial API call is lost.
|
||||
|
||||
```
|
||||
Current:
|
||||
POST /uploads-complete → Trace A
|
||||
↓ (Cloud Task)
|
||||
POST /internal/workers/audio → NEW Trace B (no link to A!)
|
||||
```
|
||||
|
||||
**Should be:**
|
||||
```
|
||||
POST /uploads-complete → Trace A
|
||||
↓ (Cloud Task - propagates trace context)
|
||||
POST /internal/workers/audio → Trace A, Span B (child of A)
|
||||
```
|
||||
|
||||
### Gap 2: No Job ID on Traces
|
||||
|
||||
Even when traces exist, they don't have `job_id` as a searchable attribute. You can't search Cloud Trace for "all spans for job 8f03f0ac".
|
||||
|
||||
### Gap 3: No Custom Spans in Workers
|
||||
|
||||
The workers just log text. There are no spans for:
|
||||
- Audio download duration
|
||||
- Modal API call duration
|
||||
- GCS upload duration
|
||||
- Lyrics transcription duration
|
||||
- Each processing stage
|
||||
|
||||
### Gap 4: No Structured Logging with Job Context
|
||||
|
||||
Cloud Logging receives unstructured text logs. Should be structured JSON with `job_id`, `worker`, `stage` fields for filtering.
|
||||
|
||||
### Gap 5: No Metrics
|
||||
|
||||
No custom Cloud Monitoring metrics for:
|
||||
- Jobs started/completed/failed per hour
|
||||
- Processing time by stage (p50, p95, p99)
|
||||
- Queue depth
|
||||
- Error rates
|
||||
|
||||
### Gap 6: No Dashboard
|
||||
|
||||
No Cloud Console dashboard showing:
|
||||
- Active jobs
|
||||
- Success/failure rates
|
||||
- Latency distribution
|
||||
- Worker health
|
||||
|
||||
### Gap 7: No Alerting
|
||||
|
||||
No alerts for:
|
||||
- Job stuck in status > X minutes
|
||||
- Error rate spike
|
||||
- Worker health degradation
|
||||
|
||||
---
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Fix Core Tracing (2-3 hours) ⭐ High Priority
|
||||
|
||||
**Goal:** Every job should have a single trace that spans its entire lifecycle.
|
||||
|
||||
#### 1.1 Propagate Trace Context Through Cloud Tasks
|
||||
|
||||
**File: `backend/services/worker_service.py`**
|
||||
|
||||
```python
|
||||
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
|
||||
from opentelemetry import trace
|
||||
|
||||
def _enqueue_cloud_task(self, queue: str, endpoint: str, payload: Dict, ...):
|
||||
# Get current trace context
|
||||
propagator = TraceContextTextMapPropagator()
|
||||
carrier = {}
|
||||
propagator.inject(carrier) # Injects traceparent header
|
||||
|
||||
# Add trace context to task headers
|
||||
headers = {
|
||||
"Authorization": f"Bearer {admin_token}",
|
||||
"Content-Type": "application/json",
|
||||
**carrier, # Add traceparent, tracestate
|
||||
}
|
||||
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": "POST",
|
||||
"url": url,
|
||||
"headers": headers, # Now includes trace context
|
||||
"body": base64.b64encode(json.dumps(payload).encode()).decode(),
|
||||
},
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
**File: `backend/api/routes/internal.py`**
|
||||
|
||||
```python
|
||||
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
|
||||
from opentelemetry import trace
|
||||
|
||||
@router.post("/workers/audio")
|
||||
async def trigger_audio_worker(request: Request, ...):
|
||||
# Extract trace context from headers
|
||||
propagator = TraceContextTextMapPropagator()
|
||||
context = propagator.extract(request.headers)
|
||||
|
||||
# Create span as child of propagated context
|
||||
with tracer.start_as_current_span(
|
||||
"audio-worker",
|
||||
context=context,
|
||||
attributes={"job_id": payload.job_id}
|
||||
):
|
||||
await process_audio_separation(payload.job_id)
|
||||
```
|
||||
|
||||
#### 1.2 Add Job ID to All Spans
|
||||
|
||||
Create a helper that adds job_id as a span attribute:
|
||||
|
||||
```python
|
||||
# backend/services/tracing.py
|
||||
|
||||
def job_span(name: str, job_id: str, **extra_attributes):
|
||||
"""Create a span with job_id attribute for easy filtering."""
|
||||
return create_span(name, {
|
||||
"job_id": job_id,
|
||||
"service.operation": name,
|
||||
**extra_attributes
|
||||
})
|
||||
```
|
||||
|
||||
Usage in workers:
|
||||
|
||||
```python
|
||||
# backend/workers/audio_worker.py
|
||||
|
||||
async def process_audio_separation(job_id: str):
|
||||
with job_span("audio-worker", job_id) as span:
|
||||
span.set_attribute("worker", "audio")
|
||||
|
||||
with job_span("download-audio", job_id):
|
||||
# download logic
|
||||
pass
|
||||
|
||||
with job_span("modal-separation", job_id):
|
||||
# Modal API call
|
||||
pass
|
||||
|
||||
with job_span("upload-stems", job_id):
|
||||
# GCS upload
|
||||
pass
|
||||
```
|
||||
|
||||
### Phase 2: Instrument All Workers (3-4 hours) ⭐ High Priority
|
||||
|
||||
Add spans to every worker following this pattern:
|
||||
|
||||
#### Audio Worker (`audio_worker.py`)
|
||||
|
||||
```python
|
||||
async def process_audio_separation(job_id: str):
|
||||
with job_span("audio-worker", job_id) as root:
|
||||
# Stage 1: Download/Get Audio
|
||||
with job_span("get-audio-file", job_id) as span:
|
||||
if job.audio_url:
|
||||
span.set_attribute("source", "url")
|
||||
with job_span("download-from-url", job_id):
|
||||
audio_path = await download_from_url(...)
|
||||
else:
|
||||
span.set_attribute("source", "gcs")
|
||||
with job_span("download-from-gcs", job_id):
|
||||
audio_path = storage.download_file(...)
|
||||
|
||||
# Stage 2: Audio Separation
|
||||
with job_span("audio-separation", job_id) as span:
|
||||
with job_span("clean-instrumental", job_id):
|
||||
# First separation pass
|
||||
pass
|
||||
with job_span("backing-vocals", job_id):
|
||||
# Second separation pass
|
||||
pass
|
||||
|
||||
# Stage 3: Upload Results
|
||||
with job_span("upload-stems", job_id) as span:
|
||||
span.set_attribute("stem_count", len(stems))
|
||||
for stem_name, stem_path in stems.items():
|
||||
with job_span("upload-stem", job_id, stem_name=stem_name):
|
||||
storage.upload_file(...)
|
||||
```
|
||||
|
||||
#### Lyrics Worker (`lyrics_worker.py`)
|
||||
|
||||
```python
|
||||
async def process_lyrics_transcription(job_id: str):
|
||||
with job_span("lyrics-worker", job_id) as root:
|
||||
with job_span("get-vocals", job_id):
|
||||
# Download vocals from GCS
|
||||
pass
|
||||
|
||||
with job_span("audioshake-transcription", job_id):
|
||||
# AudioShake API call
|
||||
pass
|
||||
|
||||
with job_span("fetch-genius-lyrics", job_id) as span:
|
||||
span.set_attribute("artist", artist)
|
||||
span.set_attribute("title", title)
|
||||
# Genius API call
|
||||
pass
|
||||
|
||||
with job_span("auto-correction", job_id):
|
||||
# LyricsTranscriber correction
|
||||
pass
|
||||
|
||||
with job_span("upload-corrections", job_id):
|
||||
# Upload corrections.json
|
||||
pass
|
||||
```
|
||||
|
||||
### Phase 3: Structured Logging with Job Context (1-2 hours)
|
||||
|
||||
Replace text logs with structured JSON that includes job context.
|
||||
|
||||
**File: `backend/main.py`**
|
||||
|
||||
```python
|
||||
import json
|
||||
import logging
|
||||
|
||||
class StructuredFormatter(logging.Formatter):
|
||||
"""JSON formatter with trace correlation."""
|
||||
|
||||
def format(self, record):
|
||||
from backend.services.tracing import get_current_trace_id, get_current_span_id
|
||||
|
||||
log_entry = {
|
||||
"timestamp": self.formatTime(record),
|
||||
"severity": record.levelname,
|
||||
"message": record.getMessage(),
|
||||
"logger": record.name,
|
||||
# Trace correlation
|
||||
"logging.googleapis.com/trace": f"projects/nomadkaraoke/traces/{get_current_trace_id()}" if get_current_trace_id() else None,
|
||||
"logging.googleapis.com/spanId": get_current_span_id(),
|
||||
# Custom fields (if available in log record)
|
||||
"job_id": getattr(record, 'job_id', None),
|
||||
"worker": getattr(record, 'worker', None),
|
||||
}
|
||||
return json.dumps({k: v for k, v in log_entry.items() if v is not None})
|
||||
|
||||
# Configure in Cloud Run
|
||||
if os.environ.get("K_SERVICE"):
|
||||
handler = logging.StreamHandler()
|
||||
handler.setFormatter(StructuredFormatter())
|
||||
logging.root.handlers = [handler]
|
||||
```
|
||||
|
||||
Now logs in Cloud Logging can be filtered:
|
||||
```
|
||||
jsonPayload.job_id="8f03f0ac"
|
||||
```
|
||||
|
||||
### Phase 4: Custom Metrics (2-3 hours) 🟡 Medium Priority
|
||||
|
||||
Add Cloud Monitoring metrics for key indicators.
|
||||
|
||||
**File: `backend/services/metrics.py`**
|
||||
|
||||
```python
|
||||
from opentelemetry import metrics
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.exporter.cloud_monitoring import CloudMonitoringMetricsExporter
|
||||
|
||||
def setup_metrics():
|
||||
exporter = CloudMonitoringMetricsExporter(project_id="nomadkaraoke")
|
||||
provider = MeterProvider(metric_readers=[...])
|
||||
metrics.set_meter_provider(provider)
|
||||
|
||||
meter = metrics.get_meter("karaoke-backend")
|
||||
|
||||
# Counter: Jobs by status
|
||||
jobs_counter = meter.create_counter(
|
||||
"jobs_total",
|
||||
description="Total jobs by status",
|
||||
unit="1"
|
||||
)
|
||||
|
||||
# Histogram: Processing duration by stage
|
||||
duration_histogram = meter.create_histogram(
|
||||
"job_stage_duration_seconds",
|
||||
description="Duration of job processing stages",
|
||||
unit="s"
|
||||
)
|
||||
|
||||
return {
|
||||
"jobs": jobs_counter,
|
||||
"duration": duration_histogram,
|
||||
}
|
||||
|
||||
# Usage
|
||||
metrics["jobs"].add(1, {"status": "completed", "worker": "audio"})
|
||||
metrics["duration"].record(45.2, {"stage": "audio-separation", "job_id": job_id})
|
||||
```
|
||||
|
||||
**Metrics to track:**
|
||||
|
||||
| Metric | Type | Labels |
|
||||
|--------|------|--------|
|
||||
| `jobs_total` | Counter | status, source (upload/url/search) |
|
||||
| `job_stage_duration_seconds` | Histogram | stage, worker |
|
||||
| `worker_invocations_total` | Counter | worker, success/failure |
|
||||
| `gcs_operations_total` | Counter | operation (upload/download), bucket |
|
||||
| `external_api_duration_seconds` | Histogram | api (modal/audioshake/genius) |
|
||||
|
||||
### Phase 5: Dashboard & Alerting (2-3 hours) 🟡 Medium Priority
|
||||
|
||||
#### Create Cloud Monitoring Dashboard
|
||||
|
||||
**File: `infrastructure/monitoring/dashboard.json`**
|
||||
|
||||
```json
|
||||
{
|
||||
"displayName": "Karaoke Backend Overview",
|
||||
"dashboardFilters": [],
|
||||
"mosaicLayout": {
|
||||
"columns": 12,
|
||||
"tiles": [
|
||||
{
|
||||
"title": "Active Jobs by Status",
|
||||
"scorecard": {
|
||||
"timeSeriesQuery": {
|
||||
"prometheusQuery": "sum by (status) (jobs_total)"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Job Processing Time (p95)",
|
||||
"xyChart": {
|
||||
"dataSets": [{
|
||||
"timeSeriesQuery": {
|
||||
"prometheusQuery": "histogram_quantile(0.95, rate(job_stage_duration_seconds_bucket[5m]))"
|
||||
}
|
||||
}]
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Error Rate",
|
||||
"xyChart": {
|
||||
"dataSets": [{
|
||||
"timeSeriesQuery": {
|
||||
"prometheusQuery": "rate(worker_invocations_total{success=\"false\"}[5m]) / rate(worker_invocations_total[5m])"
|
||||
}
|
||||
}]
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Cloud Tasks Queue Depth",
|
||||
"xyChart": {
|
||||
"dataSets": [{
|
||||
"timeSeriesQuery": {
|
||||
"prometheusQuery": "cloud_tasks_queue_depth"
|
||||
}
|
||||
}]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Create Alerts
|
||||
|
||||
**File: `infrastructure/__main__.py`** (Pulumi)
|
||||
|
||||
```python
|
||||
# Alert: Job failure rate > 10%
|
||||
job_failure_alert = gcp.monitoring.AlertPolicy(
|
||||
"job-failure-rate-alert",
|
||||
display_name="High Job Failure Rate",
|
||||
conditions=[
|
||||
gcp.monitoring.AlertPolicyConditionArgs(
|
||||
display_name="Failure rate > 10%",
|
||||
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
|
||||
filter='metric.type="custom.googleapis.com/karaoke/job_failure_rate"',
|
||||
comparison="COMPARISON_GT",
|
||||
threshold_value=0.1,
|
||||
duration="300s",
|
||||
),
|
||||
),
|
||||
],
|
||||
notification_channels=[...], # Slack, email, etc.
|
||||
)
|
||||
|
||||
# Alert: Job stuck in processing > 30 min
|
||||
stuck_job_alert = gcp.monitoring.AlertPolicy(
|
||||
"stuck-job-alert",
|
||||
display_name="Job Stuck in Processing",
|
||||
conditions=[
|
||||
gcp.monitoring.AlertPolicyConditionArgs(
|
||||
display_name="Job > 30 min in processing state",
|
||||
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
|
||||
filter='metric.type="custom.googleapis.com/karaoke/job_age_seconds" AND metric.labels.status!="complete"',
|
||||
comparison="COMPARISON_GT",
|
||||
threshold_value=1800,
|
||||
duration="60s",
|
||||
),
|
||||
),
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Wins (Do First)
|
||||
|
||||
These give the most debugging value with minimal effort:
|
||||
|
||||
### 1. Add Job ID to Worker Logs (30 min)
|
||||
|
||||
Simplest possible fix - add job_id to every log message:
|
||||
|
||||
```python
|
||||
# backend/workers/audio_worker.py
|
||||
logger.info(f"[job:{job_id}] Starting audio separation")
|
||||
logger.info(f"[job:{job_id}] Downloaded audio file: {path}")
|
||||
logger.info(f"[job:{job_id}] Modal separation complete")
|
||||
```
|
||||
|
||||
Now you can grep:
|
||||
```bash
|
||||
gcloud logging read 'textPayload:"[job:8f03f0ac]"' --project=nomadkaraoke
|
||||
```
|
||||
|
||||
### 2. Log Worker Start/End with Timing (30 min)
|
||||
|
||||
```python
|
||||
import time
|
||||
|
||||
async def process_audio_separation(job_id: str):
|
||||
start = time.time()
|
||||
logger.info(f"[job:{job_id}] WORKER_START worker=audio")
|
||||
|
||||
try:
|
||||
# ... processing ...
|
||||
|
||||
duration = time.time() - start
|
||||
logger.info(f"[job:{job_id}] WORKER_END worker=audio status=success duration={duration:.1f}s")
|
||||
except Exception as e:
|
||||
duration = time.time() - start
|
||||
logger.error(f"[job:{job_id}] WORKER_END worker=audio status=error duration={duration:.1f}s error={e}")
|
||||
raise
|
||||
```
|
||||
|
||||
Now you can easily find failed workers:
|
||||
```bash
|
||||
gcloud logging read 'textPayload:"WORKER_END" AND textPayload:"status=error"' --project=nomadkaraoke
|
||||
```
|
||||
|
||||
### 3. Enable Cloud Trace Viewing (0 min - just fixed!)
|
||||
|
||||
With the IAM permission fix deployed today, Cloud Trace should start working. Visit:
|
||||
https://console.cloud.google.com/traces/list?project=nomadkaraoke
|
||||
|
||||
---
|
||||
|
||||
## Summary: Implementation Priority
|
||||
|
||||
| Phase | Effort | Value | Priority |
|
||||
|-------|--------|-------|----------|
|
||||
| Quick Win #1: Job ID in logs | 30 min | High | ⭐ Do now |
|
||||
| Quick Win #2: Worker timing | 30 min | High | ⭐ Do now |
|
||||
| Phase 1: Trace propagation | 2-3 hrs | Very High | ⭐ Do soon |
|
||||
| Phase 2: Instrument workers | 3-4 hrs | Very High | ⭐ Do soon |
|
||||
| Phase 3: Structured logging | 1-2 hrs | Medium | 🟡 Nice to have |
|
||||
| Phase 4: Custom metrics | 2-3 hrs | Medium | 🟡 Nice to have |
|
||||
| Phase 5: Dashboard & alerts | 2-3 hrs | Medium | 🟡 Nice to have |
|
||||
|
||||
**Total for full observability: ~12-15 hours**
|
||||
|
||||
**Minimum viable observability (Quick Wins + Phase 1-2): ~5-6 hours**
|
||||
|
||||
---
|
||||
|
||||
## Appendix: GCP Observability Resources
|
||||
|
||||
- **Cloud Trace:** https://console.cloud.google.com/traces/list?project=nomadkaraoke
|
||||
- **Cloud Logging:** https://console.cloud.google.com/logs?project=nomadkaraoke
|
||||
- **Cloud Monitoring:** https://console.cloud.google.com/monitoring?project=nomadkaraoke
|
||||
- **OpenTelemetry Python Docs:** https://opentelemetry.io/docs/instrumentation/python/
|
||||
- **Cloud Trace Python:** https://cloud.google.com/trace/docs/setup/python-ot
|
||||
|
||||
### Useful Log Queries
|
||||
|
||||
```
|
||||
# All logs for a specific job
|
||||
resource.type="cloud_run_revision" AND textPayload:"JOB_ID"
|
||||
|
||||
# All worker starts
|
||||
resource.type="cloud_run_revision" AND textPayload:"WORKER_START"
|
||||
|
||||
# All errors
|
||||
resource.type="cloud_run_revision" AND severity>=ERROR
|
||||
|
||||
# Cloud Tasks failures
|
||||
resource.type="cloud_tasks_queue"
|
||||
|
||||
# Trace-correlated logs (after structured logging)
|
||||
jsonPayload.job_id="JOB_ID"
|
||||
```
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
# Refactor: Unify Audio Fetcher Code
|
||||
|
||||
**Created:** 2025-12-20
|
||||
**Completed:** 2025-12-20
|
||||
**Priority:** MEDIUM (prevents future bugs like YouTubeProvider -> YoutubeProvider)
|
||||
**Status:** ✅ Completed
|
||||
|
||||
## Problem
|
||||
|
||||
The codebase has **two separate implementations** of flacfetch integration:
|
||||
1. `karaoke_gen/audio_fetcher.py` - Used by local CLI
|
||||
2. `backend/services/audio_search_service.py` - Used by cloud backend
|
||||
|
||||
This violates DRY and caused a bug where the backend's import wasn't updated when flacfetch renamed `YouTubeProvider` to `YoutubeProvider`.
|
||||
|
||||
## Current State
|
||||
|
||||
### Duplicated Code
|
||||
|
||||
| Component | Local (`audio_fetcher.py`) | Backend (`audio_search_service.py`) |
|
||||
|-----------|---------------------------|-------------------------------------|
|
||||
| `_get_manager()` | ✅ 40 lines | ❌ 35 lines (copy) |
|
||||
| Provider setup | ✅ RED, OPS, YouTube | ❌ Same, diverged |
|
||||
| `search()` | ✅ Full implementation | ❌ Similar, diverged |
|
||||
| `download()` | ✅ With downloaders | ⚠️ Different, simpler |
|
||||
| `AudioSearchResult` | ✅ Dataclass | ❌ Duplicate, different fields |
|
||||
| `AudioDownloadResult` | ✅ (`AudioFetchResult`) | ❌ Duplicate |
|
||||
| Exceptions | ✅ 4 types | ❌ 3 types, different names |
|
||||
|
||||
### Drift Examples
|
||||
- Backend missing `raw_result` field on `AudioSearchResult`
|
||||
- Backend has `index` field, local doesn't
|
||||
- Backend exceptions inherit from `AudioSearchError`, local from `AudioFetcherError`
|
||||
- Backend missing `UserCancelledError`
|
||||
- Backend missing downloader registration (TorrentDownloader, YoutubeDownloader)
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
### Option 1: Backend imports from karaoke_gen (Recommended)
|
||||
|
||||
The backend already depends on `karaoke_gen` (installed via `pip install -e .` in Dockerfile).
|
||||
|
||||
```python
|
||||
# backend/services/audio_search_service.py - REFACTORED
|
||||
|
||||
from karaoke_gen.audio_fetcher import (
|
||||
FlacFetcher,
|
||||
AudioSearchResult,
|
||||
AudioFetchResult,
|
||||
AudioFetcherError,
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
|
||||
class AudioSearchService:
|
||||
"""
|
||||
Thin wrapper around karaoke_gen.audio_fetcher.FlacFetcher.
|
||||
|
||||
Adds backend-specific functionality:
|
||||
- Firestore serialization (to_dict/from_dict)
|
||||
- GCS integration for downloads
|
||||
- Job state management
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
red_api_key: Optional[str] = None,
|
||||
red_api_url: Optional[str] = None,
|
||||
ops_api_key: Optional[str] = None,
|
||||
):
|
||||
self._fetcher = FlacFetcher(
|
||||
red_api_key=red_api_key,
|
||||
red_api_url=red_api_url,
|
||||
ops_api_key=ops_api_key,
|
||||
)
|
||||
|
||||
def search(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""Delegate to FlacFetcher.search()."""
|
||||
return self._fetcher.search(artist, title)
|
||||
|
||||
def select_best(self, results: List[AudioSearchResult]) -> int:
|
||||
"""Delegate to FlacFetcher.select_best()."""
|
||||
return self._fetcher.select_best(results)
|
||||
|
||||
def download(
|
||||
self,
|
||||
result: AudioSearchResult,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
) -> AudioFetchResult:
|
||||
"""Delegate to FlacFetcher.download()."""
|
||||
return self._fetcher.download(result, output_dir, output_filename)
|
||||
```
|
||||
|
||||
### Changes Required to karaoke_gen.audio_fetcher
|
||||
|
||||
1. **Add `to_dict()` method to `AudioSearchResult`** - For Firestore serialization
|
||||
2. **Add `from_dict()` classmethod** - For deserialization
|
||||
3. **Ensure `select_best()` is a standalone method** - Currently buried in interactive flow
|
||||
4. **Export all public classes from `__init__.py`**
|
||||
|
||||
### Changes Required to backend
|
||||
|
||||
1. **Remove duplicate dataclasses** - Use imports from karaoke_gen
|
||||
2. **Remove duplicate `_get_manager()`** - Delegate to FlacFetcher
|
||||
3. **Remove duplicate exception classes** - Use karaoke_gen exceptions
|
||||
4. **Update tests** - Point to real classes, not mocks
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Single source of truth** - Bug fixes apply everywhere
|
||||
2. **No more drift** - Dataclasses, exceptions stay in sync
|
||||
3. **Better test coverage** - karaoke_gen has integration tests for flacfetch imports
|
||||
4. **Less code to maintain** - Backend becomes thin wrapper
|
||||
|
||||
## Migration Steps
|
||||
|
||||
1. [x] Add `to_dict()` / `from_dict()` to `AudioSearchResult` in karaoke_gen
|
||||
2. [x] Extract `select_best()` as standalone method in FlacFetcher
|
||||
3. [x] Update karaoke_gen `__init__.py` exports (with lazy imports)
|
||||
4. [x] Refactor backend to import from karaoke_gen
|
||||
5. [x] Update backend tests
|
||||
6. [x] Delete duplicate code from backend (~150 lines removed)
|
||||
7. [x] Add integration test that catches import divergence
|
||||
|
||||
## Related Files
|
||||
|
||||
- `karaoke_gen/audio_fetcher.py` - Source of truth (keep)
|
||||
- `backend/services/audio_search_service.py` - Refactor to thin wrapper
|
||||
- `backend/api/routes/audio_search.py` - Update imports
|
||||
- `backend/tests/test_audio_search.py` - Update tests
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
- **Time:** 2-4 hours
|
||||
- **Risk:** Low (well-defined changes, good test coverage)
|
||||
- **Dependencies:** None
|
||||
|
||||
## Related Architecture Docs
|
||||
|
||||
- `docs/01-reference/ARCHITECTURE.md` - "LyricsTranscriber Integration" section shows similar pattern
|
||||
- `docs/00-current-plan/SCALABLE-ARCHITECTURE-PLAN.md` - "Shared pipeline architecture" goal
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
# Backend Testing Gaps Analysis
|
||||
|
||||
**Created:** 2025-12-22
|
||||
**Updated:** 2025-12-22
|
||||
**Status:** Mostly Complete ✅
|
||||
**Priority:** High
|
||||
|
||||
## Background
|
||||
|
||||
During end-to-end testing of the remote karaoke-gen CLI, multiple bugs were discovered:
|
||||
|
||||
### Bug #1: Missing `extra_info` Attribute (Fixed 2025-12-22)
|
||||
```
|
||||
Error: 'AudioSearchResult' object has no attribute 'extra_info'
|
||||
```
|
||||
|
||||
This bug existed in production code but wasn't caught by automated tests because the remote download code path had no test coverage.
|
||||
|
||||
### Bug #2: Missing `gcs_path` Parameter in API Route (Fixed 2025-12-22)
|
||||
```
|
||||
Error handling audio selection: Error selecting audio: {'detail': "Download failed: [Errno 2] No such file or directory: '/var/lib/transmission-daemon/downloads/Avril Lavigne - Unwanted (Album, 2002, CD, 16bit, RED).flac'"}
|
||||
```
|
||||
|
||||
The backend API route `_download_and_start_processing` in `audio_search.py` was not passing the `gcs_path` parameter to `audio_search_service.download()`. Without this parameter, the remote `flacfetch` VM would download the file locally but not upload it to GCS. The backend then tried to open the VM's local path (`/var/lib/transmission-daemon/downloads/...`) which doesn't exist on Cloud Run.
|
||||
|
||||
**Why this wasn't caught:**
|
||||
- Service-level tests for `AudioSearchService.download()` existed and tested `gcs_path` parameter
|
||||
- But the **API route caller** (`_download_and_start_processing`) that invokes this service was never tested
|
||||
- Tests validated the service layer but not the route layer's usage of the service
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
1. **Missing remote code path tests**: The download routing logic for `provider in ["RED", "OPS"]` was never exercised
|
||||
2. **State not set up for remote scenario**: Tests never configured `_remote_search_id` and `_remote_client` together
|
||||
3. **`FlacfetchClient` had zero tests**: The entire HTTP client was untested
|
||||
4. **Tests mocked at wrong level**: Tests mocked `service._fetcher` completely, bypassing the routing logic
|
||||
5. **API route orchestration untested**: The `_download_and_start_processing` function in `api/routes/audio_search.py` orchestrates multiple services but had no dedicated tests verifying correct parameter passing
|
||||
|
||||
## Services Testing Status
|
||||
|
||||
### ✅ Fully Tested
|
||||
|
||||
| Service | Test File | Test Count | Notes |
|
||||
|---------|-----------|------------|-------|
|
||||
| `credential_manager.py` | `test_credential_manager.py` | 27 | Good coverage |
|
||||
| `job_manager.py` | `test_job_manager.py` | 15 | Good coverage |
|
||||
| `flacfetch_client.py` | `test_flacfetch_client.py` | 26 | **Added 2025-12-22** |
|
||||
| `audio_search_service.py` | `test_audio_search.py` | 43 | **+11 tests 2025-12-22** |
|
||||
| `api/routes/audio_search.py` | `test_audio_search.py` | 4 | **API route tests added 2025-12-22** |
|
||||
| `storage_service.py` | `test_storage_service.py` | 17 | **Added 2025-12-22** |
|
||||
| `audio_analysis_service.py` | `test_audio_analysis_service.py` | 9 | **Added 2025-12-22** |
|
||||
| `audio_editing_service.py` | `test_audio_editing_service.py` | 13 | **Added 2025-12-22** |
|
||||
| `youtube_service.py` | `test_youtube_service.py` | 12 | **Added 2025-12-22** |
|
||||
| `dropbox_service.py` | `test_dropbox_service.py` | 14 | **Added 2025-12-22** |
|
||||
| `gdrive_service.py` | `test_gdrive_service.py` | 13 | **Added 2025-12-22** |
|
||||
|
||||
### ❌ No Dedicated Test File (Lower Priority)
|
||||
|
||||
| Service | Priority | Complexity | Notes |
|
||||
|---------|----------|------------|-------|
|
||||
| `rclone_service.py` | Low | Low | Wrapper around rclone CLI |
|
||||
| `firestore_service.py` | Low | Medium | Firestore operations |
|
||||
| `auth_service.py` | Low | Low | Auth utilities |
|
||||
| `job_logging.py` | Low | Low | Logging utilities |
|
||||
| `tracing.py` | Low | Low | OpenTelemetry setup |
|
||||
| `worker_service.py` | Medium | High | Worker orchestration |
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Core Audio Services (High Priority) ✅
|
||||
|
||||
- [x] `flacfetch_client.py` - Remote audio download client
|
||||
- [x] `audio_search_service.py` - Remote download path tests
|
||||
|
||||
### Phase 2: Audio Processing Services (High Priority) ✅
|
||||
|
||||
- [x] `audio_analysis_service.py` - Waveform analysis, peak detection
|
||||
- [x] `audio_editing_service.py` - Mute regions, audio manipulation
|
||||
|
||||
### Phase 3: Distribution Services (Medium Priority) ✅
|
||||
|
||||
- [x] `youtube_service.py` - YouTube credential management
|
||||
- [x] `dropbox_service.py` - Dropbox upload API
|
||||
- [x] `gdrive_service.py` - Google Drive upload API
|
||||
- [x] `storage_service.py` - GCS operations
|
||||
|
||||
### Phase 4: Infrastructure Services (Low Priority)
|
||||
|
||||
- [ ] `worker_service.py` - Worker orchestration
|
||||
- [ ] `firestore_service.py` - Firestore operations
|
||||
- [ ] `rclone_service.py` - Rclone wrapper
|
||||
- [ ] `auth_service.py` - Auth utilities
|
||||
- [ ] `job_logging.py` - Logging
|
||||
- [ ] `tracing.py` - Tracing
|
||||
|
||||
## Test Coverage Goals
|
||||
|
||||
- **Target:** 80%+ line coverage for high-priority services
|
||||
- **Minimum:** Key code paths tested (happy path + error handling)
|
||||
- **Integration:** Mock external APIs (YouTube, Dropbox, GCS, etc.)
|
||||
|
||||
## Testing Patterns to Follow
|
||||
|
||||
### 1. Mock External Services
|
||||
|
||||
```python
|
||||
@patch("backend.services.youtube_service.build")
|
||||
def test_upload_video(self, mock_build):
|
||||
mock_service = Mock()
|
||||
mock_build.return_value = mock_service
|
||||
# ...
|
||||
```
|
||||
|
||||
### 2. Test Error Handling
|
||||
|
||||
```python
|
||||
def test_upload_handles_network_error(self):
|
||||
with pytest.raises(ServiceError) as exc_info:
|
||||
service.upload(...)
|
||||
assert "network" in str(exc_info.value).lower()
|
||||
```
|
||||
|
||||
### 3. Test State Transitions
|
||||
|
||||
```python
|
||||
def test_download_routes_to_remote_for_torrent_providers(self):
|
||||
service._remote_client = Mock()
|
||||
service._remote_search_id = "search_123"
|
||||
service._cached_results = [result_with_provider_red]
|
||||
|
||||
service.download(0, "/tmp")
|
||||
|
||||
service._download_remote.assert_called_once()
|
||||
```
|
||||
|
||||
### 4. Verify Request/Response Format
|
||||
|
||||
```python
|
||||
def test_search_sends_correct_payload(self):
|
||||
# ...
|
||||
call_args = mock_client.post.call_args
|
||||
assert call_args.kwargs["json"] == {"artist": "ABBA", "title": "Waterloo"}
|
||||
```
|
||||
|
||||
### 5. Test API Route Orchestration (CRITICAL)
|
||||
|
||||
**Key Lesson from Bug #2:** Service-level tests are not sufficient. API routes that orchestrate multiple services must also be tested to verify correct parameter passing.
|
||||
|
||||
```python
|
||||
class TestAudioSearchApiRouteDownload:
|
||||
"""Tests for the _download_and_start_processing function in audio_search.py."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_remote_download_passes_gcs_path(self):
|
||||
"""Verify that gcs_path is passed to audio_search_service.download for torrent providers."""
|
||||
# Mock the audio_search_service.download to capture the call
|
||||
mock_audio_search_service = Mock()
|
||||
mock_audio_search_service.download.return_value = AudioDownloadResult(
|
||||
filepath="gs://bucket/path.flac", artist="A", title="T", provider="RED", quality="FLAC"
|
||||
)
|
||||
|
||||
result = await _download_and_start_processing(job_id="job1", ...)
|
||||
|
||||
# Assert gcs_path was passed
|
||||
call_kwargs = mock_audio_search_service.download.call_args.kwargs
|
||||
assert call_kwargs.get('gcs_path') is not None # CRITICAL check!
|
||||
assert call_kwargs['gcs_path'].startswith('uploads/')
|
||||
```
|
||||
|
||||
This pattern ensures that when service APIs change or new required parameters are added, API route tests will catch missing parameter passing.
|
||||
|
||||
## Files Changed
|
||||
|
||||
**Bug Fixes:**
|
||||
- `backend/services/audio_search_service.py` - Removed `extra_info` attribute access (Bug #1)
|
||||
- `backend/api/routes/audio_search.py` - Added `gcs_path` parameter to `audio_search_service.download()` call (Bug #2)
|
||||
|
||||
**Tests Added (2025-12-22):**
|
||||
- `backend/tests/test_audio_search.py` - +11 tests for service + 4 tests for API route orchestration
|
||||
- `backend/tests/test_flacfetch_client.py` - 26 new tests (new file)
|
||||
- `backend/tests/test_storage_service.py` - 17 new tests (new file)
|
||||
- `backend/tests/test_audio_analysis_service.py` - 9 new tests (new file)
|
||||
- `backend/tests/test_audio_editing_service.py` - 13 new tests (new file)
|
||||
- `backend/tests/test_youtube_service.py` - 12 new tests (new file)
|
||||
- `backend/tests/test_dropbox_service.py` - 14 new tests (new file)
|
||||
- `backend/tests/test_gdrive_service.py` - 13 new tests (new file)
|
||||
|
||||
**Total New Tests:** 119 tests added
|
||||
|
||||
**Version:** 0.74.3
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# All backend tests
|
||||
cd backend && python -m pytest tests/ -v
|
||||
|
||||
# Specific service tests
|
||||
python -m pytest tests/test_flacfetch_client.py -v
|
||||
python -m pytest tests/test_audio_search.py -v
|
||||
|
||||
# With coverage
|
||||
python -m pytest tests/ -v --cov=services --cov-report=term-missing
|
||||
```
|
||||
|
||||
## Notes for Future Sessions
|
||||
|
||||
When continuing this work:
|
||||
|
||||
1. **Test API route orchestration, not just services**: Bug #2 showed that service tests alone don't catch parameter passing bugs at the route level
|
||||
2. Distribution services need OAuth mocking - check existing patterns in `test_credential_manager.py`
|
||||
3. Use `pytest.mark.asyncio` for async service methods
|
||||
4. Follow existing test patterns in `test_flacfetch_client.py` for HTTP client mocking
|
||||
5. **For remote operations**: Always verify that `gcs_path` parameters are passed when code paths involve remote VM operations
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
### Testing Layers Matter
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ API Routes (api/routes/*.py) │
|
||||
│ └─> These orchestrate services and MUST be tested separately │
|
||||
│ to verify correct parameter passing │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Services (services/*.py) │
|
||||
│ └─> Business logic with proper unit tests │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Clients (FlacfetchClient, etc.) │
|
||||
│ └─> HTTP/external API interactions │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Bug #2 was missed because:
|
||||
- `FlacfetchClient` tests ✅ verified `gcs_path` is sent in HTTP requests
|
||||
- `AudioSearchService` tests ✅ verified `gcs_path` is passed to client
|
||||
- **API route tests ❌ didn't exist** - so nobody verified the route passes `gcs_path` to the service
|
||||
|
||||
Now we have `TestAudioSearchApiRouteDownload` to catch this class of bugs.
|
||||
|
||||
@@ -0,0 +1,837 @@
|
||||
# Flacfetch HTTP API Service Implementation Plan
|
||||
|
||||
**Created:** 2025-12-20
|
||||
**Updated:** 2025-12-22
|
||||
**Status:** Complete ✅
|
||||
**Priority:** Medium
|
||||
**Estimated Effort:** 3-4 days (actual: ~3 days)
|
||||
|
||||
## Problem Statement
|
||||
|
||||
BitTorrent downloads don't work reliably in Cloud Run due to:
|
||||
- No inbound peer connections (NAT/firewall)
|
||||
- Ephemeral containers with no stable IP
|
||||
- DHT peer discovery blocked
|
||||
- Container timeout limits (max 60 min)
|
||||
|
||||
This prevents downloading lossless audio from private trackers (RED, OPS) via the remote backend.
|
||||
|
||||
## Solution Overview
|
||||
|
||||
Add an HTTP API mode to **flacfetch** (our audio fetching library) and deploy it on a dedicated GCP VM with:
|
||||
- Full BitTorrent peer connectivity via Transmission daemon
|
||||
- Static IP for tracker whitelist (seedbox-like)
|
||||
- Indefinite seeding of downloaded torrents
|
||||
- Automatic disk cleanup when space runs low
|
||||
- YouTube downloads also handled (single service for all audio acquisition)
|
||||
|
||||
This approach provides clean separation of concerns:
|
||||
- **flacfetch**: Handles ALL audio acquisition (torrents + YouTube)
|
||||
- **karaoke-gen backend**: Orchestrates karaoke generation pipeline, calls flacfetch API
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ FLACFETCH SERVICE ARCHITECTURE │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Cloud Run Backend Flacfetch VM (e2-small) │
|
||||
│ ┌────────────────────┐ ┌─────────────────────────┐ │
|
||||
│ │ │ │ │ │
|
||||
│ │ karaoke-gen │ 1. Search │ flacfetch HTTP API │ │
|
||||
│ │ AudioSearchService │ ────────────▶ │ POST /search │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ │ 2. Results │ Providers: │ │
|
||||
│ │ │ ◀──────────── │ • RED (FLAC) │ │
|
||||
│ │ │ │ • OPS (FLAC) │ │
|
||||
│ │ │ 3. Download │ • YouTube (fallback) │ │
|
||||
│ │ │ ────────────▶ │ │ │
|
||||
│ │ │ │ POST /download │ │
|
||||
│ │ │ 4. Status │ GET /download/{id} │ │
|
||||
│ │ │ ◀──────────── │ │ │
|
||||
│ │ │ (polling) │ Transmission Daemon │ │
|
||||
│ │ │ │ • Full peer connect │ │
|
||||
│ │ │ 5. Complete │ • DHT enabled │ │
|
||||
│ │ │ ◀──────────── │ • Indefinite seeding │ │
|
||||
│ │ │ (file path) │ │ │
|
||||
│ └────────────────────┘ └───────────┬─────────────┘ │
|
||||
│ │ │ │
|
||||
│ │ │ 6. Upload to GCS │
|
||||
│ │ │ │
|
||||
│ ▼ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Google Cloud Storage │ │
|
||||
│ │ │ │
|
||||
│ │ uploads/{job_id}/audio/{filename}.flac │ │
|
||||
│ │ │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. HTTP API in flacfetch (not separate service)
|
||||
- **Why**: Clean separation of concerns, reusable by future projects, single source of truth
|
||||
- **How**: Add `flacfetch/api/` module with FastAPI, new CLI mode `flacfetch serve`
|
||||
|
||||
### 2. Indefinite Seeding
|
||||
- **Why**: VM is always-on anyway, helps maintain tracker ratio, acts like a seedbox
|
||||
- **Current behavior**: Torrent removed after download (line 317-321 in torrent.py)
|
||||
- **New behavior**: Torrent kept seeding, only removed by cleanup or explicit delete
|
||||
|
||||
### 3. Static IP
|
||||
- **Why**: Required for tracker account IP whitelist, consistent for firewall rules
|
||||
- **Cost**: ~$3/month additional
|
||||
|
||||
### 4. Disk Space Management
|
||||
- **Strategy**: Auto-cleanup oldest torrents when free space < 5GB threshold
|
||||
- **API**: Manual cleanup endpoints + automatic background cleanup
|
||||
|
||||
### 5. Instance Type: e2-small
|
||||
- **Specs**: 0.5 vCPU, 2GB RAM, ~$12/month
|
||||
- **Why**: Sufficient for I/O-bound workload, newer C3/N4 require 2+ vCPUs (overkill)
|
||||
- **Disk**: 30GB SSD (20GB for torrents + headroom)
|
||||
|
||||
### 6. Authentication
|
||||
- **Method**: Static API key in `X-API-Key` header
|
||||
- **Firewall**: Permissive (allows future web UI), auth enforced at app level
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Add HTTP API to flacfetch
|
||||
|
||||
**Location:** `./flacfetch/flacfetch/api/`
|
||||
|
||||
**New Directory Structure:**
|
||||
```
|
||||
flacfetch/
|
||||
├── flacfetch/
|
||||
│ ├── api/ # NEW
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── main.py # FastAPI app entry point
|
||||
│ │ ├── auth.py # API key authentication
|
||||
│ │ ├── models.py # Pydantic request/response models
|
||||
│ │ ├── routes/
|
||||
│ │ │ ├── __init__.py
|
||||
│ │ │ ├── search.py # POST /search
|
||||
│ │ │ ├── download.py # POST /download, GET /download/{id}/status
|
||||
│ │ │ ├── torrents.py # GET /torrents, DELETE /torrents/{id}
|
||||
│ │ │ └── health.py # GET /health, GET /disk-usage
|
||||
│ │ └── services/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── download_manager.py # Track active downloads
|
||||
│ │ └── disk_manager.py # Disk cleanup logic
|
||||
│ ├── core/
|
||||
│ ├── downloaders/
|
||||
│ │ └── torrent.py # MODIFIED: optional keep-seeding mode
|
||||
│ ├── interface/
|
||||
│ │ └── cli.py # MODIFIED: add 'serve' subcommand
|
||||
│ └── providers/
|
||||
└── pyproject.toml # MODIFIED: add fastapi, uvicorn deps
|
||||
```
|
||||
|
||||
**API Endpoints:**
|
||||
|
||||
```yaml
|
||||
# All endpoints require X-API-Key header
|
||||
|
||||
# Search for audio
|
||||
POST /search
|
||||
Request:
|
||||
{ "artist": "ABBA", "title": "Waterloo" }
|
||||
Response:
|
||||
{
|
||||
"search_id": "abc123",
|
||||
"results": [
|
||||
{
|
||||
"index": 0,
|
||||
"title": "Waterloo",
|
||||
"artist": "ABBA",
|
||||
"provider": "RED",
|
||||
"quality": "FLAC 16bit CD",
|
||||
"seeders": 45,
|
||||
"size_bytes": 31457280,
|
||||
"target_file": "02. Waterloo.flac",
|
||||
...
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
|
||||
# Start download
|
||||
POST /download
|
||||
Request:
|
||||
{
|
||||
"search_id": "abc123", # From previous search
|
||||
"result_index": 0, # Which result to download
|
||||
"output_filename": "ABBA - Waterloo", # Optional
|
||||
"upload_to_gcs": true, # Upload to GCS when complete
|
||||
"gcs_path": "uploads/job123/audio/" # Required if upload_to_gcs
|
||||
}
|
||||
Response:
|
||||
{
|
||||
"download_id": "dl_xyz789",
|
||||
"status": "queued"
|
||||
}
|
||||
|
||||
# Check download status
|
||||
GET /download/{download_id}/status
|
||||
Response:
|
||||
{
|
||||
"download_id": "dl_xyz789",
|
||||
"status": "downloading", # queued, downloading, uploading, seeding, complete, failed
|
||||
"progress": 45.2,
|
||||
"peers": 3,
|
||||
"download_speed_kbps": 1250.5,
|
||||
"eta_seconds": 120,
|
||||
"provider": "RED",
|
||||
"title": "ABBA - Waterloo",
|
||||
"output_path": null, # Set when complete (local path)
|
||||
"gcs_path": null, # Set when uploaded to GCS
|
||||
"error": null
|
||||
}
|
||||
|
||||
# List all torrents (for management)
|
||||
GET /torrents
|
||||
Response:
|
||||
{
|
||||
"torrents": [
|
||||
{
|
||||
"id": 1,
|
||||
"name": "ABBA - Waterloo [FLAC]",
|
||||
"status": "seeding",
|
||||
"progress": 100.0,
|
||||
"size_bytes": 314572800,
|
||||
"uploaded_bytes": 628145600, # Seeding stats
|
||||
"ratio": 2.0,
|
||||
"added_at": "2025-12-20T10:30:00Z"
|
||||
},
|
||||
...
|
||||
],
|
||||
"total_size_bytes": 5368709120,
|
||||
"count": 15
|
||||
}
|
||||
|
||||
# Delete a torrent (manual cleanup)
|
||||
DELETE /torrents/{torrent_id}
|
||||
Query params: ?delete_data=true # Also delete downloaded files
|
||||
Response:
|
||||
{ "status": "deleted" }
|
||||
|
||||
# Trigger cleanup (free disk space)
|
||||
POST /torrents/cleanup
|
||||
Request:
|
||||
{
|
||||
"strategy": "oldest", # oldest, largest, lowest_ratio
|
||||
"target_free_gb": 10 # Keep removing until this much free
|
||||
}
|
||||
Response:
|
||||
{
|
||||
"removed_count": 3,
|
||||
"freed_bytes": 1073741824,
|
||||
"free_space_gb": 10.5
|
||||
}
|
||||
|
||||
# Health check
|
||||
GET /health
|
||||
Response:
|
||||
{
|
||||
"status": "healthy",
|
||||
"transmission": {
|
||||
"available": true,
|
||||
"version": "4.0.5",
|
||||
"active_torrents": 5
|
||||
},
|
||||
"disk": {
|
||||
"total_gb": 30,
|
||||
"used_gb": 18,
|
||||
"free_gb": 12
|
||||
},
|
||||
"providers": {
|
||||
"red": true,
|
||||
"ops": true,
|
||||
"youtube": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Torrent Downloader Changes:**
|
||||
|
||||
```python
|
||||
# flacfetch/downloaders/torrent.py
|
||||
|
||||
class TorrentDownloader(Downloader):
|
||||
def __init__(self, ..., keep_seeding: bool = False):
|
||||
"""
|
||||
Args:
|
||||
keep_seeding: If True, don't remove torrent after download.
|
||||
Used in server mode for indefinite seeding.
|
||||
"""
|
||||
self.keep_seeding = keep_seeding
|
||||
|
||||
def download(self, release, output_path, output_filename=None):
|
||||
# ... existing download logic ...
|
||||
|
||||
# After download complete:
|
||||
if self.keep_seeding:
|
||||
# Don't remove, just return the file path
|
||||
logger.info(f"Download complete, keeping torrent for seeding: {torrent.name}")
|
||||
else:
|
||||
# Original behavior: stop and remove
|
||||
self.client.stop_torrent(torrent.id)
|
||||
self.client.remove_torrent(torrent.id, delete_data=True)
|
||||
```
|
||||
|
||||
**CLI Changes:**
|
||||
|
||||
```bash
|
||||
# New serve subcommand
|
||||
flacfetch serve --port 8080 --host 0.0.0.0
|
||||
|
||||
# Or via environment
|
||||
FLACFETCH_API_PORT=8080 flacfetch serve
|
||||
```
|
||||
|
||||
### Phase 2: Infrastructure (Pulumi)
|
||||
|
||||
**Location:** `./infrastructure/__main__.py`
|
||||
|
||||
**New Resources:**
|
||||
```python
|
||||
# Static IP for flacfetch service (for tracker whitelist)
|
||||
flacfetch_ip = gcp.compute.Address(
|
||||
"flacfetch-ip",
|
||||
name="flacfetch-static-ip",
|
||||
region="us-central1",
|
||||
)
|
||||
|
||||
# Service account for flacfetch VM
|
||||
flacfetch_sa = gcp.serviceaccount.Account(
|
||||
"flacfetch-sa",
|
||||
account_id="flacfetch-service",
|
||||
display_name="Flacfetch Service Account",
|
||||
)
|
||||
|
||||
# GCS write permissions for flacfetch
|
||||
gcp.storage.BucketIAMMember(
|
||||
"flacfetch-gcs-writer",
|
||||
bucket=bucket.name,
|
||||
role="roles/storage.objectCreator",
|
||||
member=pulumi.Output.concat("serviceAccount:", flacfetch_sa.email),
|
||||
)
|
||||
|
||||
# Flacfetch VM
|
||||
flacfetch_vm = gcp.compute.Instance(
|
||||
"flacfetch-service",
|
||||
machine_type="e2-small",
|
||||
zone="us-central1-a",
|
||||
boot_disk=gcp.compute.InstanceBootDiskArgs(
|
||||
initialize_params=gcp.compute.InstanceBootDiskInitializeParamsArgs(
|
||||
image="debian-cloud/debian-12",
|
||||
size=30, # GB
|
||||
type="pd-ssd",
|
||||
),
|
||||
),
|
||||
network_interfaces=[gcp.compute.InstanceNetworkInterfaceArgs(
|
||||
network="default",
|
||||
access_configs=[gcp.compute.InstanceNetworkInterfaceAccessConfigArgs(
|
||||
nat_ip=flacfetch_ip.address, # Static IP
|
||||
)],
|
||||
)],
|
||||
service_account=gcp.compute.InstanceServiceAccountArgs(
|
||||
email=flacfetch_sa.email,
|
||||
scopes=["cloud-platform"],
|
||||
),
|
||||
metadata_startup_script=FLACFETCH_STARTUP_SCRIPT,
|
||||
tags=["flacfetch-service"],
|
||||
)
|
||||
|
||||
# Firewall for flacfetch
|
||||
flacfetch_firewall = gcp.compute.Firewall(
|
||||
"flacfetch-firewall",
|
||||
network="default",
|
||||
allows=[
|
||||
# BitTorrent peer connections
|
||||
gcp.compute.FirewallAllowArgs(protocol="tcp", ports=["51413"]),
|
||||
gcp.compute.FirewallAllowArgs(protocol="udp", ports=["51413"]),
|
||||
# Flacfetch HTTP API (auth required at app level)
|
||||
gcp.compute.FirewallAllowArgs(protocol="tcp", ports=["8080"]),
|
||||
],
|
||||
source_ranges=["0.0.0.0/0"], # Permissive, auth at app level
|
||||
target_tags=["flacfetch-service"],
|
||||
)
|
||||
|
||||
# Store API key in Secret Manager
|
||||
flacfetch_api_key = gcp.secretmanager.Secret(
|
||||
"flacfetch-api-key",
|
||||
secret_id="flacfetch-api-key",
|
||||
replication=gcp.secretmanager.SecretReplicationArgs(
|
||||
auto=gcp.secretmanager.SecretReplicationAutoArgs(),
|
||||
),
|
||||
)
|
||||
|
||||
# Export for use
|
||||
pulumi.export("flacfetch_static_ip", flacfetch_ip.address)
|
||||
pulumi.export("flacfetch_service_url", pulumi.Output.concat("http://", flacfetch_ip.address, ":8080"))
|
||||
```
|
||||
|
||||
**Startup Script:**
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Install dependencies
|
||||
apt-get update
|
||||
apt-get install -y python3-pip python3-venv transmission-daemon ffmpeg git
|
||||
|
||||
# Configure Transmission
|
||||
systemctl stop transmission-daemon
|
||||
cat > /etc/transmission-daemon/settings.json << 'EOF'
|
||||
{
|
||||
"download-dir": "/var/lib/transmission-daemon/downloads",
|
||||
"incomplete-dir": "/var/lib/transmission-daemon/.incomplete",
|
||||
"incomplete-dir-enabled": true,
|
||||
"rpc-authentication-required": false,
|
||||
"rpc-bind-address": "127.0.0.1",
|
||||
"rpc-enabled": true,
|
||||
"rpc-port": 9091,
|
||||
"rpc-whitelist-enabled": false,
|
||||
"peer-port": 51413,
|
||||
"port-forwarding-enabled": false,
|
||||
"speed-limit-down": 0,
|
||||
"speed-limit-down-enabled": false,
|
||||
"speed-limit-up": 0,
|
||||
"speed-limit-up-enabled": false,
|
||||
"ratio-limit-enabled": false,
|
||||
"umask": 2
|
||||
}
|
||||
EOF
|
||||
systemctl start transmission-daemon
|
||||
|
||||
# Install flacfetch
|
||||
cd /opt
|
||||
git clone https://github.com/nomadkaraoke/flacfetch.git
|
||||
cd flacfetch
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -e ".[api]"
|
||||
|
||||
# Get secrets from metadata/Secret Manager
|
||||
FLACFETCH_API_KEY=$(gcloud secrets versions access latest --secret=flacfetch-api-key)
|
||||
RED_API_KEY=$(gcloud secrets versions access latest --secret=red-api-key)
|
||||
RED_API_URL=$(gcloud secrets versions access latest --secret=red-api-url)
|
||||
OPS_API_KEY=$(gcloud secrets versions access latest --secret=ops-api-key)
|
||||
GCS_BUCKET=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/attributes/gcs-bucket" -H "Metadata-Flavor: Google")
|
||||
|
||||
# Create systemd service
|
||||
cat > /etc/systemd/system/flacfetch.service << EOF
|
||||
[Unit]
|
||||
Description=Flacfetch HTTP API Service
|
||||
After=network.target transmission-daemon.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/flacfetch
|
||||
Environment="FLACFETCH_API_KEY=${FLACFETCH_API_KEY}"
|
||||
Environment="RED_API_KEY=${RED_API_KEY}"
|
||||
Environment="RED_API_URL=${RED_API_URL}"
|
||||
Environment="OPS_API_KEY=${OPS_API_KEY}"
|
||||
Environment="GCS_BUCKET=${GCS_BUCKET}"
|
||||
Environment="FLACFETCH_KEEP_SEEDING=true"
|
||||
Environment="FLACFETCH_MIN_FREE_GB=5"
|
||||
ExecStart=/opt/flacfetch/venv/bin/flacfetch serve --host 0.0.0.0 --port 8080
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
systemctl daemon-reload
|
||||
systemctl enable flacfetch
|
||||
systemctl start flacfetch
|
||||
```
|
||||
|
||||
### Phase 3: Backend Integration
|
||||
|
||||
**New File:** `backend/services/flacfetch_client.py`
|
||||
|
||||
```python
|
||||
"""
|
||||
Client for the remote flacfetch HTTP API service.
|
||||
|
||||
Used when FLACFETCH_API_URL is configured, otherwise falls back to local flacfetch.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional, List, Dict, Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FlacfetchServiceError(Exception):
|
||||
"""Error communicating with flacfetch service."""
|
||||
pass
|
||||
|
||||
|
||||
class FlacfetchClient:
|
||||
"""Client for remote flacfetch HTTP API."""
|
||||
|
||||
def __init__(self, base_url: str, api_key: str, timeout: int = 30):
|
||||
self.base_url = base_url.rstrip('/')
|
||||
self.api_key = api_key
|
||||
self.timeout = timeout
|
||||
|
||||
def _headers(self) -> Dict[str, str]:
|
||||
return {"X-API-Key": self.api_key}
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""Check if service is healthy."""
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/health",
|
||||
headers=self._headers(),
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
async def search(self, artist: str, title: str) -> Dict[str, Any]:
|
||||
"""Search for audio."""
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/search",
|
||||
headers=self._headers(),
|
||||
json={"artist": artist, "title": title},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
async def download(
|
||||
self,
|
||||
search_id: str,
|
||||
result_index: int,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Start download and return download_id.
|
||||
|
||||
If gcs_path is provided, file will be uploaded to GCS.
|
||||
"""
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"search_id": search_id,
|
||||
"result_index": result_index,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()["download_id"]
|
||||
|
||||
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
|
||||
"""Get download status."""
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/download/{download_id}/status",
|
||||
headers=self._headers(),
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
async def wait_for_download(
|
||||
self,
|
||||
download_id: str,
|
||||
timeout: int = 600,
|
||||
poll_interval: int = 2,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Wait for download to complete.
|
||||
|
||||
Returns final status dict with output_path or gcs_path.
|
||||
Raises FlacfetchServiceError on failure or timeout.
|
||||
"""
|
||||
elapsed = 0
|
||||
while elapsed < timeout:
|
||||
status = await self.get_download_status(download_id)
|
||||
|
||||
if status["status"] == "complete":
|
||||
return status
|
||||
elif status["status"] == "failed":
|
||||
raise FlacfetchServiceError(f"Download failed: {status.get('error')}")
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
elapsed += poll_interval
|
||||
|
||||
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
|
||||
|
||||
|
||||
# Singleton
|
||||
_client: Optional[FlacfetchClient] = None
|
||||
|
||||
|
||||
def get_flacfetch_client() -> Optional[FlacfetchClient]:
|
||||
"""Get flacfetch client if configured."""
|
||||
global _client
|
||||
if _client is None:
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
if settings.flacfetch_api_url and settings.flacfetch_api_key:
|
||||
_client = FlacfetchClient(
|
||||
base_url=settings.flacfetch_api_url,
|
||||
api_key=settings.flacfetch_api_key,
|
||||
)
|
||||
return _client
|
||||
```
|
||||
|
||||
**Modify:** `backend/services/audio_search_service.py` to use remote client when available.
|
||||
|
||||
**Modify:** `backend/config.py` to add new settings:
|
||||
```python
|
||||
flacfetch_api_url: Optional[str] = None # e.g., http://10.0.0.5:8080
|
||||
flacfetch_api_key: Optional[str] = None
|
||||
```
|
||||
|
||||
### Phase 4: Monitoring & Reliability
|
||||
|
||||
**Tasks:**
|
||||
- [ ] Add Cloud Monitoring alert for VM CPU/memory
|
||||
- [ ] Add alert for disk space < 5GB
|
||||
- [ ] Add alert for flacfetch service down (health check fails)
|
||||
- [ ] Configure Cloud Logging export from VM
|
||||
- [ ] Add Grafana dashboard (optional)
|
||||
|
||||
**Automatic Disk Cleanup:**
|
||||
The disk manager runs as a background task in the flacfetch API service:
|
||||
```python
|
||||
# Check every 5 minutes, cleanup if free space < threshold
|
||||
async def background_cleanup_task():
|
||||
while True:
|
||||
await asyncio.sleep(300)
|
||||
free_gb = get_free_space_gb()
|
||||
if free_gb < MIN_FREE_GB:
|
||||
cleanup_oldest_torrents(target_free_gb=MIN_FREE_GB + 5)
|
||||
```
|
||||
|
||||
### Phase 5: Testing & Documentation
|
||||
|
||||
**Testing:**
|
||||
- [ ] Unit tests for flacfetch API endpoints
|
||||
- [ ] Unit tests for disk manager
|
||||
- [ ] Integration test: search → download → GCS upload
|
||||
- [ ] Mock tests for karaoke-gen backend client
|
||||
|
||||
**Documentation:**
|
||||
- [ ] Update `docs/01-reference/ARCHITECTURE.md` with flacfetch service
|
||||
- [ ] Add `flacfetch/README.md` HTTP API section
|
||||
- [ ] Document deployment process
|
||||
- [ ] Document environment variables
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Flacfetch Service (VM)
|
||||
```bash
|
||||
# API Configuration
|
||||
FLACFETCH_API_KEY=<secret> # Required for API auth
|
||||
FLACFETCH_API_PORT=8080 # Default: 8080
|
||||
FLACFETCH_API_HOST=0.0.0.0 # Default: 0.0.0.0
|
||||
|
||||
# Tracker API Keys
|
||||
RED_API_KEY=<key> # For RED provider
|
||||
RED_API_URL=<url> # For RED provider base URL
|
||||
OPS_API_KEY=<key> # For OPS provider
|
||||
|
||||
# GCS Upload (optional)
|
||||
GCS_BUCKET=karaoke-uploads # For uploading completed downloads
|
||||
GOOGLE_CLOUD_PROJECT=nomadkaraoke
|
||||
|
||||
# Transmission
|
||||
TRANSMISSION_HOST=localhost # Default: localhost
|
||||
TRANSMISSION_PORT=9091 # Default: 9091
|
||||
|
||||
# Behavior
|
||||
FLACFETCH_KEEP_SEEDING=true # Keep torrents seeding after download
|
||||
FLACFETCH_MIN_FREE_GB=5 # Minimum free disk space before cleanup
|
||||
FLACFETCH_DOWNLOAD_DIR=/var/lib/transmission-daemon/downloads
|
||||
```
|
||||
|
||||
### Cloud Run Backend
|
||||
```bash
|
||||
# New variables
|
||||
FLACFETCH_API_URL=http://10.128.0.5:8080 # Internal IP of flacfetch VM
|
||||
FLACFETCH_API_KEY=<same-secret>
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **API Authentication**: Static API key in `X-API-Key` header
|
||||
2. **Firewall**: Permissive for future web UI, auth enforced at application level
|
||||
3. **GCS**: Service account with `storage.objectCreator` role only
|
||||
4. **Transmission**: RPC bound to localhost only, no external access
|
||||
5. **Static IP**: Can be whitelisted on tracker accounts
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If flacfetch service fails:
|
||||
1. Set `FLACFETCH_API_URL=""` in Cloud Run (disables remote service)
|
||||
2. Backend falls back to local flacfetch (YouTube only in Cloud Run)
|
||||
3. Users see warning that lossless downloads may be slower/unavailable
|
||||
|
||||
## Cost Estimate
|
||||
|
||||
| Resource | Monthly Cost |
|
||||
|----------|--------------|
|
||||
| e2-small VM | ~$12 |
|
||||
| Static IP | ~$3 |
|
||||
| 30GB SSD | ~$5 |
|
||||
| Network egress | ~$2-5 |
|
||||
| **Total** | **~$22-25/month** |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [x] Flacfetch HTTP API accepts search requests and returns results
|
||||
- [x] Downloads complete successfully for RED/OPS sources
|
||||
- [x] Downloaded files upload to GCS automatically
|
||||
- [x] Torrents continue seeding after download
|
||||
- [x] Disk cleanup triggers automatically when space is low
|
||||
- [x] Remote CLI shows download progress from flacfetch service
|
||||
- [x] Fallback to local flacfetch works when service unavailable
|
||||
- [x] Static IP can be whitelisted on tracker accounts
|
||||
|
||||
## Timeline
|
||||
|
||||
| Phase | Duration | Dependencies |
|
||||
|-------|----------|--------------|
|
||||
| Phase 1: Flacfetch HTTP API | 1.5 days | None |
|
||||
| Phase 2: Infrastructure | 0.5 day | Phase 1 |
|
||||
| Phase 3: Backend Integration | 0.5 day | Phase 1, 2 |
|
||||
| Phase 4: Monitoring | 0.5 day | Phase 2 |
|
||||
| Phase 5: Testing & Docs | 0.5 day | Phase 1-3 |
|
||||
| **Total** | **3.5 days** | |
|
||||
|
||||
## Implementation Status (2025-12-22)
|
||||
|
||||
### Completed Work
|
||||
|
||||
#### Phase 1: Flacfetch HTTP API ✅
|
||||
- Full HTTP API implemented in `flacfetch/flacfetch/api/`
|
||||
- Endpoints: `/search`, `/download`, `/download/{id}/status`, `/download/{id}/file`, `/torrents`, `/health`
|
||||
- API key authentication via `X-API-Key` header
|
||||
- Disk manager with automatic cleanup when space < 5GB
|
||||
- Download manager with search caching and progress tracking
|
||||
- Dynamic version reporting via `importlib.metadata`
|
||||
- **Current Version:** 0.7.3
|
||||
|
||||
#### Phase 2: Infrastructure ✅
|
||||
- Pulumi infrastructure in `infrastructure/__main__.py`
|
||||
- GCP VM: `flacfetch-service` (e2-small, us-central1-a)
|
||||
- Static IP: `104.198.214.26` (whitelisted with trackers)
|
||||
- Firewall rules: TCP/UDP 51413 (BitTorrent), TCP 8080 (API)
|
||||
- Secrets: `flacfetch-api-key`, `red-api-key`, `red-api-url`, `ops-api-key`, `ops-api-url`, `flacfetch-api-url`
|
||||
- GitHub Actions deployer has `compute.admin` and `compute.osLogin` roles
|
||||
|
||||
#### Phase 3: Backend Integration ✅
|
||||
- `FlacfetchClient` in `backend/services/flacfetch_client.py`
|
||||
- `AudioSearchService` updated to use remote flacfetch when configured
|
||||
- Cloud Run configured with `FLACFETCH_API_URL` and `FLACFETCH_API_KEY` from Secret Manager
|
||||
- `--loop asyncio` added to uvicorn CMD (required for `nest_asyncio` compatibility)
|
||||
- `nest_asyncio` applied only when async loop is running (not at import time)
|
||||
|
||||
#### Phase 4: Monitoring ✅
|
||||
- Health endpoint reports Transmission status, disk usage, provider availability
|
||||
- Version verification in deployment workflows
|
||||
- Automatic deployment on release via GitHub Actions
|
||||
|
||||
#### Phase 5: CLI & Testing ✅
|
||||
- `flacfetch-remote` CLI added for testing remote API
|
||||
- Downloads file locally with descriptive filenames (e.g., `Artist - Title (Album, 2022, CD, 16bit, RED).flac`)
|
||||
- 70%+ unit test coverage requirement enforced
|
||||
- Integration tests for API routes
|
||||
|
||||
### Deployment Automation
|
||||
|
||||
#### Flacfetch VM (automatic)
|
||||
- `.github/workflows/release-on-version-bump.yml` in flacfetch repo
|
||||
- Triggers on version bump in `pyproject.toml`
|
||||
- Runs tests → Creates GitHub release → SSHs to VM → Pulls latest → Restarts service
|
||||
- Verifies deployed version matches release version
|
||||
|
||||
#### Karaoke-gen Backend (version-gated)
|
||||
- `.github/workflows/ci.yml` deploy-backend job
|
||||
- Only deploys if version tag doesn't exist (requires version bump)
|
||||
- Verifies deployed version via `/api/health/detailed` endpoint
|
||||
- Retries version check with backoff for Cloud Run routing delays
|
||||
|
||||
### Key Files Modified
|
||||
|
||||
**Flacfetch repo (`./flacfetch/`):**
|
||||
- `flacfetch/api/` - New HTTP API module
|
||||
- `flacfetch/interface/cli_remote.py` - New remote CLI
|
||||
- `flacfetch/core/manager.py` - Improved sorting (artist match, release type, seeders)
|
||||
- `flacfetch/downloaders/torrent.py` - Keep-seeding mode, file permissions fix
|
||||
- `.github/workflows/release-on-version-bump.yml` - Auto-deploy to VM
|
||||
|
||||
**Karaoke-gen repo:**
|
||||
- `backend/services/flacfetch_client.py` - Remote API client
|
||||
- `backend/services/audio_search_service.py` - Remote integration with nest_asyncio
|
||||
- `backend/Dockerfile` - `--loop asyncio` for uvicorn
|
||||
- `infrastructure/__main__.py` - VM, firewall, secrets, WIF bindings
|
||||
- `cloudbuild.yaml` - FLACFETCH_API_URL/KEY from Secret Manager
|
||||
- `.github/workflows/ci.yml` - Version-gated deployments
|
||||
|
||||
### Testing the Integration
|
||||
|
||||
```bash
|
||||
# Test flacfetch-remote CLI (requires env vars)
|
||||
export FLACFETCH_API_URL=http://104.198.214.26:8080
|
||||
export FLACFETCH_API_KEY=<from-secret-manager>
|
||||
flacfetch-remote "Avril Lavigne" "Unwanted"
|
||||
|
||||
# Test backend health
|
||||
curl -s https://api.nomadkaraoke.com/api/health/detailed | jq '.dependencies.flacfetch_remote'
|
||||
|
||||
# Test backend search
|
||||
curl -s -X POST https://api.nomadkaraoke.com/api/audio-search/search \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"artist": "Avril Lavigne", "title": "Unwanted"}' | jq '.results | length'
|
||||
```
|
||||
|
||||
### Known Issues / Future Work
|
||||
|
||||
1. **Transmission removal from Cloud Run**: Transmission daemon and startup script removed from Docker images as torrent downloads are now handled by dedicated VM.
|
||||
|
||||
2. **URL field in search results**: Made optional in `AudioSearchResultResponse` since remote results don't have direct download URLs.
|
||||
|
||||
3. **Sorting improvements**: Results now prioritize:
|
||||
- Lossless first
|
||||
- Artist name match (exact match wins)
|
||||
- Release type (Album > Single > EP > Compilation > Anthology > Remix)
|
||||
- Seeders (more = higher)
|
||||
- Quality (24bit > 16bit, FLAC > MP3)
|
||||
|
||||
4. **Future enhancements**:
|
||||
- Cloud Monitoring alerts for disk/CPU/memory
|
||||
- Grafana dashboard for torrent stats
|
||||
- Web UI for manual torrent management
|
||||
|
||||
## References
|
||||
|
||||
- [Transmission RPC Spec](https://github.com/transmission/transmission/blob/main/docs/rpc-spec.md)
|
||||
- [transmission-rpc Python library](https://github.com/Trim21/transmission-rpc)
|
||||
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
|
||||
- [flacfetch source code](./flacfetch/)
|
||||
@@ -0,0 +1,265 @@
|
||||
# Countdown Padding Feature
|
||||
|
||||
This document explains how the countdown padding feature works across the karaoke-gen system, including both the standalone CLI tool and the cloud backend.
|
||||
|
||||
## Overview
|
||||
|
||||
When a song's vocals start within the first 3 seconds, karaoke singers don't have enough time to prepare. The countdown padding feature addresses this by:
|
||||
|
||||
1. Adding 3 seconds of silence to the start of the audio
|
||||
2. Shifting all lyrics timestamps forward by 3 seconds
|
||||
3. Adding a "3... 2... 1..." countdown segment (displayed at 0.1s - 2.9s)
|
||||
|
||||
This gives singers a visual countdown before the lyrics begin.
|
||||
|
||||
## Technical Details
|
||||
|
||||
### CountdownProcessor (lyrics-transcriber)
|
||||
|
||||
The core countdown logic lives in the `lyrics-transcriber` library:
|
||||
|
||||
**File:** `lyrics_transcriber/output/countdown_processor.py`
|
||||
|
||||
**Constants:**
|
||||
- `COUNTDOWN_THRESHOLD_SECONDS = 3.0` - Trigger countdown if first word starts within this time
|
||||
- `COUNTDOWN_PADDING_SECONDS = 3.0` - Amount of silence to add
|
||||
- `COUNTDOWN_START_TIME = 0.1` - When countdown text starts displaying
|
||||
- `COUNTDOWN_END_TIME = 2.9` - When countdown text ends
|
||||
- `COUNTDOWN_TEXT = "3... 2... 1..."` - The countdown text displayed
|
||||
|
||||
**Key Methods:**
|
||||
- `process(correction_result, audio_filepath)` - Main entry point. Returns `(modified_result, padded_audio_path, padding_added, padding_seconds)`
|
||||
- `_needs_countdown(correction_result)` - Checks if first word starts within 3 seconds
|
||||
- `_create_padded_audio(audio_filepath)` - Uses ffmpeg to prepend silence
|
||||
- `_add_countdown_to_result(correction_result)` - Shifts timestamps and adds countdown segment
|
||||
- `has_countdown(correction_result)` - Detects if countdown already exists (for loading saved corrections)
|
||||
|
||||
**FFmpeg Command Used:**
|
||||
```bash
|
||||
ffmpeg -y -hide_banner -loglevel error \
|
||||
-f lavfi -t 3.0 -i anullsrc=channel_layout=stereo:sample_rate=44100 \
|
||||
-i input_audio.flac \
|
||||
-filter_complex "[0:a][1:a]concat=n=2:v=0:a=1[out]" \
|
||||
-map "[out]" \
|
||||
-c:a flac \
|
||||
output_padded.flac
|
||||
```
|
||||
|
||||
This prepends 3 seconds of stereo silence (44.1kHz) and uses FLAC to preserve audio quality.
|
||||
|
||||
## Workflow: Standalone CLI (karaoke-gen)
|
||||
|
||||
When using the karaoke-gen CLI tool locally:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ STANDALONE CLI WORKFLOW │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ 1. LyricsTranscriber.process() │
|
||||
│ └─> add_countdown=True (default) │
|
||||
│ └─> CountdownProcessor adds countdown + pads vocals audio │
|
||||
│ └─> Returns: padded_audio_filepath, countdown_padding_added │
|
||||
│ │
|
||||
│ 2. Audio Separation │
|
||||
│ └─> Creates instrumental files (original timing) │
|
||||
│ │
|
||||
│ 3. Apply Padding to Instrumentals │
|
||||
│ └─> AudioProcessor.apply_countdown_padding_to_instrumentals() │
|
||||
│ └─> Pads all instrumental variants to match vocals │
|
||||
│ └─> Creates files with "(Padded)" suffix │
|
||||
│ │
|
||||
│ 4. Final Video Generation │
|
||||
│ └─> Uses padded vocals + padded instrumental │
|
||||
│ └─> Everything is synchronized │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Key Files:**
|
||||
- `karaoke_gen/lyrics_processor.py` - Calls LyricsTranscriber, captures padding info
|
||||
- `karaoke_gen/audio_processor.py` - `apply_countdown_padding_to_instrumentals()` and `pad_audio_file()`
|
||||
- `karaoke_gen/karaoke_gen.py` - Orchestrates the workflow, applies padding after separation
|
||||
|
||||
**Instrumental File Naming:**
|
||||
- Original: `Artist - Song (Instrumental model.ckpt).flac`
|
||||
- Padded: `Artist - Song (Instrumental model.ckpt) (Padded).flac`
|
||||
|
||||
## Workflow: Cloud Backend
|
||||
|
||||
The cloud backend has a two-phase workflow with human review in between. This requires special handling to avoid sync issues.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ CLOUD BACKEND WORKFLOW │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ PHASE 1: Lyrics Worker (lyrics_worker.py) │
|
||||
│ ───────────────────────────────────────── │
|
||||
│ └─> LyricsTranscriber.process() │
|
||||
│ └─> add_countdown=FALSE <── Deferred to Phase 2 │
|
||||
│ └─> Corrections saved WITHOUT countdown │
|
||||
│ └─> Original audio timing preserved │
|
||||
│ │
|
||||
│ HUMAN REVIEW (Review UI) │
|
||||
│ ──────────────────────── │
|
||||
│ └─> User reviews/edits lyrics │
|
||||
│ └─> Audio playback uses original (un-padded) audio │
|
||||
│ └─> Timestamps match audio perfectly │
|
||||
│ │
|
||||
│ PHASE 2: Render Video Worker (render_video_worker.py) │
|
||||
│ ───────────────────────────────────────────────────── │
|
||||
│ └─> CountdownProcessor.process() called HERE │
|
||||
│ └─> Adds countdown segment + shifts timestamps │
|
||||
│ └─> Creates padded audio for video rendering │
|
||||
│ └─> Final video has countdown + synced audio │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Why Defer Countdown to Phase 2?**
|
||||
|
||||
Previously, countdown was added during Phase 1 (lyrics transcription). This caused a problem:
|
||||
|
||||
1. Lyrics corrections were saved with countdown segment + shifted timestamps
|
||||
2. The padded audio file was created in a temp directory that was deleted
|
||||
3. Review UI served the original (un-padded) audio
|
||||
4. **Result:** 3-second sync mismatch between lyrics and audio in review UI
|
||||
|
||||
By deferring countdown to Phase 2:
|
||||
- Review UI shows original timing (audio and lyrics are perfectly synced)
|
||||
- Users can accurately review and edit lyrics
|
||||
- Countdown is added just before final video rendering
|
||||
|
||||
**Key Files:**
|
||||
- `karaoke_gen/lyrics_processor.py:405` - Sets `add_countdown=not is_serverless`
|
||||
- `backend/workers/render_video_worker.py:188-200` - Adds countdown using `CountdownProcessor.process()`
|
||||
|
||||
**Serverless Detection:**
|
||||
```python
|
||||
is_serverless = (
|
||||
os.getenv("MODAL_TASK_ID") is not None or
|
||||
os.getenv("MODAL_FUNCTION_NAME") is not None or
|
||||
os.path.exists("/.modal")
|
||||
)
|
||||
```
|
||||
|
||||
## Finalization Step (karaoke_finalise)
|
||||
|
||||
The `KaraokeFinalise` class handles creating the final karaoke video with instrumental audio.
|
||||
|
||||
**File:** `karaoke_gen/karaoke_finalise/karaoke_finalise.py`
|
||||
|
||||
It includes a safety net in `remux_with_instrumental()`:
|
||||
- If `countdown_padding_seconds` is set but instrumental doesn't have "(Padded)" in the name
|
||||
- It automatically pads the instrumental before remuxing
|
||||
- This ensures vocals and instrumental stay synchronized
|
||||
|
||||
## Detecting Existing Countdown
|
||||
|
||||
When loading saved corrections (e.g., from a JSON file), you can detect if countdown was applied:
|
||||
|
||||
```python
|
||||
countdown_processor = CountdownProcessor(cache_dir=temp_dir)
|
||||
has_countdown = countdown_processor.has_countdown(correction_result)
|
||||
|
||||
# Or check for countdown text in first segment
|
||||
first_segment = correction_result.corrected_segments[0]
|
||||
is_countdown = first_segment.text == "3... 2... 1..."
|
||||
```
|
||||
|
||||
You can also detect from LRC files:
|
||||
```python
|
||||
# LyricsProcessor._detect_countdown_padding_from_lrc()
|
||||
# Checks for "3... 2... 1..." text or first timestamp >= 2.5s
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Disabling Countdown
|
||||
|
||||
To disable countdown for a specific transcription:
|
||||
|
||||
```python
|
||||
from lyrics_transcriber.core.config import OutputConfig
|
||||
|
||||
output_config = OutputConfig(
|
||||
# ... other options
|
||||
add_countdown=False, # Disable countdown
|
||||
)
|
||||
```
|
||||
|
||||
### Padding Amount
|
||||
|
||||
The padding amount is currently fixed at 3.0 seconds. To change this, you would need to modify the constants in `CountdownProcessor`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Countdown appears but audio is out of sync
|
||||
|
||||
**Cause:** Instrumental audio wasn't padded to match vocals.
|
||||
|
||||
**Solution:**
|
||||
- Check that `countdown_padding_added` was captured from transcription
|
||||
- Verify `apply_countdown_padding_to_instrumentals()` was called
|
||||
- Look for "(Padded)" in instrumental filenames
|
||||
|
||||
### Issue: Review UI shows 3-second offset
|
||||
|
||||
**Cause:** Countdown was added during transcription instead of rendering phase.
|
||||
|
||||
**Solution:**
|
||||
- This was fixed in v0.76.1
|
||||
- Ensure `add_countdown=False` for serverless mode
|
||||
- Countdown should only be added in `render_video_worker.py`
|
||||
|
||||
### Issue: No countdown even though song starts immediately
|
||||
|
||||
**Cause:** First word timestamp might be exactly at or after 3.0 seconds.
|
||||
|
||||
**Check:**
|
||||
- Look at the first segment's first word start_time
|
||||
- Countdown triggers only if `first_word_start < 3.0`
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
Countdown padding tests are in `tests/unit/test_countdown_padding.py`:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/unit/test_countdown_padding.py -v
|
||||
```
|
||||
|
||||
Tests cover:
|
||||
- `pad_audio_file()` success, failure, and timeout
|
||||
- `apply_countdown_padding_to_instrumentals()` for clean and combined instrumentals
|
||||
- Integration with `LyricsProcessor` and `KaraokeGen`
|
||||
- Custom instrumental padding
|
||||
- Idempotency (skipping already-padded files)
|
||||
|
||||
## Data Flow Summary
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ DATA FLOW │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Input: Song with vocals starting at 1.5s │
|
||||
│ │
|
||||
│ After Countdown Processing: │
|
||||
│ ───────────────────────────── │
|
||||
│ • Audio: 3s silence + original audio │
|
||||
│ • First segment: "3... 2... 1..." at 0.1s-2.9s │
|
||||
│ • Original first word: moved from 1.5s → 4.5s │
|
||||
│ • All subsequent timestamps: shifted +3.0s │
|
||||
│ │
|
||||
│ Result: Singer sees countdown, has time to prepare │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Version History
|
||||
|
||||
- **v0.76.1** - Fixed cloud backend sync issue by deferring countdown to render phase
|
||||
- **v0.70.x** - Initial countdown padding implementation integrated from lyrics-transcriber
|
||||
@@ -0,0 +1,481 @@
|
||||
# Frontend Guide - Karaoke Generator Web UI
|
||||
|
||||
## Overview
|
||||
|
||||
The karaoke-gen frontend is a Next.js-based static web application deployed to GitHub Pages at **https://gen.nomadkaraoke.com**. It provides a web interface for submitting jobs to the backend API, monitoring job progress, and managing karaoke video generation.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Deployment Topology
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ gen.nomadkaraoke.com (GitHub Pages) │
|
||||
│ - Next.js static export │
|
||||
│ - Pure client-side React application │
|
||||
│ - No server-side rendering │
|
||||
└────────────────┬────────────────────────────────────────┘
|
||||
│
|
||||
│ API calls (HTTPS)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ api.nomadkaraoke.com (Cloud Run) │
|
||||
│ - FastAPI backend │
|
||||
│ - Requires Bearer token authentication │
|
||||
│ - Manages jobs, storage, workers │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
- **Frontend Domain**: `gen.nomadkaraoke.com` → GitHub Pages (static HTML/CSS/JS)
|
||||
- **Backend API**: `api.nomadkaraoke.com` → Google Cloud Run
|
||||
- **Framework**: Next.js 16.0.7 with Turbopack (static export mode)
|
||||
- **UI Library**: React 19 + shadcn/ui components
|
||||
- **State Management**: React hooks (useState, useEffect)
|
||||
- **Storage**: localStorage for auth tokens (client-side only)
|
||||
|
||||
## Authentication
|
||||
|
||||
### How It Works
|
||||
|
||||
The backend API requires **Bearer token authentication** for all requests. The frontend implements client-side token management:
|
||||
|
||||
1. **Token Storage**: Tokens are stored in browser localStorage
|
||||
2. **Token Input**: Users enter tokens via the AuthBanner component
|
||||
3. **API Requests**: All API calls include `Authorization: Bearer <token>` header
|
||||
4. **Token Persistence**: Tokens persist across browser sessions until cleared
|
||||
|
||||
### Getting a Token
|
||||
|
||||
Tokens are managed by administrators. Users need to contact an admin to receive an access token. There is no self-service signup.
|
||||
|
||||
**Admin token management**: Admin tokens are stored in Google Cloud Secret Manager (`ADMIN_TOKENS` secret).
|
||||
|
||||
### Using Authentication in the UI
|
||||
|
||||
```typescript
|
||||
// Import auth functions
|
||||
import { setAccessToken, getAccessToken, clearAccessToken } from '@/lib/api';
|
||||
|
||||
// Set token (after user input)
|
||||
setAccessToken('your-token-here');
|
||||
|
||||
// Get current token
|
||||
const token = getAccessToken();
|
||||
|
||||
// Clear token (logout)
|
||||
clearAccessToken();
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
### 1. Job Submission
|
||||
|
||||
Three methods for submitting karaoke generation jobs:
|
||||
|
||||
#### File Upload
|
||||
- Upload audio files directly (FLAC, WAV, MP3, etc.)
|
||||
- Provide artist and title metadata
|
||||
- Automatic processing starts immediately
|
||||
|
||||
#### YouTube URL
|
||||
- Submit a YouTube video URL
|
||||
- Backend downloads audio automatically
|
||||
- Provide artist and title metadata
|
||||
|
||||
#### Audio Search
|
||||
- Search by artist and title
|
||||
- Backend finds high-quality audio sources
|
||||
- Select from multiple results
|
||||
- Automatic download and processing
|
||||
|
||||
### 2. Job Monitoring
|
||||
|
||||
Real-time job status tracking with:
|
||||
- **Status badges**: Visual indicators (Pending, Processing, Awaiting Review, etc.)
|
||||
- **Progress bars**: Visual progress for active jobs
|
||||
- **Auto-refresh**: Updates every 10 seconds
|
||||
- **Manual refresh**: Click refresh button anytime
|
||||
|
||||
**Job Statuses**:
|
||||
- `PENDING` - Job created, waiting to start
|
||||
- `DOWNLOADING` - Fetching audio
|
||||
- `PROCESSING_AUDIO` - Separating vocals/instrumental
|
||||
- `PROCESSING_LYRICS` - Transcribing lyrics
|
||||
- `AWAITING_REVIEW` - Ready for human review
|
||||
- `AWAITING_INSTRUMENTAL_SELECTION` - User needs to select instrumental
|
||||
- `RENDERING_VIDEO` - Creating final video
|
||||
- `UPLOADING` - Distributing to YouTube/Dropbox
|
||||
- `COMPLETE` - Finished successfully
|
||||
- `FAILED` - Error occurred
|
||||
- `CANCELLED` - User cancelled
|
||||
|
||||
### 3. Interactive Elements
|
||||
|
||||
#### Lyrics Review
|
||||
- Launch lyrics correction UI in new tab
|
||||
- Make timing and text corrections
|
||||
- Submit corrections to trigger video rendering
|
||||
|
||||
#### Instrumental Selection
|
||||
- Preview multiple instrumental options
|
||||
- Select best quality instrumental
|
||||
- Audio player for A/B comparison
|
||||
|
||||
#### Job Actions
|
||||
- **Cancel**: Stop a running job
|
||||
- **Retry**: Restart a failed job
|
||||
- **Delete**: Remove job and associated files
|
||||
- **View Logs**: See detailed processing logs
|
||||
|
||||
### 4. Output Access
|
||||
|
||||
When jobs complete, access outputs via:
|
||||
- **YouTube**: Direct link to uploaded video
|
||||
- **Dropbox**: Link to Dropbox folder with files
|
||||
- **Direct Downloads**: Download MP4, CDG, TXT files
|
||||
|
||||
## Component Structure
|
||||
|
||||
### Modular Architecture
|
||||
|
||||
```
|
||||
frontend/
|
||||
├── app/
|
||||
│ ├── page.tsx # Main application page
|
||||
│ ├── layout.tsx # Root layout with metadata
|
||||
│ └── globals.css # Global styles and CSS variables
|
||||
│
|
||||
├── components/
|
||||
│ ├── auth-banner.tsx # Authentication UI
|
||||
│ │
|
||||
│ ├── job/ # Job-related components
|
||||
│ │ ├── JobCard.tsx # Individual job display
|
||||
│ │ ├── JobSubmission.tsx # Job creation forms
|
||||
│ │ ├── JobActions.tsx # Job management buttons
|
||||
│ │ ├── JobLogs.tsx # Log viewer
|
||||
│ │ ├── OutputLinks.tsx # Download/distribution links
|
||||
│ │ └── InstrumentalSelector.tsx # Instrumental picker
|
||||
│ │
|
||||
│ ├── audio-search/ # Audio search feature
|
||||
│ │ └── AudioSearchDialog.tsx
|
||||
│ │
|
||||
│ └── ui/ # Reusable UI primitives (shadcn/ui)
|
||||
│ ├── button.tsx
|
||||
│ ├── card.tsx
|
||||
│ ├── dialog.tsx
|
||||
│ ├── tabs.tsx
|
||||
│ └── ... (more)
|
||||
│
|
||||
└── lib/
|
||||
├── api.ts # API client (all backend calls)
|
||||
├── types.ts # TypeScript type definitions
|
||||
└── utils.ts # Utility functions
|
||||
```
|
||||
|
||||
### Key Files
|
||||
|
||||
#### `lib/api.ts` - API Client
|
||||
|
||||
The central hub for all backend communication:
|
||||
|
||||
```typescript
|
||||
const API_BASE_URL = process.env.NEXT_PUBLIC_API_URL || 'https://api.nomadkaraoke.com';
|
||||
|
||||
export const api = {
|
||||
// Authentication
|
||||
setAccessToken(token: string): void
|
||||
getAccessToken(): string | null
|
||||
clearAccessToken(): void
|
||||
|
||||
// Job Management
|
||||
listJobs(params?: { status?: string; limit?: number }): Promise<Job[]>
|
||||
getJob(jobId: string): Promise<Job>
|
||||
uploadJob(file: File, artist: string, title: string): Promise<JobResponse>
|
||||
createJobFromUrl(body: CreateJobFromUrlRequest): Promise<JobResponse>
|
||||
cancelJob(jobId: string, reason?: string): Promise<void>
|
||||
retryJob(jobId: string): Promise<void>
|
||||
deleteJob(jobId: string): Promise<void>
|
||||
|
||||
// Audio Search
|
||||
searchAudio(request: AudioSearchRequest): Promise<AudioSearchResponse>
|
||||
getAudioSearchResults(jobId: string): Promise<AudioSearchResult[]>
|
||||
selectAudioResult(jobId: string, resultIndex: number): Promise<void>
|
||||
|
||||
// Lyrics Review
|
||||
getReviewData(jobId: string): Promise<ReviewData>
|
||||
completeReview(jobId: string, corrections: any): Promise<void>
|
||||
|
||||
// Instrumental Selection
|
||||
getInstrumentalOptions(jobId: string): Promise<InstrumentalOption[]>
|
||||
selectInstrumental(jobId: string, sourceIndex: number): Promise<void>
|
||||
|
||||
// Outputs
|
||||
getDownloadUrls(jobId: string): Promise<DownloadUrls>
|
||||
}
|
||||
```
|
||||
|
||||
#### `components/auth-banner.tsx` - Authentication UI
|
||||
|
||||
Prominent banner at top of page for token management:
|
||||
- Shows "Authentication Required" when no token
|
||||
- Displays truncated token when authenticated
|
||||
- Password-masked input field
|
||||
- Persistent across page reloads
|
||||
|
||||
#### `components/job/JobCard.tsx` - Job Display
|
||||
|
||||
Main component for displaying individual jobs:
|
||||
- Collapsible details view
|
||||
- Status badges and progress bars
|
||||
- Conditional rendering based on job state
|
||||
- Integrates all sub-components (actions, logs, outputs, etc.)
|
||||
|
||||
## Styling
|
||||
|
||||
### Color Scheme
|
||||
|
||||
Matches the instrumental review UI for consistency:
|
||||
|
||||
```css
|
||||
/* Background colors */
|
||||
--bg: #0f0f0f; /* Very dark, neutral */
|
||||
--card: #1a1a1a; /* Card backgrounds */
|
||||
|
||||
/* Text colors */
|
||||
--text: #e5e5e5; /* Primary text */
|
||||
--text-muted: #999999; /* Secondary text */
|
||||
|
||||
/* UI colors */
|
||||
--primary: #3b82f6; /* Blue (actions, links) */
|
||||
--success: #10b981; /* Green (success states) */
|
||||
--warning: #f59e0b; /* Orange (warnings) */
|
||||
--error: #ef4444; /* Red (errors) */
|
||||
--border: #2a2a2a; /* Borders */
|
||||
```
|
||||
|
||||
### Typography
|
||||
|
||||
System font stack for fast loading and native feel:
|
||||
|
||||
```css
|
||||
--font-sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
--font-mono: "SF Mono", Monaco, "Cascadia Code", "Courier New", monospace;
|
||||
```
|
||||
|
||||
## Local Development
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
|
||||
# Install dependencies
|
||||
npm ci --legacy-peer-deps
|
||||
|
||||
# Start dev server with hot-reload
|
||||
npm run dev
|
||||
# → http://localhost:3000
|
||||
|
||||
# Build static export
|
||||
npm run build
|
||||
# → Output: frontend/out/
|
||||
|
||||
# Run tests
|
||||
npm test
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Create `.env.local` for local development:
|
||||
|
||||
```bash
|
||||
# Optional: Override API URL for local backend testing
|
||||
NEXT_PUBLIC_API_URL=http://localhost:8080
|
||||
```
|
||||
|
||||
### Testing Against Production API
|
||||
|
||||
The default configuration connects to the production API at `api.nomadkaraoke.com`, so you can develop locally while using the real backend.
|
||||
|
||||
**Important**: You'll need a valid access token to make API requests.
|
||||
|
||||
## Deployment
|
||||
|
||||
### Automatic Deployment (CI/CD)
|
||||
|
||||
Changes to `main` branch automatically deploy via GitHub Actions:
|
||||
|
||||
1. **Trigger**: Push to `main` or merge PR
|
||||
2. **Build**: `npm ci --legacy-peer-deps && npm run build`
|
||||
3. **Deploy**: Upload `frontend/out/` to GitHub Pages
|
||||
4. **Live**: Available at `gen.nomadkaraoke.com` within ~2 minutes
|
||||
|
||||
### CI Workflow
|
||||
|
||||
`.github/workflows/ci.yml`:
|
||||
|
||||
```yaml
|
||||
deploy-frontend:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
- name: Build
|
||||
working-directory: frontend
|
||||
run: npm run build
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-pages-artifact@v3
|
||||
with:
|
||||
path: frontend/out
|
||||
```
|
||||
|
||||
### Manual Deployment
|
||||
|
||||
If needed, manually deploy via GitHub Pages settings:
|
||||
1. Build: `cd frontend && npm run build`
|
||||
2. Upload `frontend/out/` contents to GitHub Pages
|
||||
3. Wait for GitHub Pages to rebuild
|
||||
|
||||
## Configuration Files
|
||||
|
||||
### `next.config.mjs`
|
||||
|
||||
```javascript
|
||||
/** @type {import('next').NextConfig} */
|
||||
const nextConfig = {
|
||||
output: 'export', // Static export for GitHub Pages
|
||||
trailingSlash: true, // Ensures paths work on static hosts
|
||||
typescript: {
|
||||
ignoreBuildErrors: false // Enforce type safety
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### `package.json`
|
||||
|
||||
Key dependencies:
|
||||
- `next@16.0.7` - Framework
|
||||
- `react@19.0.0` - UI library
|
||||
- `lucide-react` - Icon library
|
||||
- `tailwindcss` - Utility-first CSS
|
||||
- `@testing-library/react` - Testing
|
||||
|
||||
## Common Issues
|
||||
|
||||
### Issue: API requests return 404
|
||||
|
||||
**Cause**: Frontend trying to call `gen.nomadkaraoke.com/api` instead of `api.nomadkaraoke.com/api`
|
||||
|
||||
**Solution**: Verify `API_BASE_URL` in `lib/api.ts` is set to `https://api.nomadkaraoke.com`
|
||||
|
||||
### Issue: API requests return 401 Unauthorized
|
||||
|
||||
**Cause**: No token or invalid token
|
||||
|
||||
**Solution**: Enter a valid access token via the AuthBanner
|
||||
|
||||
### Issue: Changes not appearing after deployment
|
||||
|
||||
**Cause**: GitHub Pages caching or CDN propagation delay
|
||||
|
||||
**Solutions**:
|
||||
- Wait 2-5 minutes for CDN to update
|
||||
- Hard refresh browser (Cmd/Ctrl + Shift + R)
|
||||
- Check GitHub Actions for deployment status
|
||||
|
||||
### Issue: Build fails in CI with dependency conflicts
|
||||
|
||||
**Cause**: React 19 peer dependency mismatch with testing libraries
|
||||
|
||||
**Solution**: Use `npm ci --legacy-peer-deps` (already configured in CI)
|
||||
|
||||
## Testing
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
|
||||
# Run all tests
|
||||
npm test
|
||||
|
||||
# Run with coverage
|
||||
npm run test:coverage
|
||||
|
||||
# Run in CI mode (non-interactive)
|
||||
npm run test:ci
|
||||
```
|
||||
|
||||
### Test Structure
|
||||
|
||||
```
|
||||
frontend/
|
||||
├── components/__tests__/
|
||||
│ ├── JobCard.test.tsx
|
||||
│ └── ...
|
||||
└── lib/__tests__/
|
||||
├── api.test.ts
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Coverage Requirements
|
||||
|
||||
Minimum 70% code coverage enforced in CI (though frontend tests are still being expanded).
|
||||
|
||||
## API Integration
|
||||
|
||||
### Request Flow
|
||||
|
||||
1. User action in UI (e.g., click "Create Job")
|
||||
2. Component calls API function from `lib/api.ts`
|
||||
3. API function adds auth header if token exists
|
||||
4. Fetch request to `api.nomadkaraoke.com`
|
||||
5. Response parsed and returned
|
||||
6. Component updates UI with response data
|
||||
|
||||
### Error Handling
|
||||
|
||||
All API errors are wrapped in `ApiError` class:
|
||||
|
||||
```typescript
|
||||
try {
|
||||
const jobs = await api.listJobs({ limit: 20 });
|
||||
setJobs(jobs);
|
||||
} catch (error) {
|
||||
if (error instanceof ApiError) {
|
||||
console.error(`API Error ${error.status}: ${error.message}`);
|
||||
// error.data contains full response body
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential improvements for future iterations:
|
||||
|
||||
1. **OAuth Integration**: Replace manual token entry with OAuth flow
|
||||
2. **Real-time Updates**: WebSocket connection for live job updates
|
||||
3. **Admin Dashboard**: User management, token generation, system stats
|
||||
4. **Job History**: Pagination and filtering for large job lists
|
||||
5. **Batch Operations**: Select and manage multiple jobs at once
|
||||
6. **Mobile Optimization**: Responsive design improvements for mobile devices
|
||||
7. **Offline Support**: Service worker for offline functionality
|
||||
8. **Enhanced Testing**: Increase test coverage to 90%+
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Backend API Reference](../README.md#backend-api-reference)
|
||||
- [Cloud Run Deployment](../03-deployment/CLOUD-RUN-DEPLOYMENT.md)
|
||||
- [Architecture Overview](./ARCHITECTURE.md)
|
||||
- [GitHub Pages Setup](.github/workflows/ci.yml)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: December 2025
|
||||
**Frontend Version**: Deployed from `main` branch commit `91f51d6`
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
# Instrumental Review: Original Audio Tab Support
|
||||
|
||||
**Date:** 2025-12-22
|
||||
**Version:** 0.75.12
|
||||
|
||||
## Summary
|
||||
|
||||
Added support for an "Original Audio" tab in the Instrumental Review UI, allowing users to compare the original audio (with lead vocals) alongside the backing vocals, clean instrumental, and instrumental with backing vocals options. This helps users make better decisions about whether certain audio regions are actually backing vocals or need to be muted.
|
||||
|
||||
## Problem
|
||||
|
||||
When reviewing backing vocals in the Instrumental Review UI, users couldn't hear the original full song to compare and determine whether detected regions were actually backing vocals or if they needed to be muted. This made it difficult to make informed decisions about the instrumental selection.
|
||||
|
||||
## Solution
|
||||
|
||||
The Instrumental Review HTML UI already had built-in support for displaying an "Original Audio" tab when `hasOriginal` flag was set, but this feature was not being activated because the `original_audio_path` parameter was not being passed to the `InstrumentalReviewServer`.
|
||||
|
||||
### Changes Made
|
||||
|
||||
1. **`karaoke_gen/utils/gen_cli.py`** (lines 203-217, 239):
|
||||
- Added code to locate the original audio file from the track data (`track.get("input_audio_wav")`)
|
||||
- Resolved the path relative to the track directory
|
||||
- Passed `original_audio_path` parameter when initializing `InstrumentalReviewServer`
|
||||
|
||||
2. **`scripts/test-instrumental-review.py`**:
|
||||
- Added optional `--original` command-line argument
|
||||
- Added validation for the original audio file path
|
||||
- Passed `original_audio_path` parameter to `InstrumentalReviewServer`
|
||||
|
||||
3. **`tests/unit/test_instrumental_review/test_server.py`**:
|
||||
- Added `test_server_initialization_with_original_audio()` to verify server properly stores the original audio path
|
||||
- Added `test_get_analysis_with_original_audio()` to verify the API properly exposes `has_original` flag and original audio URL
|
||||
|
||||
4. **`pyproject.toml`**:
|
||||
- Bumped version from 0.75.11 to 0.75.12
|
||||
|
||||
## How It Works
|
||||
|
||||
### Backend Flow
|
||||
|
||||
1. When the instrumental review is triggered, the system:
|
||||
- Locates the original audio file (`input_audio_wav` from the track data)
|
||||
- Resolves the path relative to the working directory
|
||||
- Passes it to `InstrumentalReviewServer` as `original_audio_path`
|
||||
|
||||
2. The server exposes the original audio through:
|
||||
- `/api/audio/original` endpoint (serves the audio file)
|
||||
- `has_original: true` flag in the analysis response
|
||||
|
||||
### Frontend Flow
|
||||
|
||||
The HTML UI (already implemented in `karaoke_gen/instrumental_review/static/index.html`):
|
||||
- Checks `hasOriginal` flag from the API
|
||||
- Conditionally shows "Original" tab in the audio toggle group (line 697-699)
|
||||
- Provides audio player that switches between:
|
||||
- **Original Audio** - Full song with lead vocals
|
||||
- **Backing Vocals Only** - Isolated backing vocals
|
||||
- **Pure Instrumental** - Clean instrumental (no vocals)
|
||||
- **Instrumental + Backing** - Instrumental with backing vocals included
|
||||
- **Custom** - User-created muted regions
|
||||
|
||||
### Usage
|
||||
|
||||
When running the local CLI:
|
||||
```bash
|
||||
karaoke-gen "Artist" "Song Title"
|
||||
```
|
||||
|
||||
During the instrumental review phase, the UI will now automatically show the "Original Audio" tab if the original audio file is available.
|
||||
|
||||
For testing:
|
||||
```bash
|
||||
python scripts/test-instrumental-review.py \
|
||||
backing_vocals.flac \
|
||||
clean_instrumental.flac \
|
||||
--with_backing instrumental_with_backing.flac \
|
||||
--original original.wav
|
||||
```
|
||||
|
||||
## Files Modified
|
||||
|
||||
- `karaoke_gen/utils/gen_cli.py` - Added original audio path detection and passing
|
||||
- `scripts/test-instrumental-review.py` - Added --original argument support
|
||||
- `tests/unit/test_instrumental_review/test_server.py` - Added tests for original audio functionality
|
||||
- `pyproject.toml` - Version bump
|
||||
|
||||
## Testing
|
||||
|
||||
All 139 tests in `tests/unit/test_instrumental_review/` pass, including:
|
||||
- `TestServerInitialization::test_server_initialization_with_original_audio` - Verifies server stores original audio path
|
||||
- `TestServerAPIEndpoints::test_get_analysis_with_original_audio` - Verifies API exposes original audio properly
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Better Decision Making**: Users can hear the full original song to compare against backing vocals
|
||||
2. **Improved Accuracy**: Easier to identify if detected regions are truly backing vocals or artifacts
|
||||
3. **No UI Changes Required**: Leveraged existing UI capabilities that were already built-in
|
||||
4. **Backward Compatible**: If no original audio is provided, the tab simply doesn't appear
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The original audio feature is optional - if the file is not available, the tab won't appear
|
||||
- The server already had the route handler for serving original audio (`/api/audio/original`)
|
||||
- The HTML UI already had the conditional rendering logic for the "Original" tab
|
||||
- The main change was ensuring the `original_audio_path` parameter was properly populated and passed through
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
# CLI + Backend Integration Testing
|
||||
|
||||
## The Problem
|
||||
|
||||
We had multiple bugs that unit tests didn't catch:
|
||||
- Content-type mismatch in signed URL uploads (403 errors)
|
||||
- Missing auth headers in download requests (all downloads failed)
|
||||
- YouTube description field name mismatch (uploads silently failed)
|
||||
- Style file path handling issues
|
||||
|
||||
These bugs required:
|
||||
1. Full CI/CD deploy (~12 min)
|
||||
2. Full karaoke generation (~10 min)
|
||||
3. Manual inspection of logs
|
||||
|
||||
**Total: 20+ minutes per iteration** 😱
|
||||
|
||||
## The Solution
|
||||
|
||||
New **CLI + Backend Integration Tests** that:
|
||||
- Run the ACTUAL CLI code against a LOCAL backend
|
||||
- Use GCP emulators (Firestore + GCS)
|
||||
- Test the full signed URL upload flow
|
||||
- Run in **< 30 seconds**
|
||||
|
||||
## Test Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ Unit Tests (Mocked) │ ← Fast, isolated
|
||||
│ tests/unit/ │ ~0.5s, 100+ tests
|
||||
├─────────────────────────────────────┤
|
||||
│ Backend Emulator Tests │ ← Backend-only integration
|
||||
│ backend/tests/emulator/ │ ~10s, uses Firestore/GCS
|
||||
│ - test_emulator_integration.py │ Basic CRUD operations
|
||||
│ - test_e2e_cli_backend.py │ Full CLI → Backend flow
|
||||
├─────────────────────────────────────┤
|
||||
│ Cloud Integration Tests │ ← Against deployed backend
|
||||
│ backend/tests/test_api_integration│ ~10min, pre-prod only
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Run All Integration Tests
|
||||
|
||||
```bash
|
||||
./scripts/run-cli-integration-tests.sh
|
||||
```
|
||||
|
||||
This automatically:
|
||||
1. Starts GCP emulators
|
||||
2. Starts local backend
|
||||
3. Runs tests
|
||||
4. Cleans up
|
||||
|
||||
### Interactive Development Mode
|
||||
|
||||
```bash
|
||||
# Terminal 1: Start environment
|
||||
./scripts/start-local-test-env.sh
|
||||
|
||||
# Terminal 2: Run specific tests repeatedly
|
||||
pytest tests/integration/test_cli_backend_integration.py -v -k "test_style_params"
|
||||
```
|
||||
|
||||
## What These Tests Catch
|
||||
|
||||
### ✅ Content-Type Mismatches
|
||||
```python
|
||||
def test_style_params_json_upload(self, test_style_files, cli_config):
|
||||
"""Test uploading style_params.json via signed URL."""
|
||||
# This would have caught the Path('.json').suffix bug!
|
||||
result = client.search_audio(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
style_params_path=test_style_files['style_params_path'],
|
||||
)
|
||||
# If we get 403, the test fails with a clear message
|
||||
```
|
||||
|
||||
### ✅ Request/Response Format Mismatches
|
||||
Tests verify CLI and backend agree on API contract.
|
||||
|
||||
### ✅ GCS Path Issues
|
||||
Uploads go to real (emulated) GCS, catches path/encoding bugs.
|
||||
|
||||
### ✅ Auth Token Handling
|
||||
Tests run with actual token flow.
|
||||
|
||||
### ✅ Download Auth Headers
|
||||
```python
|
||||
def test_download_uses_session_not_requests(self, cli_client, tmp_path):
|
||||
"""Verify downloads use session.get() not requests.get()."""
|
||||
# This catches the bug where downloads failed with 401/403
|
||||
# because auth headers weren't included
|
||||
```
|
||||
|
||||
### ✅ Cross-Component Field Mapping
|
||||
```python
|
||||
def test_job_has_youtube_description_template(self, client, auth_headers):
|
||||
"""Test youtube_description AND youtube_description_template are set."""
|
||||
# audio_search sets youtube_description
|
||||
# video_worker reads youtube_description_template
|
||||
# Both must be set for YouTube upload to work!
|
||||
```
|
||||
|
||||
## When to Run
|
||||
|
||||
| Situation | Run This |
|
||||
|-----------|----------|
|
||||
| Quick check before commit | `./scripts/run-emulator-tests.sh` |
|
||||
| Testing CLI changes | `./scripts/run-cli-integration-tests.sh` |
|
||||
| Testing backend API changes | `./scripts/run-cli-integration-tests.sh` |
|
||||
| Before creating PR | Both of the above |
|
||||
| Pre-production | Cloud integration tests |
|
||||
|
||||
## Test Files
|
||||
|
||||
### `backend/tests/emulator/test_e2e_cli_backend.py` (Primary)
|
||||
|
||||
Key test classes:
|
||||
- `TestStyleFileUploadE2E` - Full style file upload flow with content-type verification
|
||||
- `TestYouTubeDescriptionFieldMapping` - Verifies youtube_description_template is set
|
||||
- `TestDownloadAuthHeaders` - Verifies auth headers are sent with downloads
|
||||
- `TestFullAudioSearchFlow` - Complete audio search → job creation flow
|
||||
- `TestCLIClientIntegration` - Tests actual CLI client code
|
||||
|
||||
### `tests/integration/test_cli_backend_integration.py` (Legacy)
|
||||
|
||||
- `TestSignedUrlUploadRoundTrip` - Full upload flow
|
||||
- `TestContentTypeMatching` - Content-type bug documentation
|
||||
- `TestAudioSearchWithStyles` - API contract tests
|
||||
|
||||
### Test Fixtures
|
||||
|
||||
- `test_style_files` - Creates minimal valid PNG/TTF/JSON files
|
||||
- `cli_config` - Points CLI at local backend
|
||||
|
||||
## Extending Tests
|
||||
|
||||
### Adding a New Integration Test
|
||||
|
||||
```python
|
||||
class TestMyNewFeature:
|
||||
def test_my_feature(self, cli_config, tmp_path):
|
||||
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, RemoteConfig
|
||||
import logging
|
||||
|
||||
config = RemoteConfig(
|
||||
service_url=cli_config['service_url'],
|
||||
auth_token=cli_config['auth_token'],
|
||||
environment='test'
|
||||
)
|
||||
client = RemoteKaraokeClient(config, logging.getLogger("test"))
|
||||
|
||||
# Test your feature
|
||||
result = client.some_method(...)
|
||||
assert result['expected_field'] == expected_value
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Docker** (for fake-gcs-server)
|
||||
2. **gcloud CLI** with Firestore emulator:
|
||||
```bash
|
||||
gcloud components install cloud-firestore-emulator
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Emulators not running"
|
||||
```bash
|
||||
./scripts/start-emulators.sh
|
||||
```
|
||||
|
||||
### "Backend failed to start"
|
||||
Check if port 8000 is already in use:
|
||||
```bash
|
||||
lsof -i :8000
|
||||
```
|
||||
|
||||
### Tests pass locally but fail in CI
|
||||
The integration tests require Docker, which may not be available in all CI environments. Consider:
|
||||
- Running in a CI job with Docker support
|
||||
- Skipping in CI and running locally before merge
|
||||
|
||||
## Benefits
|
||||
|
||||
| Before | After |
|
||||
|--------|-------|
|
||||
| 20+ min per iteration | < 30 sec per iteration |
|
||||
| Manual log inspection | Automated assertions |
|
||||
| Bugs found in production | Bugs caught locally |
|
||||
| Expensive cloud resources | Free local emulators |
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
# Backend API Authentication - Remaining Work
|
||||
|
||||
## Overview
|
||||
|
||||
This document tracks the progress of adding authentication requirements to ALL backend API endpoints. Currently, many endpoints are publicly accessible without any authentication, which is a **major security vulnerability**.
|
||||
|
||||
## Completed Endpoints
|
||||
|
||||
### ✅ jobs.py (20 endpoints) - DONE
|
||||
- All job management endpoints now require `Depends(require_auth)`
|
||||
- Includes: create, get, list, delete, bulk_delete, review endpoints, instrumental selection, download, cancel, retry, logs
|
||||
|
||||
### ✅ file_upload.py (6 endpoints) - DONE
|
||||
- upload_and_create_job
|
||||
- create_job_with_upload_urls
|
||||
- mark_uploads_complete
|
||||
- create_job_from_url
|
||||
- create_finalise_only_job
|
||||
- mark_finalise_uploads_complete
|
||||
|
||||
### ✅ audio_search.py (3 endpoints) - DONE
|
||||
- search_audio
|
||||
- get_audio_search_results
|
||||
- select_audio_source
|
||||
|
||||
### ⚠️ review.py (12 endpoints) - PARTIAL (4/12 done)
|
||||
|
||||
**Completed:**
|
||||
- get_correction_data ✅
|
||||
- complete_review ✅
|
||||
- add_lyrics ✅
|
||||
- generate_preview_video ✅
|
||||
|
||||
**Remaining - Need Auth:**
|
||||
- ❌ ping - Health check (maybe OK without auth?)
|
||||
- ❌ get_audio_with_hash - Audio streaming for review UI
|
||||
- ❌ get_audio_no_hash - Audio streaming for review UI
|
||||
- ❌ update_handlers - Correction handlers config
|
||||
- ❌ get_preview_video - Preview video streaming
|
||||
- ❌ submit_annotation - ML training data
|
||||
- ❌ get_annotation_stats - ML stats
|
||||
- ❌ (1 more endpoint not yet reviewed)
|
||||
|
||||
**Special Considerations:**
|
||||
- Audio streaming endpoints (`get_audio_*`) are used by the review UI frontend. May need token passed via query param or special handling.
|
||||
- `ping` endpoint is just a health check - possibly OK without auth
|
||||
|
||||
## Remaining Routes
|
||||
|
||||
### ❌ auth.py (8 endpoints) - TODO
|
||||
|
||||
These are auth-related endpoints themselves, so they have special requirements:
|
||||
|
||||
- `/auth/status` - Get OAuth credentials status (probably needs auth)
|
||||
- `/auth/status/{service}` - Get service credential status (probably needs auth)
|
||||
- `/auth/validate` - Validate credentials (probably needs auth)
|
||||
- `/auth/youtube/device` - Start YouTube device auth (admin only)
|
||||
- `/auth/youtube/device/{device_code}` - Poll YouTube auth (admin only)
|
||||
- `/auth/gdrive/device` - Start GDrive device auth (admin only)
|
||||
- `/auth/gdrive/device/{device_code}` - Poll GDrive auth (admin only)
|
||||
- (1 more endpoint)
|
||||
|
||||
**Recommendation:** Most/all of these should require admin auth (`Depends(require_admin)`)
|
||||
|
||||
### ❌ health.py (3 endpoints) - TODO
|
||||
|
||||
- `/health` - Basic health check (probably OK without auth for k8s/monitoring)
|
||||
- `/health/detailed` - Detailed health (should require auth)
|
||||
- `/readiness` - Readiness check for Cloud Run (probably OK without auth)
|
||||
|
||||
**Recommendation:**
|
||||
- Keep `/health` and `/readiness` unauthenticated for infrastructure monitoring
|
||||
- Require auth for `/health/detailed` as it may expose sensitive info
|
||||
|
||||
### ❌ internal.py (6 endpoints) - CRITICAL
|
||||
|
||||
**All worker trigger endpoints - MUST have auth!** Currently using `X-Admin-Token` header but needs formal validation:
|
||||
|
||||
- `/internal/workers/audio` - Trigger audio worker
|
||||
- `/internal/workers/lyrics` - Trigger lyrics worker
|
||||
- `/internal/workers/screens` - Trigger screens worker
|
||||
- `/internal/workers/video` - Trigger video worker
|
||||
- `/internal/workers/render-video` - Trigger render worker
|
||||
- `/internal/health` - Internal health (already has `require_admin`)
|
||||
|
||||
**Current State:** These endpoints check for `X-Admin-Token` in the request, but this should use the formal `require_admin` dependency.
|
||||
|
||||
**Recommendation:** ALL internal endpoints should use `Depends(require_admin)`
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
### Phase 1: Critical Security Fixes (HIGH PRIORITY)
|
||||
1. ✅ Complete jobs.py, file_upload.py, audio_search.py
|
||||
2. ❌ **Secure internal.py worker triggers** - These are the most critical as they allow arbitrary job manipulation
|
||||
3. ❌ **Secure auth.py admin endpoints** - Prevent unauthorized OAuth credential management
|
||||
|
||||
### Phase 2: User-Facing Endpoints
|
||||
1. ❌ Complete review.py endpoints (with special handling for streaming)
|
||||
2. ❌ health.py detailed endpoint
|
||||
|
||||
### Phase 3: Testing & Documentation
|
||||
1. ❌ Update frontend to always include auth token
|
||||
2. ❌ Test all endpoints with/without auth
|
||||
3. ❌ Update API documentation
|
||||
4. ❌ Add integration tests for auth
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
After adding auth to all endpoints:
|
||||
|
||||
- [ ] Test job creation without token → should return 401
|
||||
- [ ] Test job listing without token → should return 401
|
||||
- [ ] Test worker triggers without admin token → should return 401 or 403
|
||||
- [ ] Test OAuth endpoints without admin token → should return 401 or 403
|
||||
- [ ] Test health endpoint without token → should work (by design)
|
||||
- [ ] Test authenticated requests with valid user token → should work
|
||||
- [ ] Test authenticated requests with valid admin token → should work
|
||||
- [ ] Update frontend auth flow to handle 401 errors gracefully
|
||||
|
||||
## Notes
|
||||
|
||||
- The authentication system is already implemented in `backend/api/dependencies.py`
|
||||
- Functions available: `require_auth()`, `require_admin()`, `optional_auth()`
|
||||
- Tokens are validated against `ADMIN_TOKENS` env var or Firestore user tokens
|
||||
- Frontend already has token storage in localStorage via `AuthBanner` component
|
||||
|
||||
## Migration Path for Frontend
|
||||
|
||||
Once all endpoints require auth:
|
||||
|
||||
1. Frontend must obtain a token (either admin token or user token from a signup flow)
|
||||
2. Token must be included in Authorization header: `Authorization: Bearer <token>`
|
||||
3. Alternatively, some endpoints support `?token=<token>` query parameter
|
||||
4. Update frontend API client to always include auth
|
||||
|
||||
##Frontend Update Required
|
||||
|
||||
Current state: Frontend has AuthBanner that prompts for token, but some endpoints worked without auth (security issue).
|
||||
New state: ALL endpoints require auth, so users MUST provide a token before using the app.
|
||||
|
||||
Update `frontend/lib/api.ts` to:
|
||||
- Throw user-friendly error if token is missing when calling authenticated endpoints
|
||||
- Handle 401 responses by prompting user to re-authenticate
|
||||
- Consider implementing a "logged out" state in the UI
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
# Test Fixes Needed for Authentication
|
||||
|
||||
## Problem
|
||||
|
||||
After adding `Depends(require_auth)` to 33+ backend endpoints, all unit and integration tests are failing because they don't include authentication headers in their requests.
|
||||
|
||||
## Solution
|
||||
|
||||
All test requests to authenticated endpoints need to include:
|
||||
```python
|
||||
headers={"Authorization": "Bearer test-admin-token"}
|
||||
```
|
||||
|
||||
## Files Need Updating
|
||||
|
||||
### 1. Global Fixture (DONE)
|
||||
✅ `backend/tests/conftest.py` - Added mock_auth_service and auth_headers fixtures
|
||||
|
||||
### 2. Test Files to Update
|
||||
|
||||
Each test file needs:
|
||||
1. Mock auth service in the client fixture
|
||||
2. Add `auth_headers` parameter to test methods
|
||||
3. Pass `headers=auth_headers` to all client requests (get, post, put, delete)
|
||||
|
||||
Files:
|
||||
- `backend/tests/test_jobs_api.py` - ~15 test methods
|
||||
- `backend/tests/test_upload_api.py` - ~10 test methods
|
||||
- `backend/tests/test_audio_search.py` - ~5 test methods
|
||||
- `backend/tests/test_api_routes.py` - ~10 test methods
|
||||
- `backend/tests/test_instrumental_api.py` - ~5 test methods
|
||||
- `backend/tests/emulator/test_emulator_integration.py` - ~20 test methods
|
||||
|
||||
## Quick Fix Script
|
||||
|
||||
Rather than manually updating 60+ test methods, here's a Python script to auto-fix them:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Auto-fix test files to include authentication headers.
|
||||
"""
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
def fix_test_file(filepath):
|
||||
"""Add auth_headers to client requests in test file."""
|
||||
with open(filepath, 'r') as f:
|
||||
content = f.read()
|
||||
|
||||
# Pattern: client.METHOD(
|
||||
# Replace with: client.METHOD(..., headers=auth_headers)
|
||||
patterns = [
|
||||
# client.get("/path")
|
||||
(r'client\.(get|post|put|delete|patch)\((["\'][^"\']+["\'])\)',
|
||||
r'client.\1(\2, headers=auth_headers)'),
|
||||
|
||||
# client.get("/path", json={...})
|
||||
(r'client\.(get|post|put|delete|patch)\((["\'][^"\']+["\']),\s*json=',
|
||||
r'client.\1(\2, headers=auth_headers, json='),
|
||||
|
||||
# client.post("/path", data={...})
|
||||
(r'client\.(get|post|put|delete|patch)\((["\'][^"\']+["\']),\s*data=',
|
||||
r'client.\1(\2, headers=auth_headers, data='),
|
||||
|
||||
# client.post("/path", files={...})
|
||||
(r'client\.(post)\((["\'][^"\']+["\']),\s*files=',
|
||||
r'client.\1(\2, headers=auth_headers, files='),
|
||||
]
|
||||
|
||||
modified = content
|
||||
for pattern, replacement in patterns:
|
||||
modified = re.sub(pattern, replacement, modified)
|
||||
|
||||
# Add auth_headers parameter to test methods that don't have it
|
||||
# def test_something(self, client, mock_job):
|
||||
# -> def test_something(self, client, mock_job, auth_headers):
|
||||
modified = re.sub(
|
||||
r'def (test_\w+)\(self,\s*client([^)]*)\):',
|
||||
r'def \1(self, client\2, auth_headers):',
|
||||
modified
|
||||
)
|
||||
|
||||
with open(filepath, 'w') as f:
|
||||
f.write(modified)
|
||||
|
||||
print(f"Fixed: {filepath}")
|
||||
|
||||
# Find all test files
|
||||
test_dir = Path("backend/tests")
|
||||
for test_file in test_dir.rglob("test_*.py"):
|
||||
if 'conftest' not in str(test_file):
|
||||
fix_test_file(test_file)
|
||||
|
||||
print("Done!")
|
||||
```
|
||||
|
||||
## Manual Approach
|
||||
|
||||
For each test method:
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
def test_get_job(self, client, mock_job_manager):
|
||||
response = client.get("/api/jobs/test123")
|
||||
assert response.status_code == 200
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
def test_get_job(self, client, mock_job_manager, auth_headers):
|
||||
response = client.get("/api/jobs/test123", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
```
|
||||
|
||||
## Emulator Tests
|
||||
|
||||
Emulator tests already have `auth_headers` fixture in `backend/tests/emulator/conftest.py`.
|
||||
Just need to:
|
||||
1. Add `auth_headers` parameter to test methods
|
||||
2. Pass `headers=auth_headers` to client requests
|
||||
|
||||
## Health Endpoints
|
||||
|
||||
Health endpoints (`/api/health`, `/api/readiness`) should remain unauthenticated, so those tests don't need auth headers.
|
||||
|
||||
## Internal API Tests
|
||||
|
||||
Internal API tests in `test_internal_api.py` already have auth handling via `X-Admin-Token` header. May need updating if we migrate to formal `require_admin`.
|
||||
|
||||
## Test Coverage
|
||||
|
||||
After fixes, run:
|
||||
```bash
|
||||
cd backend
|
||||
python -m pytest tests/ -v --ignore=tests/integration
|
||||
```
|
||||
|
||||
Should see all tests passing.
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
# Karaoke Generator Frontend
|
||||
|
||||
Next.js frontend for the Karaoke Generator application.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
npm install
|
||||
|
||||
# Run development server
|
||||
npm run dev
|
||||
```
|
||||
|
||||
The app will be available at http://localhost:3000
|
||||
|
||||
## Development Setup
|
||||
|
||||
### Connecting to the Cloud Backend
|
||||
|
||||
In development, the frontend proxies API requests to the cloud backend (`api.nomadkaraoke.com`) via Next.js rewrites. This avoids CORS issues and allows local development without running a local backend.
|
||||
|
||||
The proxy is configured in `next.config.mjs`:
|
||||
- All `/api/*` requests are forwarded to the cloud backend
|
||||
- You can override the backend URL with `BACKEND_URL` environment variable
|
||||
|
||||
### Authentication
|
||||
|
||||
The backend requires authentication. To authenticate:
|
||||
|
||||
1. Create a `.env.local` file in the frontend directory:
|
||||
```
|
||||
KARAOKE_ACCESS_TOKEN=your-token-here
|
||||
```
|
||||
|
||||
2. The token is automatically loaded by Playwright tests
|
||||
3. In the browser, the token is stored in `localStorage` as `karaoke_access_token`
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests (Jest)
|
||||
|
||||
```bash
|
||||
# Run tests in watch mode
|
||||
npm test
|
||||
|
||||
# Run tests with coverage
|
||||
npm run test:coverage
|
||||
|
||||
# Run tests in CI mode
|
||||
npm run test:ci
|
||||
```
|
||||
|
||||
### E2E Tests (Playwright)
|
||||
|
||||
E2E tests run against the local dev server connected to the real cloud backend.
|
||||
|
||||
```bash
|
||||
# Run all E2E tests
|
||||
npm run test:e2e
|
||||
|
||||
# Run with UI (interactive mode)
|
||||
npm run test:e2e:ui
|
||||
|
||||
# Run in headed mode (see the browser)
|
||||
npm run test:e2e:headed
|
||||
|
||||
# Run in debug mode
|
||||
npm run test:e2e:debug
|
||||
```
|
||||
|
||||
#### Authentication for E2E Tests
|
||||
|
||||
E2E tests require a valid access token. Set it in `.env.local`:
|
||||
|
||||
```
|
||||
KARAOKE_ACCESS_TOKEN=your-token-here
|
||||
```
|
||||
|
||||
The Playwright config (`playwright.config.ts`) automatically loads this file.
|
||||
|
||||
#### Test Structure
|
||||
|
||||
Tests are located in `e2e/`:
|
||||
|
||||
- `karaoke-generation.spec.ts` - Main test suite covering:
|
||||
- Homepage loading and form display
|
||||
- Search tab functionality
|
||||
- Audio search submission
|
||||
- Audio selection dialog flow
|
||||
- API health checks
|
||||
|
||||
#### Test Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────────┐
|
||||
│ Playwright │────▶│ Next.js Dev │────▶│ Cloud Backend │
|
||||
│ Browser │ │ Server (:3000) │ │ api.nomadkaraoke │
|
||||
└─────────────────┘ └──────────────────┘ └─────────────────────┘
|
||||
│
|
||||
▼
|
||||
API Proxy
|
||||
/api/* → backend
|
||||
```
|
||||
|
||||
#### Debugging Test Failures
|
||||
|
||||
1. **Screenshots**: Failed tests save screenshots to `test-results/`
|
||||
2. **Videos**: First retry captures video (when retries enabled)
|
||||
3. **Traces**: View detailed traces with `npx playwright show-trace <trace.zip>`
|
||||
4. **HTML Report**: After running tests, view `npx playwright show-report`
|
||||
|
||||
#### Common Issues
|
||||
|
||||
- **401 Unauthorized**: Token missing or invalid - check `.env.local`
|
||||
- **Socket hang up**: Backend operation taking too long - some operations (like audio download) can take minutes
|
||||
- **Element outside viewport**: Dialog scrolling issues - tests use JavaScript click to handle this
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
frontend/
|
||||
├── app/ # Next.js app router pages
|
||||
├── components/ # React components
|
||||
│ ├── ui/ # shadcn/ui components
|
||||
│ └── audio-search/ # Audio search feature components
|
||||
├── lib/ # Utilities and API client
|
||||
│ └── api.ts # Backend API client
|
||||
├── e2e/ # Playwright E2E tests
|
||||
├── playwright.config.ts # Playwright configuration
|
||||
└── next.config.mjs # Next.js configuration
|
||||
```
|
||||
|
||||
## Scripts
|
||||
|
||||
| Script | Description |
|
||||
|--------|-------------|
|
||||
| `npm run dev` | Start development server |
|
||||
| `npm run build` | Build for production |
|
||||
| `npm run start` | Start production server |
|
||||
| `npm run lint` | Run ESLint |
|
||||
| `npm test` | Run unit tests (watch mode) |
|
||||
| `npm run test:ci` | Run unit tests (CI mode with coverage) |
|
||||
| `npm run test:e2e` | Run E2E tests |
|
||||
| `npm run test:e2e:ui` | Run E2E tests with interactive UI |
|
||||
@@ -1,348 +0,0 @@
|
||||
"use client"
|
||||
|
||||
import { useState } from "react"
|
||||
import { useJobs } from "@/lib/jobs"
|
||||
import type { JobStatus } from "@/lib/types"
|
||||
import { AppHeader } from "@/components/app-header"
|
||||
import { ProtectedRoute } from "@/components/protected-route"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Input } from "@/components/ui/input"
|
||||
import { Label } from "@/components/ui/label"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"
|
||||
import { StatusBadge } from "@/components/status-badge"
|
||||
import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table"
|
||||
import { BarChart3, FileVideo, Ticket, TrendingUp, Activity, ChevronRight } from "lucide-react"
|
||||
import Link from "next/link"
|
||||
|
||||
function AdminContent() {
|
||||
const { jobs } = useJobs()
|
||||
const [promoCode, setPromoCode] = useState("")
|
||||
const [promoCredits, setPromoCredits] = useState("5")
|
||||
const [generatedCode, setGeneratedCode] = useState("")
|
||||
|
||||
const totalJobs = jobs.length
|
||||
const completedJobs = jobs.filter((j) => j.status === "completed").length
|
||||
const processingJobs = jobs.filter((j) => j.status === "processing" || j.status === "queued").length
|
||||
const failedJobs = jobs.filter((j) => j.status === "failed").length
|
||||
|
||||
const handleGeneratePromo = () => {
|
||||
const code = `NOMAD${Math.random().toString(36).substr(2, 6).toUpperCase()}`
|
||||
setGeneratedCode(code)
|
||||
setPromoCode("")
|
||||
}
|
||||
|
||||
const statusCounts = {
|
||||
queued: jobs.filter((j) => j.status === "queued").length,
|
||||
processing: jobs.filter((j) => j.status === "processing").length,
|
||||
awaiting_review: jobs.filter((j) => j.status === "awaiting_review").length,
|
||||
awaiting_instrumental: jobs.filter((j) => j.status === "awaiting_instrumental").length,
|
||||
completed: completedJobs,
|
||||
failed: failedJobs,
|
||||
}
|
||||
|
||||
const recentJobs = [...jobs]
|
||||
.sort((a, b) => new Date(b.createdAt).getTime() - new Date(a.createdAt).getTime())
|
||||
.slice(0, 10)
|
||||
|
||||
return (
|
||||
<div className="min-h-screen bg-background">
|
||||
<AppHeader />
|
||||
|
||||
<main className="container mx-auto px-4 py-8">
|
||||
<div className="mb-8">
|
||||
<h1 className="text-4xl font-bold mb-2">Admin Dashboard</h1>
|
||||
<p className="text-muted-foreground">Manage users, jobs, and system analytics</p>
|
||||
</div>
|
||||
|
||||
{/* Stats Grid */}
|
||||
<div className="grid gap-4 md:grid-cols-2 lg:grid-cols-4 mb-8">
|
||||
<Card className="bg-gradient-to-br from-primary/10 to-transparent border-primary/20">
|
||||
<CardHeader className="flex flex-row items-center justify-between pb-2">
|
||||
<CardTitle className="text-sm font-medium text-muted-foreground">Total Jobs</CardTitle>
|
||||
<FileVideo className="w-4 h-4 text-primary" />
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="text-3xl font-bold text-primary">{totalJobs}</div>
|
||||
<p className="text-xs text-muted-foreground mt-1">All time</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card>
|
||||
<CardHeader className="flex flex-row items-center justify-between pb-2">
|
||||
<CardTitle className="text-sm font-medium text-muted-foreground">Completed</CardTitle>
|
||||
<TrendingUp className="w-4 h-4 text-green-400" />
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="text-3xl font-bold text-green-400">{completedJobs}</div>
|
||||
<p className="text-xs text-muted-foreground mt-1">
|
||||
{((completedJobs / totalJobs) * 100).toFixed(1)}% success rate
|
||||
</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card>
|
||||
<CardHeader className="flex flex-row items-center justify-between pb-2">
|
||||
<CardTitle className="text-sm font-medium text-muted-foreground">Processing</CardTitle>
|
||||
<Activity className="w-4 h-4 text-blue-400" />
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="text-3xl font-bold text-blue-400">{processingJobs}</div>
|
||||
<p className="text-xs text-muted-foreground mt-1">Active jobs</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card>
|
||||
<CardHeader className="flex flex-row items-center justify-between pb-2">
|
||||
<CardTitle className="text-sm font-medium text-muted-foreground">Failed</CardTitle>
|
||||
<BarChart3 className="w-4 h-4 text-destructive" />
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="text-3xl font-bold text-destructive">{failedJobs}</div>
|
||||
<p className="text-xs text-muted-foreground mt-1">
|
||||
{((failedJobs / totalJobs) * 100).toFixed(1)}% failure rate
|
||||
</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
{/* Tabs */}
|
||||
<Tabs defaultValue="overview" className="space-y-6">
|
||||
<TabsList className="grid w-full max-w-md grid-cols-3">
|
||||
<TabsTrigger value="overview">Overview</TabsTrigger>
|
||||
<TabsTrigger value="jobs">All Jobs</TabsTrigger>
|
||||
<TabsTrigger value="promo">Promo Codes</TabsTrigger>
|
||||
</TabsList>
|
||||
|
||||
<TabsContent value="overview" className="space-y-6">
|
||||
{/* Status Breakdown */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Job Status Breakdown</CardTitle>
|
||||
<CardDescription>Current distribution of job statuses</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-3">
|
||||
{Object.entries(statusCounts).map(([status, count]) => (
|
||||
<div
|
||||
key={status}
|
||||
className="flex items-center justify-between p-3 rounded-lg border border-border bg-card"
|
||||
>
|
||||
<div className="flex items-center gap-3">
|
||||
<StatusBadge status={status as JobStatus} size="sm" />
|
||||
<span className="text-sm text-muted-foreground">Jobs</span>
|
||||
</div>
|
||||
<span className="text-lg font-bold">{count}</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Recent Activity */}
|
||||
<Card>
|
||||
<CardHeader className="flex flex-row items-center justify-between">
|
||||
<div>
|
||||
<CardTitle>Recent Activity</CardTitle>
|
||||
<CardDescription>Latest jobs across all users</CardDescription>
|
||||
</div>
|
||||
<Button variant="outline" size="sm" asChild>
|
||||
<Link href="#" onClick={(e) => e.preventDefault()}>
|
||||
View All
|
||||
</Link>
|
||||
</Button>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="space-y-2">
|
||||
{recentJobs.slice(0, 5).map((job) => (
|
||||
<Link
|
||||
key={job.id}
|
||||
href={`/jobs/${job.id}`}
|
||||
className="flex items-center justify-between p-3 rounded-lg border border-border hover:border-primary/50 hover:bg-card/50 transition-all group"
|
||||
>
|
||||
<div className="flex-1 min-w-0">
|
||||
<p className="font-medium text-sm truncate group-hover:text-primary transition-colors">
|
||||
{job.artist} - {job.title}
|
||||
</p>
|
||||
<p className="text-xs text-muted-foreground">{new Date(job.createdAt).toLocaleString()}</p>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<StatusBadge status={job.status} size="sm" />
|
||||
<ChevronRight className="w-4 h-4 text-muted-foreground group-hover:text-primary transition-colors" />
|
||||
</div>
|
||||
</Link>
|
||||
))}
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="jobs" className="space-y-6">
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>All Jobs</CardTitle>
|
||||
<CardDescription>Complete list of all karaoke video jobs in the system</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="rounded-md border border-border overflow-auto">
|
||||
<Table>
|
||||
<TableHeader>
|
||||
<TableRow>
|
||||
<TableHead>Artist - Title</TableHead>
|
||||
<TableHead>User ID</TableHead>
|
||||
<TableHead>Status</TableHead>
|
||||
<TableHead>Created</TableHead>
|
||||
<TableHead>Progress</TableHead>
|
||||
<TableHead></TableHead>
|
||||
</TableRow>
|
||||
</TableHeader>
|
||||
<TableBody>
|
||||
{recentJobs.map((job) => (
|
||||
<TableRow key={job.id} className="hover:bg-muted/50">
|
||||
<TableCell className="font-medium">
|
||||
{job.artist} - {job.title}
|
||||
</TableCell>
|
||||
<TableCell className="text-muted-foreground text-sm">{job.userId}</TableCell>
|
||||
<TableCell>
|
||||
<StatusBadge status={job.status} size="sm" />
|
||||
</TableCell>
|
||||
<TableCell className="text-sm text-muted-foreground">
|
||||
{new Date(job.createdAt).toLocaleDateString()}
|
||||
</TableCell>
|
||||
<TableCell className="text-sm">{job.progress}%</TableCell>
|
||||
<TableCell>
|
||||
<Button variant="ghost" size="sm" asChild>
|
||||
<Link href={`/jobs/${job.id}`}>
|
||||
<ChevronRight className="w-4 h-4" />
|
||||
</Link>
|
||||
</Button>
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
))}
|
||||
</TableBody>
|
||||
</Table>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="promo" className="space-y-6">
|
||||
<div className="grid gap-6 lg:grid-cols-2">
|
||||
{/* Generate Promo Code */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle className="flex items-center gap-2">
|
||||
<Ticket className="w-5 h-5 text-primary" />
|
||||
Generate Promo Code
|
||||
</CardTitle>
|
||||
<CardDescription>Create promotional codes for users to redeem credits</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent className="space-y-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="promo-credits">Credit Amount</Label>
|
||||
<Input
|
||||
id="promo-credits"
|
||||
type="number"
|
||||
min="1"
|
||||
max="100"
|
||||
value={promoCredits}
|
||||
onChange={(e) => setPromoCredits(e.target.value)}
|
||||
/>
|
||||
<p className="text-xs text-muted-foreground">Number of credits this code will provide</p>
|
||||
</div>
|
||||
|
||||
<Button onClick={handleGeneratePromo} className="w-full bg-primary hover:bg-primary/90">
|
||||
Generate Code
|
||||
</Button>
|
||||
|
||||
{generatedCode && (
|
||||
<div className="p-4 rounded-lg bg-primary/10 border border-primary/20">
|
||||
<p className="text-sm text-muted-foreground mb-2">Generated Promo Code:</p>
|
||||
<p className="text-2xl font-mono font-bold text-primary">{generatedCode}</p>
|
||||
<p className="text-xs text-muted-foreground mt-2">Worth {promoCredits} credits</p>
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Active Promo Codes */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Active Promo Codes</CardTitle>
|
||||
<CardDescription>Currently available promotional codes</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="space-y-3">
|
||||
<div className="flex items-center justify-between p-3 rounded-lg border border-border bg-card">
|
||||
<div>
|
||||
<p className="font-mono font-medium">NOMADWELCOME</p>
|
||||
<p className="text-xs text-muted-foreground">5 credits • Used 23 times</p>
|
||||
</div>
|
||||
<Button variant="ghost" size="sm" className="text-destructive">
|
||||
Disable
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
<div className="flex items-center justify-between p-3 rounded-lg border border-border bg-card">
|
||||
<div>
|
||||
<p className="font-mono font-medium">LAUNCH2024</p>
|
||||
<p className="text-xs text-muted-foreground">10 credits • Used 8 times</p>
|
||||
</div>
|
||||
<Button variant="ghost" size="sm" className="text-destructive">
|
||||
Disable
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{generatedCode && (
|
||||
<div className="flex items-center justify-between p-3 rounded-lg border border-primary/20 bg-primary/5">
|
||||
<div>
|
||||
<p className="font-mono font-medium text-primary">{generatedCode}</p>
|
||||
<p className="text-xs text-muted-foreground">{promoCredits} credits • Not used yet</p>
|
||||
</div>
|
||||
<Button variant="ghost" size="sm" className="text-destructive">
|
||||
Disable
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
{/* Usage Statistics */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Promo Code Statistics</CardTitle>
|
||||
<CardDescription>Usage analytics and performance metrics</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="grid gap-4 sm:grid-cols-3">
|
||||
<div className="space-y-2">
|
||||
<p className="text-sm text-muted-foreground">Total Codes</p>
|
||||
<p className="text-3xl font-bold">12</p>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<p className="text-sm text-muted-foreground">Total Redemptions</p>
|
||||
<p className="text-3xl font-bold">156</p>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<p className="text-sm text-muted-foreground">Credits Distributed</p>
|
||||
<p className="text-3xl font-bold text-primary">892</p>
|
||||
</div>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</TabsContent>
|
||||
</Tabs>
|
||||
</main>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default function AdminPage() {
|
||||
return (
|
||||
<ProtectedRoute requireAdmin>
|
||||
<AdminContent />
|
||||
</ProtectedRoute>
|
||||
)
|
||||
}
|
||||
@@ -1,171 +0,0 @@
|
||||
"use client"
|
||||
|
||||
import { useRouter } from "next/navigation"
|
||||
import { useAuth } from "@/lib/auth"
|
||||
import { useJobs } from "@/lib/jobs"
|
||||
import { AppHeader } from "@/components/app-header"
|
||||
import { ProtectedRoute } from "@/components/protected-route"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import { StatusBadge } from "@/components/status-badge"
|
||||
import { Plus, FileVideo, Clock, CheckCircle2, Loader2 } from "lucide-react"
|
||||
import Link from "next/link"
|
||||
|
||||
function DashboardContent() {
|
||||
const { user } = useAuth()
|
||||
const { jobs, getUserJobs } = useJobs()
|
||||
const router = useRouter()
|
||||
|
||||
if (!user) return null
|
||||
|
||||
const userJobs = getUserJobs(user.token.includes("demo") ? "demo" : user.token)
|
||||
const completedJobs = userJobs.filter((j) => j.status === "completed").length
|
||||
const inProgressJobs = userJobs.filter((j) => j.status === "processing" || j.status === "queued").length
|
||||
const totalJobs = userJobs.length
|
||||
const recentJobs = userJobs.slice(0, 5)
|
||||
|
||||
return (
|
||||
<div className="min-h-screen bg-background">
|
||||
<AppHeader />
|
||||
|
||||
<main className="container mx-auto px-4 py-8">
|
||||
{/* Welcome Section */}
|
||||
<div className="mb-8">
|
||||
<h1 className="text-4xl font-bold mb-2">
|
||||
Welcome back<span className="text-primary">!</span>
|
||||
</h1>
|
||||
<p className="text-muted-foreground">Manage your karaoke video projects</p>
|
||||
</div>
|
||||
|
||||
{/* Stats Grid */}
|
||||
<div className="grid gap-4 md:grid-cols-4 mb-8">
|
||||
<Card>
|
||||
<CardHeader className="flex flex-row items-center justify-between pb-2">
|
||||
<CardTitle className="text-sm font-medium text-muted-foreground">Total Jobs</CardTitle>
|
||||
<FileVideo className="w-4 h-4 text-muted-foreground" />
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="text-3xl font-bold">{totalJobs}</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card>
|
||||
<CardHeader className="flex flex-row items-center justify-between pb-2">
|
||||
<CardTitle className="text-sm font-medium text-muted-foreground">Completed</CardTitle>
|
||||
<CheckCircle2 className="w-4 h-4 text-green-400" />
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="text-3xl font-bold text-green-400">{completedJobs}</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card>
|
||||
<CardHeader className="flex flex-row items-center justify-between pb-2">
|
||||
<CardTitle className="text-sm font-medium text-muted-foreground">In Progress</CardTitle>
|
||||
<Loader2 className="w-4 h-4 text-blue-400 animate-spin" />
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="text-3xl font-bold text-blue-400">{inProgressJobs}</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card className="bg-gradient-to-br from-primary/10 to-secondary/10 border-primary/20">
|
||||
<CardHeader className="flex flex-row items-center justify-between pb-2">
|
||||
<CardTitle className="text-sm font-medium">Credits Remaining</CardTitle>
|
||||
<div className="w-2 h-2 rounded-full bg-primary animate-pulse" />
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="text-3xl font-bold text-primary">{user.credits}</div>
|
||||
<p className="text-xs text-muted-foreground mt-1">
|
||||
<Link href="#" className="hover:text-primary transition-colors">
|
||||
Purchase more
|
||||
</Link>
|
||||
</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
{/* Create New Job CTA */}
|
||||
<Card className="mb-8 bg-gradient-to-r from-primary/5 via-card to-secondary/5 border-primary/20">
|
||||
<CardHeader>
|
||||
<CardTitle className="text-2xl">Create New Karaoke Video</CardTitle>
|
||||
<CardDescription>
|
||||
Transform any song into a professional karaoke video with synchronized lyrics
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<Button size="lg" className="bg-primary hover:bg-primary/90" onClick={() => router.push("/jobs/new")}>
|
||||
<Plus className="w-5 h-5 mr-2" />
|
||||
Create New Job
|
||||
</Button>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Recent Jobs */}
|
||||
<Card>
|
||||
<CardHeader className="flex flex-row items-center justify-between">
|
||||
<div>
|
||||
<CardTitle>Recent Jobs</CardTitle>
|
||||
<CardDescription>Your latest karaoke video projects</CardDescription>
|
||||
</div>
|
||||
<Button variant="outline" size="sm" asChild>
|
||||
<Link href="/jobs">View All</Link>
|
||||
</Button>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
{recentJobs.length === 0 ? (
|
||||
<div className="text-center py-12">
|
||||
<FileVideo className="w-12 h-12 mx-auto mb-4 text-muted-foreground" />
|
||||
<p className="text-muted-foreground mb-4">No jobs yet</p>
|
||||
<Button onClick={() => router.push("/jobs/new")}>
|
||||
<Plus className="w-4 h-4 mr-2" />
|
||||
Create Your First Job
|
||||
</Button>
|
||||
</div>
|
||||
) : (
|
||||
<div className="space-y-3">
|
||||
{recentJobs.map((job) => (
|
||||
<Link
|
||||
key={job.id}
|
||||
href={`/jobs/${job.id}`}
|
||||
className="flex items-center justify-between p-4 rounded-lg border border-border hover:border-primary/50 hover:bg-card/50 transition-all cursor-pointer group"
|
||||
>
|
||||
<div className="flex-1">
|
||||
<div className="flex items-center gap-3 mb-1">
|
||||
<h3 className="font-medium group-hover:text-primary transition-colors">
|
||||
{job.artist} - {job.title}
|
||||
</h3>
|
||||
<StatusBadge status={job.status} size="sm" />
|
||||
</div>
|
||||
<div className="flex items-center gap-4 text-sm text-muted-foreground">
|
||||
<span className="flex items-center gap-1">
|
||||
<Clock className="w-3 h-3" />
|
||||
{new Date(job.createdAt).toLocaleDateString()}
|
||||
</span>
|
||||
{job.status === "processing" && <span>{job.progress}% complete</span>}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{job.status === "completed" && (
|
||||
<Button size="sm" variant="outline" className="ml-4 bg-transparent">
|
||||
Download
|
||||
</Button>
|
||||
)}
|
||||
</Link>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
</main>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default function DashboardPage() {
|
||||
return (
|
||||
<ProtectedRoute>
|
||||
<DashboardContent />
|
||||
</ProtectedRoute>
|
||||
)
|
||||
}
|
||||
+55
-48
@@ -4,45 +4,65 @@
|
||||
@custom-variant dark (&:is(.dark *));
|
||||
|
||||
:root {
|
||||
/* Nomad Karaoke brand colors - matching nomadkaraoke.com */
|
||||
--background: oklch(0.22 0.04 260); /* Dark slate blue #2b3544 */
|
||||
--foreground: oklch(0.95 0.01 0); /* White text */
|
||||
--card: oklch(0.26 0.04 260); /* Slightly lighter slate for cards */
|
||||
--card-foreground: oklch(0.95 0.01 0);
|
||||
--popover: oklch(0.26 0.04 260);
|
||||
--popover-foreground: oklch(0.95 0.01 0);
|
||||
--primary: oklch(0.68 0.22 340); /* Pink #ff7acc */
|
||||
--primary-foreground: oklch(0.98 0.01 0);
|
||||
--secondary: oklch(0.78 0.15 85); /* Gold/yellow #ffc857 */
|
||||
--secondary-foreground: oklch(0.15 0.01 270);
|
||||
--muted: oklch(0.3 0.04 260);
|
||||
--muted-foreground: oklch(0.65 0.01 260);
|
||||
--accent: oklch(0.68 0.22 340);
|
||||
--accent-foreground: oklch(0.98 0.01 0);
|
||||
--destructive: oklch(0.55 0.22 25);
|
||||
--destructive-foreground: oklch(0.98 0.01 0);
|
||||
--border: oklch(0.32 0.04 260);
|
||||
--input: oklch(0.32 0.04 260);
|
||||
--ring: oklch(0.68 0.22 340);
|
||||
--chart-1: oklch(0.68 0.22 340);
|
||||
--chart-2: oklch(0.78 0.15 85);
|
||||
--chart-3: oklch(0.55 0.2 280);
|
||||
--chart-4: oklch(0.6 0.18 180);
|
||||
--chart-5: oklch(0.65 0.2 60);
|
||||
/* Dark theme (default) - Matching instrumental review UI color scheme */
|
||||
--bg: #0f0f0f;
|
||||
--card: #1a1a1a;
|
||||
--card-border: #2a2a2a;
|
||||
--waveform-bg: #0d1117;
|
||||
--text: #e5e5e5;
|
||||
--text-muted: #888;
|
||||
--primary: #3b82f6;
|
||||
--primary-hover: #2563eb;
|
||||
--secondary: #252525;
|
||||
--secondary-hover: #333;
|
||||
--success: #22c55e;
|
||||
--warning: #f59e0b;
|
||||
--danger: #ef4444;
|
||||
--pink: #ec4899;
|
||||
--blue: #60a5fa;
|
||||
|
||||
/* Tailwind compatibility mappings */
|
||||
--background: var(--bg);
|
||||
--foreground: var(--text);
|
||||
--card-foreground: var(--text);
|
||||
--popover: var(--card);
|
||||
--popover-foreground: var(--text);
|
||||
--primary-foreground: #ffffff;
|
||||
--secondary-foreground: var(--text);
|
||||
--muted: var(--secondary);
|
||||
--muted-foreground: var(--text-muted);
|
||||
--accent: var(--primary);
|
||||
--accent-foreground: #ffffff;
|
||||
--destructive: var(--danger);
|
||||
--destructive-foreground: #ffffff;
|
||||
--border: var(--card-border);
|
||||
--input: var(--secondary);
|
||||
--ring: var(--primary);
|
||||
--radius: 0.5rem;
|
||||
--sidebar: oklch(0.26 0.04 260);
|
||||
--sidebar-foreground: oklch(0.95 0.01 0);
|
||||
--sidebar-primary: oklch(0.68 0.22 340);
|
||||
--sidebar-primary-foreground: oklch(0.98 0.01 0);
|
||||
--sidebar-accent: oklch(0.3 0.04 260);
|
||||
--sidebar-accent-foreground: oklch(0.95 0.01 0);
|
||||
--sidebar-border: oklch(0.32 0.04 260);
|
||||
--sidebar-ring: oklch(0.68 0.22 340);
|
||||
}
|
||||
|
||||
:root.light {
|
||||
/* Light theme */
|
||||
--bg: #f8fafc;
|
||||
--card: #ffffff;
|
||||
--card-border: #e2e8f0;
|
||||
--waveform-bg: #f1f5f9;
|
||||
--text: #1e293b;
|
||||
--text-muted: #64748b;
|
||||
--primary: #3b82f6;
|
||||
--primary-hover: #2563eb;
|
||||
--secondary: #f1f5f9;
|
||||
--secondary-hover: #e2e8f0;
|
||||
--success: #22c55e;
|
||||
--warning: #f59e0b;
|
||||
--danger: #ef4444;
|
||||
--pink: #ec4899;
|
||||
--blue: #60a5fa;
|
||||
}
|
||||
|
||||
@theme inline {
|
||||
--font-sans: "Geist", "Geist Fallback";
|
||||
--font-mono: "Geist Mono", "Geist Mono Fallback";
|
||||
--font-sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
--font-mono: "SF Mono", Monaco, "Cascadia Code", monospace;
|
||||
--color-background: var(--background);
|
||||
--color-foreground: var(--foreground);
|
||||
--color-card: var(--card);
|
||||
@@ -62,23 +82,10 @@
|
||||
--color-border: var(--border);
|
||||
--color-input: var(--input);
|
||||
--color-ring: var(--ring);
|
||||
--color-chart-1: var(--chart-1);
|
||||
--color-chart-2: var(--chart-2);
|
||||
--color-chart-3: var(--chart-3);
|
||||
--color-chart-4: var(--chart-4);
|
||||
--color-chart-5: var(--chart-5);
|
||||
--radius-sm: calc(var(--radius) - 4px);
|
||||
--radius-md: calc(var(--radius) - 2px);
|
||||
--radius-lg: var(--radius);
|
||||
--radius-xl: calc(var(--radius) + 4px);
|
||||
--color-sidebar: var(--sidebar);
|
||||
--color-sidebar-foreground: var(--sidebar-foreground);
|
||||
--color-sidebar-primary: var(--sidebar-primary);
|
||||
--color-sidebar-primary-foreground: var(--sidebar-primary-foreground);
|
||||
--color-sidebar-accent: var(--sidebar-accent);
|
||||
--color-sidebar-accent-foreground: var(--sidebar-accent-foreground);
|
||||
--color-sidebar-border: var(--sidebar-border);
|
||||
--color-sidebar-ring: var(--sidebar-ring);
|
||||
}
|
||||
|
||||
@layer base {
|
||||
|
||||
@@ -1,603 +0,0 @@
|
||||
"use client"
|
||||
|
||||
import { useParams, useRouter } from "next/navigation"
|
||||
import { useState, useEffect, useCallback } from "react"
|
||||
import { AppHeader } from "@/components/app-header"
|
||||
import { ProtectedRoute } from "@/components/protected-route"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import { Alert, AlertDescription } from "@/components/ui/alert"
|
||||
import { Badge } from "@/components/ui/badge"
|
||||
import { RadioGroup, RadioGroupItem } from "@/components/ui/radio-group"
|
||||
import { Label } from "@/components/ui/label"
|
||||
import { Skeleton } from "@/components/ui/skeleton"
|
||||
import {
|
||||
WaveformDisplay,
|
||||
AudioPlayer,
|
||||
RegionSelector,
|
||||
} from "@/components/instrumental-review"
|
||||
import {
|
||||
ArrowLeft,
|
||||
AlertCircle,
|
||||
CheckCircle2,
|
||||
Loader2,
|
||||
Music,
|
||||
VolumeX,
|
||||
Volume2,
|
||||
} from "lucide-react"
|
||||
|
||||
interface AudibleSegment {
|
||||
start_seconds: number
|
||||
end_seconds: number
|
||||
duration_seconds: number
|
||||
avg_amplitude_db: number
|
||||
peak_amplitude_db?: number
|
||||
}
|
||||
|
||||
interface MuteRegion {
|
||||
start_seconds: number
|
||||
end_seconds: number
|
||||
}
|
||||
|
||||
interface AnalysisData {
|
||||
has_audible_content: boolean
|
||||
total_duration_seconds: number
|
||||
audible_segments: AudibleSegment[]
|
||||
recommended_selection: string
|
||||
total_audible_duration_seconds: number
|
||||
audible_percentage: number
|
||||
silence_threshold_db: number
|
||||
}
|
||||
|
||||
interface InstrumentalAnalysisResponse {
|
||||
job_id: string
|
||||
artist: string
|
||||
title: string
|
||||
status: string
|
||||
analysis: AnalysisData
|
||||
audio_urls: {
|
||||
clean_instrumental: string | null
|
||||
backing_vocals: string | null
|
||||
with_backing: string | null
|
||||
custom_instrumental: string | null
|
||||
}
|
||||
waveform_url: string | null
|
||||
has_custom_instrumental: boolean
|
||||
}
|
||||
|
||||
// Get API base URL from environment or default to relative path
|
||||
const API_BASE = process.env.NEXT_PUBLIC_API_URL || "/api"
|
||||
|
||||
function InstrumentalReviewContent() {
|
||||
const params = useParams()
|
||||
const router = useRouter()
|
||||
const jobId = params.id as string
|
||||
|
||||
// State
|
||||
const [isLoading, setIsLoading] = useState(true)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const [analysisData, setAnalysisData] = useState<InstrumentalAnalysisResponse | null>(null)
|
||||
const [waveformData, setWaveformData] = useState<{ amplitudes: number[]; duration: number } | null>(null)
|
||||
|
||||
const [selectedOption, setSelectedOption] = useState<string>("clean")
|
||||
const [muteRegions, setMuteRegions] = useState<MuteRegion[]>([])
|
||||
const [isSelectionMode, setIsSelectionMode] = useState(false)
|
||||
const [currentTime, setCurrentTime] = useState(0)
|
||||
const [seekTo, setSeekTo] = useState<number | undefined>(undefined)
|
||||
|
||||
const [isCreatingCustom, setIsCreatingCustom] = useState(false)
|
||||
const [isSubmitting, setIsSubmitting] = useState(false)
|
||||
const [activeAudio, setActiveAudio] = useState<"clean" | "backing" | "with_backing" | "custom">("backing")
|
||||
|
||||
// Fetch analysis data
|
||||
const fetchAnalysis = useCallback(async () => {
|
||||
try {
|
||||
setIsLoading(true)
|
||||
setError(null)
|
||||
|
||||
const response = await fetch(`${API_BASE}/jobs/${jobId}/instrumental-analysis`)
|
||||
if (!response.ok) {
|
||||
const errorData = await response.json().catch(() => ({}))
|
||||
throw new Error(errorData.detail || `Failed to fetch analysis: ${response.status}`)
|
||||
}
|
||||
|
||||
const data: InstrumentalAnalysisResponse = await response.json()
|
||||
setAnalysisData(data)
|
||||
|
||||
// Set initial selection based on recommendation
|
||||
const recommendation = data.analysis.recommended_selection
|
||||
if (recommendation === "clean") {
|
||||
setSelectedOption("clean")
|
||||
} else if (recommendation === "with_backing") {
|
||||
setSelectedOption("with_backing")
|
||||
} else {
|
||||
// "review_needed" - default to with_backing but user should review
|
||||
setSelectedOption("with_backing")
|
||||
}
|
||||
|
||||
// Fetch waveform data for interactive display
|
||||
const waveformResponse = await fetch(`${API_BASE}/jobs/${jobId}/waveform-data?num_points=600`)
|
||||
if (waveformResponse.ok) {
|
||||
const waveform = await waveformResponse.json()
|
||||
setWaveformData(waveform)
|
||||
}
|
||||
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : "Failed to load analysis data")
|
||||
} finally {
|
||||
setIsLoading(false)
|
||||
}
|
||||
}, [jobId])
|
||||
|
||||
useEffect(() => {
|
||||
fetchAnalysis()
|
||||
}, [fetchAnalysis])
|
||||
|
||||
// Handle seeking from waveform click
|
||||
const handleSeek = (time: number) => {
|
||||
setSeekTo(time)
|
||||
// Reset seekTo after a short delay
|
||||
setTimeout(() => setSeekTo(undefined), 100)
|
||||
}
|
||||
|
||||
// Handle adding a mute region
|
||||
const handleAddRegion = (region: MuteRegion) => {
|
||||
setMuteRegions((prev) => {
|
||||
// Check for overlaps and merge if needed
|
||||
const newRegions = [...prev, region].sort((a, b) => a.start_seconds - b.start_seconds)
|
||||
|
||||
// Merge overlapping regions
|
||||
const merged: MuteRegion[] = []
|
||||
for (const r of newRegions) {
|
||||
if (merged.length === 0) {
|
||||
merged.push(r)
|
||||
} else {
|
||||
const last = merged[merged.length - 1]
|
||||
if (r.start_seconds <= last.end_seconds) {
|
||||
last.end_seconds = Math.max(last.end_seconds, r.end_seconds)
|
||||
} else {
|
||||
merged.push(r)
|
||||
}
|
||||
}
|
||||
}
|
||||
return merged
|
||||
})
|
||||
setIsSelectionMode(false)
|
||||
}
|
||||
|
||||
// Handle removing a mute region
|
||||
const handleRemoveRegion = (index: number) => {
|
||||
setMuteRegions((prev) => prev.filter((_, i) => i !== index))
|
||||
}
|
||||
|
||||
// Handle clearing all mute regions
|
||||
const handleClearAll = () => {
|
||||
setMuteRegions([])
|
||||
}
|
||||
|
||||
// Create custom instrumental
|
||||
const handleCreateCustom = async () => {
|
||||
if (muteRegions.length === 0) {
|
||||
setError("Please select at least one region to mute")
|
||||
return
|
||||
}
|
||||
|
||||
try {
|
||||
setIsCreatingCustom(true)
|
||||
setError(null)
|
||||
|
||||
const response = await fetch(`${API_BASE}/jobs/${jobId}/create-custom-instrumental`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
mute_regions: muteRegions.map((r) => ({
|
||||
start_seconds: r.start_seconds,
|
||||
end_seconds: r.end_seconds,
|
||||
})),
|
||||
}),
|
||||
})
|
||||
|
||||
if (!response.ok) {
|
||||
const errorData = await response.json().catch(() => ({}))
|
||||
throw new Error(errorData.detail || "Failed to create custom instrumental")
|
||||
}
|
||||
|
||||
// Refresh analysis to get updated URLs
|
||||
await fetchAnalysis()
|
||||
setSelectedOption("custom")
|
||||
setActiveAudio("custom")
|
||||
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : "Failed to create custom instrumental")
|
||||
} finally {
|
||||
setIsCreatingCustom(false)
|
||||
}
|
||||
}
|
||||
|
||||
// Submit selection
|
||||
const handleSubmit = async () => {
|
||||
let selection = selectedOption
|
||||
|
||||
// If "review" is selected, determine based on mute regions
|
||||
if (selectedOption === "review") {
|
||||
if (muteRegions.length > 0 && analysisData?.has_custom_instrumental) {
|
||||
selection = "custom"
|
||||
} else {
|
||||
selection = "with_backing"
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
setIsSubmitting(true)
|
||||
setError(null)
|
||||
|
||||
const response = await fetch(`${API_BASE}/jobs/${jobId}/select-instrumental`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ selection }),
|
||||
})
|
||||
|
||||
if (!response.ok) {
|
||||
const errorData = await response.json().catch(() => ({}))
|
||||
throw new Error(errorData.detail || "Failed to submit selection")
|
||||
}
|
||||
|
||||
// Navigate back to job page
|
||||
router.push(`/jobs/${jobId}`)
|
||||
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : "Failed to submit selection")
|
||||
} finally {
|
||||
setIsSubmitting(false)
|
||||
}
|
||||
}
|
||||
|
||||
// Get current audio URL based on active selection
|
||||
const getCurrentAudioUrl = () => {
|
||||
if (!analysisData) return null
|
||||
switch (activeAudio) {
|
||||
case "clean":
|
||||
return analysisData.audio_urls.clean_instrumental
|
||||
case "backing":
|
||||
return analysisData.audio_urls.backing_vocals
|
||||
case "with_backing":
|
||||
return analysisData.audio_urls.with_backing
|
||||
case "custom":
|
||||
return analysisData.audio_urls.custom_instrumental
|
||||
default:
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
// Loading state
|
||||
if (isLoading) {
|
||||
return (
|
||||
<div className="min-h-screen bg-background">
|
||||
<AppHeader />
|
||||
<main className="container mx-auto px-4 py-8 max-w-5xl">
|
||||
<Skeleton className="h-8 w-48 mb-4" />
|
||||
<Skeleton className="h-4 w-96 mb-8" />
|
||||
<Skeleton className="h-[200px] w-full mb-4" />
|
||||
<Skeleton className="h-32 w-full" />
|
||||
</main>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
// Error state
|
||||
if (error && !analysisData) {
|
||||
return (
|
||||
<div className="min-h-screen bg-background">
|
||||
<AppHeader />
|
||||
<main className="container mx-auto px-4 py-8 max-w-5xl">
|
||||
<Alert variant="destructive">
|
||||
<AlertCircle className="h-4 w-4" />
|
||||
<AlertDescription>{error}</AlertDescription>
|
||||
</Alert>
|
||||
<Button variant="outline" className="mt-4" onClick={() => router.push(`/jobs/${jobId}`)}>
|
||||
<ArrowLeft className="w-4 h-4 mr-2" />
|
||||
Back to Job
|
||||
</Button>
|
||||
</main>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
const analysis = analysisData?.analysis
|
||||
|
||||
return (
|
||||
<div className="min-h-screen bg-background">
|
||||
<AppHeader />
|
||||
|
||||
<main className="container mx-auto px-4 py-8 max-w-6xl">
|
||||
{/* Header */}
|
||||
<Button variant="ghost" className="mb-6" onClick={() => router.push(`/jobs/${jobId}`)}>
|
||||
<ArrowLeft className="w-4 h-4 mr-2" />
|
||||
Back to Job
|
||||
</Button>
|
||||
|
||||
<div className="mb-8">
|
||||
<h1 className="text-3xl font-bold mb-2">
|
||||
Select Instrumental Track
|
||||
</h1>
|
||||
<p className="text-muted-foreground">
|
||||
{analysisData?.artist} - {analysisData?.title}
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{/* Error Alert */}
|
||||
{error && (
|
||||
<Alert variant="destructive" className="mb-6">
|
||||
<AlertCircle className="h-4 w-4" />
|
||||
<AlertDescription>{error}</AlertDescription>
|
||||
</Alert>
|
||||
)}
|
||||
|
||||
{/* Analysis Summary */}
|
||||
{analysis && (
|
||||
<Card className="mb-6">
|
||||
<CardHeader>
|
||||
<CardTitle className="text-lg flex items-center gap-2">
|
||||
<Music className="h-5 w-5" />
|
||||
Backing Vocals Analysis
|
||||
</CardTitle>
|
||||
<CardDescription>
|
||||
Automated analysis of the backing vocals stem
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="flex flex-wrap gap-4">
|
||||
<div className="flex items-center gap-2">
|
||||
{analysis.has_audible_content ? (
|
||||
<Volume2 className="h-4 w-4 text-yellow-500" />
|
||||
) : (
|
||||
<VolumeX className="h-4 w-4 text-green-500" />
|
||||
)}
|
||||
<span className="text-sm">
|
||||
{analysis.has_audible_content
|
||||
? "Backing vocals detected"
|
||||
: "No backing vocals detected"}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{analysis.has_audible_content && (
|
||||
<>
|
||||
<Badge variant="outline">
|
||||
{analysis.audible_segments.length} audible segments
|
||||
</Badge>
|
||||
<Badge variant="outline">
|
||||
{analysis.audible_percentage.toFixed(1)}% of track
|
||||
</Badge>
|
||||
<Badge variant="outline">
|
||||
{analysis.total_audible_duration_seconds.toFixed(1)}s total
|
||||
</Badge>
|
||||
</>
|
||||
)}
|
||||
|
||||
<Badge
|
||||
variant={analysis.recommended_selection === "clean" ? "default" : "secondary"}
|
||||
>
|
||||
Recommended: {analysis.recommended_selection === "clean" ? "Clean Instrumental" : "Review Needed"}
|
||||
</Badge>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
)}
|
||||
|
||||
{/* Waveform Display */}
|
||||
{waveformData && analysis && (
|
||||
<Card className="mb-6">
|
||||
<CardHeader>
|
||||
<CardTitle className="text-lg">Backing Vocals Waveform</CardTitle>
|
||||
<CardDescription>
|
||||
Click to seek • {isSelectionMode ? "Drag to select mute region" : "Click 'Add Region' to start selecting"}
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<WaveformDisplay
|
||||
amplitudes={waveformData.amplitudes}
|
||||
duration={waveformData.duration}
|
||||
currentTime={currentTime}
|
||||
segments={analysis.audible_segments}
|
||||
muteRegions={muteRegions}
|
||||
onSeek={handleSeek}
|
||||
onRegionSelect={handleAddRegion}
|
||||
isSelecting={isSelectionMode}
|
||||
width={900}
|
||||
height={150}
|
||||
className="w-full overflow-x-auto"
|
||||
/>
|
||||
</CardContent>
|
||||
</Card>
|
||||
)}
|
||||
|
||||
<div className="grid lg:grid-cols-2 gap-6 mb-6">
|
||||
{/* Audio Players */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle className="text-lg">Audio Preview</CardTitle>
|
||||
<CardDescription>
|
||||
Listen to different instrumental options
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent className="space-y-4">
|
||||
{/* Audio source selector */}
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button
|
||||
variant={activeAudio === "backing" ? "default" : "outline"}
|
||||
size="sm"
|
||||
onClick={() => setActiveAudio("backing")}
|
||||
>
|
||||
Backing Vocals
|
||||
</Button>
|
||||
<Button
|
||||
variant={activeAudio === "clean" ? "default" : "outline"}
|
||||
size="sm"
|
||||
onClick={() => setActiveAudio("clean")}
|
||||
>
|
||||
Clean Instrumental
|
||||
</Button>
|
||||
<Button
|
||||
variant={activeAudio === "with_backing" ? "default" : "outline"}
|
||||
size="sm"
|
||||
onClick={() => setActiveAudio("with_backing")}
|
||||
>
|
||||
With Backing
|
||||
</Button>
|
||||
{analysisData?.has_custom_instrumental && (
|
||||
<Button
|
||||
variant={activeAudio === "custom" ? "default" : "outline"}
|
||||
size="sm"
|
||||
onClick={() => setActiveAudio("custom")}
|
||||
>
|
||||
Custom
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Audio player */}
|
||||
{getCurrentAudioUrl() && (
|
||||
<AudioPlayer
|
||||
src={getCurrentAudioUrl()!}
|
||||
onTimeUpdate={setCurrentTime}
|
||||
seekTo={seekTo}
|
||||
label={
|
||||
activeAudio === "backing"
|
||||
? "Backing Vocals Only"
|
||||
: activeAudio === "clean"
|
||||
? "Clean Instrumental"
|
||||
: activeAudio === "custom"
|
||||
? "Custom Instrumental"
|
||||
: "Instrumental + Backing Vocals"
|
||||
}
|
||||
/>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Region Selector */}
|
||||
<RegionSelector
|
||||
muteRegions={muteRegions}
|
||||
audibleSegments={analysis?.audible_segments}
|
||||
onAddRegion={handleAddRegion}
|
||||
onRemoveRegion={handleRemoveRegion}
|
||||
onClearAll={handleClearAll}
|
||||
onSeekTo={handleSeek}
|
||||
isSelectionMode={isSelectionMode}
|
||||
onToggleSelectionMode={() => setIsSelectionMode(!isSelectionMode)}
|
||||
/>
|
||||
</div>
|
||||
|
||||
{/* Create Custom Button */}
|
||||
{muteRegions.length > 0 && !analysisData?.has_custom_instrumental && (
|
||||
<Card className="mb-6">
|
||||
<CardContent className="py-4">
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<p className="font-medium">Ready to create custom instrumental</p>
|
||||
<p className="text-sm text-muted-foreground">
|
||||
{muteRegions.length} region{muteRegions.length > 1 ? "s" : ""} will be muted
|
||||
</p>
|
||||
</div>
|
||||
<Button onClick={handleCreateCustom} disabled={isCreatingCustom}>
|
||||
{isCreatingCustom ? (
|
||||
<>
|
||||
<Loader2 className="h-4 w-4 mr-2 animate-spin" />
|
||||
Creating...
|
||||
</>
|
||||
) : (
|
||||
"Create Custom Instrumental"
|
||||
)}
|
||||
</Button>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
)}
|
||||
|
||||
{/* Selection Options */}
|
||||
<Card className="mb-6">
|
||||
<CardHeader>
|
||||
<CardTitle className="text-lg">Final Selection</CardTitle>
|
||||
<CardDescription>
|
||||
Choose which instrumental to use for your karaoke video
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<RadioGroup value={selectedOption} onValueChange={setSelectedOption}>
|
||||
<div className="space-y-4">
|
||||
<div className="flex items-center space-x-3">
|
||||
<RadioGroupItem value="clean" id="clean" />
|
||||
<Label htmlFor="clean" className="flex-1 cursor-pointer">
|
||||
<div className="font-medium">Clean Instrumental</div>
|
||||
<div className="text-sm text-muted-foreground">
|
||||
Use the instrumental with no backing vocals at all
|
||||
</div>
|
||||
</Label>
|
||||
{analysis?.recommended_selection === "clean" && (
|
||||
<Badge variant="default" className="bg-green-600">
|
||||
<CheckCircle2 className="h-3 w-3 mr-1" />
|
||||
Recommended
|
||||
</Badge>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex items-center space-x-3">
|
||||
<RadioGroupItem value="with_backing" id="with_backing" />
|
||||
<Label htmlFor="with_backing" className="flex-1 cursor-pointer">
|
||||
<div className="font-medium">Instrumental with Backing Vocals</div>
|
||||
<div className="text-sm text-muted-foreground">
|
||||
Use the instrumental with all backing vocals included
|
||||
</div>
|
||||
</Label>
|
||||
</div>
|
||||
|
||||
{analysisData?.has_custom_instrumental && (
|
||||
<div className="flex items-center space-x-3">
|
||||
<RadioGroupItem value="custom" id="custom" />
|
||||
<Label htmlFor="custom" className="flex-1 cursor-pointer">
|
||||
<div className="font-medium">Custom Instrumental</div>
|
||||
<div className="text-sm text-muted-foreground">
|
||||
Use your custom instrumental with {muteRegions.length} muted region{muteRegions.length > 1 ? "s" : ""}
|
||||
</div>
|
||||
</Label>
|
||||
<Badge variant="secondary">Custom</Badge>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</RadioGroup>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Submit Button */}
|
||||
<div className="flex gap-4">
|
||||
<Button
|
||||
size="lg"
|
||||
onClick={handleSubmit}
|
||||
disabled={isSubmitting}
|
||||
className="flex-1 max-w-md"
|
||||
>
|
||||
{isSubmitting ? (
|
||||
<>
|
||||
<Loader2 className="h-4 w-4 mr-2 animate-spin" />
|
||||
Submitting...
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<CheckCircle2 className="h-4 w-4 mr-2" />
|
||||
Confirm Selection & Continue
|
||||
</>
|
||||
)}
|
||||
</Button>
|
||||
</div>
|
||||
</main>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default function InstrumentalReviewPage() {
|
||||
return (
|
||||
<ProtectedRoute>
|
||||
<InstrumentalReviewContent />
|
||||
</ProtectedRoute>
|
||||
)
|
||||
}
|
||||
@@ -1,216 +0,0 @@
|
||||
"use client"
|
||||
|
||||
import { useParams, useRouter } from "next/navigation"
|
||||
import { useJobs } from "@/lib/jobs"
|
||||
import { AppHeader } from "@/components/app-header"
|
||||
import { ProtectedRoute } from "@/components/protected-route"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import { Progress } from "@/components/ui/progress"
|
||||
import { StatusBadge } from "@/components/status-badge"
|
||||
import { Alert, AlertDescription } from "@/components/ui/alert"
|
||||
import {
|
||||
ArrowLeft,
|
||||
Download,
|
||||
Youtube,
|
||||
FileAudio,
|
||||
Calendar,
|
||||
Clock,
|
||||
CheckCircle2,
|
||||
Loader2,
|
||||
XCircle,
|
||||
AlertCircle,
|
||||
} from "lucide-react"
|
||||
|
||||
function JobDetailContent() {
|
||||
const params = useParams()
|
||||
const router = useRouter()
|
||||
const { getJobById } = useJobs()
|
||||
|
||||
const job = getJobById(params.id as string)
|
||||
|
||||
if (!job) {
|
||||
return (
|
||||
<div className="min-h-screen bg-background">
|
||||
<AppHeader />
|
||||
<main className="container mx-auto px-4 py-8">
|
||||
<Alert variant="destructive">
|
||||
<AlertCircle className="h-4 w-4" />
|
||||
<AlertDescription>Job not found</AlertDescription>
|
||||
</Alert>
|
||||
<Button variant="outline" className="mt-4 bg-transparent" onClick={() => router.push("/jobs")}>
|
||||
<ArrowLeft className="w-4 h-4 mr-2" />
|
||||
Back to Jobs
|
||||
</Button>
|
||||
</main>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
const getStageIcon = (status: string) => {
|
||||
switch (status) {
|
||||
case "completed":
|
||||
return <CheckCircle2 className="w-5 h-5 text-green-400" />
|
||||
case "in_progress":
|
||||
return <Loader2 className="w-5 h-5 text-blue-400 animate-spin" />
|
||||
case "failed":
|
||||
return <XCircle className="w-5 h-5 text-destructive" />
|
||||
default:
|
||||
return <div className="w-5 h-5 rounded-full border-2 border-muted" />
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="min-h-screen bg-background">
|
||||
<AppHeader />
|
||||
|
||||
<main className="container mx-auto px-4 py-8 max-w-5xl">
|
||||
<Button variant="ghost" className="mb-6" onClick={() => router.push("/jobs")}>
|
||||
<ArrowLeft className="w-4 h-4 mr-2" />
|
||||
Back to Jobs
|
||||
</Button>
|
||||
|
||||
{/* Header */}
|
||||
<div className="mb-8">
|
||||
<div className="flex items-start justify-between gap-4 mb-4">
|
||||
<div className="flex-1">
|
||||
<h1 className="text-4xl font-bold mb-2 text-balance">
|
||||
{job.artist} - {job.title}
|
||||
</h1>
|
||||
<div className="flex flex-wrap items-center gap-4 text-muted-foreground">
|
||||
<span className="flex items-center gap-1.5">
|
||||
<Calendar className="w-4 h-4" />
|
||||
Created {new Date(job.createdAt).toLocaleDateString()}
|
||||
</span>
|
||||
<span className="flex items-center gap-1.5">
|
||||
<Clock className="w-4 h-4" />
|
||||
Updated {new Date(job.updatedAt).toLocaleDateString()}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
<StatusBadge status={job.status} size="lg" />
|
||||
</div>
|
||||
|
||||
{job.status === "completed" && (
|
||||
<Button size="lg" className="bg-primary hover:bg-primary/90">
|
||||
<Download className="w-5 h-5 mr-2" />
|
||||
Download Karaoke Video
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Source Info */}
|
||||
<Card className="mb-6">
|
||||
<CardHeader>
|
||||
<CardTitle className="text-lg">Source Information</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="flex items-center gap-3">
|
||||
{job.sourceType === "youtube" ? (
|
||||
<>
|
||||
<div className="w-10 h-10 rounded-lg bg-red-500/10 flex items-center justify-center">
|
||||
<Youtube className="w-5 h-5 text-red-500" />
|
||||
</div>
|
||||
<div className="flex-1 min-w-0">
|
||||
<p className="text-sm font-medium">YouTube Video</p>
|
||||
<p className="text-sm text-muted-foreground truncate">{job.sourceUrl}</p>
|
||||
</div>
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<div className="w-10 h-10 rounded-lg bg-primary/10 flex items-center justify-center">
|
||||
<FileAudio className="w-5 h-5 text-primary" />
|
||||
</div>
|
||||
<div className="flex-1 min-w-0">
|
||||
<p className="text-sm font-medium">Uploaded Audio File</p>
|
||||
<p className="text-sm text-muted-foreground truncate">{job.fileName}</p>
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Progress Section */}
|
||||
<Card className="mb-6">
|
||||
<CardHeader>
|
||||
<CardTitle className="text-lg">Processing Pipeline</CardTitle>
|
||||
<CardDescription>Track the progress of your karaoke video generation</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="space-y-6">
|
||||
{job.stages.map((stage, index) => (
|
||||
<div key={index} className="flex items-start gap-4">
|
||||
<div className="pt-1">{getStageIcon(stage.status)}</div>
|
||||
<div className="flex-1">
|
||||
<div className="flex items-center justify-between mb-2">
|
||||
<h3 className="font-medium">{stage.name}</h3>
|
||||
{stage.status === "in_progress" && (
|
||||
<span className="text-sm text-muted-foreground">{stage.progress}%</span>
|
||||
)}
|
||||
</div>
|
||||
{stage.status === "in_progress" && <Progress value={stage.progress} className="h-2 bg-muted" />}
|
||||
{stage.message && <p className="text-sm text-muted-foreground mt-1">{stage.message}</p>}
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
{job.status === "processing" && (
|
||||
<div className="mt-6 pt-6 border-t border-border">
|
||||
<div className="flex items-center justify-between mb-2">
|
||||
<p className="text-sm font-medium">Overall Progress</p>
|
||||
<span className="text-sm text-muted-foreground">{job.progress}%</span>
|
||||
</div>
|
||||
<Progress value={job.progress} className="h-3" />
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Error Message */}
|
||||
{job.status === "failed" && job.errorMessage && (
|
||||
<Alert variant="destructive">
|
||||
<AlertCircle className="h-4 w-4" />
|
||||
<AlertDescription>{job.errorMessage}</AlertDescription>
|
||||
</Alert>
|
||||
)}
|
||||
|
||||
{/* Action Alerts */}
|
||||
{job.status === "awaiting_review" && (
|
||||
<Alert className="bg-secondary/10 border-secondary/30">
|
||||
<AlertCircle className="h-4 w-4 text-secondary" />
|
||||
<AlertDescription className="text-foreground">
|
||||
Your karaoke video is ready for review. Please check the lyrics synchronization and approve or request
|
||||
changes.
|
||||
</AlertDescription>
|
||||
</Alert>
|
||||
)}
|
||||
|
||||
{job.status === "awaiting_instrumental" && (
|
||||
<Alert className="bg-secondary/10 border-secondary/30">
|
||||
<AlertCircle className="h-4 w-4 text-secondary" />
|
||||
<AlertDescription className="text-foreground flex items-center justify-between">
|
||||
<span>Multiple instrumental versions were detected. Please select your preferred version to continue.</span>
|
||||
<Button
|
||||
size="sm"
|
||||
onClick={() => router.push(`/jobs/${job.id}/instrumental-review`)}
|
||||
className="ml-4"
|
||||
>
|
||||
Select Instrumental
|
||||
</Button>
|
||||
</AlertDescription>
|
||||
</Alert>
|
||||
)}
|
||||
</main>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default function JobDetailPage() {
|
||||
return (
|
||||
<ProtectedRoute>
|
||||
<JobDetailContent />
|
||||
</ProtectedRoute>
|
||||
)
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
export default function Loading() {
|
||||
return null
|
||||
}
|
||||
@@ -1,366 +0,0 @@
|
||||
"use client"
|
||||
|
||||
import type React from "react"
|
||||
|
||||
import { useState } from "react"
|
||||
import { useRouter } from "next/navigation"
|
||||
import { useAuth } from "@/lib/auth"
|
||||
import { useJobs } from "@/lib/jobs"
|
||||
import { AppHeader } from "@/components/app-header"
|
||||
import { ProtectedRoute } from "@/components/protected-route"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Input } from "@/components/ui/input"
|
||||
import { Label } from "@/components/ui/label"
|
||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import { Alert, AlertDescription } from "@/components/ui/alert"
|
||||
import { Youtube, Upload, AlertCircle, Sparkles } from "lucide-react"
|
||||
|
||||
function NewJobContent() {
|
||||
const { user, updateCredits } = useAuth()
|
||||
const { addJob } = useJobs()
|
||||
const router = useRouter()
|
||||
|
||||
const [activeTab, setActiveTab] = useState("youtube")
|
||||
|
||||
// YouTube form state
|
||||
const [youtubeUrl, setYoutubeUrl] = useState("")
|
||||
const [youtubeArtist, setYoutubeArtist] = useState("")
|
||||
const [youtubeTitle, setYoutubeTitle] = useState("")
|
||||
|
||||
// Upload form state
|
||||
const [uploadFile, setUploadFile] = useState<File | null>(null)
|
||||
const [uploadArtist, setUploadArtist] = useState("")
|
||||
const [uploadTitle, setUploadTitle] = useState("")
|
||||
|
||||
const [isSubmitting, setIsSubmitting] = useState(false)
|
||||
const [error, setError] = useState("")
|
||||
|
||||
if (!user) return null
|
||||
|
||||
const insufficientCredits = user.credits < 1
|
||||
|
||||
const handleYoutubeSubmit = async (e: React.FormEvent) => {
|
||||
e.preventDefault()
|
||||
setError("")
|
||||
|
||||
if (!youtubeUrl.trim()) {
|
||||
setError("Please enter a YouTube URL")
|
||||
return
|
||||
}
|
||||
if (!youtubeArtist.trim() || !youtubeTitle.trim()) {
|
||||
setError("Please enter both artist and title")
|
||||
return
|
||||
}
|
||||
|
||||
if (insufficientCredits) {
|
||||
setError("Insufficient credits. Please purchase more credits to continue.")
|
||||
return
|
||||
}
|
||||
|
||||
setIsSubmitting(true)
|
||||
|
||||
// Simulate processing
|
||||
setTimeout(() => {
|
||||
addJob({
|
||||
userId: user.token.includes("demo") ? "demo" : user.token,
|
||||
artist: youtubeArtist,
|
||||
title: youtubeTitle,
|
||||
status: "queued",
|
||||
sourceType: "youtube",
|
||||
sourceUrl: youtubeUrl,
|
||||
})
|
||||
|
||||
updateCredits(user.credits - 1)
|
||||
setIsSubmitting(false)
|
||||
router.push("/jobs")
|
||||
}, 1000)
|
||||
}
|
||||
|
||||
const handleUploadSubmit = async (e: React.FormEvent) => {
|
||||
e.preventDefault()
|
||||
setError("")
|
||||
|
||||
if (!uploadFile) {
|
||||
setError("Please select an audio file")
|
||||
return
|
||||
}
|
||||
if (!uploadArtist.trim() || !uploadTitle.trim()) {
|
||||
setError("Please enter both artist and title")
|
||||
return
|
||||
}
|
||||
|
||||
if (insufficientCredits) {
|
||||
setError("Insufficient credits. Please purchase more credits to continue.")
|
||||
return
|
||||
}
|
||||
|
||||
setIsSubmitting(true)
|
||||
|
||||
// Simulate processing
|
||||
setTimeout(() => {
|
||||
addJob({
|
||||
userId: user.token.includes("demo") ? "demo" : user.token,
|
||||
artist: uploadArtist,
|
||||
title: uploadTitle,
|
||||
status: "queued",
|
||||
sourceType: "upload",
|
||||
fileName: uploadFile.name,
|
||||
})
|
||||
|
||||
updateCredits(user.credits - 1)
|
||||
setIsSubmitting(false)
|
||||
router.push("/jobs")
|
||||
}, 1000)
|
||||
}
|
||||
|
||||
const handleFileChange = (e: React.ChangeEvent<HTMLInputElement>) => {
|
||||
const file = e.target.files?.[0]
|
||||
if (file) {
|
||||
const validTypes = ["audio/mpeg", "audio/wav", "audio/flac", "audio/mp4", "audio/ogg"]
|
||||
if (!validTypes.includes(file.type) && !file.name.match(/\.(mp3|wav|flac|m4a|ogg)$/i)) {
|
||||
setError("Invalid file type. Please upload MP3, WAV, FLAC, M4A, or OGG files.")
|
||||
return
|
||||
}
|
||||
if (file.size > 50 * 1024 * 1024) {
|
||||
// 50MB limit
|
||||
setError("File too large. Maximum size is 50MB.")
|
||||
return
|
||||
}
|
||||
setUploadFile(file)
|
||||
setError("")
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="min-h-screen bg-background">
|
||||
<AppHeader />
|
||||
|
||||
<main className="container mx-auto px-4 py-8 max-w-3xl">
|
||||
<div className="mb-8">
|
||||
<h1 className="text-4xl font-bold mb-2">Create New Karaoke Video</h1>
|
||||
<p className="text-muted-foreground">Start from a YouTube URL or upload your own audio file</p>
|
||||
</div>
|
||||
|
||||
{insufficientCredits && (
|
||||
<Alert variant="destructive" className="mb-6">
|
||||
<AlertCircle className="h-4 w-4" />
|
||||
<AlertDescription>
|
||||
You don't have enough credits to create a new job.{" "}
|
||||
<a href="#" className="underline font-medium">
|
||||
Purchase more credits
|
||||
</a>{" "}
|
||||
to continue.
|
||||
</AlertDescription>
|
||||
</Alert>
|
||||
)}
|
||||
|
||||
<Card className="border-border/50">
|
||||
<CardHeader>
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<CardTitle>Source Selection</CardTitle>
|
||||
<CardDescription>Choose how you want to provide the audio</CardDescription>
|
||||
</div>
|
||||
<div className="flex items-center gap-2 px-3 py-1.5 rounded-lg bg-primary/10 border border-primary/20">
|
||||
<Sparkles className="w-4 h-4 text-primary" />
|
||||
<span className="text-sm font-medium text-primary">Cost: 1 credit</span>
|
||||
</div>
|
||||
</div>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<Tabs value={activeTab} onValueChange={setActiveTab}>
|
||||
<TabsList className="grid w-full grid-cols-2">
|
||||
<TabsTrigger value="youtube" className="gap-2">
|
||||
<Youtube className="w-4 h-4" />
|
||||
From YouTube URL
|
||||
</TabsTrigger>
|
||||
<TabsTrigger value="upload" className="gap-2">
|
||||
<Upload className="w-4 h-4" />
|
||||
Upload Audio File
|
||||
</TabsTrigger>
|
||||
</TabsList>
|
||||
|
||||
<TabsContent value="youtube" className="mt-6">
|
||||
<form onSubmit={handleYoutubeSubmit} className="space-y-6">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="youtube-url">YouTube URL</Label>
|
||||
<div className="relative">
|
||||
<Youtube className="absolute left-3 top-1/2 -translate-y-1/2 w-4 h-4 text-muted-foreground" />
|
||||
<Input
|
||||
id="youtube-url"
|
||||
type="url"
|
||||
placeholder="https://youtube.com/watch?v=..."
|
||||
value={youtubeUrl}
|
||||
onChange={(e) => setYoutubeUrl(e.target.value)}
|
||||
className="pl-10"
|
||||
disabled={isSubmitting}
|
||||
/>
|
||||
</div>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
Paste the URL of the YouTube video you want to convert
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div className="grid md:grid-cols-2 gap-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="youtube-artist">Artist Name</Label>
|
||||
<Input
|
||||
id="youtube-artist"
|
||||
type="text"
|
||||
placeholder="Artist name"
|
||||
value={youtubeArtist}
|
||||
onChange={(e) => setYoutubeArtist(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="youtube-title">Song Title</Label>
|
||||
<Input
|
||||
id="youtube-title"
|
||||
type="text"
|
||||
placeholder="Song title"
|
||||
value={youtubeTitle}
|
||||
onChange={(e) => setYoutubeTitle(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{error && (
|
||||
<Alert variant="destructive">
|
||||
<AlertCircle className="h-4 w-4" />
|
||||
<AlertDescription>{error}</AlertDescription>
|
||||
</Alert>
|
||||
)}
|
||||
|
||||
<div className="flex gap-3">
|
||||
<Button
|
||||
type="submit"
|
||||
className="flex-1 bg-primary hover:bg-primary/90"
|
||||
disabled={isSubmitting || insufficientCredits}
|
||||
>
|
||||
{isSubmitting ? "Creating Job..." : "Create Karaoke Video"}
|
||||
</Button>
|
||||
<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
onClick={() => router.push("/dashboard")}
|
||||
disabled={isSubmitting}
|
||||
>
|
||||
Cancel
|
||||
</Button>
|
||||
</div>
|
||||
</form>
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="upload" className="mt-6">
|
||||
<form onSubmit={handleUploadSubmit} className="space-y-6">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="audio-file">Audio File</Label>
|
||||
<div
|
||||
className={`relative border-2 border-dashed rounded-lg p-8 text-center transition-colors ${
|
||||
uploadFile
|
||||
? "border-primary bg-primary/5"
|
||||
: "border-border hover:border-primary/50 hover:bg-card/50"
|
||||
}`}
|
||||
>
|
||||
<Input
|
||||
id="audio-file"
|
||||
type="file"
|
||||
accept=".mp3,.wav,.flac,.m4a,.ogg,audio/*"
|
||||
onChange={handleFileChange}
|
||||
className="absolute inset-0 w-full h-full opacity-0 cursor-pointer"
|
||||
disabled={isSubmitting}
|
||||
/>
|
||||
<Upload className="w-8 h-8 mx-auto mb-2 text-muted-foreground" />
|
||||
{uploadFile ? (
|
||||
<div>
|
||||
<p className="font-medium text-foreground">{uploadFile.name}</p>
|
||||
<p className="text-sm text-muted-foreground">
|
||||
{(uploadFile.size / 1024 / 1024).toFixed(2)} MB
|
||||
</p>
|
||||
</div>
|
||||
) : (
|
||||
<div>
|
||||
<p className="font-medium text-foreground mb-1">Click to upload or drag and drop</p>
|
||||
<p className="text-sm text-muted-foreground">MP3, WAV, FLAC, M4A, or OGG (max 50MB)</p>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="grid md:grid-cols-2 gap-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="upload-artist">
|
||||
Artist Name <span className="text-destructive">*</span>
|
||||
</Label>
|
||||
<Input
|
||||
id="upload-artist"
|
||||
type="text"
|
||||
placeholder="Artist name"
|
||||
value={uploadArtist}
|
||||
onChange={(e) => setUploadArtist(e.target.value)}
|
||||
required
|
||||
disabled={isSubmitting}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="upload-title">
|
||||
Song Title <span className="text-destructive">*</span>
|
||||
</Label>
|
||||
<Input
|
||||
id="upload-title"
|
||||
type="text"
|
||||
placeholder="Song title"
|
||||
value={uploadTitle}
|
||||
onChange={(e) => setUploadTitle(e.target.value)}
|
||||
required
|
||||
disabled={isSubmitting}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{error && (
|
||||
<Alert variant="destructive">
|
||||
<AlertCircle className="h-4 w-4" />
|
||||
<AlertDescription>{error}</AlertDescription>
|
||||
</Alert>
|
||||
)}
|
||||
|
||||
<div className="flex gap-3">
|
||||
<Button
|
||||
type="submit"
|
||||
className="flex-1 bg-primary hover:bg-primary/90"
|
||||
disabled={isSubmitting || insufficientCredits}
|
||||
>
|
||||
{isSubmitting ? "Creating Job..." : "Create Karaoke Video"}
|
||||
</Button>
|
||||
<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
onClick={() => router.push("/dashboard")}
|
||||
disabled={isSubmitting}
|
||||
>
|
||||
Cancel
|
||||
</Button>
|
||||
</div>
|
||||
</form>
|
||||
</TabsContent>
|
||||
</Tabs>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</main>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default function NewJobPage() {
|
||||
return (
|
||||
<ProtectedRoute>
|
||||
<NewJobContent />
|
||||
</ProtectedRoute>
|
||||
)
|
||||
}
|
||||
@@ -1,193 +0,0 @@
|
||||
"use client"
|
||||
|
||||
import { useState } from "react"
|
||||
import { useAuth } from "@/lib/auth"
|
||||
import { useJobs } from "@/lib/jobs"
|
||||
import type { JobStatus } from "@/lib/types"
|
||||
import { AppHeader } from "@/components/app-header"
|
||||
import { ProtectedRoute } from "@/components/protected-route"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Input } from "@/components/ui/input"
|
||||
import { Card, CardContent } from "@/components/ui/card"
|
||||
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select"
|
||||
import { StatusBadge } from "@/components/status-badge"
|
||||
import { Plus, Search, Clock, Download, ChevronRight } from "lucide-react"
|
||||
import Link from "next/link"
|
||||
|
||||
function JobsListContent() {
|
||||
const { user } = useAuth()
|
||||
const { jobs, getUserJobs } = useJobs()
|
||||
|
||||
const [searchQuery, setSearchQuery] = useState("")
|
||||
const [statusFilter, setStatusFilter] = useState<JobStatus | "all">("all")
|
||||
|
||||
if (!user) return null
|
||||
|
||||
const userJobs = getUserJobs(user.token.includes("demo") ? "demo" : user.token)
|
||||
|
||||
const filteredJobs = userJobs.filter((job) => {
|
||||
const matchesSearch =
|
||||
searchQuery === "" ||
|
||||
job.artist.toLowerCase().includes(searchQuery.toLowerCase()) ||
|
||||
job.title.toLowerCase().includes(searchQuery.toLowerCase())
|
||||
|
||||
const matchesStatus = statusFilter === "all" || job.status === statusFilter
|
||||
|
||||
return matchesSearch && matchesStatus
|
||||
})
|
||||
|
||||
return (
|
||||
<div className="min-h-screen bg-background">
|
||||
<AppHeader />
|
||||
|
||||
<main className="container mx-auto px-4 py-8">
|
||||
<div className="flex items-center justify-between mb-8">
|
||||
<div>
|
||||
<h1 className="text-4xl font-bold mb-2">My Jobs</h1>
|
||||
<p className="text-muted-foreground">Manage and track your karaoke video projects</p>
|
||||
</div>
|
||||
|
||||
<Button asChild className="bg-primary hover:bg-primary/90">
|
||||
<Link href="/jobs/new">
|
||||
<Plus className="w-4 h-4 mr-2" />
|
||||
New Job
|
||||
</Link>
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{/* Filters */}
|
||||
<Card className="mb-6">
|
||||
<CardContent className="pt-6">
|
||||
<div className="flex flex-col sm:flex-row gap-4">
|
||||
<div className="flex-1 relative">
|
||||
<Search className="absolute left-3 top-1/2 -translate-y-1/2 w-4 h-4 text-muted-foreground" />
|
||||
<Input
|
||||
placeholder="Search by artist or title..."
|
||||
value={searchQuery}
|
||||
onChange={(e) => setSearchQuery(e.target.value)}
|
||||
className="pl-10"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<Select value={statusFilter} onValueChange={(value) => setStatusFilter(value as JobStatus | "all")}>
|
||||
<SelectTrigger className="w-full sm:w-[200px]">
|
||||
<SelectValue placeholder="Filter by status" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
<SelectItem value="all">All Status</SelectItem>
|
||||
<SelectItem value="queued">Queued</SelectItem>
|
||||
<SelectItem value="processing">Processing</SelectItem>
|
||||
<SelectItem value="awaiting_review">Awaiting Review</SelectItem>
|
||||
<SelectItem value="awaiting_instrumental">Select Instrumental</SelectItem>
|
||||
<SelectItem value="completed">Completed</SelectItem>
|
||||
<SelectItem value="failed">Failed</SelectItem>
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Results count */}
|
||||
<div className="mb-4">
|
||||
<p className="text-sm text-muted-foreground">
|
||||
Showing {filteredJobs.length} of {userJobs.length} jobs
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{/* Jobs List */}
|
||||
{filteredJobs.length === 0 ? (
|
||||
<Card>
|
||||
<CardContent className="text-center py-12">
|
||||
<div className="w-16 h-16 mx-auto mb-4 rounded-full bg-muted flex items-center justify-center">
|
||||
<Search className="w-8 h-8 text-muted-foreground" />
|
||||
</div>
|
||||
<h3 className="font-semibold mb-2">No jobs found</h3>
|
||||
<p className="text-muted-foreground mb-4">
|
||||
{searchQuery || statusFilter !== "all"
|
||||
? "Try adjusting your filters"
|
||||
: "Create your first karaoke video to get started"}
|
||||
</p>
|
||||
{!searchQuery && statusFilter === "all" && (
|
||||
<Button asChild>
|
||||
<Link href="/jobs/new">
|
||||
<Plus className="w-4 h-4 mr-2" />
|
||||
Create Job
|
||||
</Link>
|
||||
</Button>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
) : (
|
||||
<div className="space-y-3">
|
||||
{filteredJobs.map((job) => (
|
||||
<Link key={job.id} href={`/jobs/${job.id}`}>
|
||||
<Card className="hover:border-primary/50 hover:bg-card/80 transition-all group cursor-pointer">
|
||||
<CardContent className="p-6">
|
||||
<div className="flex items-center justify-between gap-4">
|
||||
<div className="flex-1 min-w-0">
|
||||
<div className="flex items-center gap-3 mb-2">
|
||||
<h3 className="font-semibold text-lg group-hover:text-primary transition-colors truncate">
|
||||
{job.artist} - {job.title}
|
||||
</h3>
|
||||
<StatusBadge status={job.status} />
|
||||
</div>
|
||||
|
||||
<div className="flex flex-wrap items-center gap-4 text-sm text-muted-foreground">
|
||||
<span className="flex items-center gap-1.5">
|
||||
<Clock className="w-3.5 h-3.5" />
|
||||
{new Date(job.createdAt).toLocaleDateString("en-US", {
|
||||
month: "short",
|
||||
day: "numeric",
|
||||
year: "numeric",
|
||||
})}
|
||||
</span>
|
||||
|
||||
<span className="text-xs">
|
||||
{job.sourceType === "youtube" ? "From YouTube" : "Uploaded File"}
|
||||
</span>
|
||||
|
||||
{(job.status === "processing" || job.status === "queued") && (
|
||||
<span className="text-xs font-medium">{job.progress}% complete</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{(job.status === "processing" || job.status === "queued") && (
|
||||
<div className="mt-3 w-full max-w-xs">
|
||||
<div className="h-1.5 bg-muted rounded-full overflow-hidden">
|
||||
<div
|
||||
className="h-full bg-primary transition-all duration-300"
|
||||
style={{ width: `${job.progress}%` }}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex items-center gap-2">
|
||||
{job.status === "completed" && (
|
||||
<Button size="sm" variant="outline" onClick={(e) => e.preventDefault()}>
|
||||
<Download className="w-4 h-4 mr-2" />
|
||||
Download
|
||||
</Button>
|
||||
)}
|
||||
<ChevronRight className="w-5 h-5 text-muted-foreground group-hover:text-primary transition-colors" />
|
||||
</div>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</Link>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</main>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default function JobsPage() {
|
||||
return (
|
||||
<ProtectedRoute>
|
||||
<JobsListContent />
|
||||
</ProtectedRoute>
|
||||
)
|
||||
}
|
||||
+8
-24
@@ -1,32 +1,17 @@
|
||||
import type React from "react"
|
||||
import type { Metadata } from "next"
|
||||
import { Geist, Geist_Mono } from "next/font/google"
|
||||
import { Analytics } from "@vercel/analytics/next"
|
||||
import "./globals.css"
|
||||
|
||||
const _geist = Geist({ subsets: ["latin"] })
|
||||
const _geistMono = Geist_Mono({ subsets: ["latin"] })
|
||||
|
||||
export const metadata: Metadata = {
|
||||
title: "Nomad Karaoke - Wherever you are, sing!",
|
||||
description: "Cloud-based karaoke video generation service",
|
||||
generator: "v0.app",
|
||||
title: "Nomad Karaoke: Generator",
|
||||
description: "Generate professional karaoke videos with AI-powered vocal separation and synchronized lyrics",
|
||||
icons: {
|
||||
icon: [
|
||||
{
|
||||
url: "/icon-light-32x32.png",
|
||||
media: "(prefers-color-scheme: light)",
|
||||
},
|
||||
{
|
||||
url: "/icon-dark-32x32.png",
|
||||
media: "(prefers-color-scheme: dark)",
|
||||
},
|
||||
{
|
||||
url: "/icon.svg",
|
||||
type: "image/svg+xml",
|
||||
},
|
||||
{ url: "/favicon.ico", sizes: "any" },
|
||||
{ url: "/favicon-16x16.png", sizes: "16x16", type: "image/png" },
|
||||
{ url: "/favicon-32x32.png", sizes: "32x32", type: "image/png" },
|
||||
],
|
||||
apple: "/apple-icon.png",
|
||||
apple: "/apple-touch-icon.png",
|
||||
},
|
||||
}
|
||||
|
||||
@@ -36,10 +21,9 @@ export default function RootLayout({
|
||||
children: React.ReactNode
|
||||
}>) {
|
||||
return (
|
||||
<html lang="en">
|
||||
<body className={`font-sans antialiased`}>
|
||||
<html lang="en" className="dark">
|
||||
<body className="font-sans antialiased" style={{ background: 'var(--bg)', color: 'var(--text)' }}>
|
||||
{children}
|
||||
<Analytics />
|
||||
</body>
|
||||
</html>
|
||||
)
|
||||
|
||||
+221
-121
@@ -1,143 +1,243 @@
|
||||
"use client"
|
||||
|
||||
import type React from "react"
|
||||
|
||||
import { useState } from "react"
|
||||
import { useRouter } from "next/navigation"
|
||||
import { useAuth } from "@/lib/auth"
|
||||
import { Logo } from "@/components/logo"
|
||||
import { useState, useEffect } from "react"
|
||||
import { api, Job, getAccessToken } from "@/lib/api"
|
||||
import { useAutoMode, getAutoModeFromUrl } from "@/lib/auto-mode"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Input } from "@/components/ui/input"
|
||||
import { Label } from "@/components/ui/label"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import { Sparkles } from "lucide-react"
|
||||
import { Music2, RefreshCw, Loader2, Zap, ZapOff, KeyRound, Moon, Sun } from "lucide-react"
|
||||
import { JobCard } from "@/components/job"
|
||||
import { JobSubmission } from "@/components/job/JobSubmission"
|
||||
import { AuthStatus } from "@/components/auth"
|
||||
import { AutoProcessor } from "@/components/AutoProcessor"
|
||||
import { useTheme } from "@/lib/theme"
|
||||
import {
|
||||
Tooltip,
|
||||
TooltipContent,
|
||||
TooltipProvider,
|
||||
TooltipTrigger,
|
||||
} from "@/components/ui/tooltip"
|
||||
|
||||
export default function LandingPage() {
|
||||
const [token, setToken] = useState("")
|
||||
const [error, setError] = useState("")
|
||||
const [isLoading, setIsLoading] = useState(false)
|
||||
const { login } = useAuth()
|
||||
const router = useRouter()
|
||||
export default function HomePage() {
|
||||
const [jobs, setJobs] = useState<Job[]>([])
|
||||
const [isLoadingJobs, setIsLoadingJobs] = useState(true)
|
||||
const [isAuthenticated, setIsAuthenticated] = useState(false)
|
||||
const { enabled: autoModeEnabled, setEnabled: setAutoMode, toggle: toggleAutoMode } = useAutoMode()
|
||||
const { isDarkMode, toggleTheme, mounted } = useTheme()
|
||||
|
||||
const handleLogin = async (e: React.FormEvent) => {
|
||||
e.preventDefault()
|
||||
setError("")
|
||||
setIsLoading(true)
|
||||
// Check auth status on mount
|
||||
useEffect(() => {
|
||||
setIsAuthenticated(!!getAccessToken())
|
||||
}, [])
|
||||
|
||||
if (!token.trim()) {
|
||||
setError("Please enter your access token")
|
||||
setIsLoading(false)
|
||||
// Initialize auto-mode from URL parameter on mount
|
||||
useEffect(() => {
|
||||
const urlAutoMode = getAutoModeFromUrl()
|
||||
if (urlAutoMode) {
|
||||
setAutoMode(true)
|
||||
}
|
||||
}, [setAutoMode])
|
||||
|
||||
// Load jobs on mount (only if authenticated)
|
||||
useEffect(() => {
|
||||
if (!isAuthenticated) {
|
||||
setIsLoadingJobs(false)
|
||||
return
|
||||
}
|
||||
loadJobs()
|
||||
// Poll for updates every 10 seconds
|
||||
const interval = setInterval(loadJobs, 10000)
|
||||
return () => clearInterval(interval)
|
||||
}, [isAuthenticated])
|
||||
|
||||
const success = await login(token)
|
||||
setIsLoading(false)
|
||||
|
||||
if (success) {
|
||||
router.push("/dashboard")
|
||||
} else {
|
||||
setError("Invalid token. Please try again.")
|
||||
async function loadJobs() {
|
||||
if (!getAccessToken()) {
|
||||
setIsLoadingJobs(false)
|
||||
return
|
||||
}
|
||||
try {
|
||||
const data = await api.listJobs({ limit: 20 })
|
||||
setJobs(data)
|
||||
} catch (err: any) {
|
||||
// Don't log auth errors - user just needs to authenticate
|
||||
if (err?.status !== 401) {
|
||||
console.error("Failed to load jobs:", err)
|
||||
}
|
||||
} finally {
|
||||
setIsLoadingJobs(false)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="min-h-screen bg-gradient-to-b from-background via-background to-card">
|
||||
{/* Animated background effects */}
|
||||
<div className="fixed inset-0 -z-10 overflow-hidden">
|
||||
<div className="absolute top-1/4 left-1/4 w-96 h-96 bg-primary/10 rounded-full blur-3xl animate-pulse" />
|
||||
<div className="absolute bottom-1/4 right-1/4 w-96 h-96 bg-secondary/10 rounded-full blur-3xl animate-pulse delay-1000" />
|
||||
</div>
|
||||
<div className="min-h-screen" style={{ background: 'var(--bg)' }}>
|
||||
{/* AutoProcessor - handles non-interactive mode */}
|
||||
<AutoProcessor jobs={jobs} onJobsChanged={loadJobs} />
|
||||
|
||||
<div className="container mx-auto px-4 py-8 lg:py-12">
|
||||
{/* Hero Section - horizontal layout on desktop */}
|
||||
<div className="max-w-6xl mx-auto">
|
||||
<div className="grid lg:grid-cols-2 gap-8 lg:gap-12 items-center">
|
||||
{/* Left: Hero Content */}
|
||||
<div className="text-center lg:text-left">
|
||||
<Logo size="lg" className="mb-8 justify-center lg:justify-start" />
|
||||
|
||||
<div className="inline-flex items-center gap-2 px-4 py-2 rounded-full bg-primary/10 border border-primary/20 mb-6">
|
||||
<Sparkles className="w-4 h-4 text-primary" />
|
||||
<span className="text-sm text-primary font-medium">AI-Powered Karaoke Generation</span>
|
||||
</div>
|
||||
|
||||
<h1 className="text-4xl md:text-4xl lg:text-4xl font-bold mb-4 text-balance leading-tight">
|
||||
Wherever you are, <span className="text-primary">sing!</span>
|
||||
</h1>
|
||||
|
||||
<p className="text-base md:text-lg text-muted-foreground text-pretty leading-relaxed">
|
||||
Transform any song into professional karaoke videos with AI-powered vocal separation, synchronized
|
||||
lyrics, and stunning visuals.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{/* Right: Login Card */}
|
||||
{/* Header */}
|
||||
<header className="border-b backdrop-blur-sm sticky top-0 z-10" style={{ borderColor: 'var(--card-border)', backgroundColor: 'var(--card)' }}>
|
||||
<div className="px-4 py-3 flex items-center justify-between">
|
||||
<div className="flex items-center gap-3">
|
||||
<img src="/nomad-karaoke-logo.svg" alt="Nomad Karaoke" className="h-10" />
|
||||
<div>
|
||||
<Card className="border-border/50 backdrop-blur">
|
||||
<CardHeader>
|
||||
<CardTitle className="text-2xl">Get Started</CardTitle>
|
||||
<CardDescription>Enter your access token to begin creating karaoke videos</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<form onSubmit={handleLogin} className="space-y-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="token">Access Token</Label>
|
||||
<Input
|
||||
id="token"
|
||||
type="text"
|
||||
placeholder="Enter your token"
|
||||
value={token}
|
||||
onChange={(e) => setToken(e.target.value)}
|
||||
className="bg-background"
|
||||
/>
|
||||
{error && <p className="text-sm text-destructive">{error}</p>}
|
||||
</div>
|
||||
|
||||
<Button type="submit" className="w-full bg-primary hover:bg-primary/90" disabled={isLoading}>
|
||||
{isLoading ? "Logging in..." : "Login"}
|
||||
</Button>
|
||||
|
||||
<div className="text-center text-sm text-muted-foreground">
|
||||
Don't have access?{" "}
|
||||
<a href="#" className="text-primary hover:underline">
|
||||
Request a token
|
||||
</a>{" "}
|
||||
or{" "}
|
||||
<a href="#" className="text-primary hover:underline">
|
||||
purchase access
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<div className="pt-4 border-t border-border">
|
||||
<p className="text-xs text-muted-foreground mb-2">Demo tokens:</p>
|
||||
<div className="flex gap-2">
|
||||
<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
size="sm"
|
||||
className="flex-1 text-xs bg-transparent"
|
||||
onClick={() => setToken("demo-user-token")}
|
||||
>
|
||||
User Demo
|
||||
</Button>
|
||||
<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
size="sm"
|
||||
className="flex-1 text-xs bg-transparent"
|
||||
onClick={() => setToken("demo-admin-token")}
|
||||
>
|
||||
Admin Demo
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</form>
|
||||
</CardContent>
|
||||
</Card>
|
||||
<h1 className="text-xl font-bold" style={{ color: 'var(--text)' }}>Karaoke Generator</h1>
|
||||
</div>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<TooltipProvider>
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={toggleAutoMode}
|
||||
className={autoModeEnabled ? "text-amber-400 hover:text-amber-300 bg-amber-500/10" : ""}
|
||||
style={!autoModeEnabled ? { color: 'var(--text-muted)' } : undefined}
|
||||
>
|
||||
{autoModeEnabled ? (
|
||||
<Zap className="w-4 h-4 mr-2" />
|
||||
) : (
|
||||
<ZapOff className="w-4 h-4 mr-2" />
|
||||
)}
|
||||
Auto
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent side="bottom" className="max-w-xs">
|
||||
<p className="font-medium">
|
||||
{autoModeEnabled ? "Auto Mode Enabled" : "Auto Mode Disabled"}
|
||||
</p>
|
||||
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
|
||||
{autoModeEnabled
|
||||
? "Jobs will auto-complete review and select clean instrumental (like -y flag)"
|
||||
: "Click to enable non-interactive mode"}
|
||||
</p>
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
</TooltipProvider>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={loadJobs}
|
||||
disabled={isLoadingJobs || !isAuthenticated}
|
||||
style={{ color: 'var(--text-muted)' }}
|
||||
>
|
||||
<RefreshCw className={`w-4 h-4 mr-2 ${isLoadingJobs ? 'animate-spin' : ''}`} />
|
||||
Refresh
|
||||
</Button>
|
||||
<AuthStatus />
|
||||
{mounted && (
|
||||
<TooltipProvider>
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={toggleTheme}
|
||||
style={{ color: 'var(--text-muted)' }}
|
||||
>
|
||||
{isDarkMode ? (
|
||||
<Sun className="w-4 h-4" />
|
||||
) : (
|
||||
<Moon className="w-4 h-4" />
|
||||
)}
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent side="bottom">
|
||||
<p>{isDarkMode ? "Switch to light mode" : "Switch to dark mode"}</p>
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
</TooltipProvider>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main className="px-4 py-8 space-y-6">
|
||||
{/* Auto Mode Banner */}
|
||||
{autoModeEnabled && (
|
||||
<div className="bg-amber-500/10 border border-amber-500/30 rounded-lg p-4 flex items-center gap-3">
|
||||
<Zap className="w-5 h-5 text-amber-400 flex-shrink-0" />
|
||||
<div className="flex-1">
|
||||
<p className="text-sm text-amber-200 font-medium">Non-Interactive Mode Active</p>
|
||||
<p className="text-xs text-amber-200/70 mt-0.5">
|
||||
Jobs will automatically accept lyrics and select clean instrumental (equivalent to -y flag)
|
||||
</p>
|
||||
</div>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={toggleAutoMode}
|
||||
className="text-amber-200 hover:text-amber-100 hover:bg-amber-500/20"
|
||||
>
|
||||
Disable
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Unauthenticated state */}
|
||||
{!isAuthenticated && (
|
||||
<div className="bg-amber-500/10 border border-amber-500/30 rounded-lg p-6 text-center">
|
||||
<KeyRound className="w-8 h-8 text-amber-400 mx-auto mb-3" />
|
||||
<h3 className="text-lg font-medium mb-2" style={{ color: 'var(--text)' }}>Authentication Required</h3>
|
||||
<p className="text-sm mb-4" style={{ color: 'var(--text-muted)' }}>
|
||||
Click the Login button in the header to enter your access token
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="grid lg:grid-cols-2 gap-8">
|
||||
{/* Submit Job Card */}
|
||||
<Card className="backdrop-blur" style={{ borderColor: 'var(--card-border)', backgroundColor: 'var(--card)' }}>
|
||||
<CardHeader>
|
||||
<CardTitle style={{ color: 'var(--text)' }}>Create Karaoke Video</CardTitle>
|
||||
<CardDescription style={{ color: 'var(--text-muted)' }}>
|
||||
Upload an audio file, provide a YouTube URL, or search for audio by artist & title
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<JobSubmission onJobCreated={loadJobs} />
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Jobs List Card */}
|
||||
<Card className="backdrop-blur" style={{ borderColor: 'var(--card-border)', backgroundColor: 'var(--card)' }}>
|
||||
<CardHeader>
|
||||
<CardTitle className="flex items-center justify-between" style={{ color: 'var(--text)' }}>
|
||||
Recent Jobs
|
||||
{isLoadingJobs && <Loader2 className="w-4 h-4 animate-spin" style={{ color: 'var(--text-muted)' }} />}
|
||||
</CardTitle>
|
||||
<CardDescription style={{ color: 'var(--text-muted)' }}>
|
||||
{jobs.length} job{jobs.length !== 1 ? 's' : ''} found
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
{jobs.length === 0 ? (
|
||||
<div className="text-center py-8" style={{ color: 'var(--text-muted)' }}>
|
||||
<Music2 className="w-12 h-12 mx-auto mb-3 opacity-50" />
|
||||
<p>No jobs yet. Create one to get started!</p>
|
||||
</div>
|
||||
) : (
|
||||
<div className="space-y-3 max-h-[500px] overflow-y-auto pr-2">
|
||||
{jobs.map((job) => (
|
||||
<JobCard key={job.job_id} job={job} onRefresh={loadJobs} />
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
{/* Footer */}
|
||||
<footer className="border-t mt-12 py-6" style={{ borderColor: 'var(--card-border)' }}>
|
||||
<div className="px-4 text-center text-sm" style={{ color: 'var(--text-muted)' }}>
|
||||
<p>
|
||||
Powered by{" "}
|
||||
<a href="https://github.com/nomadkaraoke/karaoke-gen" className="text-amber-500 hover:underline">
|
||||
karaoke-gen
|
||||
</a>
|
||||
</p>
|
||||
</div>
|
||||
</footer>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useCallback, useRef } from "react"
|
||||
import { api, Job } from "@/lib/api"
|
||||
import { useAutoMode } from "@/lib/auto-mode"
|
||||
|
||||
interface AutoProcessorProps {
|
||||
/** List of jobs to monitor */
|
||||
jobs: Job[]
|
||||
/** Callback to refresh jobs list after auto-processing */
|
||||
onJobsChanged: () => void
|
||||
}
|
||||
|
||||
/**
|
||||
* AutoProcessor component that automatically handles interactive states
|
||||
* when auto-mode is enabled.
|
||||
*
|
||||
* This is equivalent to the -y flag in karaoke-gen-remote CLI:
|
||||
* - Auto-completes lyrics review (accepts auto-corrected lyrics)
|
||||
* - Auto-selects "clean" instrumental
|
||||
* - Auto-selects first/best audio search result
|
||||
*/
|
||||
export function AutoProcessor({ jobs, onJobsChanged }: AutoProcessorProps) {
|
||||
const { enabled, markProcessing, unmarkProcessing, isProcessing } = useAutoMode()
|
||||
const processedRef = useRef<Set<string>>(new Set())
|
||||
|
||||
const processJob = useCallback(async (job: Job, action: string) => {
|
||||
const processingKey = `${job.job_id}-${action}`
|
||||
|
||||
// Skip if already being processed
|
||||
if (isProcessing(processingKey) || processedRef.current.has(processingKey)) {
|
||||
return
|
||||
}
|
||||
|
||||
// Mark as processing
|
||||
markProcessing(processingKey)
|
||||
processedRef.current.add(processingKey)
|
||||
|
||||
console.log(`[AutoProcessor] Auto-processing ${action} for job ${job.job_id}`)
|
||||
|
||||
try {
|
||||
switch (action) {
|
||||
case 'review':
|
||||
// Auto-complete lyrics review
|
||||
await api.completeReview(job.job_id)
|
||||
console.log(`[AutoProcessor] Auto-completed review for job ${job.job_id}`)
|
||||
break
|
||||
|
||||
case 'instrumental':
|
||||
// Auto-select "clean" instrumental
|
||||
await api.selectInstrumental(job.job_id, 'clean')
|
||||
console.log(`[AutoProcessor] Auto-selected clean instrumental for job ${job.job_id}`)
|
||||
break
|
||||
|
||||
case 'audio':
|
||||
// Auto-select first audio search result (index 0)
|
||||
await api.selectAudioResult(job.job_id, 0)
|
||||
console.log(`[AutoProcessor] Auto-selected first audio result for job ${job.job_id}`)
|
||||
break
|
||||
}
|
||||
|
||||
// Refresh jobs to reflect the change
|
||||
onJobsChanged()
|
||||
} catch (err) {
|
||||
console.error(`[AutoProcessor] Failed to auto-process ${action} for job ${job.job_id}:`, err)
|
||||
// Remove from processed so it can be retried
|
||||
processedRef.current.delete(processingKey)
|
||||
} finally {
|
||||
unmarkProcessing(processingKey)
|
||||
}
|
||||
}, [isProcessing, markProcessing, unmarkProcessing, onJobsChanged])
|
||||
|
||||
useEffect(() => {
|
||||
if (!enabled) return
|
||||
|
||||
// Find jobs that need auto-processing
|
||||
for (const job of jobs) {
|
||||
switch (job.status) {
|
||||
case 'awaiting_review':
|
||||
processJob(job, 'review')
|
||||
break
|
||||
case 'awaiting_instrumental_selection':
|
||||
processJob(job, 'instrumental')
|
||||
break
|
||||
case 'awaiting_audio_selection':
|
||||
processJob(job, 'audio')
|
||||
break
|
||||
}
|
||||
}
|
||||
}, [enabled, jobs, processJob])
|
||||
|
||||
// This component doesn't render anything - it's purely for side effects
|
||||
return null
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
/**
|
||||
* Tests for AuthStatus and AuthDialog components
|
||||
*/
|
||||
|
||||
import { render, screen, fireEvent, waitFor } from '@testing-library/react'
|
||||
import { AuthStatus } from '../auth/AuthStatus'
|
||||
import { AuthDialog } from '../auth/AuthDialog'
|
||||
import { getAccessToken, setAccessToken, clearAccessToken } from '@/lib/api'
|
||||
|
||||
// Mock the api module
|
||||
jest.mock('@/lib/api', () => ({
|
||||
getAccessToken: jest.fn(),
|
||||
setAccessToken: jest.fn(),
|
||||
clearAccessToken: jest.fn(),
|
||||
}))
|
||||
|
||||
const mockGetAccessToken = getAccessToken as jest.Mock
|
||||
const mockSetAccessToken = setAccessToken as jest.Mock
|
||||
const mockClearAccessToken = clearAccessToken as jest.Mock
|
||||
|
||||
// Mock window.location.reload
|
||||
const mockReload = jest.fn()
|
||||
Object.defineProperty(window, 'location', {
|
||||
value: { reload: mockReload },
|
||||
writable: true,
|
||||
})
|
||||
|
||||
describe('AuthStatus', () => {
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks()
|
||||
})
|
||||
|
||||
it('shows Login button when not authenticated', () => {
|
||||
mockGetAccessToken.mockReturnValue(null)
|
||||
render(<AuthStatus />)
|
||||
|
||||
expect(screen.getByRole('button', { name: /login/i })).toBeInTheDocument()
|
||||
expect(screen.queryByRole('button', { name: /logout/i })).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('shows Logout button when authenticated', () => {
|
||||
mockGetAccessToken.mockReturnValue('test-token')
|
||||
render(<AuthStatus />)
|
||||
|
||||
expect(screen.getByRole('button', { name: /logout/i })).toBeInTheDocument()
|
||||
expect(screen.queryByRole('button', { name: /login/i })).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('opens auth dialog when Login is clicked', () => {
|
||||
mockGetAccessToken.mockReturnValue(null)
|
||||
render(<AuthStatus />)
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: /login/i }))
|
||||
|
||||
// Dialog should appear with "Authentication Required" title
|
||||
expect(screen.getByText('Authentication Required')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('clears token and reloads when Logout is clicked', () => {
|
||||
mockGetAccessToken.mockReturnValue('test-token')
|
||||
render(<AuthStatus />)
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: /logout/i }))
|
||||
|
||||
expect(mockClearAccessToken).toHaveBeenCalled()
|
||||
expect(mockReload).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('calls onAuthChange callback when logging out', () => {
|
||||
mockGetAccessToken.mockReturnValue('test-token')
|
||||
const mockOnAuthChange = jest.fn()
|
||||
render(<AuthStatus onAuthChange={mockOnAuthChange} />)
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: /logout/i }))
|
||||
|
||||
expect(mockOnAuthChange).toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
describe('AuthDialog', () => {
|
||||
const mockOnClose = jest.fn()
|
||||
const mockOnSuccess = jest.fn()
|
||||
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks()
|
||||
})
|
||||
|
||||
it('renders when open is true', () => {
|
||||
render(<AuthDialog open={true} onClose={mockOnClose} onSuccess={mockOnSuccess} />)
|
||||
|
||||
expect(screen.getByText('Authentication Required')).toBeInTheDocument()
|
||||
expect(screen.getByPlaceholderText('Enter access token')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('does not render when open is false', () => {
|
||||
render(<AuthDialog open={false} onClose={mockOnClose} onSuccess={mockOnSuccess} />)
|
||||
|
||||
expect(screen.queryByText('Authentication Required')).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('calls onClose when Cancel is clicked', () => {
|
||||
render(<AuthDialog open={true} onClose={mockOnClose} onSuccess={mockOnSuccess} />)
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: /cancel/i }))
|
||||
|
||||
expect(mockOnClose).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('disables Authenticate button when token is empty', () => {
|
||||
render(<AuthDialog open={true} onClose={mockOnClose} onSuccess={mockOnSuccess} />)
|
||||
|
||||
const authenticateBtn = screen.getByRole('button', { name: /authenticate/i })
|
||||
expect(authenticateBtn).toBeDisabled()
|
||||
})
|
||||
|
||||
it('enables Authenticate button when token is entered', () => {
|
||||
render(<AuthDialog open={true} onClose={mockOnClose} onSuccess={mockOnSuccess} />)
|
||||
|
||||
const input = screen.getByPlaceholderText('Enter access token')
|
||||
fireEvent.change(input, { target: { value: 'my-token' } })
|
||||
|
||||
const authenticateBtn = screen.getByRole('button', { name: /authenticate/i })
|
||||
expect(authenticateBtn).not.toBeDisabled()
|
||||
})
|
||||
|
||||
it('sets token and calls onSuccess when form is submitted', async () => {
|
||||
render(<AuthDialog open={true} onClose={mockOnClose} onSuccess={mockOnSuccess} />)
|
||||
|
||||
const input = screen.getByPlaceholderText('Enter access token')
|
||||
fireEvent.change(input, { target: { value: 'my-token' } })
|
||||
|
||||
const authenticateBtn = screen.getByRole('button', { name: /authenticate/i })
|
||||
fireEvent.click(authenticateBtn)
|
||||
|
||||
expect(mockSetAccessToken).toHaveBeenCalledWith('my-token')
|
||||
expect(mockOnSuccess).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('trims whitespace from token', () => {
|
||||
render(<AuthDialog open={true} onClose={mockOnClose} onSuccess={mockOnSuccess} />)
|
||||
|
||||
const input = screen.getByPlaceholderText('Enter access token')
|
||||
fireEvent.change(input, { target: { value: ' my-token ' } })
|
||||
|
||||
const authenticateBtn = screen.getByRole('button', { name: /authenticate/i })
|
||||
fireEvent.click(authenticateBtn)
|
||||
|
||||
expect(mockSetAccessToken).toHaveBeenCalledWith('my-token')
|
||||
})
|
||||
|
||||
it('shows error for empty token submission', () => {
|
||||
render(<AuthDialog open={true} onClose={mockOnClose} onSuccess={mockOnSuccess} />)
|
||||
|
||||
// Token with only whitespace
|
||||
const input = screen.getByPlaceholderText('Enter access token')
|
||||
fireEvent.change(input, { target: { value: ' ' } })
|
||||
|
||||
// Button should still be disabled for whitespace-only input
|
||||
const authenticateBtn = screen.getByRole('button', { name: /authenticate/i })
|
||||
expect(authenticateBtn).toBeDisabled()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,211 @@
|
||||
/**
|
||||
* Tests for JobCard component
|
||||
*
|
||||
* Note: Job cards are no longer collapsible - details are always visible
|
||||
*/
|
||||
|
||||
import { render, screen } from '@testing-library/react'
|
||||
import { JobCard } from '../job/JobCard'
|
||||
import { Job } from '@/lib/api'
|
||||
import { useAutoMode } from '@/lib/auto-mode'
|
||||
|
||||
// Mock the auto-mode hook
|
||||
jest.mock('@/lib/auto-mode', () => ({
|
||||
useAutoMode: jest.fn()
|
||||
}))
|
||||
|
||||
const mockUseAutoMode = useAutoMode as jest.Mock
|
||||
|
||||
// Mock the child components
|
||||
jest.mock('../job/JobActions', () => ({
|
||||
JobActions: () => <div data-testid="job-actions">Job Actions</div>
|
||||
}))
|
||||
|
||||
jest.mock('../job/JobLogs', () => ({
|
||||
JobLogs: () => <div data-testid="job-logs">Job Logs</div>
|
||||
}))
|
||||
|
||||
jest.mock('../job/OutputLinks', () => ({
|
||||
OutputLinks: () => <div data-testid="output-links">Output Links</div>
|
||||
}))
|
||||
|
||||
jest.mock('../job/InstrumentalSelector', () => ({
|
||||
InstrumentalSelector: () => <div data-testid="instrumental-selector">Instrumental Selector</div>
|
||||
}))
|
||||
|
||||
jest.mock('../audio-search/AudioSearchDialog', () => ({
|
||||
AudioSearchDialog: () => <div data-testid="audio-search-dialog">Audio Search Dialog</div>
|
||||
}))
|
||||
|
||||
describe('JobCard', () => {
|
||||
const mockOnRefresh = jest.fn()
|
||||
|
||||
const mockJob: Job = {
|
||||
job_id: '123',
|
||||
status: 'processing',
|
||||
progress: 50,
|
||||
created_at: '2025-01-01T00:00:00Z',
|
||||
updated_at: '2025-01-01T00:00:00Z',
|
||||
artist: 'Test Artist',
|
||||
title: 'Test Song',
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks()
|
||||
// Default mock: auto-mode disabled
|
||||
mockUseAutoMode.mockReturnValue({
|
||||
enabled: false,
|
||||
isProcessing: jest.fn().mockReturnValue(false)
|
||||
})
|
||||
})
|
||||
|
||||
it('renders job information', () => {
|
||||
render(<JobCard job={mockJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText('Test Artist - Test Song')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('shows job ID (details always visible)', () => {
|
||||
render(<JobCard job={mockJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
// Details are always visible now (no need to click to expand)
|
||||
// ID is shown in format: "ID: 123 • date • status"
|
||||
expect(screen.getByText(/ID:/)).toBeInTheDocument()
|
||||
expect(screen.getByText(/123/)).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('shows job actions (details always visible)', () => {
|
||||
render(<JobCard job={mockJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
// Actions should be immediately visible
|
||||
expect(screen.getByTestId('job-actions')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
// Note: Progress bars have been removed from JobCard in favor of a simpler design
|
||||
|
||||
it('shows error message for failed jobs', () => {
|
||||
const failedJob = { ...mockJob, status: 'failed', error_message: 'Something went wrong' }
|
||||
render(<JobCard job={failedJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText('Something went wrong')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
// Note: Orange border for interactive jobs removed - uses default slate border now
|
||||
|
||||
it('applies green border for complete jobs', () => {
|
||||
const completeJob = { ...mockJob, status: 'complete' }
|
||||
const { container } = render(<JobCard job={completeJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
const card = container.firstChild
|
||||
expect(card).toHaveClass('border-green-500/30')
|
||||
})
|
||||
|
||||
it('applies red border for failed jobs', () => {
|
||||
const failedJob = { ...mockJob, status: 'failed' }
|
||||
const { container } = render(<JobCard job={failedJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
const card = container.firstChild
|
||||
expect(card).toHaveClass('border-red-500/30')
|
||||
})
|
||||
|
||||
describe('when auto-mode is disabled', () => {
|
||||
beforeEach(() => {
|
||||
mockUseAutoMode.mockReturnValue({
|
||||
enabled: false,
|
||||
isProcessing: jest.fn().mockReturnValue(false)
|
||||
})
|
||||
})
|
||||
|
||||
it('shows lyrics review link for awaiting_review status', () => {
|
||||
const reviewJob = { ...mockJob, status: 'awaiting_review', review_token: 'test-token-123' }
|
||||
render(<JobCard job={reviewJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
const reviewLink = screen.getByText('Review Lyrics')
|
||||
expect(reviewLink).toBeInTheDocument()
|
||||
const href = reviewLink.closest('a')?.getAttribute('href') || ''
|
||||
expect(href).toContain('gen.nomadkaraoke.com/lyrics')
|
||||
expect(href).toContain('baseApiUrl=')
|
||||
expect(href).toContain('reviewToken=test-token-123')
|
||||
})
|
||||
|
||||
it('shows audio search button for awaiting_audio_selection status', () => {
|
||||
const audioSelectionJob = { ...mockJob, status: 'awaiting_audio_selection' }
|
||||
render(<JobCard job={audioSelectionJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByRole('button', { name: 'Select Audio' })).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('shows instrumental selection button for awaiting_instrumental_selection status', () => {
|
||||
const instrumentalJob = { ...mockJob, status: 'awaiting_instrumental_selection' }
|
||||
render(<JobCard job={instrumentalJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByRole('button', { name: 'Select Instrumental' })).toBeInTheDocument()
|
||||
})
|
||||
})
|
||||
|
||||
describe('when auto-mode is enabled', () => {
|
||||
beforeEach(() => {
|
||||
mockUseAutoMode.mockReturnValue({
|
||||
enabled: true,
|
||||
isProcessing: jest.fn().mockReturnValue(false)
|
||||
})
|
||||
})
|
||||
|
||||
it('shows auto-accept message for awaiting_review status', () => {
|
||||
const reviewJob = { ...mockJob, status: 'awaiting_review' }
|
||||
render(<JobCard job={reviewJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText(/Will auto-accept/)).toBeInTheDocument()
|
||||
expect(screen.queryByText('Review Lyrics')).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('shows auto-select message for awaiting_instrumental_selection status', () => {
|
||||
const instrumentalJob = { ...mockJob, status: 'awaiting_instrumental_selection' }
|
||||
render(<JobCard job={instrumentalJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText(/Will auto-select clean/)).toBeInTheDocument()
|
||||
expect(screen.queryByRole('button', { name: 'Select Instrumental' })).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('shows auto-select message for awaiting_audio_selection status', () => {
|
||||
const audioJob = { ...mockJob, status: 'awaiting_audio_selection' }
|
||||
render(<JobCard job={audioJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText(/Will auto-select first/)).toBeInTheDocument()
|
||||
expect(screen.queryByRole('button', { name: 'Select Audio' })).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('shows auto-processing indicator when processing', () => {
|
||||
mockUseAutoMode.mockReturnValue({
|
||||
enabled: true,
|
||||
isProcessing: jest.fn().mockReturnValue(true)
|
||||
})
|
||||
|
||||
const reviewJob = { ...mockJob, status: 'awaiting_review' }
|
||||
render(<JobCard job={reviewJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText(/Auto-processing.../)).toBeInTheDocument()
|
||||
})
|
||||
})
|
||||
|
||||
it('shows output links for complete jobs', () => {
|
||||
const completeJob = { ...mockJob, status: 'complete' }
|
||||
render(<JobCard job={completeJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByTestId('output-links')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('handles unknown artist gracefully', () => {
|
||||
const noArtistJob = { ...mockJob, artist: undefined }
|
||||
render(<JobCard job={noArtistJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText('Unknown - Test Song')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('handles unknown title gracefully', () => {
|
||||
const noTitleJob = { ...mockJob, title: undefined }
|
||||
render(<JobCard job={noTitleJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText('Test Artist - Unknown')).toBeInTheDocument()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,416 @@
|
||||
"use client"
|
||||
|
||||
import { useState, useEffect, useMemo } from "react"
|
||||
import { api, AudioSearchResult } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Dialog, DialogContent, DialogTitle } from "@/components/ui/dialog"
|
||||
import { Loader2, Music2, ChevronDown, ChevronUp } from "lucide-react"
|
||||
|
||||
// Version from pyproject.toml (single source of truth)
|
||||
const APP_VERSION = process.env.NEXT_PUBLIC_APP_VERSION || "0.0.0"
|
||||
|
||||
interface AudioSearchDialogProps {
|
||||
jobId: string
|
||||
open: boolean
|
||||
onClose: () => void
|
||||
onSelect: () => void
|
||||
}
|
||||
|
||||
// Extended result type to include all backend fields
|
||||
interface ExtendedAudioSearchResult extends AudioSearchResult {
|
||||
channel?: string
|
||||
view_count?: number
|
||||
formatted_duration?: string
|
||||
formatted_size?: string
|
||||
release_type?: string
|
||||
label?: string
|
||||
edition_info?: string
|
||||
target_file?: string
|
||||
quality_data?: {
|
||||
format?: string
|
||||
bit_depth?: number
|
||||
sample_rate?: number
|
||||
bitrate?: number
|
||||
media?: string
|
||||
}
|
||||
}
|
||||
|
||||
// Category configuration matching flacfetch
|
||||
interface CategoryConfig {
|
||||
name: string
|
||||
maxDisplay: number
|
||||
color: string
|
||||
borderColor: string
|
||||
}
|
||||
|
||||
const CATEGORY_CONFIG: Record<string, CategoryConfig> = {
|
||||
'TOP SEEDED': { name: 'TOP SEEDED', maxDisplay: 3, color: 'text-amber-400', borderColor: 'border-amber-500/30' },
|
||||
'HI-RES 24-BIT': { name: 'HI-RES 24-BIT', maxDisplay: 3, color: 'text-purple-400', borderColor: 'border-purple-500/30' },
|
||||
'STUDIO ALBUMS': { name: 'STUDIO ALBUMS', maxDisplay: 3, color: 'text-blue-400', borderColor: 'border-blue-500/30' },
|
||||
'SINGLES': { name: 'SINGLES', maxDisplay: 2, color: 'text-cyan-400', borderColor: 'border-cyan-500/30' },
|
||||
'LIVE VERSIONS': { name: 'LIVE VERSIONS', maxDisplay: 2, color: 'text-pink-400', borderColor: 'border-pink-500/30' },
|
||||
'COMPILATIONS': { name: 'COMPILATIONS', maxDisplay: 2, color: 'text-teal-400', borderColor: 'border-teal-500/30' },
|
||||
'VINYL RIPS': { name: 'VINYL RIPS', maxDisplay: 2, color: 'text-orange-400', borderColor: 'border-orange-500/30' },
|
||||
'YOUTUBE/LOSSY': { name: 'YOUTUBE/LOSSY', maxDisplay: 3, color: 'text-red-400', borderColor: 'border-red-500/30' },
|
||||
'OTHER': { name: 'OTHER', maxDisplay: 3, color: 'text-slate-400', borderColor: 'border-slate-500/30' },
|
||||
}
|
||||
|
||||
const CATEGORY_ORDER = [
|
||||
'TOP SEEDED',
|
||||
'HI-RES 24-BIT',
|
||||
'STUDIO ALBUMS',
|
||||
'SINGLES',
|
||||
'LIVE VERSIONS',
|
||||
'COMPILATIONS',
|
||||
'VINYL RIPS',
|
||||
'YOUTUBE/LOSSY',
|
||||
'OTHER'
|
||||
]
|
||||
|
||||
// Categorize a single result based on flacfetch logic
|
||||
function categorizeResult(result: ExtendedAudioSearchResult): string {
|
||||
const isLossless = result.is_lossless === true
|
||||
const is24Bit = result.quality_data?.bit_depth === 24
|
||||
const seeders = result.seeders ?? 0
|
||||
const provider = result.provider?.toLowerCase() ?? ''
|
||||
const releaseType = result.release_type?.toLowerCase() ?? ''
|
||||
const media = result.quality_data?.media?.toLowerCase() ?? ''
|
||||
|
||||
// YouTube and lossy sources
|
||||
if (provider === 'youtube' || !isLossless) {
|
||||
return 'YOUTUBE/LOSSY'
|
||||
}
|
||||
|
||||
// Top seeded (50+ seeders, lossless)
|
||||
if (isLossless && seeders >= 50) {
|
||||
return 'TOP SEEDED'
|
||||
}
|
||||
|
||||
// Hi-res 24-bit
|
||||
if (isLossless && is24Bit) {
|
||||
return 'HI-RES 24-BIT'
|
||||
}
|
||||
|
||||
// Vinyl rips
|
||||
if (isLossless && media === 'vinyl') {
|
||||
return 'VINYL RIPS'
|
||||
}
|
||||
|
||||
// Live versions
|
||||
if (isLossless && (releaseType === 'live album' || releaseType === 'bootleg' || releaseType.includes('live'))) {
|
||||
return 'LIVE VERSIONS'
|
||||
}
|
||||
|
||||
// Compilations
|
||||
if (isLossless && (releaseType === 'compilation' || releaseType === 'soundtrack' || releaseType === 'anthology')) {
|
||||
return 'COMPILATIONS'
|
||||
}
|
||||
|
||||
// Singles/EPs
|
||||
if (isLossless && (releaseType === 'single' || releaseType === 'ep')) {
|
||||
return 'SINGLES'
|
||||
}
|
||||
|
||||
// Studio albums
|
||||
if (isLossless && (releaseType === 'album' || !releaseType)) {
|
||||
return 'STUDIO ALBUMS'
|
||||
}
|
||||
|
||||
return 'OTHER'
|
||||
}
|
||||
|
||||
export function AudioSearchDialog({ jobId, open, onClose, onSelect }: AudioSearchDialogProps) {
|
||||
const [results, setResults] = useState<ExtendedAudioSearchResult[]>([])
|
||||
const [isLoading, setIsLoading] = useState(true)
|
||||
const [isSelecting, setIsSelecting] = useState<number | null>(null)
|
||||
const [error, setError] = useState("")
|
||||
const [expandedCategories, setExpandedCategories] = useState<Set<string>>(new Set())
|
||||
|
||||
// Debug: Log component version on mount
|
||||
useEffect(() => {
|
||||
console.log(`[AudioSearchDialog] App version: ${APP_VERSION}`)
|
||||
}, [])
|
||||
|
||||
// Load results when dialog opens
|
||||
useEffect(() => {
|
||||
if (open) {
|
||||
console.log(`[AudioSearchDialog v${APP_VERSION}] Dialog opened for job: ${jobId}`)
|
||||
loadResults()
|
||||
setExpandedCategories(new Set()) // Reset expanded state
|
||||
}
|
||||
}, [open, jobId])
|
||||
|
||||
async function loadResults() {
|
||||
setIsLoading(true)
|
||||
setError("")
|
||||
try {
|
||||
console.log(`[AudioSearchDialog v${APP_VERSION}] Loading results...`)
|
||||
const data = await api.getAudioSearchResults(jobId)
|
||||
console.log(`[AudioSearchDialog v${APP_VERSION}] Raw API response:`, data)
|
||||
console.log(`[AudioSearchDialog v${APP_VERSION}] Results count: ${data.results?.length || 0}`)
|
||||
setResults((data.results || []) as ExtendedAudioSearchResult[])
|
||||
} catch (err: any) {
|
||||
console.error(`[AudioSearchDialog v${APP_VERSION}] Failed to load search results:`, err)
|
||||
setError(err.message || "Failed to load search results")
|
||||
} finally {
|
||||
setIsLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
// Group results by category
|
||||
const groupedResults = useMemo(() => {
|
||||
const groups: Record<string, ExtendedAudioSearchResult[]> = {}
|
||||
|
||||
for (const result of results) {
|
||||
const category = categorizeResult(result)
|
||||
if (!groups[category]) {
|
||||
groups[category] = []
|
||||
}
|
||||
groups[category].push(result)
|
||||
}
|
||||
|
||||
// Sort by category order
|
||||
const sortedGroups: Array<{ category: string; results: ExtendedAudioSearchResult[]; config: CategoryConfig }> = []
|
||||
for (const cat of CATEGORY_ORDER) {
|
||||
if (groups[cat] && groups[cat].length > 0) {
|
||||
sortedGroups.push({
|
||||
category: cat,
|
||||
results: groups[cat],
|
||||
config: CATEGORY_CONFIG[cat]
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return sortedGroups
|
||||
}, [results])
|
||||
|
||||
// Count total categories
|
||||
const totalCategories = groupedResults.length
|
||||
|
||||
async function handleSelect(index: number) {
|
||||
setIsSelecting(index)
|
||||
setError("")
|
||||
try {
|
||||
await api.selectAudioResult(jobId, index)
|
||||
onSelect()
|
||||
onClose()
|
||||
} catch (err: any) {
|
||||
console.error("Failed to select audio:", err)
|
||||
setError(err.message || "Failed to select audio")
|
||||
} finally {
|
||||
setIsSelecting(null)
|
||||
}
|
||||
}
|
||||
|
||||
function toggleCategory(category: string) {
|
||||
setExpandedCategories(prev => {
|
||||
const next = new Set(prev)
|
||||
if (next.has(category)) {
|
||||
next.delete(category)
|
||||
} else {
|
||||
next.add(category)
|
||||
}
|
||||
return next
|
||||
})
|
||||
}
|
||||
|
||||
// Get display name - use channel for YouTube, artist for others
|
||||
function getDisplayName(result: ExtendedAudioSearchResult): string {
|
||||
if (result.provider === "YouTube" && result.channel) {
|
||||
return result.channel
|
||||
}
|
||||
return result.artist || ""
|
||||
}
|
||||
|
||||
// Format views/seeders compactly
|
||||
function formatCount(count?: number): string {
|
||||
if (!count && count !== 0) return "-"
|
||||
if (count >= 1000000) return `${(count / 1000000).toFixed(1)}M`
|
||||
if (count >= 1000) return `${(count / 1000).toFixed(1)}K`
|
||||
return count.toString()
|
||||
}
|
||||
|
||||
// Format release metadata line
|
||||
function formatMetadata(result: ExtendedAudioSearchResult): string {
|
||||
const parts: string[] = []
|
||||
|
||||
if (result.release_type) parts.push(result.release_type)
|
||||
if (result.year) parts.push(result.year.toString())
|
||||
if (result.label) parts.push(result.label)
|
||||
if (result.edition_info) parts.push(result.edition_info)
|
||||
if (result.quality_data?.media) parts.push(result.quality_data.media)
|
||||
|
||||
return parts.length > 0 ? `[${parts.join(' / ')}]` : ''
|
||||
}
|
||||
|
||||
// Get quality display string
|
||||
function formatQuality(result: ExtendedAudioSearchResult): string {
|
||||
if (result.quality) return result.quality
|
||||
|
||||
const qd = result.quality_data
|
||||
if (!qd) return '-'
|
||||
|
||||
const parts: string[] = []
|
||||
if (qd.format) parts.push(qd.format)
|
||||
if (qd.bit_depth) parts.push(`${qd.bit_depth}bit`)
|
||||
if (qd.bitrate) parts.push(`${qd.bitrate}kbps`)
|
||||
if (qd.media) parts.push(qd.media)
|
||||
|
||||
return parts.join(' ') || '-'
|
||||
}
|
||||
|
||||
return (
|
||||
<Dialog open={open} onOpenChange={(open) => !open && onClose()}>
|
||||
<DialogContent className="max-w-7xl w-[95vw] max-h-[90vh] flex flex-col bg-slate-900 border-slate-700 !p-0 gap-0 overflow-hidden">
|
||||
<div className="px-4 py-2.5 border-b border-slate-700 shrink-0 bg-slate-900 flex items-center justify-between">
|
||||
<DialogTitle className="text-white text-sm font-semibold flex items-center gap-2">
|
||||
Select Audio Source
|
||||
<span className="text-[10px] px-1.5 py-0.5 rounded bg-slate-700 text-slate-400 font-mono">v{APP_VERSION}</span>
|
||||
<span className="text-slate-400 font-normal text-xs">
|
||||
({results.length} results across {totalCategories} categories)
|
||||
</span>
|
||||
</DialogTitle>
|
||||
</div>
|
||||
|
||||
{error && (
|
||||
<div className="px-4 py-2 text-xs text-red-400 bg-red-500/10 border-b border-slate-800">
|
||||
{error}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{isLoading ? (
|
||||
<div className="flex-1 flex items-center justify-center py-12">
|
||||
<Loader2 className="w-5 h-5 animate-spin text-slate-400" />
|
||||
<span className="ml-2 text-sm text-slate-400">Loading...</span>
|
||||
</div>
|
||||
) : results.length === 0 ? (
|
||||
<div className="flex-1 flex flex-col items-center justify-center py-12 text-slate-500">
|
||||
<Music2 className="w-8 h-8 mb-2 opacity-50" />
|
||||
<p className="text-sm">No audio sources found</p>
|
||||
</div>
|
||||
) : (
|
||||
<div className="flex-1 overflow-y-auto min-h-0 p-4">
|
||||
{/* Two-column grid on desktop */}
|
||||
<div className="grid grid-cols-1 xl:grid-cols-2 gap-4">
|
||||
{groupedResults.map(({ category, results: catResults, config }) => {
|
||||
const isExpanded = expandedCategories.has(category)
|
||||
const displayResults = isExpanded ? catResults : catResults.slice(0, config.maxDisplay)
|
||||
const hiddenCount = catResults.length - config.maxDisplay
|
||||
const hasMore = hiddenCount > 0
|
||||
|
||||
return (
|
||||
<div key={category} className="border border-slate-700 rounded-lg overflow-hidden bg-slate-800/20">
|
||||
{/* Category Header */}
|
||||
<div className={`px-3 py-2 bg-slate-800/70 flex items-center justify-between ${config.color}`}>
|
||||
<div className="flex items-center gap-2 text-xs font-semibold">
|
||||
<span>{category}</span>
|
||||
<span className="text-slate-500 font-normal">({catResults.length})</span>
|
||||
</div>
|
||||
{hasMore && (
|
||||
<button
|
||||
onClick={() => toggleCategory(category)}
|
||||
className="text-[10px] text-slate-400 hover:text-white flex items-center gap-1"
|
||||
>
|
||||
{isExpanded ? (
|
||||
<>Show less <ChevronUp className="w-3 h-3" /></>
|
||||
) : (
|
||||
<>+{hiddenCount} more <ChevronDown className="w-3 h-3" /></>
|
||||
)}
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Results in category */}
|
||||
<div className="divide-y divide-slate-700/50">
|
||||
{displayResults.map((result) => (
|
||||
<div
|
||||
key={result.index}
|
||||
className="px-3 py-1.5 hover:bg-slate-800/50 flex items-center gap-2 text-xs"
|
||||
>
|
||||
{/* Index */}
|
||||
<span className="w-5 text-slate-500 font-mono shrink-0 text-[10px]">
|
||||
{result.index + 1}.
|
||||
</span>
|
||||
|
||||
{/* Main content - 2 lines */}
|
||||
<div className="flex-1 min-w-0">
|
||||
{/* Line 1: badges + artist + title + quality + size + seeders */}
|
||||
<div className="flex items-center gap-1 flex-wrap">
|
||||
{result.is_lossless && (
|
||||
<span className="text-[8px] px-1 py-0.5 rounded bg-green-600/20 text-green-400 font-medium">
|
||||
LOSSLESS
|
||||
</span>
|
||||
)}
|
||||
<span className={`text-[8px] px-1 py-0.5 rounded font-medium ${
|
||||
result.provider === "YouTube" ? "bg-red-600/20 text-red-400" :
|
||||
result.provider === "RED" ? "bg-red-600/30 text-red-300" :
|
||||
result.provider === "OPS" ? "bg-blue-600/20 text-blue-400" :
|
||||
"bg-slate-600/20 text-slate-400"
|
||||
}`}>
|
||||
{result.provider}
|
||||
</span>
|
||||
<span className="text-green-400 font-medium">{getDisplayName(result)}</span>
|
||||
<span className="text-slate-500">-</span>
|
||||
<span className="text-white">{result.title}</span>
|
||||
<span className={`text-[10px] ${result.is_lossless ? "text-green-400" : "text-slate-400"}`}>
|
||||
({formatQuality(result)})
|
||||
</span>
|
||||
<span className="text-[10px] text-slate-400">{result.formatted_size || '-'}</span>
|
||||
{result.seeders !== undefined && result.seeders !== null ? (
|
||||
<span className={`text-[10px] ${
|
||||
result.seeders >= 50 ? 'text-green-400' :
|
||||
result.seeders >= 10 ? 'text-yellow-400' :
|
||||
'text-red-400'
|
||||
}`}>
|
||||
{result.seeders} seed
|
||||
</span>
|
||||
) : result.view_count !== undefined ? (
|
||||
<span className={`text-[10px] ${
|
||||
result.view_count >= 1000000 ? 'text-green-400' :
|
||||
result.view_count >= 10000 ? 'text-yellow-400' :
|
||||
'text-slate-400'
|
||||
}`}>
|
||||
{formatCount(result.view_count)} views
|
||||
</span>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
{/* Line 2: metadata + filename */}
|
||||
{(formatMetadata(result) || result.target_file) && (
|
||||
<div className="flex items-center gap-1.5 text-[10px] text-slate-500">
|
||||
{formatMetadata(result) && (
|
||||
<span className="truncate">{formatMetadata(result)}</span>
|
||||
)}
|
||||
{result.target_file && (
|
||||
<span className="font-mono truncate">"{result.target_file}"</span>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Select button */}
|
||||
<Button
|
||||
size="sm"
|
||||
onClick={() => handleSelect(result.index)}
|
||||
disabled={isSelecting !== null}
|
||||
className="w-14 h-6 text-[9px] bg-amber-600 hover:bg-amber-500 text-white px-2 shrink-0"
|
||||
>
|
||||
{isSelecting === result.index ? (
|
||||
<Loader2 className="w-3 h-3 animate-spin" />
|
||||
) : (
|
||||
"Select"
|
||||
)}
|
||||
</Button>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
export { AudioSearchDialog } from "./AudioSearchDialog"
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
'use client';
|
||||
|
||||
import { useState, useEffect } from 'react';
|
||||
import { X, KeyRound } from 'lucide-react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Input } from '@/components/ui/input';
|
||||
import { setAccessToken, getAccessToken, clearAccessToken } from '@/lib/api';
|
||||
|
||||
export function AuthBanner() {
|
||||
const [token, setToken] = useState('');
|
||||
const [isAuthenticated, setIsAuthenticated] = useState(false);
|
||||
const [showInput, setShowInput] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
const storedToken = getAccessToken();
|
||||
if (storedToken) {
|
||||
setIsAuthenticated(true);
|
||||
setToken(storedToken.substring(0, 8) + '...');
|
||||
} else {
|
||||
setShowInput(true);
|
||||
}
|
||||
}, []);
|
||||
|
||||
const handleSubmit = (e: React.FormEvent) => {
|
||||
e.preventDefault();
|
||||
if (token.trim()) {
|
||||
setAccessToken(token.trim());
|
||||
setIsAuthenticated(true);
|
||||
setShowInput(false);
|
||||
// Reload to fetch jobs with auth
|
||||
window.location.reload();
|
||||
}
|
||||
};
|
||||
|
||||
const handleLogout = () => {
|
||||
clearAccessToken();
|
||||
setIsAuthenticated(false);
|
||||
setToken('');
|
||||
setShowInput(true);
|
||||
window.location.reload();
|
||||
};
|
||||
|
||||
if (isAuthenticated && !showInput) {
|
||||
return (
|
||||
<div className="bg-green-500/10 border border-green-500/20 text-green-400 px-4 py-2 rounded-lg flex items-center justify-between text-sm">
|
||||
<div className="flex items-center gap-2">
|
||||
<KeyRound className="h-4 w-4" />
|
||||
<span>Authenticated</span>
|
||||
</div>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={handleLogout}
|
||||
className="h-6 px-2 text-xs text-green-400 hover:text-green-300"
|
||||
>
|
||||
Logout
|
||||
</Button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="bg-yellow-500/10 border border-yellow-500/20 text-yellow-400 px-4 py-3 rounded-lg">
|
||||
<div className="flex items-start gap-3">
|
||||
<KeyRound className="h-5 w-5 mt-0.5 flex-shrink-0" />
|
||||
<div className="flex-1">
|
||||
<p className="font-medium mb-2">Authentication Required</p>
|
||||
{showInput ? (
|
||||
<form onSubmit={handleSubmit} className="flex gap-2">
|
||||
<Input
|
||||
type="password"
|
||||
placeholder="Enter access token"
|
||||
value={token}
|
||||
onChange={(e) => setToken(e.target.value)}
|
||||
className="flex-1 bg-[var(--secondary)] border-[var(--card-border)] text-sm"
|
||||
/>
|
||||
<Button type="submit" size="sm" disabled={!token.trim()}>
|
||||
Authenticate
|
||||
</Button>
|
||||
</form>
|
||||
) : (
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
onClick={() => setShowInput(true)}
|
||||
className="mt-1"
|
||||
>
|
||||
Enter Token
|
||||
</Button>
|
||||
)}
|
||||
<p className="text-xs mt-2 text-[var(--text-muted)]">
|
||||
Contact an admin to get an access token
|
||||
</p>
|
||||
</div>
|
||||
{showInput && (
|
||||
<button
|
||||
onClick={() => setShowInput(false)}
|
||||
className="text-[var(--text-muted)] hover:text-[var(--text)]"
|
||||
>
|
||||
<X className="h-4 w-4" />
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
"use client"
|
||||
|
||||
import { useState } from "react"
|
||||
import { KeyRound } from "lucide-react"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Input } from "@/components/ui/input"
|
||||
import {
|
||||
Dialog,
|
||||
DialogContent,
|
||||
DialogDescription,
|
||||
DialogHeader,
|
||||
DialogTitle,
|
||||
} from "@/components/ui/dialog"
|
||||
import { setAccessToken } from "@/lib/api"
|
||||
|
||||
interface AuthDialogProps {
|
||||
open: boolean
|
||||
onClose: () => void
|
||||
onSuccess: () => void
|
||||
}
|
||||
|
||||
export function AuthDialog({ open, onClose, onSuccess }: AuthDialogProps) {
|
||||
const [token, setToken] = useState("")
|
||||
const [error, setError] = useState("")
|
||||
|
||||
const handleSubmit = (e: React.FormEvent) => {
|
||||
e.preventDefault()
|
||||
if (token.trim()) {
|
||||
setAccessToken(token.trim())
|
||||
setError("")
|
||||
setToken("") // Clear token from state for security
|
||||
onSuccess()
|
||||
} else {
|
||||
setError("Please enter a valid token")
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<Dialog open={open} onOpenChange={(open) => !open && onClose()}>
|
||||
<DialogContent className="sm:max-w-md bg-slate-900 border-slate-700">
|
||||
<DialogHeader>
|
||||
<DialogTitle className="text-white flex items-center gap-2">
|
||||
<KeyRound className="w-5 h-5 text-amber-400" />
|
||||
Authentication Required
|
||||
</DialogTitle>
|
||||
<DialogDescription className="text-slate-400">
|
||||
Enter your access token to use the Karaoke Generator
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
<form onSubmit={handleSubmit} className="space-y-4">
|
||||
<div>
|
||||
<Input
|
||||
type="password"
|
||||
placeholder="Enter access token"
|
||||
value={token}
|
||||
onChange={(e) => {
|
||||
setToken(e.target.value)
|
||||
setError("")
|
||||
}}
|
||||
className="bg-slate-800 border-slate-600 text-white"
|
||||
autoFocus
|
||||
/>
|
||||
{error && (
|
||||
<p className="text-xs text-red-400 mt-1">{error}</p>
|
||||
)}
|
||||
</div>
|
||||
<div className="flex justify-end gap-2">
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
onClick={onClose}
|
||||
className="text-slate-400"
|
||||
>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
type="submit"
|
||||
disabled={!token.trim()}
|
||||
className="bg-amber-600 hover:bg-amber-500"
|
||||
>
|
||||
Authenticate
|
||||
</Button>
|
||||
</div>
|
||||
<p className="text-xs text-slate-500 text-center">
|
||||
Contact an admin to get an access token
|
||||
</p>
|
||||
</form>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
"use client"
|
||||
|
||||
import { useState, useEffect } from "react"
|
||||
import { KeyRound, LogOut } from "lucide-react"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { getAccessToken, clearAccessToken } from "@/lib/api"
|
||||
import { AuthDialog } from "./AuthDialog"
|
||||
|
||||
interface AuthStatusProps {
|
||||
onAuthChange?: () => void
|
||||
}
|
||||
|
||||
export function AuthStatus({ onAuthChange }: AuthStatusProps) {
|
||||
const [isAuthenticated, setIsAuthenticated] = useState(false)
|
||||
const [showAuthDialog, setShowAuthDialog] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
setIsAuthenticated(!!getAccessToken())
|
||||
}, [])
|
||||
|
||||
const handleLogout = () => {
|
||||
clearAccessToken()
|
||||
setIsAuthenticated(false)
|
||||
onAuthChange?.()
|
||||
window.location.reload()
|
||||
}
|
||||
|
||||
const handleAuthSuccess = () => {
|
||||
setIsAuthenticated(true)
|
||||
setShowAuthDialog(false)
|
||||
onAuthChange?.()
|
||||
window.location.reload()
|
||||
}
|
||||
|
||||
if (isAuthenticated) {
|
||||
return (
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={handleLogout}
|
||||
className="text-slate-400 hover:text-white"
|
||||
>
|
||||
<LogOut className="w-4 h-4 mr-2" />
|
||||
Logout
|
||||
</Button>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={() => setShowAuthDialog(true)}
|
||||
className="text-amber-400 hover:text-amber-300"
|
||||
>
|
||||
<KeyRound className="w-4 h-4 mr-2" />
|
||||
Login
|
||||
</Button>
|
||||
<AuthDialog
|
||||
open={showAuthDialog}
|
||||
onClose={() => setShowAuthDialog(false)}
|
||||
onSuccess={handleAuthSuccess}
|
||||
/>
|
||||
</>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
export { AuthStatus } from "./AuthStatus"
|
||||
export { AuthDialog } from "./AuthDialog"
|
||||
@@ -0,0 +1,166 @@
|
||||
"use client"
|
||||
|
||||
import { useState, useEffect } from "react"
|
||||
import { api, InstrumentalOption } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Dialog, DialogContent, DialogDescription, DialogHeader, DialogTitle } from "@/components/ui/dialog"
|
||||
import { Loader2, Music2, Play, Pause } from "lucide-react"
|
||||
|
||||
interface InstrumentalSelectorProps {
|
||||
jobId: string
|
||||
open: boolean
|
||||
onClose: () => void
|
||||
onSelect: () => void
|
||||
}
|
||||
|
||||
export function InstrumentalSelector({ jobId, open, onClose, onSelect }: InstrumentalSelectorProps) {
|
||||
const [options, setOptions] = useState<InstrumentalOption[]>([])
|
||||
const [isLoading, setIsLoading] = useState(true)
|
||||
const [isSelecting, setIsSelecting] = useState(false)
|
||||
const [error, setError] = useState("")
|
||||
const [playingId, setPlayingId] = useState<string | null>(null)
|
||||
const [audioElements, setAudioElements] = useState<Map<string, HTMLAudioElement>>(new Map())
|
||||
|
||||
useEffect(() => {
|
||||
if (open) {
|
||||
loadOptions()
|
||||
}
|
||||
return () => {
|
||||
// Cleanup audio elements
|
||||
audioElements.forEach(audio => {
|
||||
audio.pause()
|
||||
audio.src = ""
|
||||
})
|
||||
}
|
||||
}, [open])
|
||||
|
||||
async function loadOptions() {
|
||||
setIsLoading(true)
|
||||
setError("")
|
||||
try {
|
||||
const data = await api.getInstrumentalOptions(jobId)
|
||||
setOptions(data.options || [])
|
||||
} catch (err: any) {
|
||||
console.error("Failed to load instrumental options:", err)
|
||||
setError(err.message || "Failed to load options")
|
||||
} finally {
|
||||
setIsLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
async function handleSelect(selection: 'clean' | 'with_backing') {
|
||||
setIsSelecting(true)
|
||||
try {
|
||||
await api.selectInstrumental(jobId, selection)
|
||||
onSelect()
|
||||
onClose()
|
||||
} catch (err: any) {
|
||||
console.error("Failed to select instrumental:", err)
|
||||
setError(err.message || "Failed to select instrumental")
|
||||
} finally {
|
||||
setIsSelecting(false)
|
||||
}
|
||||
}
|
||||
|
||||
function togglePlayback(optionId: string, audioUrl: string) {
|
||||
if (playingId === optionId) {
|
||||
// Pause current
|
||||
const audio = audioElements.get(optionId)
|
||||
if (audio) {
|
||||
audio.pause()
|
||||
}
|
||||
setPlayingId(null)
|
||||
} else {
|
||||
// Stop any currently playing audio
|
||||
audioElements.forEach(audio => audio.pause())
|
||||
|
||||
// Play new audio
|
||||
let audio = audioElements.get(optionId)
|
||||
if (!audio) {
|
||||
audio = new Audio(audioUrl)
|
||||
audio.addEventListener('ended', () => setPlayingId(null))
|
||||
audioElements.set(optionId, audio)
|
||||
setAudioElements(new Map(audioElements))
|
||||
}
|
||||
audio.play()
|
||||
setPlayingId(optionId)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<Dialog open={open} onOpenChange={(open) => !open && onClose()}>
|
||||
<DialogContent className="max-w-2xl bg-slate-900 border-slate-700">
|
||||
<DialogHeader>
|
||||
<DialogTitle className="text-white">Select Instrumental Version</DialogTitle>
|
||||
<DialogDescription className="text-slate-400">
|
||||
Choose between a clean instrumental or one with backing vocals
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
|
||||
{isLoading ? (
|
||||
<div className="flex items-center justify-center py-8">
|
||||
<Loader2 className="w-6 h-6 animate-spin text-slate-400" />
|
||||
<span className="ml-2 text-slate-400">Loading options...</span>
|
||||
</div>
|
||||
) : error ? (
|
||||
<div className="text-center py-8 text-red-400">{error}</div>
|
||||
) : options.length === 0 ? (
|
||||
<div className="text-center py-8 text-slate-500">
|
||||
<Music2 className="w-12 h-12 mx-auto mb-3 opacity-50" />
|
||||
<p>No instrumental options available</p>
|
||||
</div>
|
||||
) : (
|
||||
<div className="space-y-3">
|
||||
{options.map((option) => (
|
||||
<div
|
||||
key={option.id}
|
||||
className="border border-slate-700 rounded-lg p-4 hover:bg-slate-800/50 transition-colors"
|
||||
>
|
||||
<div className="flex items-center justify-between gap-3">
|
||||
<div className="flex-1">
|
||||
<p className="font-medium text-white">{option.label}</p>
|
||||
{option.duration_seconds && (
|
||||
<p className="text-xs text-slate-400 mt-1">
|
||||
Duration: {Math.floor(option.duration_seconds / 60)}:
|
||||
{(option.duration_seconds % 60).toString().padStart(2, '0')}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
{option.audio_url && (
|
||||
<Button
|
||||
size="sm"
|
||||
variant="ghost"
|
||||
onClick={() => togglePlayback(option.id, option.audio_url)}
|
||||
className="text-slate-400 hover:text-white"
|
||||
>
|
||||
{playingId === option.id ? (
|
||||
<Pause className="w-4 h-4" />
|
||||
) : (
|
||||
<Play className="w-4 h-4" />
|
||||
)}
|
||||
</Button>
|
||||
)}
|
||||
<Button
|
||||
size="sm"
|
||||
onClick={() => handleSelect(option.id as 'clean' | 'with_backing')}
|
||||
disabled={isSelecting}
|
||||
className="bg-pink-600 hover:bg-pink-500 text-white"
|
||||
>
|
||||
{isSelecting ? (
|
||||
<Loader2 className="w-3 h-3 animate-spin" />
|
||||
) : (
|
||||
"Select"
|
||||
)}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
"use client"
|
||||
|
||||
import { useState } from "react"
|
||||
import { Job, api } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import {
|
||||
Loader2, RotateCcw, XCircle, Trash2, FileText
|
||||
} from "lucide-react"
|
||||
|
||||
interface JobActionsProps {
|
||||
job: Job
|
||||
onRefresh: () => void
|
||||
showLogs: boolean
|
||||
onToggleLogs: () => void
|
||||
}
|
||||
|
||||
export function JobActions({ job, onRefresh, showLogs, onToggleLogs }: JobActionsProps) {
|
||||
const [isRetrying, setIsRetrying] = useState(false)
|
||||
const [isCancelling, setIsCancelling] = useState(false)
|
||||
const [isDeleting, setIsDeleting] = useState(false)
|
||||
|
||||
const canRetry = job.status === "failed"
|
||||
const canCancel = !["complete", "failed", "cancelled"].includes(job.status)
|
||||
const canDelete = ["complete", "failed", "cancelled"].includes(job.status)
|
||||
|
||||
async function handleRetry() {
|
||||
setIsRetrying(true)
|
||||
try {
|
||||
await api.retryJob(job.job_id)
|
||||
onRefresh()
|
||||
} catch (error) {
|
||||
console.error("Failed to retry job:", error)
|
||||
} finally {
|
||||
setIsRetrying(false)
|
||||
}
|
||||
}
|
||||
|
||||
async function handleCancel() {
|
||||
if (!confirm("Are you sure you want to cancel this job?")) return
|
||||
|
||||
setIsCancelling(true)
|
||||
try {
|
||||
await api.cancelJob(job.job_id)
|
||||
onRefresh()
|
||||
} catch (error) {
|
||||
console.error("Failed to cancel job:", error)
|
||||
} finally {
|
||||
setIsCancelling(false)
|
||||
}
|
||||
}
|
||||
|
||||
async function handleDelete() {
|
||||
if (!confirm("Are you sure you want to delete this job? This will remove all associated files.")) return
|
||||
|
||||
setIsDeleting(true)
|
||||
try {
|
||||
await api.deleteJob(job.job_id, true)
|
||||
onRefresh()
|
||||
} catch (error) {
|
||||
console.error("Failed to delete job:", error)
|
||||
} finally {
|
||||
setIsDeleting(false)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex flex-wrap gap-2" onClick={(e) => e.stopPropagation()}>
|
||||
{/* Cancel button first (red) */}
|
||||
{canCancel && (
|
||||
<Button
|
||||
size="sm"
|
||||
variant="ghost"
|
||||
onClick={handleCancel}
|
||||
disabled={isCancelling}
|
||||
className="text-xs text-red-400 hover:text-red-300"
|
||||
>
|
||||
{isCancelling ? (
|
||||
<Loader2 className="w-3 h-3 mr-1 animate-spin" />
|
||||
) : (
|
||||
<XCircle className="w-3 h-3 mr-1" />
|
||||
)}
|
||||
Cancel
|
||||
</Button>
|
||||
)}
|
||||
|
||||
{/* Delete button (red) */}
|
||||
{canDelete && (
|
||||
<Button
|
||||
size="sm"
|
||||
variant="ghost"
|
||||
onClick={handleDelete}
|
||||
disabled={isDeleting}
|
||||
className="text-xs text-red-400 hover:text-red-300"
|
||||
>
|
||||
{isDeleting ? (
|
||||
<Loader2 className="w-3 h-3 mr-1 animate-spin" />
|
||||
) : (
|
||||
<Trash2 className="w-3 h-3 mr-1" />
|
||||
)}
|
||||
Delete
|
||||
</Button>
|
||||
)}
|
||||
|
||||
{/* Retry button */}
|
||||
{canRetry && (
|
||||
<Button
|
||||
size="sm"
|
||||
variant="ghost"
|
||||
onClick={handleRetry}
|
||||
disabled={isRetrying}
|
||||
className="text-xs text-blue-400 hover:text-blue-300"
|
||||
>
|
||||
{isRetrying ? (
|
||||
<Loader2 className="w-3 h-3 mr-1 animate-spin" />
|
||||
) : (
|
||||
<RotateCcw className="w-3 h-3 mr-1" />
|
||||
)}
|
||||
Retry
|
||||
</Button>
|
||||
)}
|
||||
|
||||
{/* Show/Hide Logs */}
|
||||
<Button
|
||||
size="sm"
|
||||
variant="ghost"
|
||||
onClick={onToggleLogs}
|
||||
className="text-xs text-slate-400 hover:text-white"
|
||||
>
|
||||
<FileText className="w-3 h-3 mr-1" />
|
||||
{showLogs ? "Hide" : "Show"} Logs
|
||||
</Button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
"use client"
|
||||
|
||||
import { useState } from "react"
|
||||
import { Job } from "@/lib/api"
|
||||
import { useAutoMode } from "@/lib/auto-mode"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Loader2, ExternalLink, Zap } from "lucide-react"
|
||||
import { JobActions } from "./JobActions"
|
||||
import { JobLogs } from "./JobLogs"
|
||||
import { OutputLinks } from "./OutputLinks"
|
||||
import { InstrumentalSelector } from "./InstrumentalSelector"
|
||||
import { AudioSearchDialog } from "../audio-search/AudioSearchDialog"
|
||||
|
||||
// Status text styling (subtle colored text instead of badges)
|
||||
const statusConfig: Record<string, { textColor: string; label: string }> = {
|
||||
pending: { textColor: "text-slate-400", label: "Pending" },
|
||||
searching_audio: { textColor: "text-blue-400", label: "Searching..." },
|
||||
awaiting_audio_selection: { textColor: "text-amber-400", label: "Needs audio" },
|
||||
downloading: { textColor: "text-blue-400", label: "Downloading..." },
|
||||
downloading_audio: { textColor: "text-blue-400", label: "Downloading..." },
|
||||
separating_stage1: { textColor: "text-purple-400", label: "Separating 1/2..." },
|
||||
separating_stage2: { textColor: "text-purple-400", label: "Separating 2/2..." },
|
||||
audio_complete: { textColor: "text-purple-400", label: "Audio ready" },
|
||||
transcribing: { textColor: "text-blue-400", label: "Transcribing..." },
|
||||
correcting: { textColor: "text-blue-400", label: "Correcting..." },
|
||||
lyrics_complete: { textColor: "text-teal-400", label: "Lyrics ready" },
|
||||
generating_screens: { textColor: "text-cyan-400", label: "Generating screens..." },
|
||||
awaiting_review: { textColor: "text-amber-400", label: "Needs review" },
|
||||
in_review: { textColor: "text-blue-400", label: "In review..." },
|
||||
review_complete: { textColor: "text-teal-400", label: "Review complete" },
|
||||
rendering_video: { textColor: "text-indigo-400", label: "Rendering..." },
|
||||
awaiting_instrumental_selection: { textColor: "text-amber-400", label: "Needs instrumental" },
|
||||
instrumental_selected: { textColor: "text-pink-400", label: "Instrumental selected" },
|
||||
generating_video: { textColor: "text-violet-400", label: "Generating video..." },
|
||||
encoding: { textColor: "text-violet-400", label: "Encoding..." },
|
||||
packaging: { textColor: "text-violet-400", label: "Packaging..." },
|
||||
uploading: { textColor: "text-green-400", label: "Uploading..." },
|
||||
complete: { textColor: "text-green-400", label: "Complete" },
|
||||
failed: { textColor: "text-red-400", label: "Failed" },
|
||||
cancelled: { textColor: "text-slate-500", label: "Cancelled" },
|
||||
}
|
||||
|
||||
function StatusText({ status }: { status: string }) {
|
||||
const config = statusConfig[status] || { textColor: "text-slate-400", label: status }
|
||||
return <span className={config.textColor}>{config.label}</span>
|
||||
}
|
||||
|
||||
interface JobCardProps {
|
||||
job: Job
|
||||
onRefresh: () => void
|
||||
}
|
||||
|
||||
export function JobCard({ job, onRefresh }: JobCardProps) {
|
||||
const [showLogs, setShowLogs] = useState(false)
|
||||
const [showAudioSearch, setShowAudioSearch] = useState(false)
|
||||
const [showInstrumentalSelector, setShowInstrumentalSelector] = useState(false)
|
||||
const { enabled: autoModeEnabled, isProcessing: isAutoProcessing } = useAutoMode()
|
||||
|
||||
const createdAt = new Date(job.created_at).toLocaleString()
|
||||
const isInteractive = job.status === "awaiting_review" ||
|
||||
job.status === "in_review" ||
|
||||
job.status === "awaiting_instrumental_selection" ||
|
||||
job.status === "awaiting_audio_selection"
|
||||
const isComplete = job.status === "complete"
|
||||
const isFailed = job.status === "failed"
|
||||
const isProcessing = !isComplete && !isFailed && job.status !== "cancelled"
|
||||
|
||||
// Check if this job is being auto-processed
|
||||
const isBeingAutoProcessed = autoModeEnabled && isInteractive && (
|
||||
isAutoProcessing(`${job.job_id}-review`) ||
|
||||
isAutoProcessing(`${job.job_id}-instrumental`) ||
|
||||
isAutoProcessing(`${job.job_id}-audio`)
|
||||
)
|
||||
|
||||
// Build review URL for awaiting_review status
|
||||
const getReviewUrl = () => {
|
||||
const backendUrl = process.env.NEXT_PUBLIC_API_URL || 'https://api.nomadkaraoke.com'
|
||||
const reviewUiUrl = process.env.NEXT_PUBLIC_REVIEW_UI_URL || 'https://gen.nomadkaraoke.com/lyrics/'
|
||||
const baseApiUrl = `${backendUrl}/api/review/${job.job_id}`
|
||||
const encodedApiUrl = encodeURIComponent(baseApiUrl)
|
||||
let url = `${reviewUiUrl}/?baseApiUrl=${encodedApiUrl}`
|
||||
if (job.audio_hash) {
|
||||
url += `&audioHash=${encodeURIComponent(job.audio_hash)}`
|
||||
}
|
||||
if (job.review_token) {
|
||||
url += `&reviewToken=${encodeURIComponent(job.review_token)}`
|
||||
}
|
||||
return url
|
||||
}
|
||||
|
||||
// Render primary action button based on status
|
||||
const renderPrimaryAction = () => {
|
||||
if (autoModeEnabled && isInteractive) {
|
||||
return (
|
||||
<div className="flex items-center gap-1 text-xs text-amber-400">
|
||||
{isBeingAutoProcessed ? (
|
||||
<>
|
||||
<Loader2 className="w-3 h-3 animate-spin" />
|
||||
Auto-processing...
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<Zap className="w-3 h-3" />
|
||||
Will auto-{job.status === "awaiting_review" || job.status === "in_review" ? "accept" :
|
||||
job.status === "awaiting_instrumental_selection" ? "select clean" :
|
||||
"select first"}
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
if (job.status === "awaiting_review" || job.status === "in_review") {
|
||||
return (
|
||||
<a
|
||||
href={getReviewUrl()}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-blue-600 hover:bg-blue-500 text-white"
|
||||
>
|
||||
<ExternalLink className="w-3 h-3" />
|
||||
Review Lyrics
|
||||
</a>
|
||||
)
|
||||
}
|
||||
|
||||
if (job.status === "awaiting_audio_selection") {
|
||||
return (
|
||||
<Button
|
||||
size="sm"
|
||||
onClick={() => setShowAudioSearch(true)}
|
||||
className="text-xs h-7 px-3 bg-blue-600 hover:bg-blue-500"
|
||||
>
|
||||
Select Audio
|
||||
</Button>
|
||||
)
|
||||
}
|
||||
|
||||
if (job.status === "awaiting_instrumental_selection") {
|
||||
return (
|
||||
<Button
|
||||
size="sm"
|
||||
onClick={() => setShowInstrumentalSelector(true)}
|
||||
className="text-xs h-7 px-3 bg-blue-600 hover:bg-blue-500"
|
||||
>
|
||||
Select Instrumental
|
||||
</Button>
|
||||
)
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
return (
|
||||
<div
|
||||
className={`rounded-lg border p-3 transition-colors border-slate-700 bg-slate-800/30
|
||||
${isComplete ? "border-green-500/30" : ""}
|
||||
${isFailed ? "border-red-500/30" : ""}`}
|
||||
>
|
||||
{/* Header row with title */}
|
||||
<p className="font-medium text-white truncate">
|
||||
{job.artist || "Unknown"} - {job.title || "Unknown"}
|
||||
</p>
|
||||
|
||||
{/* Meta row: ID, date, status */}
|
||||
<p className="text-xs text-slate-500 mt-1">
|
||||
<span className="text-slate-600">ID:</span> {job.job_id} <span className="text-slate-600">•</span> {createdAt} <span className="text-slate-600">•</span> <StatusText status={job.status} />
|
||||
</p>
|
||||
|
||||
{/* Error message if any */}
|
||||
{job.error_message && (
|
||||
<div className="mt-2 text-xs text-red-400 bg-red-500/10 rounded p-2 break-words">
|
||||
{job.error_message}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Output links for completed jobs */}
|
||||
{isComplete && (
|
||||
<div className="mt-2">
|
||||
<OutputLinks jobId={job.job_id} />
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Actions row: Cancel/secondary on left, primary action on right */}
|
||||
<div className="mt-2 flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<JobActions
|
||||
job={job}
|
||||
onRefresh={onRefresh}
|
||||
showLogs={showLogs}
|
||||
onToggleLogs={() => setShowLogs(!showLogs)}
|
||||
/>
|
||||
</div>
|
||||
{renderPrimaryAction()}
|
||||
</div>
|
||||
|
||||
{/* Expandable logs */}
|
||||
{showLogs && (
|
||||
<div className="mt-2">
|
||||
<JobLogs jobId={job.job_id} />
|
||||
</div>
|
||||
)}
|
||||
|
||||
<AudioSearchDialog
|
||||
jobId={job.job_id}
|
||||
open={showAudioSearch}
|
||||
onClose={() => setShowAudioSearch(false)}
|
||||
onSelect={onRefresh}
|
||||
/>
|
||||
|
||||
<InstrumentalSelector
|
||||
jobId={job.job_id}
|
||||
open={showInstrumentalSelector}
|
||||
onClose={() => setShowInstrumentalSelector(false)}
|
||||
onSelect={onRefresh}
|
||||
/>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState } from "react"
|
||||
import { api, JobLog } from "@/lib/api"
|
||||
import { Loader2, AlertCircle } from "lucide-react"
|
||||
|
||||
interface JobLogsProps {
|
||||
jobId: string
|
||||
limit?: number
|
||||
}
|
||||
|
||||
export function JobLogs({ jobId, limit = 50 }: JobLogsProps) {
|
||||
const [logs, setLogs] = useState<JobLog[]>([])
|
||||
const [isLoading, setIsLoading] = useState(true)
|
||||
const [error, setError] = useState("")
|
||||
|
||||
useEffect(() => {
|
||||
loadLogs()
|
||||
}, [jobId])
|
||||
|
||||
async function loadLogs() {
|
||||
setIsLoading(true)
|
||||
setError("")
|
||||
try {
|
||||
const data = await api.getJobLogs(jobId, limit)
|
||||
setLogs(data)
|
||||
} catch (err) {
|
||||
console.error("Failed to load logs:", err)
|
||||
setError("Failed to load logs")
|
||||
} finally {
|
||||
setIsLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
if (isLoading) {
|
||||
return (
|
||||
<div className="flex items-center justify-center py-4 text-slate-400">
|
||||
<Loader2 className="w-4 h-4 animate-spin mr-2" />
|
||||
Loading logs...
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
if (error) {
|
||||
return (
|
||||
<div className="flex items-center py-4 text-red-400 text-sm">
|
||||
<AlertCircle className="w-4 h-4 mr-2" />
|
||||
{error}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
if (logs.length === 0) {
|
||||
return (
|
||||
<div className="text-center py-4 text-slate-500 text-sm">
|
||||
No logs available
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="bg-slate-900 rounded border border-slate-700 p-3 max-h-64 overflow-y-auto">
|
||||
<div className="space-y-1 font-mono text-xs">
|
||||
{logs.map((log, index) => (
|
||||
<div
|
||||
key={index}
|
||||
className={`flex gap-2 ${
|
||||
log.level === "ERROR" ? "text-red-400" :
|
||||
log.level === "WARNING" ? "text-yellow-400" :
|
||||
log.level === "INFO" ? "text-blue-400" :
|
||||
"text-slate-400"
|
||||
}`}
|
||||
>
|
||||
<span className="text-slate-600 shrink-0">
|
||||
{new Date(log.timestamp).toLocaleTimeString()}
|
||||
</span>
|
||||
<span className="shrink-0 font-semibold w-12">
|
||||
{log.level}
|
||||
</span>
|
||||
<span className="break-all">{log.message}</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,338 @@
|
||||
"use client"
|
||||
|
||||
import { useState } from "react"
|
||||
import { api, ApiError } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Input } from "@/components/ui/input"
|
||||
import { Label } from "@/components/ui/label"
|
||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"
|
||||
import { Upload, Youtube, Music, Loader2 } from "lucide-react"
|
||||
|
||||
interface JobSubmissionProps {
|
||||
onJobCreated: () => void
|
||||
}
|
||||
|
||||
export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
const [activeTab, setActiveTab] = useState("upload")
|
||||
const [error, setError] = useState("")
|
||||
const [isSubmitting, setIsSubmitting] = useState(false)
|
||||
|
||||
// Upload form
|
||||
const [uploadFile, setUploadFile] = useState<File | null>(null)
|
||||
const [uploadArtist, setUploadArtist] = useState("")
|
||||
const [uploadTitle, setUploadTitle] = useState("")
|
||||
|
||||
// URL form
|
||||
const [youtubeUrl, setYoutubeUrl] = useState("")
|
||||
const [youtubeArtist, setYoutubeArtist] = useState("")
|
||||
const [youtubeTitle, setYoutubeTitle] = useState("")
|
||||
|
||||
// Audio search form
|
||||
const [searchArtist, setSearchArtist] = useState("")
|
||||
const [searchTitle, setSearchTitle] = useState("")
|
||||
|
||||
async function handleUploadSubmit(e: React.FormEvent) {
|
||||
e.preventDefault()
|
||||
setError("")
|
||||
|
||||
if (!uploadFile) {
|
||||
setError("Please select an audio file")
|
||||
return
|
||||
}
|
||||
if (!uploadArtist.trim() || !uploadTitle.trim()) {
|
||||
setError("Please enter both artist and title")
|
||||
return
|
||||
}
|
||||
|
||||
setIsSubmitting(true)
|
||||
try {
|
||||
await api.uploadJob(uploadFile, uploadArtist.trim(), uploadTitle.trim())
|
||||
setUploadFile(null)
|
||||
setUploadArtist("")
|
||||
setUploadTitle("")
|
||||
onJobCreated()
|
||||
} catch (err) {
|
||||
if (err instanceof ApiError) {
|
||||
setError(err.message)
|
||||
} else {
|
||||
setError("Failed to create job")
|
||||
}
|
||||
} finally {
|
||||
setIsSubmitting(false)
|
||||
}
|
||||
}
|
||||
|
||||
async function handleUrlSubmit(e: React.FormEvent) {
|
||||
e.preventDefault()
|
||||
setError("")
|
||||
|
||||
if (!youtubeUrl.trim()) {
|
||||
setError("Please enter a URL")
|
||||
return
|
||||
}
|
||||
|
||||
setIsSubmitting(true)
|
||||
try {
|
||||
await api.createJobFromUrl(
|
||||
youtubeUrl.trim(),
|
||||
youtubeArtist.trim() || undefined,
|
||||
youtubeTitle.trim() || undefined
|
||||
)
|
||||
setYoutubeUrl("")
|
||||
setYoutubeArtist("")
|
||||
setYoutubeTitle("")
|
||||
onJobCreated()
|
||||
} catch (err) {
|
||||
if (err instanceof ApiError) {
|
||||
setError(err.message)
|
||||
} else {
|
||||
setError("Failed to create job")
|
||||
}
|
||||
} finally {
|
||||
setIsSubmitting(false)
|
||||
}
|
||||
}
|
||||
|
||||
async function handleSearchSubmit(e: React.FormEvent) {
|
||||
e.preventDefault()
|
||||
setError("")
|
||||
|
||||
if (!searchArtist.trim() || !searchTitle.trim()) {
|
||||
setError("Please enter both artist and title")
|
||||
return
|
||||
}
|
||||
|
||||
setIsSubmitting(true)
|
||||
try {
|
||||
await api.searchAudio(searchArtist.trim(), searchTitle.trim())
|
||||
setSearchArtist("")
|
||||
setSearchTitle("")
|
||||
onJobCreated()
|
||||
} catch (err) {
|
||||
if (err instanceof ApiError) {
|
||||
setError(err.message)
|
||||
} else {
|
||||
setError("Failed to search for audio")
|
||||
}
|
||||
} finally {
|
||||
setIsSubmitting(false)
|
||||
}
|
||||
}
|
||||
|
||||
function handleFileChange(e: React.ChangeEvent<HTMLInputElement>) {
|
||||
const file = e.target.files?.[0]
|
||||
if (file) {
|
||||
setUploadFile(file)
|
||||
setError("")
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<Tabs value={activeTab} onValueChange={setActiveTab}>
|
||||
<TabsList className="grid w-full grid-cols-3 bg-slate-800">
|
||||
<TabsTrigger value="upload" className="gap-2 data-[state=active]:bg-slate-700">
|
||||
<Upload className="w-4 h-4" />
|
||||
Upload
|
||||
</TabsTrigger>
|
||||
<TabsTrigger value="url" className="gap-2 data-[state=active]:bg-slate-700">
|
||||
<Youtube className="w-4 h-4" />
|
||||
URL
|
||||
</TabsTrigger>
|
||||
<TabsTrigger value="search" className="gap-2 data-[state=active]:bg-slate-700">
|
||||
<Music className="w-4 h-4" />
|
||||
Search
|
||||
</TabsTrigger>
|
||||
</TabsList>
|
||||
|
||||
<TabsContent value="upload" className="mt-4">
|
||||
<form onSubmit={handleUploadSubmit} className="space-y-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="audio-file" className="text-slate-200">Audio File</Label>
|
||||
<div
|
||||
className={`relative border-2 border-dashed rounded-lg p-6 text-center transition-colors cursor-pointer
|
||||
${uploadFile ? "border-amber-500/50 bg-amber-500/5" : "border-slate-700 hover:border-slate-600 bg-slate-800/50"}`}
|
||||
>
|
||||
<Input
|
||||
id="audio-file"
|
||||
type="file"
|
||||
accept=".mp3,.wav,.flac,.m4a,.ogg,audio/*"
|
||||
onChange={handleFileChange}
|
||||
className="absolute inset-0 w-full h-full opacity-0 cursor-pointer"
|
||||
disabled={isSubmitting}
|
||||
/>
|
||||
<Upload className="w-8 h-8 mx-auto mb-2 text-slate-500" />
|
||||
{uploadFile ? (
|
||||
<div>
|
||||
<p className="font-medium text-white">{uploadFile.name}</p>
|
||||
<p className="text-sm text-slate-400">{(uploadFile.size / 1024 / 1024).toFixed(2)} MB</p>
|
||||
</div>
|
||||
) : (
|
||||
<div>
|
||||
<p className="font-medium text-slate-300 mb-1">Click to upload</p>
|
||||
<p className="text-sm text-slate-500">MP3, WAV, FLAC, M4A, or OGG</p>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="upload-artist" className="text-slate-200">Artist</Label>
|
||||
<Input
|
||||
id="upload-artist"
|
||||
placeholder="Artist name"
|
||||
value={uploadArtist}
|
||||
onChange={(e) => setUploadArtist(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
|
||||
/>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="upload-title" className="text-slate-200">Title</Label>
|
||||
<Input
|
||||
id="upload-title"
|
||||
placeholder="Song title"
|
||||
value={uploadTitle}
|
||||
onChange={(e) => setUploadTitle(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{error && activeTab === "upload" && (
|
||||
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
|
||||
)}
|
||||
|
||||
<Button
|
||||
type="submit"
|
||||
className="w-full bg-amber-600 hover:bg-amber-500 text-white"
|
||||
disabled={isSubmitting}
|
||||
>
|
||||
{isSubmitting ? (
|
||||
<>
|
||||
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
|
||||
Creating Job...
|
||||
</>
|
||||
) : (
|
||||
"Create Karaoke Video"
|
||||
)}
|
||||
</Button>
|
||||
</form>
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="url" className="mt-4">
|
||||
<form onSubmit={handleUrlSubmit} className="space-y-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="youtube-url" className="text-slate-200">YouTube URL</Label>
|
||||
<div className="relative">
|
||||
<Youtube className="absolute left-3 top-1/2 -translate-y-1/2 w-4 h-4 text-slate-500" />
|
||||
<Input
|
||||
id="youtube-url"
|
||||
type="url"
|
||||
placeholder="https://youtube.com/watch?v=..."
|
||||
value={youtubeUrl}
|
||||
onChange={(e) => setYoutubeUrl(e.target.value)}
|
||||
className="pl-10 bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
|
||||
disabled={isSubmitting}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="youtube-artist" className="text-slate-200">Artist (optional)</Label>
|
||||
<Input
|
||||
id="youtube-artist"
|
||||
placeholder="Auto-detected"
|
||||
value={youtubeArtist}
|
||||
onChange={(e) => setYoutubeArtist(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
|
||||
/>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="youtube-title" className="text-slate-200">Title (optional)</Label>
|
||||
<Input
|
||||
id="youtube-title"
|
||||
placeholder="Auto-detected"
|
||||
value={youtubeTitle}
|
||||
onChange={(e) => setYoutubeTitle(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{error && activeTab === "url" && (
|
||||
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
|
||||
)}
|
||||
|
||||
<Button
|
||||
type="submit"
|
||||
className="w-full bg-amber-600 hover:bg-amber-500 text-white"
|
||||
disabled={isSubmitting}
|
||||
>
|
||||
{isSubmitting ? (
|
||||
<>
|
||||
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
|
||||
Creating Job...
|
||||
</>
|
||||
) : (
|
||||
"Create Karaoke Video"
|
||||
)}
|
||||
</Button>
|
||||
</form>
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="search" className="mt-4">
|
||||
<form onSubmit={handleSearchSubmit} className="space-y-4">
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="search-artist" className="text-slate-200">Artist</Label>
|
||||
<Input
|
||||
id="search-artist"
|
||||
placeholder="Artist name"
|
||||
value={searchArtist}
|
||||
onChange={(e) => setSearchArtist(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
|
||||
/>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="search-title" className="text-slate-200">Title</Label>
|
||||
<Input
|
||||
id="search-title"
|
||||
placeholder="Song title"
|
||||
value={searchTitle}
|
||||
onChange={(e) => setSearchTitle(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{error && activeTab === "search" && (
|
||||
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
|
||||
)}
|
||||
|
||||
<Button
|
||||
type="submit"
|
||||
className="w-full bg-amber-600 hover:bg-amber-500 text-white"
|
||||
disabled={isSubmitting}
|
||||
>
|
||||
{isSubmitting ? (
|
||||
<>
|
||||
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
|
||||
Searching...
|
||||
</>
|
||||
) : (
|
||||
"Search & Create Job"
|
||||
)}
|
||||
</Button>
|
||||
</form>
|
||||
</TabsContent>
|
||||
</Tabs>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState } from "react"
|
||||
import { api } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Download, Loader2, ExternalLink, FolderOpen } from "lucide-react"
|
||||
|
||||
interface OutputLinksProps {
|
||||
jobId: string
|
||||
}
|
||||
|
||||
export function OutputLinks({ jobId }: OutputLinksProps) {
|
||||
const [downloadUrls, setDownloadUrls] = useState<Record<string, any> | null>(null)
|
||||
const [youtubeUrl, setYoutubeUrl] = useState<string | null>(null)
|
||||
const [dropboxUrl, setDropboxUrl] = useState<string | null>(null)
|
||||
const [isLoading, setIsLoading] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
loadOutputLinks()
|
||||
}, [jobId])
|
||||
|
||||
async function loadOutputLinks() {
|
||||
setIsLoading(true)
|
||||
try {
|
||||
const result = await api.getDownloadUrls(jobId)
|
||||
setDownloadUrls(result.download_urls)
|
||||
|
||||
// Extract YouTube and Dropbox URLs from state_data if available
|
||||
const job = await api.getJob(jobId)
|
||||
if (job.state_data?.youtube_url) {
|
||||
setYoutubeUrl(job.state_data.youtube_url)
|
||||
}
|
||||
if (job.state_data?.dropbox_folder_url) {
|
||||
setDropboxUrl(job.state_data.dropbox_folder_url)
|
||||
}
|
||||
} catch (err) {
|
||||
console.error("Failed to load output links:", err)
|
||||
} finally {
|
||||
setIsLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
if (isLoading) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 text-xs text-slate-400">
|
||||
<Loader2 className="w-3 h-3 animate-spin" />
|
||||
Loading downloads...
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-2">
|
||||
{/* Distribution Links */}
|
||||
{(youtubeUrl || dropboxUrl) && (
|
||||
<div className="space-y-2">
|
||||
<p className="text-xs text-slate-400 font-medium">Distribution:</p>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
{youtubeUrl && (
|
||||
<a
|
||||
href={youtubeUrl}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-red-600 hover:bg-red-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<ExternalLink className="w-3 h-3" />
|
||||
View on YouTube
|
||||
</a>
|
||||
)}
|
||||
{dropboxUrl && (
|
||||
<a
|
||||
href={dropboxUrl}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-blue-600 hover:bg-blue-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<FolderOpen className="w-3 h-3" />
|
||||
Open Dropbox Folder
|
||||
</a>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Download Links */}
|
||||
{downloadUrls && Object.keys(downloadUrls).length > 0 && (
|
||||
<div className="space-y-2">
|
||||
<p className="text-xs text-slate-400 font-medium">Downloads:</p>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
{downloadUrls.finals?.lossy_720p && (
|
||||
<a
|
||||
href={api.getDownloadUrl(jobId, "finals", "lossy_720p")}
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-green-600 hover:bg-green-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<Download className="w-3 h-3" />
|
||||
720p Video
|
||||
</a>
|
||||
)}
|
||||
{downloadUrls.finals?.lossy_4k_mp4 && (
|
||||
<a
|
||||
href={api.getDownloadUrl(jobId, "finals", "lossy_4k_mp4")}
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-green-600 hover:bg-green-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<Download className="w-3 h-3" />
|
||||
4K Video
|
||||
</a>
|
||||
)}
|
||||
{downloadUrls.packages?.cdg_zip && (
|
||||
<a
|
||||
href={api.getDownloadUrl(jobId, "packages", "cdg_zip")}
|
||||
className="inline-flex items-center gap-1 text-xs px-2 py-1.5 rounded bg-slate-600 hover:bg-slate-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<Download className="w-3 h-3" />
|
||||
CDG+MP3
|
||||
</a>
|
||||
)}
|
||||
{downloadUrls.packages?.txt_zip && (
|
||||
<a
|
||||
href={api.getDownloadUrl(jobId, "packages", "txt_zip")}
|
||||
className="inline-flex items-center gap-1 text-xs px-2 py-1.5 rounded bg-slate-600 hover:bg-slate-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<Download className="w-3 h-3" />
|
||||
TXT+MP3
|
||||
</a>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{!downloadUrls && !youtubeUrl && !dropboxUrl && (
|
||||
<p className="text-xs text-slate-500">No outputs available yet</p>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
export { JobCard, StatusBadge } from "./JobCard"
|
||||
export { JobActions } from "./JobActions"
|
||||
export { JobLogs } from "./JobLogs"
|
||||
export { OutputLinks } from "./OutputLinks"
|
||||
export { InstrumentalSelector } from "./InstrumentalSelector"
|
||||
|
||||
@@ -1,143 +1,109 @@
|
||||
'use client'
|
||||
"use client"
|
||||
|
||||
import * as React from 'react'
|
||||
import * as DialogPrimitive from '@radix-ui/react-dialog'
|
||||
import { XIcon } from 'lucide-react'
|
||||
import * as React from "react"
|
||||
import * as DialogPrimitive from "@radix-ui/react-dialog"
|
||||
import { X } from "lucide-react"
|
||||
import { cn } from "@/lib/utils"
|
||||
|
||||
import { cn } from '@/lib/utils'
|
||||
const Dialog = DialogPrimitive.Root
|
||||
|
||||
function Dialog({
|
||||
...props
|
||||
}: React.ComponentProps<typeof DialogPrimitive.Root>) {
|
||||
return <DialogPrimitive.Root data-slot="dialog" {...props} />
|
||||
}
|
||||
const DialogTrigger = DialogPrimitive.Trigger
|
||||
|
||||
function DialogTrigger({
|
||||
...props
|
||||
}: React.ComponentProps<typeof DialogPrimitive.Trigger>) {
|
||||
return <DialogPrimitive.Trigger data-slot="dialog-trigger" {...props} />
|
||||
}
|
||||
const DialogPortal = DialogPrimitive.Portal
|
||||
|
||||
function DialogPortal({
|
||||
...props
|
||||
}: React.ComponentProps<typeof DialogPrimitive.Portal>) {
|
||||
return <DialogPrimitive.Portal data-slot="dialog-portal" {...props} />
|
||||
}
|
||||
const DialogClose = DialogPrimitive.Close
|
||||
|
||||
function DialogClose({
|
||||
...props
|
||||
}: React.ComponentProps<typeof DialogPrimitive.Close>) {
|
||||
return <DialogPrimitive.Close data-slot="dialog-close" {...props} />
|
||||
}
|
||||
const DialogOverlay = React.forwardRef<
|
||||
React.ElementRef<typeof DialogPrimitive.Overlay>,
|
||||
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Overlay>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<DialogPrimitive.Overlay
|
||||
ref={ref}
|
||||
className="fixed inset-0 z-50 bg-black/80 data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0"
|
||||
{...props}
|
||||
/>
|
||||
))
|
||||
DialogOverlay.displayName = DialogPrimitive.Overlay.displayName
|
||||
|
||||
function DialogOverlay({
|
||||
className,
|
||||
...props
|
||||
}: React.ComponentProps<typeof DialogPrimitive.Overlay>) {
|
||||
return (
|
||||
<DialogPrimitive.Overlay
|
||||
data-slot="dialog-overlay"
|
||||
const DialogContent = React.forwardRef<
|
||||
React.ElementRef<typeof DialogPrimitive.Content>,
|
||||
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Content>
|
||||
>(({ className, children, ...props }, ref) => (
|
||||
<DialogPortal>
|
||||
<DialogOverlay />
|
||||
<DialogPrimitive.Content
|
||||
ref={ref}
|
||||
className={cn(
|
||||
'data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 fixed inset-0 z-50 bg-black/50',
|
||||
className,
|
||||
"fixed left-[50%] top-[50%] z-50 grid w-full max-w-lg translate-x-[-50%] translate-y-[-50%] gap-4 border bg-background p-6 shadow-lg duration-200 data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[state=closed]:slide-out-to-left-1/2 data-[state=closed]:slide-out-to-top-[48%] data-[state=open]:slide-in-from-left-1/2 data-[state=open]:slide-in-from-top-[48%] sm:rounded-lg",
|
||||
className
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
)
|
||||
}
|
||||
>
|
||||
{children}
|
||||
<DialogPrimitive.Close className="absolute right-3 top-2.5 rounded-sm opacity-70 ring-offset-background transition-opacity hover:opacity-100 focus:outline-none focus:ring-2 focus:ring-ring focus:ring-offset-2 disabled:pointer-events-none data-[state=open]:bg-accent data-[state=open]:text-muted-foreground">
|
||||
<X className="h-4 w-4" />
|
||||
<span className="sr-only">Close</span>
|
||||
</DialogPrimitive.Close>
|
||||
</DialogPrimitive.Content>
|
||||
</DialogPortal>
|
||||
))
|
||||
DialogContent.displayName = DialogPrimitive.Content.displayName
|
||||
|
||||
function DialogContent({
|
||||
className,
|
||||
children,
|
||||
showCloseButton = true,
|
||||
...props
|
||||
}: React.ComponentProps<typeof DialogPrimitive.Content> & {
|
||||
showCloseButton?: boolean
|
||||
}) {
|
||||
return (
|
||||
<DialogPortal data-slot="dialog-portal">
|
||||
<DialogOverlay />
|
||||
<DialogPrimitive.Content
|
||||
data-slot="dialog-content"
|
||||
className={cn(
|
||||
'bg-background data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 fixed top-[50%] left-[50%] z-50 grid w-full max-w-[calc(100%-2rem)] translate-x-[-50%] translate-y-[-50%] gap-4 rounded-lg border p-6 shadow-lg duration-200 sm:max-w-lg',
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
>
|
||||
{children}
|
||||
{showCloseButton && (
|
||||
<DialogPrimitive.Close
|
||||
data-slot="dialog-close"
|
||||
className="ring-offset-background focus:ring-ring data-[state=open]:bg-accent data-[state=open]:text-muted-foreground absolute top-4 right-4 rounded-xs opacity-70 transition-opacity hover:opacity-100 focus:ring-2 focus:ring-offset-2 focus:outline-hidden disabled:pointer-events-none [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4"
|
||||
>
|
||||
<XIcon />
|
||||
<span className="sr-only">Close</span>
|
||||
</DialogPrimitive.Close>
|
||||
)}
|
||||
</DialogPrimitive.Content>
|
||||
</DialogPortal>
|
||||
)
|
||||
}
|
||||
|
||||
function DialogHeader({ className, ...props }: React.ComponentProps<'div'>) {
|
||||
return (
|
||||
<div
|
||||
data-slot="dialog-header"
|
||||
className={cn('flex flex-col gap-2 text-center sm:text-left', className)}
|
||||
{...props}
|
||||
/>
|
||||
)
|
||||
}
|
||||
|
||||
function DialogFooter({ className, ...props }: React.ComponentProps<'div'>) {
|
||||
return (
|
||||
<div
|
||||
data-slot="dialog-footer"
|
||||
className={cn(
|
||||
'flex flex-col-reverse gap-2 sm:flex-row sm:justify-end',
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
)
|
||||
}
|
||||
|
||||
function DialogTitle({
|
||||
const DialogHeader = ({
|
||||
className,
|
||||
...props
|
||||
}: React.ComponentProps<typeof DialogPrimitive.Title>) {
|
||||
return (
|
||||
<DialogPrimitive.Title
|
||||
data-slot="dialog-title"
|
||||
className={cn('text-lg leading-none font-semibold', className)}
|
||||
{...props}
|
||||
/>
|
||||
)
|
||||
}
|
||||
}: React.HTMLAttributes<HTMLDivElement>) => (
|
||||
<div
|
||||
className="flex flex-col space-y-1.5 text-center sm:text-left"
|
||||
{...props}
|
||||
/>
|
||||
)
|
||||
DialogHeader.displayName = "DialogHeader"
|
||||
|
||||
function DialogDescription({
|
||||
const DialogFooter = ({
|
||||
className,
|
||||
...props
|
||||
}: React.ComponentProps<typeof DialogPrimitive.Description>) {
|
||||
return (
|
||||
<DialogPrimitive.Description
|
||||
data-slot="dialog-description"
|
||||
className={cn('text-muted-foreground text-sm', className)}
|
||||
{...props}
|
||||
/>
|
||||
)
|
||||
}
|
||||
}: React.HTMLAttributes<HTMLDivElement>) => (
|
||||
<div
|
||||
className="flex flex-col-reverse sm:flex-row sm:justify-end sm:space-x-2"
|
||||
{...props}
|
||||
/>
|
||||
)
|
||||
DialogFooter.displayName = "DialogFooter"
|
||||
|
||||
const DialogTitle = React.forwardRef<
|
||||
React.ElementRef<typeof DialogPrimitive.Title>,
|
||||
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Title>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<DialogPrimitive.Title
|
||||
ref={ref}
|
||||
className={cn("text-lg font-semibold leading-none tracking-tight", className)}
|
||||
{...props}
|
||||
/>
|
||||
))
|
||||
DialogTitle.displayName = DialogPrimitive.Title.displayName
|
||||
|
||||
const DialogDescription = React.forwardRef<
|
||||
React.ElementRef<typeof DialogPrimitive.Description>,
|
||||
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Description>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<DialogPrimitive.Description
|
||||
ref={ref}
|
||||
className="text-sm text-muted-foreground"
|
||||
{...props}
|
||||
/>
|
||||
))
|
||||
DialogDescription.displayName = DialogPrimitive.Description.displayName
|
||||
|
||||
export {
|
||||
Dialog,
|
||||
DialogClose,
|
||||
DialogContent,
|
||||
DialogDescription,
|
||||
DialogFooter,
|
||||
DialogHeader,
|
||||
DialogOverlay,
|
||||
DialogPortal,
|
||||
DialogTitle,
|
||||
DialogOverlay,
|
||||
DialogClose,
|
||||
DialogTrigger,
|
||||
DialogContent,
|
||||
DialogHeader,
|
||||
DialogFooter,
|
||||
DialogTitle,
|
||||
DialogDescription,
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
/**
|
||||
* Example E2E test demonstrating the API fixture system
|
||||
*
|
||||
* This test shows how to use the fixture system to run tests against
|
||||
* mock data instead of a real backend.
|
||||
*
|
||||
* Run in recording mode to capture new fixtures:
|
||||
* RECORD_FIXTURES=true npm run test:e2e -- fixtures-example.spec.ts
|
||||
*
|
||||
* Then review and approve fixtures:
|
||||
* npm run fixtures:review
|
||||
*/
|
||||
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { setupApiFixtures, setupMocks, setAuthToken } from './fixtures';
|
||||
|
||||
test.describe('API Fixtures Example', () => {
|
||||
test('shows empty state when no jobs exist', async ({ page }) => {
|
||||
// Set up mock API responses (frontend uses /api/jobs, not /api/v1/jobs)
|
||||
await setupMocks(page, [
|
||||
{
|
||||
method: 'GET',
|
||||
path: '/api/jobs',
|
||||
response: { body: [] },
|
||||
},
|
||||
]);
|
||||
|
||||
// Set auth token before navigating
|
||||
await setAuthToken(page, 'test-token');
|
||||
|
||||
await page.goto('/');
|
||||
|
||||
// Verify the empty state message
|
||||
await expect(page.getByText(/No jobs yet/i)).toBeVisible();
|
||||
});
|
||||
|
||||
test('shows jobs when they exist', async ({ page }) => {
|
||||
// Set up mock with sample job data (frontend uses /api/jobs)
|
||||
await setupMocks(page, [
|
||||
{
|
||||
method: 'GET',
|
||||
path: '/api/jobs',
|
||||
response: {
|
||||
body: [
|
||||
{
|
||||
job_id: 'job-123',
|
||||
title: 'Test Song',
|
||||
artist: 'Test Artist',
|
||||
status: 'completed',
|
||||
created_at: '2024-01-15T10:00:00Z',
|
||||
updated_at: '2024-01-15T10:05:00Z',
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
]);
|
||||
|
||||
await setAuthToken(page, 'test-token');
|
||||
await page.goto('/');
|
||||
|
||||
// Verify job is displayed
|
||||
await expect(page.getByText('Test Song')).toBeVisible();
|
||||
await expect(page.getByText('Test Artist')).toBeVisible();
|
||||
});
|
||||
|
||||
test.skip('shows error when API fails', async ({ page }) => {
|
||||
// NOTE: Currently the frontend doesn't display API errors visibly
|
||||
// This test is skipped until error handling UI is added
|
||||
await setupMocks(page, [
|
||||
{
|
||||
method: 'GET',
|
||||
path: '/api/jobs',
|
||||
response: {
|
||||
status: 500,
|
||||
body: { detail: 'Internal server error' },
|
||||
},
|
||||
},
|
||||
]);
|
||||
|
||||
await setAuthToken(page, 'test-token');
|
||||
await page.goto('/');
|
||||
|
||||
// This would require the frontend to display API errors
|
||||
await expect(page.getByText(/error/i)).toBeVisible();
|
||||
});
|
||||
|
||||
test('shows login prompt when not authenticated', async ({ page }) => {
|
||||
// No mock needed - just don't set auth token
|
||||
await page.goto('/');
|
||||
|
||||
// Verify login prompt is shown (actual UI text)
|
||||
await expect(page.getByText(/Authentication Required/i)).toBeVisible();
|
||||
});
|
||||
|
||||
test.skip('records real API responses when RECORD_FIXTURES=true', async ({ page }) => {
|
||||
// This test only runs in recording mode
|
||||
// It will capture actual API responses for review
|
||||
|
||||
const api = await setupApiFixtures(page, {
|
||||
testFile: 'fixtures-example.spec.ts',
|
||||
});
|
||||
|
||||
if (!api.isRecording) {
|
||||
test.skip();
|
||||
return;
|
||||
}
|
||||
|
||||
await setAuthToken(page, process.env.TEST_ACCESS_TOKEN || 'your-token');
|
||||
await page.goto('/');
|
||||
|
||||
// Perform actions that trigger API calls
|
||||
await page.waitForLoadState('networkidle');
|
||||
|
||||
// Stop recording and show captured fixtures
|
||||
const fixtures = api.stopRecording();
|
||||
console.log(`Captured ${fixtures.length} API calls`);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,162 @@
|
||||
# API Fixtures for E2E Testing
|
||||
|
||||
This module provides tools for recording and replaying API fixtures to enable deterministic E2E testing without requiring a running backend.
|
||||
|
||||
## Overview
|
||||
|
||||
The fixture system has two modes:
|
||||
|
||||
1. **Recording Mode**: Intercepts real API calls during test runs and saves them for review
|
||||
2. **Mock Mode**: Replays approved fixtures instead of making real API calls
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Record API Responses
|
||||
|
||||
Run E2E tests in recording mode to capture real API responses:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm run test:e2e:record
|
||||
```
|
||||
|
||||
This will:
|
||||
- Run all E2E tests against the real backend
|
||||
- Intercept and save all API calls to `e2e/fixtures/data/recordings/`
|
||||
- Each session creates a new JSON file with all captured requests
|
||||
|
||||
### Step 2: Review and Approve Fixtures
|
||||
|
||||
Review each captured fixture before approving it:
|
||||
|
||||
```bash
|
||||
npm run fixtures:review
|
||||
```
|
||||
|
||||
The review CLI will:
|
||||
- Show each captured request/response pair
|
||||
- Let you approve, skip, or edit each fixture
|
||||
- Save approved fixtures to `e2e/fixtures/data/approved/`
|
||||
|
||||
For each fixture, you can:
|
||||
- **Approve (a)**: Save the fixture as-is
|
||||
- **Edit (e)**: Add a description or notes before approving
|
||||
- **Skip (s)**: Don't save this fixture
|
||||
- **Quit (q)**: Exit the review session
|
||||
|
||||
### Step 3: Use Fixtures in Tests
|
||||
|
||||
Once you have approved fixtures, update your tests to use them:
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { setupApiFixtures } from './fixtures';
|
||||
|
||||
test('shows job list', async ({ page }) => {
|
||||
// Set up mock server with approved fixtures
|
||||
const api = await setupApiFixtures(page, {
|
||||
useApprovedFixtures: true,
|
||||
});
|
||||
|
||||
await page.goto('/');
|
||||
|
||||
// Test runs against mock data - no real backend needed
|
||||
await expect(page.getByText('My Jobs')).toBeVisible();
|
||||
|
||||
// Verify all requests were handled
|
||||
api.assertAllHandled();
|
||||
});
|
||||
```
|
||||
|
||||
## Test Helper API
|
||||
|
||||
### setupApiFixtures(page, options)
|
||||
|
||||
Main function for setting up API fixtures in a test.
|
||||
|
||||
**Options:**
|
||||
- `fixtureSet?: string` - Load a named fixture set from `data/sets/`
|
||||
- `useApprovedFixtures?: boolean` - Load all approved fixtures from `data/approved/`
|
||||
- `mocks?: Array<{method, path, response}>` - Add inline mock responses
|
||||
- `testFile?: string` - Test file name (used in recording mode)
|
||||
|
||||
**Returns:** `ApiTestContext` with:
|
||||
- `isRecording: boolean` - Whether in recording mode
|
||||
- `assertAllHandled()` - Throws if any requests weren't mocked
|
||||
- `getUnmatchedRequests()` - Returns list of unhandled requests
|
||||
- `stopRecording()` - Stop recording and return captured fixtures
|
||||
|
||||
### Convenience Functions
|
||||
|
||||
```typescript
|
||||
// Quick inline mocks
|
||||
await setupMocks(page, [
|
||||
{ method: 'GET', path: '/api/v1/jobs', response: { body: [] } },
|
||||
]);
|
||||
|
||||
// Pre-configured scenarios
|
||||
await setupAuthenticatedUser(page);
|
||||
await setupUnauthenticatedUser(page);
|
||||
|
||||
// Auth helpers
|
||||
await setAuthToken(page, 'my-token');
|
||||
await clearAuthToken(page);
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
e2e/fixtures/
|
||||
├── data/
|
||||
│ ├── recordings/ # Temporary recording sessions (gitignored content)
|
||||
│ ├── approved/ # Approved fixtures (committed to repo)
|
||||
│ └── sets/ # Named fixture sets for specific scenarios
|
||||
├── types.ts # TypeScript interfaces
|
||||
├── recorder.ts # Recording logic
|
||||
├── mock-server.ts # Mock server logic
|
||||
├── test-helper.ts # Test utilities
|
||||
├── review-cli.ts # CLI for reviewing fixtures
|
||||
└── index.ts # Module exports
|
||||
```
|
||||
|
||||
## Fixture Format
|
||||
|
||||
Each approved fixture is a JSON file:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "get-api-v1-jobs",
|
||||
"description": "GET /api/v1/jobs - List all jobs",
|
||||
"capturedAt": "2024-01-15T10:30:00.000Z",
|
||||
"request": {
|
||||
"method": "GET",
|
||||
"url": "http://localhost:8000/api/v1/jobs",
|
||||
"path": "/api/v1/jobs",
|
||||
"headers": { "authorization": "[REDACTED]" }
|
||||
},
|
||||
"response": {
|
||||
"status": 200,
|
||||
"statusText": "OK",
|
||||
"body": []
|
||||
},
|
||||
"reviewed": true,
|
||||
"reviewNotes": "Empty job list for new user"
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Review every fixture**: Don't blindly approve all recordings. Check that responses are correct and don't contain sensitive data.
|
||||
|
||||
2. **Add descriptions**: Edit fixtures to add meaningful descriptions that explain what scenario the fixture represents.
|
||||
|
||||
3. **Keep fixtures minimal**: Only approve the fixtures you actually need for tests. Don't capture unnecessary API calls.
|
||||
|
||||
4. **Update fixtures when API changes**: If the backend API changes, re-record and review fixtures to ensure tests use correct data.
|
||||
|
||||
5. **Use fixture sets for scenarios**: Group related fixtures into sets for complex test scenarios (e.g., "user-with-jobs", "empty-state").
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `RECORD_FIXTURES=true` - Enable recording mode
|
||||
- `NEXT_PUBLIC_API_URL` - Backend API URL (default: http://localhost:8000)
|
||||
@@ -0,0 +1,3 @@
|
||||
# This directory contains approved API fixtures
|
||||
# Each fixture is a JSON file representing a single API request/response pair
|
||||
# These fixtures are used by the mock server during E2E tests
|
||||
@@ -0,0 +1,4 @@
|
||||
# Ignore recording session files (temporary, should be reviewed and approved)
|
||||
*.json
|
||||
# But keep the gitkeep
|
||||
!.gitkeep
|
||||
Loaded 100 of 207 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user