mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-21 05:18:08 +09:00
Compare commits
76
Commits
No files matched your search
@@ -0,0 +1,74 @@
|
||||
---
|
||||
description: Periodic documentation maintenance and organization check
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write, Bash
|
||||
---
|
||||
|
||||
# Documentation Maintenance Check
|
||||
|
||||
You are performing periodic maintenance on the project documentation.
|
||||
|
||||
## Checks to Perform
|
||||
|
||||
### 1. Structure Compliance
|
||||
Verify docs/ follows the correct structure:
|
||||
```
|
||||
docs/
|
||||
├── README.md # Current status + navigation
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Accumulated wisdom
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
Report any files that don't belong or are misplaced.
|
||||
|
||||
### 2. Freshness Check
|
||||
- Read `docs/README.md` - is the status section current?
|
||||
- Check recent git commits - do they suggest docs should be updated?
|
||||
- Look for stale information that contradicts current code
|
||||
|
||||
### 3. Archive Organization
|
||||
- Verify all archive files have YYYY-MM-DD prefix
|
||||
- Check for any docs that should be archived (completed work, old plans)
|
||||
- Remove truly obsolete content (empty files, duplicate info)
|
||||
|
||||
### 4. Cross-Reference Check
|
||||
- Verify CLAUDE.md points to current doc locations
|
||||
- Check that docs reference each other correctly
|
||||
- Ensure no broken internal links
|
||||
|
||||
### 5. Content Quality
|
||||
- Flag any docs over 500 lines (may need splitting)
|
||||
- Identify duplicate information across docs
|
||||
- Note sections that seem outdated
|
||||
|
||||
## Actions to Take
|
||||
|
||||
1. **Fix structural issues** - Move/rename misplaced files
|
||||
2. **Update stale content** - Refresh outdated information
|
||||
3. **Archive completed work** - Move old docs to archive/
|
||||
4. **Report findings** - Summarize what was found and fixed
|
||||
|
||||
## Output Format
|
||||
|
||||
Provide a maintenance report:
|
||||
```
|
||||
## Documentation Maintenance Report - YYYY-MM-DD
|
||||
|
||||
### Issues Found
|
||||
- [list of issues]
|
||||
|
||||
### Actions Taken
|
||||
- [list of fixes made]
|
||||
|
||||
### Recommendations
|
||||
- [suggestions for future attention]
|
||||
|
||||
### Status
|
||||
- [ ] Structure: OK/Fixed
|
||||
- [ ] Freshness: OK/Updated
|
||||
- [ ] Archive: OK/Organized
|
||||
- [ ] Cross-refs: OK/Fixed
|
||||
- [ ] Quality: OK/Flagged
|
||||
```
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
description: Review current session for documentation updates before merging PR
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write
|
||||
---
|
||||
|
||||
# Pre-Merge Documentation Review
|
||||
|
||||
You are reviewing this chat session to identify documentation updates needed before merging the current PR.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. **Analyze the current session** - What work was done? What was learned? What changed?
|
||||
|
||||
2. **Check if docs need updates:**
|
||||
- Does `docs/README.md` status section need updating?
|
||||
- Should anything be added to `docs/LESSONS-LEARNED.md`?
|
||||
- Does `docs/ARCHITECTURE.md` need changes (if architecture changed)?
|
||||
- Does `docs/DEVELOPMENT.md` need changes (if dev workflow changed)?
|
||||
- Does `docs/API.md` need changes (if API changed)?
|
||||
|
||||
3. **For significant completed work:**
|
||||
- Create `docs/archive/YYYY-MM-DD-topic.md` summarizing the work
|
||||
- Use today's date and a descriptive topic name
|
||||
|
||||
4. **Update docs/README.md** with:
|
||||
- Current status if it changed
|
||||
- Any new known issues discovered
|
||||
- Link to new archive doc if created
|
||||
|
||||
## Guidelines
|
||||
|
||||
- Keep updates concise and factual
|
||||
- Focus on information useful for future AI agents
|
||||
- Don't duplicate information that's in the code
|
||||
- Archive docs should capture "what was done and why" not implementation details
|
||||
|
||||
## Output
|
||||
|
||||
After reviewing, either:
|
||||
- Make the necessary documentation updates
|
||||
- Report "No documentation updates needed" with brief explanation why
|
||||
|
||||
Remember: Good documentation helps the next agent continue where you left off.
|
||||
@@ -0,0 +1,83 @@
|
||||
# .dockerignore - Reduce Docker build context size
|
||||
# This is critical for fast builds as it reduces upload time and cache invalidation
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
.gitmodules
|
||||
|
||||
# Node.js / Frontend (not needed for backend image)
|
||||
node_modules/
|
||||
frontend/
|
||||
frontend-react/
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Python virtual environments (use Docker's Python instead)
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
backend/venv/
|
||||
ENV/
|
||||
env/
|
||||
*.egg-info/
|
||||
.eggs/
|
||||
dist/
|
||||
build/
|
||||
*.egg
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
|
||||
# Testing (not needed in production image)
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
coverage.xml
|
||||
*.cover
|
||||
tests/
|
||||
backend/tests/
|
||||
|
||||
# Documentation (not needed in Docker image)
|
||||
docs/
|
||||
*.md
|
||||
!README.md
|
||||
|
||||
# Infrastructure (not needed for Docker build)
|
||||
infrastructure/
|
||||
|
||||
# CI/CD configs
|
||||
.github/
|
||||
|
||||
# Cloud Build configs (keep for reference but not in image)
|
||||
cloudbuild*.yaml
|
||||
|
||||
# Local development files
|
||||
lyrics_transcriber_local/
|
||||
*.local
|
||||
.env
|
||||
.env.*
|
||||
*.bak
|
||||
*.tmp
|
||||
Thumbs.db
|
||||
|
||||
# Large test/sample files
|
||||
tests/data/
|
||||
*.mp3
|
||||
*.wav
|
||||
*.mp4
|
||||
*.mkv
|
||||
*.flac
|
||||
+355
-141
@@ -17,12 +17,120 @@ concurrency:
|
||||
|
||||
jobs:
|
||||
# ============================================
|
||||
# PACKAGE TESTS (karaoke-gen Python CLI)
|
||||
# PATH FILTERING - Determine what changed
|
||||
# Only needed for PRs (main branch skips tests)
|
||||
# ============================================
|
||||
|
||||
changes:
|
||||
name: "Detect Changes"
|
||||
runs-on: ubuntu-latest
|
||||
if: github.ref != 'refs/heads/main'
|
||||
outputs:
|
||||
frontend: ${{ steps.filter.outputs.frontend }}
|
||||
backend: ${{ steps.filter.outputs.backend }}
|
||||
package: ${{ steps.filter.outputs.package }}
|
||||
infra: ${{ steps.filter.outputs.infra }}
|
||||
deps: ${{ steps.filter.outputs.deps }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: dorny/paths-filter@v3
|
||||
id: filter
|
||||
with:
|
||||
filters: |
|
||||
frontend:
|
||||
- 'frontend/**'
|
||||
backend:
|
||||
- 'backend/**'
|
||||
package:
|
||||
- 'karaoke_gen/**'
|
||||
- 'lyrics_transcriber_temp/**'
|
||||
- 'tests/**'
|
||||
infra:
|
||||
- 'infrastructure/**'
|
||||
deps:
|
||||
- 'poetry.lock'
|
||||
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "### Change Detection Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Area | Changed |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|------|---------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Frontend | ${{ steps.filter.outputs.frontend }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Backend | ${{ steps.filter.outputs.backend }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Package | ${{ steps.filter.outputs.package }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Infrastructure | ${{ steps.filter.outputs.infra }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Dependencies | ${{ steps.filter.outputs.deps }} |" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
# ============================================
|
||||
# FRONTEND TESTS (PRs only - main skips tests)
|
||||
# ============================================
|
||||
|
||||
frontend-unit-tests:
|
||||
name: "Frontend - Unit Tests"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js with caching
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Run unit tests with coverage
|
||||
working-directory: frontend
|
||||
run: npm run test:ci
|
||||
|
||||
- name: Upload coverage report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: frontend-coverage
|
||||
path: frontend/coverage/
|
||||
retention-days: 7
|
||||
|
||||
frontend-build:
|
||||
name: "Frontend - Build"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js with caching
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Build frontend
|
||||
working-directory: frontend
|
||||
env:
|
||||
NEXT_PUBLIC_API_URL: https://api.nomadkaraoke.com
|
||||
run: npm run build
|
||||
|
||||
# ============================================
|
||||
# PACKAGE TESTS (PRs only - main skips tests)
|
||||
# ============================================
|
||||
|
||||
package-unit-tests:
|
||||
name: "Package - Unit Tests (70% coverage)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -39,6 +147,14 @@ jobs:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Cache Poetry dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .venv
|
||||
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
|
||||
restore-keys: |
|
||||
venv-package-${{ runner.os }}-py3.13-
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install --with dev
|
||||
|
||||
@@ -47,6 +163,12 @@ jobs:
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ffmpeg
|
||||
|
||||
- name: Cache spaCy model
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.cache/spacy
|
||||
key: spacy-en_core_web_sm-${{ runner.os }}
|
||||
|
||||
- name: Install spaCy language models
|
||||
run: poetry run python -m spacy download en_core_web_sm
|
||||
|
||||
@@ -65,6 +187,8 @@ jobs:
|
||||
package-integration-tests:
|
||||
name: "Package - Integration Tests"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -81,6 +205,14 @@ jobs:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Cache Poetry dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .venv
|
||||
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
|
||||
restore-keys: |
|
||||
venv-package-${{ runner.os }}-py3.13-
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install --with dev
|
||||
|
||||
@@ -89,16 +221,27 @@ jobs:
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ffmpeg
|
||||
|
||||
- name: Cache spaCy model
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.cache/spacy
|
||||
key: spacy-en_core_web_sm-${{ runner.os }}
|
||||
|
||||
- name: Install spaCy language models
|
||||
run: poetry run python -m spacy download en_core_web_sm
|
||||
|
||||
- name: Run integration tests
|
||||
run: poetry run pytest tests/integration/ -v
|
||||
|
||||
package-build:
|
||||
name: "Package - Build & Install"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [package-unit-tests, package-integration-tests]
|
||||
package-build-test:
|
||||
name: "Package - Build & Install (Test)"
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
needs: [changes, package-unit-tests, package-integration-tests]
|
||||
if: >-
|
||||
github.ref != 'refs/heads/main' && always() &&
|
||||
(needs.package-unit-tests.result == 'success' || needs.package-unit-tests.result == 'skipped') &&
|
||||
(needs.package-integration-tests.result == 'success' || needs.package-integration-tests.result == 'skipped') &&
|
||||
(needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -118,36 +261,20 @@ jobs:
|
||||
- name: Build package
|
||||
run: poetry build
|
||||
|
||||
- name: Free disk space
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: false
|
||||
android: true
|
||||
haskell: true
|
||||
dotnet: false
|
||||
large-packages: false
|
||||
docker-images: false
|
||||
swap-storage: false
|
||||
|
||||
- name: Test package installation
|
||||
run: |
|
||||
pip install dist/*.whl
|
||||
python -m karaoke_gen.utils.gen_cli --help
|
||||
|
||||
- name: Upload build artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
retention-days: 1
|
||||
|
||||
# ============================================
|
||||
# BACKEND TESTS (Cloud Run API service)
|
||||
# BACKEND TESTS (PRs only - main skips tests)
|
||||
# ============================================
|
||||
|
||||
backend-unit-tests:
|
||||
name: "Backend - Unit Tests"
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
@@ -182,15 +309,10 @@ jobs:
|
||||
|
||||
backend-emulator-tests:
|
||||
name: "Backend - Emulator Integration Tests"
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Free disk space
|
||||
run: |
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo rm -rf /opt/ghc
|
||||
sudo rm -rf /usr/local/share/boost
|
||||
df -h
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -213,14 +335,6 @@ jobs:
|
||||
distribution: "temurin"
|
||||
java-version: "21"
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install Firestore emulator
|
||||
run: gcloud components install beta cloud-firestore-emulator --quiet
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install
|
||||
|
||||
@@ -243,6 +357,8 @@ jobs:
|
||||
backend-lint:
|
||||
name: "Backend - Code Quality"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true')
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
@@ -260,6 +376,14 @@ jobs:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Cache Poetry dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .venv
|
||||
key: venv-lint-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
|
||||
restore-keys: |
|
||||
venv-lint-${{ runner.os }}-py3.13-
|
||||
|
||||
- name: Install dependencies
|
||||
run: poetry install
|
||||
|
||||
@@ -269,9 +393,78 @@ jobs:
|
||||
echo "✅ Basic syntax validation passed"
|
||||
|
||||
# ============================================
|
||||
# DEPLOYMENTS (main branch only)
|
||||
# CI GATE - Required status check for PRs
|
||||
# Only runs on PRs, not on main
|
||||
# ============================================
|
||||
|
||||
ci-gate:
|
||||
name: "CI Gate"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [frontend-unit-tests, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
if: github.ref != 'refs/heads/main' && always()
|
||||
steps:
|
||||
- name: Check results
|
||||
run: |
|
||||
results=(
|
||||
"${{ needs.frontend-unit-tests.result }}"
|
||||
"${{ needs.frontend-build.result }}"
|
||||
"${{ needs.package-unit-tests.result }}"
|
||||
"${{ needs.package-integration-tests.result }}"
|
||||
"${{ needs.backend-unit-tests.result }}"
|
||||
"${{ needs.backend-emulator-tests.result }}"
|
||||
"${{ needs.backend-lint.result }}"
|
||||
)
|
||||
for result in "${results[@]}"; do
|
||||
if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
|
||||
echo "❌ CI failed"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "✅ CI passed"
|
||||
|
||||
# ============================================
|
||||
# DEPLOYMENTS (main branch only)
|
||||
# Tests already passed on PR, so we skip them here
|
||||
# ============================================
|
||||
|
||||
package-build:
|
||||
name: "Package - Build"
|
||||
runs-on: ubuntu-latest
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
with:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Cache Poetry dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .venv
|
||||
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
|
||||
restore-keys: |
|
||||
venv-package-${{ runner.os }}-py3.13-
|
||||
|
||||
- name: Build package
|
||||
run: poetry build
|
||||
|
||||
- name: Upload build artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
retention-days: 1
|
||||
|
||||
package-publish:
|
||||
name: "Deploy - Publish to PyPI"
|
||||
runs-on: ubuntu-latest
|
||||
@@ -363,17 +556,16 @@ jobs:
|
||||
name: "Deploy - Frontend (GitHub Pages)"
|
||||
runs-on: ubuntu-latest
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
- name: Setup Node.js with caching
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Enable Yarn (for lyrics frontend)
|
||||
run: corepack enable
|
||||
@@ -390,6 +582,14 @@ jobs:
|
||||
run: npm run build
|
||||
|
||||
# ===== Build Lyrics Review Frontend (Vite) =====
|
||||
- name: Cache Yarn dependencies (lyrics frontend)
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: lyrics_transcriber_temp/lyrics_transcriber/frontend/node_modules
|
||||
key: yarn-lyrics-${{ runner.os }}-${{ hashFiles('lyrics_transcriber_temp/lyrics_transcriber/frontend/yarn.lock') }}
|
||||
restore-keys: |
|
||||
yarn-lyrics-${{ runner.os }}-
|
||||
|
||||
- name: Install lyrics frontend dependencies
|
||||
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
|
||||
run: yarn install --frozen-lockfile
|
||||
@@ -426,20 +626,28 @@ jobs:
|
||||
|
||||
deploy-backend:
|
||||
name: "Deploy - Backend (Cloud Run)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
REPOSITORY: karaoke-repo
|
||||
BASE_IMAGE_NAME: karaoke-backend-base
|
||||
APP_IMAGE_NAME: karaoke-backend
|
||||
steps:
|
||||
# ============================================
|
||||
# Check if deployment is needed FIRST
|
||||
# ============================================
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: false
|
||||
fetch-depth: 0 # Need full history for tag check
|
||||
|
||||
- name: Set up Python (for version extraction)
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
@@ -453,7 +661,7 @@ jobs:
|
||||
VERSION=$(poetry version --short)
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
echo "📋 Current version: $VERSION"
|
||||
|
||||
|
||||
if git ls-remote --tags origin | grep -q "refs/tags/v$VERSION$"; then
|
||||
echo "needs_deploy=false" >> $GITHUB_OUTPUT
|
||||
echo "⏭️ Tag v$VERSION already exists - skipping backend deployment"
|
||||
@@ -463,6 +671,8 @@ jobs:
|
||||
echo "✨ Tag v$VERSION does not exist - proceeding with deployment"
|
||||
fi
|
||||
|
||||
# No disk cleanup needed - self-hosted runner has 200GB SSD
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: auth
|
||||
@@ -477,21 +687,37 @@ jobs:
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Check if base image needs rebuild
|
||||
- name: Configure Docker for Artifact Registry
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
# ============================================
|
||||
# Docker Build with BuildKit + Registry Cache
|
||||
# This replaces expensive Cloud Build with free GitHub Actions
|
||||
# Registry cache persists between builds for fast rebuilds
|
||||
# ============================================
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Calculate base image hash
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: base-hash
|
||||
run: |
|
||||
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
|
||||
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
|
||||
echo "hash=$CURRENT_HASH" >> $GITHUB_OUTPUT
|
||||
echo "📋 Base content hash: $CURRENT_HASH"
|
||||
|
||||
- name: Check if base image exists and is current
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: check-base
|
||||
run: |
|
||||
echo "🔍 Checking if base image needs rebuild..."
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
|
||||
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
|
||||
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
|
||||
echo "Current combined hash (poetry.lock + Dockerfile.base): $CURRENT_HASH"
|
||||
|
||||
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
|
||||
|
||||
BASE_IMAGE="us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest"
|
||||
# Try to get the existing base image's hash label
|
||||
EXISTING_HASH=""
|
||||
|
||||
if docker pull $BASE_IMAGE 2>/dev/null; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
@@ -500,10 +726,10 @@ jobs:
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
else
|
||||
echo "⚠️ No existing base image found"
|
||||
echo "⚠️ No existing base image found - will build"
|
||||
fi
|
||||
|
||||
if [ "$CURRENT_HASH" != "$EXISTING_HASH" ]; then
|
||||
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
|
||||
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
|
||||
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
@@ -511,88 +737,73 @@ jobs:
|
||||
echo "needs_rebuild=false" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Rebuild base image (if needed)
|
||||
- name: Build and push base image (if needed)
|
||||
if: steps.version_check.outputs.needs_deploy == 'true' && steps.check-base.outputs.needs_rebuild == 'true'
|
||||
run: |
|
||||
echo "🏗️ Rebuilding base image with updated dependencies..."
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: backend/Dockerfile.base
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:${{ github.sha }}
|
||||
# Registry cache for ALL layers (mode=max) - persists between builds
|
||||
# ignore-error=true allows build to continue if cache doesn't exist yet
|
||||
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,ignore-error=true
|
||||
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,mode=max,ignore-error=true
|
||||
build-args: |
|
||||
BUILD_DATE=${{ github.run_id }}
|
||||
BUILD_ID=${{ github.sha }}
|
||||
BASE_CONTENT_HASH=${{ steps.base-hash.outputs.hash }}
|
||||
# Provenance disabled for smaller image and faster push
|
||||
provenance: false
|
||||
|
||||
BASE_BUILD_ID=$(gcloud builds submit \
|
||||
--config=cloudbuild-base.yaml \
|
||||
--project=nomadkaraoke \
|
||||
--format='value(id)' \
|
||||
--async \
|
||||
.)
|
||||
- name: Build and push app image
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: backend/Dockerfile
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:latest
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:v${{ steps.version_check.outputs.version }}
|
||||
# Registry cache for app layers
|
||||
# ignore-error=true allows build to continue if cache doesn't exist yet
|
||||
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,ignore-error=true
|
||||
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,mode=max,ignore-error=true
|
||||
provenance: false
|
||||
|
||||
echo "Base Build ID: $BASE_BUILD_ID"
|
||||
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BASE_BUILD_ID?project=nomadkaraoke"
|
||||
# ============================================
|
||||
# Deploy to Cloud Run
|
||||
# ============================================
|
||||
|
||||
echo "Waiting for base image build to complete..."
|
||||
while true; do
|
||||
BUILD_STATUS=$(gcloud builds describe $BASE_BUILD_ID --project=nomadkaraoke --format='value(status)')
|
||||
echo "Base build status: $BUILD_STATUS"
|
||||
|
||||
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
|
||||
echo "✅ Base image build completed successfully"
|
||||
break
|
||||
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
|
||||
echo "❌ Base image build failed with status: $BUILD_STATUS"
|
||||
echo ""
|
||||
echo "==================== BUILD LOGS ===================="
|
||||
gcloud logging read "resource.type=build AND resource.labels.build_id=$BASE_BUILD_ID" \
|
||||
--project=nomadkaraoke \
|
||||
--format="value(textPayload)" \
|
||||
--limit=200 \
|
||||
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
|
||||
echo "===================================================="
|
||||
gcloud builds log $BASE_BUILD_ID --project=nomadkaraoke 2>&1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 30
|
||||
done
|
||||
|
||||
- name: Trigger Cloud Build
|
||||
- name: Deploy to Cloud Run
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: |
|
||||
echo "🚀 Triggering Cloud Build for deployment..."
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Version: ${{ steps.version_check.outputs.version }}"
|
||||
VERSION="${{ steps.version_check.outputs.version }}"
|
||||
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
|
||||
|
||||
BUILD_ID=$(gcloud builds submit \
|
||||
--config=cloudbuild.yaml \
|
||||
--project=nomadkaraoke \
|
||||
--format='value(id)' \
|
||||
--async \
|
||||
.)
|
||||
echo "🚀 Deploying to Cloud Run..."
|
||||
echo "Image: $IMAGE"
|
||||
echo "Version: $VERSION"
|
||||
|
||||
echo "Build ID: $BUILD_ID"
|
||||
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=nomadkaraoke"
|
||||
|
||||
echo "Waiting for build to complete..."
|
||||
while true; do
|
||||
BUILD_STATUS=$(gcloud builds describe $BUILD_ID --project=nomadkaraoke --format='value(status)')
|
||||
echo "Build status: $BUILD_STATUS"
|
||||
|
||||
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
|
||||
echo "✅ Build completed successfully"
|
||||
break
|
||||
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
|
||||
echo "❌ Build failed with status: $BUILD_STATUS"
|
||||
echo ""
|
||||
echo "==================== BUILD LOGS ===================="
|
||||
gcloud logging read "resource.type=build AND resource.labels.build_id=$BUILD_ID" \
|
||||
--project=nomadkaraoke \
|
||||
--format="value(textPayload)" \
|
||||
--limit=200 \
|
||||
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
|
||||
echo "===================================================="
|
||||
echo ""
|
||||
gcloud builds log $BUILD_ID --project=nomadkaraoke 2>&1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 30
|
||||
done
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image $IMAGE \
|
||||
--region us-central1 \
|
||||
--platform managed \
|
||||
--allow-unauthenticated \
|
||||
--memory 16Gi \
|
||||
--cpu 8 \
|
||||
--cpu-boost \
|
||||
--execution-environment gen2 \
|
||||
--timeout 1800 \
|
||||
--concurrency 1 \
|
||||
--max-instances 50 \
|
||||
--min-instances 0 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
|
||||
|
||||
- name: Verify deployment and version
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
@@ -604,22 +815,22 @@ jobs:
|
||||
|
||||
echo "Service URL: $SERVICE_URL"
|
||||
echo "Expected version: $EXPECTED_VERSION"
|
||||
|
||||
|
||||
# Wait for new revision to be ready
|
||||
echo "Waiting for new revision to be ready..."
|
||||
sleep 15
|
||||
|
||||
|
||||
# Basic health check
|
||||
curl -f "$SERVICE_URL/api/health" || exit 1
|
||||
echo "✅ Service is healthy"
|
||||
|
||||
|
||||
# Version verification with retries (Cloud Run may take time to route to new revision)
|
||||
MAX_RETRIES=6
|
||||
RETRY_DELAY=10
|
||||
for i in $(seq 1 $MAX_RETRIES); do
|
||||
echo "Checking deployed version (attempt $i/$MAX_RETRIES)..."
|
||||
DEPLOYED_VERSION=$(curl -sf "$SERVICE_URL/api/health/detailed" | jq -r '.version // empty')
|
||||
|
||||
|
||||
if [ "$DEPLOYED_VERSION" = "$EXPECTED_VERSION" ]; then
|
||||
echo "✅ Version verified: $DEPLOYED_VERSION"
|
||||
break
|
||||
@@ -643,7 +854,10 @@ jobs:
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Branch: ${{ github.ref_name }}"
|
||||
echo "Run: ${{ github.run_number }}"
|
||||
|
||||
echo ""
|
||||
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
|
||||
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
|
||||
|
||||
- name: Skipped deployment summary
|
||||
if: steps.version_check.outputs.needs_deploy == 'false'
|
||||
run: |
|
||||
|
||||
@@ -12,6 +12,9 @@ tests/outputs
|
||||
/input
|
||||
/output/
|
||||
|
||||
# Cloud Infra
|
||||
.emulator-logs
|
||||
|
||||
# Next.js / React Frontend (./frontend)
|
||||
node_modules/
|
||||
frontend/node_modules/
|
||||
@@ -224,3 +227,6 @@ cython_debug/
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
flacfetch
|
||||
|
||||
# Test output directories
|
||||
Rick Astley*/
|
||||
@@ -1,131 +1,97 @@
|
||||
# Project Guidelines for AI Assistants
|
||||
# Karaoke-Gen AI Assistant Guidelines
|
||||
|
||||
This document contains guidelines and rules for AI assistants working on this codebase.
|
||||
## Project Overview
|
||||
|
||||
## Core Principles
|
||||
**Karaoke video generation platform** - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
|
||||
|
||||
**Always be testing.** Proactively evaluate, read, run, fix, and improve tests whenever you touch any part of the system—frontend, backend, infrastructure. Every code change should be thoroughly tested. Leave everything better tested than you found it.
|
||||
- **Production**: <https://gen.nomadkaraoke.com> (frontend), <https://api.nomadkaraoke.com> (backend)
|
||||
- **CLI**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
- **Repo**: <https://github.com/nomadkaraoke/karaoke-gen>
|
||||
|
||||
**Always be documenting.** Continuously read, write, improve, update, and organize markdown documentation. Reference existing docs and keep them accurate as the codebase evolves.
|
||||
## Quick Reference
|
||||
|
||||
**Build for maintainability.** Keep code SOLID, modular, and clean. Favor simplicity and clarity. Future developers (and AI agents) will thank you.
|
||||
| What | Where |
|
||||
|------|-------|
|
||||
| Current status | `docs/README.md` |
|
||||
| Architecture | `docs/ARCHITECTURE.md` |
|
||||
| Dev setup & testing | `docs/DEVELOPMENT.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
|
||||
## Branching Workflow
|
||||
## Tech Stack
|
||||
|
||||
**Never commit directly to `main`.** Always work on a feature branch.
|
||||
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
|
||||
- **Frontend**: Next.js on Cloudflare Pages
|
||||
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
|
||||
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
|
||||
|
||||
Create descriptive branch names like `feature/add-discord-notifications`, `fix/audio-sync-issue`, or `refactor/cleanup-api-routes`.
|
||||
## Essential Rules
|
||||
|
||||
### Using Git Worktrees
|
||||
|
||||
**For non-trivial changes** (new features, multi-file edits, refactors):
|
||||
- Create a git worktree in the parent folder before making any file changes:
|
||||
### Git Workflow
|
||||
- **Never commit directly to `main`** - always use feature branches
|
||||
- **Use git worktrees** for all changes:
|
||||
```bash
|
||||
git worktree add ../karaoke-gen-feat-xyz feature/xyz
|
||||
git worktree add -b feature/xyz ../karaoke-gen-feat-xyz main
|
||||
```
|
||||
- This keeps the main directory clean on `main` and enables parallel work on multiple features
|
||||
- Name the worktree folder descriptively, e.g., `../karaoke-gen-fix-audio-sync`
|
||||
|
||||
**For trivial changes** (typos, single-line fixes):
|
||||
- Creating a branch in the current directory is acceptable
|
||||
- Use `git checkout -b fix/typo-xyz` before making changes
|
||||
|
||||
**Cleanup:** When done with a worktree, remove it with:
|
||||
```bash
|
||||
git worktree remove ../karaoke-gen-feat-xyz
|
||||
```
|
||||
|
||||
### Creating a Pull Request
|
||||
|
||||
Once all changes are complete, tests pass, and commits are pushed, create a PR with:
|
||||
|
||||
1. **Summary** - Brief overview of the changes
|
||||
2. **Changes Made** - List of specific modifications
|
||||
3. **Testing** - How changes were tested (unit tests, manual testing)
|
||||
4. **Related Issues** - Reference any GitHub issues (e.g., "Fixes #123")
|
||||
|
||||
Only merge after review and CI checks pass.
|
||||
|
||||
## Infrastructure Changes
|
||||
|
||||
All Google Cloud infrastructure (IAM, Cloud Storage, Cloud Run, Secret Manager, Artifact Registry, etc.) must be managed via **Pulumi IaC** in the `infrastructure/` folder.
|
||||
|
||||
### Rules
|
||||
|
||||
- **Reading/querying** with `gcloud` CLI is fine for investigation and debugging
|
||||
- **Temporary changes** via `gcloud` CLI or Console are acceptable for urgent fixes or experimentation, but **must be cleaned up** and properly codified in the IaC afterward
|
||||
- **Permanent changes** must go through Pulumi — never leave `gcloud` or Console changes as the final state
|
||||
|
||||
### Workflow
|
||||
|
||||
1. Make changes in `infrastructure/`
|
||||
2. Run `pulumi preview` and verify the output
|
||||
3. Run `pulumi up` to apply
|
||||
|
||||
### Authentication Issues
|
||||
|
||||
If you encounter any authentication issues with `gcloud` or `pulumi` (e.g., expired tokens, permission errors), **stop and notify the user** so they can run the appropriate auth commands in their external terminal:
|
||||
|
||||
```bash
|
||||
gcloud auth login
|
||||
gcloud auth application-default login
|
||||
```
|
||||
|
||||
Do not attempt to work around auth issues or make assumptions about credentials.
|
||||
|
||||
## Version Management
|
||||
|
||||
When committing and pushing changes, always bump the patch version in `pyproject.toml`.
|
||||
|
||||
The version is located at `tool.poetry.version` in `pyproject.toml`. For example, if the current version is `0.70.3`, bump it to `0.70.4`.
|
||||
|
||||
This ensures every commit pushed to the repository has a unique version number for tracking and deployment purposes.
|
||||
|
||||
## Testing Requirements
|
||||
|
||||
**Before committing and pushing ANY changes, you MUST run the full test suite.**
|
||||
|
||||
### Backend Test Command
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
### Testing (Required Before Commit)
|
||||
```bash
|
||||
# Backend (10 min timeout, limit output)
|
||||
make test 2>&1 | tail -n 500
|
||||
```
|
||||
|
||||
**Important for AI agents:**
|
||||
- Set a **10 minute timeout** when running tests (they can take a while with emulators)
|
||||
- Always pipe output through `tail -n 500` to limit output and preserve context window
|
||||
- The tail output will show test results and any failures at the end
|
||||
|
||||
This runs all backend tests in order and fails fast if any test fails:
|
||||
1. Unit tests (karaoke_gen package) with coverage check
|
||||
2. Backend unit tests
|
||||
3. E2E integration tests with emulators (auto-starts/stops emulators)
|
||||
|
||||
### Frontend Test Command
|
||||
|
||||
When working in the `frontend/` directory, run:
|
||||
|
||||
```bash
|
||||
# Frontend
|
||||
cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
```
|
||||
|
||||
This runs:
|
||||
1. **Unit tests** (Jest) - Component and utility tests
|
||||
2. **E2E tests** (Playwright) - Browser-based integration tests
|
||||
### Version Bumping
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
|
||||
- Skip for docs-only changes
|
||||
|
||||
**Note:** E2E tests require `KARAOKE_ACCESS_TOKEN` environment variable to be set in `frontend/.env.local`.
|
||||
### Infrastructure
|
||||
- All GCP changes via Pulumi in `infrastructure/`
|
||||
- `gcloud` CLI for reading/debugging only
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
Individual test commands:
|
||||
- `npm run test:unit` - Run unit tests only
|
||||
- `npm run test:e2e` - Run E2E tests only
|
||||
- `npm run test:ci` - Run unit tests with coverage
|
||||
## Documentation Maintenance
|
||||
|
||||
### Pre-commit Checklist
|
||||
### Structure
|
||||
```text
|
||||
docs/
|
||||
├── README.md # Current status + navigation (UPDATE THIS)
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Key insights for future agents
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
- [ ] `make test` passes (for backend changes)
|
||||
- [ ] `cd frontend && npm run test:all` passes (for frontend changes)
|
||||
- [ ] Version in `pyproject.toml` has been bumped
|
||||
- [ ] `coderabbit review --plain` has been run to get comprehensive analysis and suggestions for cleaner, more maintainable code. Apply the feedback to improve accessibility, structure, and best practices.
|
||||
### Rules
|
||||
1. **Before merging PRs**: Run `/docs-review` to check if docs need updating
|
||||
2. **Periodically**: Run `/docs-maintain` to verify docs are organized
|
||||
3. **For significant work**: Create `docs/archive/YYYY-MM-DD-topic.md`
|
||||
4. **Add learnings**: Update `docs/LESSONS-LEARNED.md` with insights
|
||||
5. **Keep status current**: Update `docs/README.md` when project state changes
|
||||
|
||||
**Do not commit or push code that fails tests. Fix any test failures before proceeding.**
|
||||
### What Goes Where
|
||||
- **README.md**: Current status, known issues, quick links
|
||||
- **ARCHITECTURE.md**: System design, data flow, tech decisions
|
||||
- **DEVELOPMENT.md**: Setup, testing, deployment, CI/CD
|
||||
- **API.md**: Endpoints, request/response formats
|
||||
- **LESSONS-LEARNED.md**: What worked, what didn't, gotchas
|
||||
- **archive/**: Completed work, session summaries, old plans
|
||||
|
||||
## Related Projects
|
||||
|
||||
### flacfetch
|
||||
Location: `/Users/andrew/Projects/flacfetch`
|
||||
|
||||
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
|
||||
|
||||
## Pre-Commit Checklist
|
||||
|
||||
- [ ] Tests pass (`make test` / `npm run test:all`)
|
||||
- [ ] Version bumped (if code changed)
|
||||
- [ ] `/docs-review` run (check for doc updates)
|
||||
- [ ] PR created with clear description
|
||||
@@ -71,8 +71,40 @@ export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
|
||||
|
||||
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
|
||||
|
||||
#### Option 2: Whisper via RunPod
|
||||
Open-source alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
#### Option 2: Local Whisper (No Cloud Required)
|
||||
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
|
||||
|
||||
```bash
|
||||
# Install with local Whisper support
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
|
||||
# Optional: Configure model size (tiny, base, small, medium, large)
|
||||
export WHISPER_MODEL_SIZE="medium"
|
||||
|
||||
# Optional: Force specific device (cpu, cuda, mps)
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
**Model Size Guide:**
|
||||
| Model | VRAM | Speed | Quality |
|
||||
|-------|------|-------|---------|
|
||||
| tiny | ~1GB | Fast | Lower |
|
||||
| base | ~1GB | Fast | Basic |
|
||||
| small | ~2GB | Medium | Good |
|
||||
| medium | ~5GB | Slower | Better |
|
||||
| large | ~10GB | Slowest | Best |
|
||||
|
||||
**CPU-Only Installation** (no GPU required):
|
||||
```bash
|
||||
# Pre-install CPU-only PyTorch first
|
||||
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
```
|
||||
|
||||
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
|
||||
|
||||
#### Option 3: Whisper via RunPod
|
||||
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
|
||||
```bash
|
||||
export RUNPOD_API_KEY="your_runpod_key"
|
||||
@@ -574,6 +606,44 @@ If the output video has quality problems:
|
||||
- Check available codecs: `ffmpeg -codecs`
|
||||
- For 4K output, ensure sufficient disk space (10GB+ per track)
|
||||
|
||||
### Local Whisper Issues
|
||||
|
||||
#### GPU Out of Memory
|
||||
If you get CUDA out of memory errors:
|
||||
```bash
|
||||
# Use a smaller model
|
||||
export WHISPER_MODEL_SIZE="small" # or "tiny"
|
||||
|
||||
# Or force CPU mode
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
#### Slow Transcription on CPU
|
||||
CPU transcription is significantly slower than GPU. For faster processing:
|
||||
- Use a smaller model (`tiny` or `base`)
|
||||
- Consider using cloud transcription (AudioShake or RunPod)
|
||||
- On Apple Silicon, the `small` model offers good speed/quality balance
|
||||
|
||||
#### Model Download Issues
|
||||
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
|
||||
- Check your internet connection
|
||||
- Set a custom cache directory: `export WHISPER_CACHE_DIR="/path/with/space"`
|
||||
- Models are cached in `~/.cache/whisper/` by default
|
||||
|
||||
#### whisper-timestamped Not Found
|
||||
If you get "whisper-timestamped is not installed":
|
||||
```bash
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
# Or install directly:
|
||||
pip install whisper-timestamped
|
||||
```
|
||||
|
||||
#### Disabling Local Whisper
|
||||
If you want to disable local Whisper (e.g., to force cloud transcription):
|
||||
```bash
|
||||
export ENABLE_LOCAL_WHISPER="false"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Development
|
||||
|
||||
@@ -85,9 +85,13 @@ class AudioSearchRequest(BaseModel):
|
||||
# Auto-download mode
|
||||
auto_download: bool = Field(False, description="Automatically select best result and download")
|
||||
|
||||
# Processing options (CDG/TXT require style config, disabled by default)
|
||||
enable_cdg: bool = Field(False, description="Generate CDG+MP3 package (requires style config)")
|
||||
enable_txt: bool = Field(False, description="Generate TXT+MP3 package (requires style config)")
|
||||
# Theme configuration
|
||||
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
|
||||
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
|
||||
|
||||
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
|
||||
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
@@ -180,6 +184,35 @@ class AudioSelectResponse(BaseModel):
|
||||
selected_provider: str
|
||||
|
||||
|
||||
def _resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool],
|
||||
enable_txt: Optional[bool]
|
||||
) -> Tuple[bool, bool]:
|
||||
"""
|
||||
Resolve CDG/TXT settings based on theme and explicit settings.
|
||||
|
||||
When a theme is selected, CDG and TXT are enabled by default.
|
||||
Explicit True/False values always override the default.
|
||||
|
||||
Args:
|
||||
theme_id: Theme identifier (if any)
|
||||
enable_cdg: Explicit CDG setting (None means use default)
|
||||
enable_txt: Explicit TXT setting (None means use default)
|
||||
|
||||
Returns:
|
||||
Tuple of (resolved_enable_cdg, resolved_enable_txt)
|
||||
"""
|
||||
# Default based on whether theme is set
|
||||
default_enabled = theme_id is not None
|
||||
|
||||
# Explicit values override defaults, None uses default
|
||||
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
|
||||
resolved_txt = enable_txt if enable_txt is not None else default_enabled
|
||||
|
||||
return resolved_cdg, resolved_txt
|
||||
|
||||
|
||||
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
|
||||
"""Extract metadata from request for job tracking."""
|
||||
headers = dict(request.headers)
|
||||
@@ -461,13 +494,20 @@ async def search_audio(
|
||||
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="audio_search")
|
||||
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
enable_cdg=body.enable_cdg,
|
||||
enable_txt=body.enable_txt,
|
||||
theme_id=body.theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=effective_brand_prefix,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=effective_youtube_description,
|
||||
@@ -497,7 +537,28 @@ async def search_audio(
|
||||
'audio_search_title': body.title,
|
||||
'auto_download': body.auto_download,
|
||||
})
|
||||
|
||||
|
||||
# If theme is set and no custom style files are being uploaded, prepare theme style now
|
||||
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
|
||||
# can access the style configuration for preview videos
|
||||
if body.theme_id and not body.style_files:
|
||||
from backend.api.routes.file_upload import _prepare_theme_for_job
|
||||
try:
|
||||
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
)
|
||||
theme_update = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
'style_assets': theme_style_assets,
|
||||
}
|
||||
if youtube_desc and not effective_youtube_description:
|
||||
theme_update['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, theme_update)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to prepare theme '{body.theme_id}' for job {job_id}: {e}")
|
||||
# Continue without theme - job can still be processed with defaults
|
||||
|
||||
# Handle style file uploads if provided
|
||||
style_upload_urls: List[StyleUploadUrl] = []
|
||||
style_assets = {}
|
||||
|
||||
@@ -22,10 +22,12 @@ from typing import Optional, List, Dict, Any, Tuple
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.models.job import JobCreate, JobStatus
|
||||
from backend.models.theme import ColorOverrides
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.services.credential_manager import get_credential_manager, CredentialStatus
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.services.metrics import metrics
|
||||
@@ -51,14 +53,18 @@ class CreateJobFromUrlRequest(BaseModel):
|
||||
"""Request to create a job from a YouTube/online URL."""
|
||||
# Required fields
|
||||
url: str = Field(..., description="YouTube or other video URL to download audio from")
|
||||
|
||||
|
||||
# Optional fields - will be auto-detected from URL if not provided
|
||||
artist: Optional[str] = Field(None, description="Artist name (auto-detected from URL if not provided)")
|
||||
title: Optional[str] = Field(None, description="Song title (auto-detected from URL if not provided)")
|
||||
|
||||
# Processing options (CDG/TXT require style config, disabled by default)
|
||||
enable_cdg: bool = Field(False, description="Generate CDG+MP3 package (requires style config)")
|
||||
enable_txt: bool = Field(False, description="Generate TXT+MP3 package (requires style config)")
|
||||
|
||||
# Theme configuration (use pre-made theme from GCS)
|
||||
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
|
||||
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
|
||||
|
||||
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
|
||||
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
@@ -103,9 +109,13 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
|
||||
title: str = Field(..., description="Song title")
|
||||
files: List[FileUploadRequest] = Field(..., description="List of files to upload")
|
||||
|
||||
# Processing options (CDG/TXT require style config, disabled by default)
|
||||
enable_cdg: bool = Field(False, description="Generate CDG+MP3 package (requires style config)")
|
||||
enable_txt: bool = Field(False, description="Generate TXT+MP3 package (requires style config)")
|
||||
# Theme configuration (use pre-made theme from GCS)
|
||||
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
|
||||
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
|
||||
|
||||
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
|
||||
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
@@ -226,7 +236,7 @@ ALLOWED_FONT_EXTENSIONS = {'.ttf', '.otf', '.woff', '.woff2'}
|
||||
async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
"""
|
||||
Trigger both audio and lyrics workers in parallel.
|
||||
|
||||
|
||||
FastAPI's BackgroundTasks runs async tasks sequentially, so we use
|
||||
asyncio.gather to ensure both workers start at the same time.
|
||||
"""
|
||||
@@ -236,6 +246,83 @@ async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool],
|
||||
enable_txt: Optional[bool]
|
||||
) -> Tuple[bool, bool]:
|
||||
"""
|
||||
Resolve CDG/TXT settings based on theme and explicit settings.
|
||||
|
||||
When a theme is selected, CDG and TXT are enabled by default.
|
||||
Explicit True/False values always override the default.
|
||||
|
||||
Args:
|
||||
theme_id: Theme identifier (if any)
|
||||
enable_cdg: Explicit CDG setting (None means use default)
|
||||
enable_txt: Explicit TXT setting (None means use default)
|
||||
|
||||
Returns:
|
||||
Tuple of (resolved_enable_cdg, resolved_enable_txt)
|
||||
"""
|
||||
# Default based on whether theme is set
|
||||
default_enabled = theme_id is not None
|
||||
|
||||
# Resolve with explicit override taking precedence
|
||||
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
|
||||
resolved_txt = enable_txt if enable_txt is not None else default_enabled
|
||||
|
||||
return resolved_cdg, resolved_txt
|
||||
|
||||
|
||||
def _prepare_theme_for_job(
|
||||
job_id: str,
|
||||
theme_id: str,
|
||||
color_overrides: Optional[Dict[str, str]] = None
|
||||
) -> Tuple[str, Dict[str, str], Optional[str]]:
|
||||
"""
|
||||
Prepare theme style files for a job.
|
||||
|
||||
Args:
|
||||
job_id: The job ID
|
||||
theme_id: Theme identifier
|
||||
color_overrides: Optional color override dict
|
||||
|
||||
Returns:
|
||||
Tuple of (style_params_gcs_path, style_assets, youtube_description)
|
||||
|
||||
Raises:
|
||||
HTTPException: If theme not found
|
||||
"""
|
||||
theme_service = get_theme_service()
|
||||
|
||||
# Verify theme exists
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Theme not found: {theme_id}. Use GET /api/themes to list available themes."
|
||||
)
|
||||
|
||||
# Convert dict to ColorOverrides model if provided
|
||||
color_overrides_model = None
|
||||
if color_overrides:
|
||||
color_overrides_model = ColorOverrides(**color_overrides)
|
||||
|
||||
# Prepare job style from theme
|
||||
style_params_path, style_assets = theme_service.prepare_job_style(
|
||||
job_id=job_id,
|
||||
theme_id=theme_id,
|
||||
color_overrides=color_overrides_model
|
||||
)
|
||||
|
||||
# Get YouTube description template if available
|
||||
youtube_desc = theme_service.get_youtube_description(theme_id)
|
||||
|
||||
logger.info(f"Prepared theme '{theme_id}' for job {job_id}")
|
||||
|
||||
return style_params_path, style_assets, youtube_desc
|
||||
|
||||
|
||||
@router.post("/jobs/upload")
|
||||
async def upload_and_create_job(
|
||||
request: Request,
|
||||
@@ -682,9 +769,13 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
|
||||
title: str = Field(..., description="Song title")
|
||||
files: List[FinaliseOnlyFileRequest] = Field(..., description="List of prep output files to upload")
|
||||
|
||||
# Processing options (CDG/TXT require style config, disabled by default)
|
||||
enable_cdg: bool = Field(False, description="Generate CDG+MP3 package (requires style config)")
|
||||
enable_txt: bool = Field(False, description="Generate TXT+MP3 package (requires style config)")
|
||||
# Theme configuration (use pre-made theme from GCS)
|
||||
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
|
||||
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
|
||||
|
||||
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
|
||||
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
@@ -859,17 +950,27 @@ async def create_job_with_upload_urls(
|
||||
|
||||
# Extract request metadata for tracking
|
||||
request_metadata = extract_request_metadata(request, created_from="signed_url_upload")
|
||||
|
||||
|
||||
# Get original audio filename
|
||||
audio_file = audio_files[0]
|
||||
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Check if style_params is being uploaded (overrides theme)
|
||||
has_style_params_upload = any(f.file_type == 'style_params' for f in body.files)
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
filename=audio_file.filename,
|
||||
enable_cdg=body.enable_cdg,
|
||||
enable_txt=body.enable_txt,
|
||||
theme_id=body.theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=body.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
@@ -889,11 +990,26 @@ async def create_job_with_upload_urls(
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job_id, source="upload")
|
||||
|
||||
|
||||
logger.info(f"Created job {job_id} for {body.artist} - {body.title} (signed URL upload flow)")
|
||||
|
||||
# If theme is set and no style_params uploaded, prepare theme style now
|
||||
if body.theme_id and not has_style_params_upload:
|
||||
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
)
|
||||
# Update job with theme style data
|
||||
update_data = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
'style_assets': style_assets,
|
||||
}
|
||||
if youtube_desc and not body.youtube_description:
|
||||
update_data['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, update_data)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
|
||||
# Generate signed upload URLs for each file
|
||||
upload_urls = []
|
||||
@@ -1245,19 +1361,26 @@ async def create_job_from_url(
|
||||
|
||||
# Extract request metadata for tracking
|
||||
request_metadata = extract_request_metadata(request, created_from="url")
|
||||
|
||||
|
||||
# Use provided artist/title or leave as None (will be auto-detected by audio worker)
|
||||
artist = body.artist
|
||||
title = body.title
|
||||
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Create job with URL
|
||||
job_create = JobCreate(
|
||||
url=body.url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
filename=None, # No file uploaded
|
||||
enable_cdg=body.enable_cdg,
|
||||
enable_txt=body.enable_txt,
|
||||
theme_id=body.theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=body.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
@@ -1277,9 +1400,24 @@ async def create_job_from_url(
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job_id, source="url")
|
||||
|
||||
# If theme is set, prepare theme style now
|
||||
if body.theme_id:
|
||||
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
)
|
||||
# Update job with theme style data
|
||||
update_data = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
'style_assets': style_assets,
|
||||
}
|
||||
if youtube_desc and not body.youtube_description:
|
||||
update_data['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, update_data)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
|
||||
logger.info(f"Created URL-based job {job_id} for URL: {body.url}")
|
||||
if artist:
|
||||
@@ -1462,14 +1600,24 @@ async def create_finalise_only_job(
|
||||
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="finalise_only_upload")
|
||||
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Check if style_params is being uploaded (overrides theme)
|
||||
has_style_params_upload = any(f.file_type == 'style_params' for f in body.files)
|
||||
|
||||
# Create job with finalise_only=True
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
filename="finalise_only", # No single audio file - using prep outputs
|
||||
enable_cdg=body.enable_cdg,
|
||||
enable_txt=body.enable_txt,
|
||||
theme_id=body.theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=body.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
@@ -1482,11 +1630,26 @@ async def create_finalise_only_job(
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job_id, source="finalise")
|
||||
|
||||
|
||||
logger.info(f"Created finalise-only job {job_id} for {body.artist} - {body.title}")
|
||||
|
||||
# If theme is set and no style_params uploaded, prepare theme style now
|
||||
if body.theme_id and not has_style_params_upload:
|
||||
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
)
|
||||
# Update job with theme style data
|
||||
update_data = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
'style_assets': style_assets,
|
||||
}
|
||||
if youtube_desc and not body.youtube_description:
|
||||
update_data['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, update_data)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
|
||||
# Generate signed upload URLs for each file
|
||||
upload_urls = []
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
"""
|
||||
Theme API routes for listing and retrieving theme information.
|
||||
|
||||
Themes are pre-created style configurations that users can select when creating jobs.
|
||||
These endpoints are public (no authentication required) to allow theme selection
|
||||
before job creation.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
|
||||
from backend.services.theme_service import get_theme_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/themes", tags=["themes"])
|
||||
|
||||
|
||||
@router.get("", response_model=ThemesListResponse)
|
||||
async def list_themes() -> ThemesListResponse:
|
||||
"""
|
||||
List all available themes.
|
||||
|
||||
Returns a list of themes with their metadata and preview image URLs.
|
||||
Preview URLs are signed and valid for 60 minutes.
|
||||
|
||||
This endpoint is public and does not require authentication.
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
themes = theme_service.list_themes()
|
||||
return ThemesListResponse(themes=themes)
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing themes: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load themes"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
|
||||
async def get_theme(theme_id: str) -> ThemeDetailResponse:
|
||||
"""
|
||||
Get detailed information about a specific theme.
|
||||
|
||||
Returns the full theme details including style parameters.
|
||||
This can be used to preview theme settings or inspect configuration.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
ThemeDetailResponse with full theme details
|
||||
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
|
||||
if theme is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
return ThemeDetailResponse(theme=theme)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/preview")
|
||||
async def get_theme_preview(theme_id: str) -> dict:
|
||||
"""
|
||||
Get a signed URL for the theme's preview image.
|
||||
|
||||
This endpoint is useful when you need just the preview URL
|
||||
without loading the full theme details.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
Object with preview_url field
|
||||
|
||||
Raises:
|
||||
404: Theme not found or no preview available
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
if theme is None or theme.preview_url is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"No preview available for theme: {theme_id}"
|
||||
)
|
||||
|
||||
return {"preview_url": theme.preview_url}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting preview for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme preview"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/youtube-description")
|
||||
async def get_theme_youtube_description(theme_id: str) -> dict:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
Some themes include a default YouTube video description template
|
||||
that can be used when uploading videos.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
Object with description field containing the template text,
|
||||
or null if no template is available
|
||||
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
description = theme_service.get_youtube_description(theme_id)
|
||||
return {"description": description}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load YouTube description"
|
||||
)
|
||||
+2
-1
@@ -7,7 +7,7 @@ from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
|
||||
@@ -108,6 +108,7 @@ app.include_router(internal.router, prefix="/api") # Internal worker endpoints
|
||||
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
|
||||
app.include_router(auth.router, prefix="/api") # OAuth credential management
|
||||
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
|
||||
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
||||
|
||||
|
||||
@app.get("/")
|
||||
|
||||
+24
-2
@@ -205,7 +205,18 @@ class Job(BaseModel):
|
||||
webhook_url: Optional[str] = None # Webhook for notifications
|
||||
user_email: Optional[str] = None # Email for notifications
|
||||
|
||||
# Style configuration (uploaded files)
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
color_overrides: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
User color overrides applied on top of theme. Keys:
|
||||
- artist_color: Hex color for artist name (#RRGGBB)
|
||||
- title_color: Hex color for song title
|
||||
- sung_lyrics_color: Hex color for highlighted lyrics
|
||||
- unsung_lyrics_color: Hex color for unhighlighted lyrics
|
||||
"""
|
||||
|
||||
# Style configuration (uploaded files - used when theme_id is not set)
|
||||
style_params_gcs_path: Optional[str] = None # GCS path to style_params.json
|
||||
style_assets: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
@@ -409,7 +420,18 @@ class JobCreate(BaseModel):
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
# Style configuration (will be populated after file upload)
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
color_overrides: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
User color overrides applied on top of theme. Keys:
|
||||
- artist_color: Hex color for artist name (#RRGGBB)
|
||||
- title_color: Hex color for song title
|
||||
- sung_lyrics_color: Hex color for highlighted lyrics
|
||||
- unsung_lyrics_color: Hex color for unhighlighted lyrics
|
||||
"""
|
||||
|
||||
# Style configuration (will be populated after file upload, or from theme)
|
||||
style_params_gcs_path: Optional[str] = None
|
||||
style_assets: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
"""
|
||||
Theme data models for karaoke-gen.
|
||||
|
||||
Themes are pre-created style configurations stored in GCS that users can select
|
||||
when creating karaoke jobs. Each theme includes:
|
||||
- Style parameters (fonts, colors, backgrounds, layouts)
|
||||
- Asset files (images, fonts)
|
||||
- Preview images
|
||||
- Optional YouTube description template
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class ThemeSummary(BaseModel):
|
||||
"""Summary of a theme for listing purposes."""
|
||||
|
||||
id: str = Field(..., description="Unique theme identifier (e.g., 'nomad', 'default')")
|
||||
name: str = Field(..., description="Human-readable theme name")
|
||||
description: str = Field(..., description="Brief description of the theme style")
|
||||
preview_url: Optional[str] = Field(None, description="Signed URL for preview image")
|
||||
thumbnail_url: Optional[str] = Field(None, description="Signed URL for smaller thumbnail")
|
||||
is_default: bool = Field(False, description="Whether this is the default theme")
|
||||
|
||||
|
||||
class ThemeMetadata(BaseModel):
|
||||
"""Metadata for a single theme in the registry."""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
is_default: bool = False
|
||||
created_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class ThemeRegistry(BaseModel):
|
||||
"""Registry of all available themes, stored as _metadata.json in GCS."""
|
||||
|
||||
version: int = 1
|
||||
themes: List[ThemeMetadata] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ThemeDetail(BaseModel):
|
||||
"""Full theme details including style parameters."""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
preview_url: Optional[str] = None
|
||||
is_default: bool = False
|
||||
style_params: Dict[str, Any] = Field(default_factory=dict)
|
||||
has_youtube_description: bool = False
|
||||
|
||||
|
||||
class ColorOverrides(BaseModel):
|
||||
"""
|
||||
User-customizable color overrides that can be applied on top of a theme.
|
||||
|
||||
All colors are hex format (#RRGGBB). When applied:
|
||||
- artist_color: Applied to intro.artist_color, end.artist_color, cdg.artist_color
|
||||
- title_color: Applied to intro.title_color, end.title_color, cdg.title_color
|
||||
- sung_lyrics_color: Applied to karaoke.primary_color (converted to RGBA), cdg.active_fill
|
||||
- unsung_lyrics_color: Applied to karaoke.secondary_color (converted to RGBA), cdg.inactive_fill
|
||||
"""
|
||||
|
||||
artist_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for artist name on intro/end screens",
|
||||
)
|
||||
title_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for song title on intro/end screens",
|
||||
)
|
||||
sung_lyrics_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for lyrics being sung (highlighted)",
|
||||
)
|
||||
unsung_lyrics_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for lyrics not yet sung",
|
||||
)
|
||||
|
||||
def has_overrides(self) -> bool:
|
||||
"""Check if any color overrides are set."""
|
||||
return any(
|
||||
[
|
||||
self.artist_color,
|
||||
self.title_color,
|
||||
self.sung_lyrics_color,
|
||||
self.unsung_lyrics_color,
|
||||
]
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, str]:
|
||||
"""Convert to dict, excluding None values."""
|
||||
return {k: v for k, v in self.model_dump().items() if v is not None}
|
||||
|
||||
|
||||
class ThemesListResponse(BaseModel):
|
||||
"""Response from GET /api/themes endpoint."""
|
||||
|
||||
themes: List[ThemeSummary]
|
||||
|
||||
|
||||
class ThemeDetailResponse(BaseModel):
|
||||
"""Response from GET /api/themes/{theme_id} endpoint."""
|
||||
|
||||
theme: ThemeDetail
|
||||
|
||||
|
||||
# Utility functions for color conversion
|
||||
|
||||
|
||||
def hex_to_rgba(hex_color: str, alpha: int = 255) -> str:
|
||||
"""
|
||||
Convert hex color (#RRGGBB) to RGBA format for ASS subtitles.
|
||||
|
||||
ASS subtitle format uses "R, G, B, A" string format.
|
||||
|
||||
Args:
|
||||
hex_color: Hex color string like "#7070F7"
|
||||
alpha: Alpha value 0-255 (default 255 = opaque)
|
||||
|
||||
Returns:
|
||||
RGBA string like "112, 112, 247, 255"
|
||||
"""
|
||||
hex_color = hex_color.lstrip("#")
|
||||
r = int(hex_color[0:2], 16)
|
||||
g = int(hex_color[2:4], 16)
|
||||
b = int(hex_color[4:6], 16)
|
||||
return f"{r}, {g}, {b}, {alpha}"
|
||||
|
||||
|
||||
def rgba_to_hex(rgba_str: str) -> str:
|
||||
"""
|
||||
Convert RGBA format to hex color.
|
||||
|
||||
Args:
|
||||
rgba_str: RGBA string like "112, 112, 247, 255"
|
||||
|
||||
Returns:
|
||||
Hex color string like "#7070F7"
|
||||
"""
|
||||
parts = [int(x.strip()) for x in rgba_str.split(",")]
|
||||
r, g, b = parts[0], parts[1], parts[2]
|
||||
return f"#{r:02x}{g:02x}{b:02x}"
|
||||
@@ -61,6 +61,11 @@ class JobManager:
|
||||
discord_webhook_url=job_create.discord_webhook_url,
|
||||
dropbox_path=job_create.dropbox_path,
|
||||
gdrive_folder_id=job_create.gdrive_folder_id,
|
||||
# Theme configuration
|
||||
theme_id=job_create.theme_id,
|
||||
color_overrides=job_create.color_overrides,
|
||||
style_params_gcs_path=job_create.style_params_gcs_path,
|
||||
style_assets=job_create.style_assets,
|
||||
# Two-phase workflow (Batch 6)
|
||||
prep_only=job_create.prep_only,
|
||||
finalise_only=job_create.finalise_only,
|
||||
|
||||
@@ -0,0 +1,469 @@
|
||||
"""
|
||||
Theme service for managing pre-created style themes.
|
||||
|
||||
Themes are stored in GCS at themes/{theme_id}/ with:
|
||||
- style_params.json: Complete style configuration
|
||||
- preview.png: Preview image for theme selection UI
|
||||
- youtube_description.txt: Optional YouTube description template
|
||||
- assets/: Fonts, backgrounds, and other assets
|
||||
|
||||
The service provides:
|
||||
- Theme listing with signed preview URLs
|
||||
- Theme detail retrieval
|
||||
- Color override application
|
||||
- Job style preparation (copying theme to job folder)
|
||||
"""
|
||||
|
||||
import copy
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import threading
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from backend.models.theme import (
|
||||
ColorOverrides,
|
||||
ThemeDetail,
|
||||
ThemeMetadata,
|
||||
ThemeRegistry,
|
||||
ThemeSummary,
|
||||
hex_to_rgba,
|
||||
)
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# GCS paths for themes
|
||||
THEMES_PREFIX = "themes"
|
||||
METADATA_FILE = f"{THEMES_PREFIX}/_metadata.json"
|
||||
|
||||
|
||||
class ThemeService:
|
||||
"""Service for managing themes from GCS."""
|
||||
|
||||
def __init__(self, storage: Optional[StorageService] = None):
|
||||
"""
|
||||
Initialize the theme service.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance (creates new one if not provided)
|
||||
"""
|
||||
self.storage = storage or StorageService()
|
||||
self._metadata_cache: Optional[ThemeRegistry] = None
|
||||
self._cache_time: Optional[datetime] = None
|
||||
self.CACHE_TTL_SECONDS = 300 # 5 minute cache for theme metadata
|
||||
|
||||
def _get_theme_path(self, theme_id: str, filename: str = "") -> str:
|
||||
"""Get the GCS path for a theme file."""
|
||||
if filename:
|
||||
return f"{THEMES_PREFIX}/{theme_id}/{filename}"
|
||||
return f"{THEMES_PREFIX}/{theme_id}"
|
||||
|
||||
def _is_cache_valid(self) -> bool:
|
||||
"""Check if the metadata cache is still valid."""
|
||||
if self._metadata_cache is None or self._cache_time is None:
|
||||
return False
|
||||
age = datetime.now() - self._cache_time
|
||||
return age.total_seconds() < self.CACHE_TTL_SECONDS
|
||||
|
||||
def _load_metadata(self, force_refresh: bool = False) -> ThemeRegistry:
|
||||
"""
|
||||
Load theme metadata from GCS with caching.
|
||||
|
||||
Args:
|
||||
force_refresh: Force reload from GCS even if cache is valid
|
||||
|
||||
Returns:
|
||||
ThemeRegistry containing all theme metadata
|
||||
"""
|
||||
if not force_refresh and self._is_cache_valid():
|
||||
return self._metadata_cache # type: ignore
|
||||
|
||||
try:
|
||||
data = self.storage.download_json(METADATA_FILE)
|
||||
self._metadata_cache = ThemeRegistry(**data)
|
||||
self._cache_time = datetime.now()
|
||||
logger.info(f"Loaded theme metadata: {len(self._metadata_cache.themes)} themes")
|
||||
return self._metadata_cache
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load theme metadata: {e}")
|
||||
# Return empty registry on error
|
||||
return ThemeRegistry(version=1, themes=[])
|
||||
|
||||
def list_themes(self) -> List[ThemeSummary]:
|
||||
"""
|
||||
List all available themes with signed preview URLs.
|
||||
|
||||
Returns:
|
||||
List of ThemeSummary objects with preview URLs
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
summaries = []
|
||||
|
||||
for theme in metadata.themes:
|
||||
try:
|
||||
# Generate signed URLs for preview images
|
||||
preview_path = self._get_theme_path(theme.id, "preview.png")
|
||||
thumbnail_path = self._get_theme_path(theme.id, "preview_thumbnail.png")
|
||||
|
||||
preview_url = None
|
||||
thumbnail_url = None
|
||||
|
||||
if self.storage.file_exists(preview_path):
|
||||
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
|
||||
|
||||
# Try thumbnail, fall back to main preview
|
||||
if self.storage.file_exists(thumbnail_path):
|
||||
thumbnail_url = self.storage.generate_signed_url(thumbnail_path, expiration_minutes=60)
|
||||
elif preview_url:
|
||||
thumbnail_url = preview_url
|
||||
|
||||
summaries.append(
|
||||
ThemeSummary(
|
||||
id=theme.id,
|
||||
name=theme.name,
|
||||
description=theme.description,
|
||||
preview_url=preview_url,
|
||||
thumbnail_url=thumbnail_url,
|
||||
is_default=theme.is_default,
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Error processing theme {theme.id}: {e}")
|
||||
# Still include theme but without preview URLs
|
||||
summaries.append(
|
||||
ThemeSummary(
|
||||
id=theme.id,
|
||||
name=theme.name,
|
||||
description=theme.description,
|
||||
is_default=theme.is_default,
|
||||
)
|
||||
)
|
||||
|
||||
return summaries
|
||||
|
||||
def get_theme(self, theme_id: str) -> Optional[ThemeDetail]:
|
||||
"""
|
||||
Get full theme details including style parameters.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
ThemeDetail if found, None otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
|
||||
# Find theme in metadata
|
||||
theme_meta = next((t for t in metadata.themes if t.id == theme_id), None)
|
||||
if not theme_meta:
|
||||
logger.warning(f"Theme not found: {theme_id}")
|
||||
return None
|
||||
|
||||
try:
|
||||
# Load style_params.json
|
||||
style_params_path = self._get_theme_path(theme_id, "style_params.json")
|
||||
style_params = self.storage.download_json(style_params_path)
|
||||
|
||||
# Check for YouTube description
|
||||
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
|
||||
has_youtube_desc = self.storage.file_exists(youtube_desc_path)
|
||||
|
||||
# Generate preview URL
|
||||
preview_path = self._get_theme_path(theme_id, "preview.png")
|
||||
preview_url = None
|
||||
if self.storage.file_exists(preview_path):
|
||||
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
|
||||
|
||||
return ThemeDetail(
|
||||
id=theme_id,
|
||||
name=theme_meta.name,
|
||||
description=theme_meta.description,
|
||||
preview_url=preview_url,
|
||||
is_default=theme_meta.is_default,
|
||||
style_params=style_params,
|
||||
has_youtube_description=has_youtube_desc,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_theme_style_params(self, theme_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get just the style_params.json for a theme.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
Style parameters dict if found, None otherwise
|
||||
"""
|
||||
try:
|
||||
style_params_path = self._get_theme_path(theme_id, "style_params.json")
|
||||
return self.storage.download_json(style_params_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading style params for theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_youtube_description(self, theme_id: str) -> Optional[str]:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
YouTube description text if found, None otherwise
|
||||
"""
|
||||
try:
|
||||
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
|
||||
if not self.storage.file_exists(youtube_desc_path):
|
||||
return None
|
||||
|
||||
# Download to temp file and read
|
||||
with tempfile.NamedTemporaryFile(mode="w", delete=False, suffix=".txt") as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
self.storage.download_file(youtube_desc_path, tmp_path)
|
||||
with open(tmp_path, "r") as f:
|
||||
return f.read()
|
||||
finally:
|
||||
if os.path.exists(tmp_path):
|
||||
os.remove(tmp_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading YouTube description for theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def apply_color_overrides(
|
||||
self, style_params: Dict[str, Any], overrides: ColorOverrides
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Apply user color overrides to theme style parameters.
|
||||
|
||||
Args:
|
||||
style_params: Base style parameters from theme
|
||||
overrides: User color overrides
|
||||
|
||||
Returns:
|
||||
Modified style parameters with overrides applied
|
||||
"""
|
||||
if not overrides.has_overrides():
|
||||
return style_params
|
||||
|
||||
# Deep copy to avoid modifying original
|
||||
result = copy.deepcopy(style_params)
|
||||
|
||||
# Apply artist_color to intro, end, and CDG sections
|
||||
if overrides.artist_color:
|
||||
if "intro" in result:
|
||||
result["intro"]["artist_color"] = overrides.artist_color
|
||||
if "end" in result:
|
||||
result["end"]["artist_color"] = overrides.artist_color
|
||||
if "cdg" in result:
|
||||
result["cdg"]["artist_color"] = overrides.artist_color
|
||||
|
||||
# Apply title_color to intro, end, and CDG sections
|
||||
if overrides.title_color:
|
||||
if "intro" in result:
|
||||
result["intro"]["title_color"] = overrides.title_color
|
||||
if "end" in result:
|
||||
result["end"]["title_color"] = overrides.title_color
|
||||
if "cdg" in result:
|
||||
result["cdg"]["title_color"] = overrides.title_color
|
||||
|
||||
# Apply sung_lyrics_color (karaoke uses RGBA, CDG uses hex)
|
||||
if overrides.sung_lyrics_color:
|
||||
if "karaoke" in result:
|
||||
result["karaoke"]["primary_color"] = hex_to_rgba(overrides.sung_lyrics_color)
|
||||
if "cdg" in result:
|
||||
result["cdg"]["active_fill"] = overrides.sung_lyrics_color
|
||||
|
||||
# Apply unsung_lyrics_color (karaoke uses RGBA, CDG uses hex)
|
||||
if overrides.unsung_lyrics_color:
|
||||
if "karaoke" in result:
|
||||
result["karaoke"]["secondary_color"] = hex_to_rgba(overrides.unsung_lyrics_color)
|
||||
if "cdg" in result:
|
||||
result["cdg"]["inactive_fill"] = overrides.unsung_lyrics_color
|
||||
|
||||
return result
|
||||
|
||||
def prepare_job_style(
|
||||
self,
|
||||
job_id: str,
|
||||
theme_id: str,
|
||||
color_overrides: Optional[ColorOverrides] = None,
|
||||
) -> Tuple[str, Dict[str, str]]:
|
||||
"""
|
||||
Prepare style files for a job by copying theme to job folder.
|
||||
|
||||
This method:
|
||||
1. Loads the theme's style_params.json
|
||||
2. Applies any color overrides
|
||||
3. Updates asset paths to point to theme assets (shared, not copied)
|
||||
4. Uploads modified style_params.json to job folder
|
||||
5. Returns the GCS path and style_assets mapping
|
||||
|
||||
Args:
|
||||
job_id: The job ID
|
||||
theme_id: The theme to use
|
||||
color_overrides: Optional color overrides
|
||||
|
||||
Returns:
|
||||
Tuple of (style_params_gcs_path, style_assets dict)
|
||||
|
||||
Raises:
|
||||
ValueError: If theme not found
|
||||
"""
|
||||
# Load theme style params
|
||||
style_params = self.get_theme_style_params(theme_id)
|
||||
if style_params is None:
|
||||
raise ValueError(f"Theme not found: {theme_id}")
|
||||
|
||||
# Apply color overrides if provided
|
||||
if color_overrides:
|
||||
style_params = self.apply_color_overrides(style_params, color_overrides)
|
||||
|
||||
# Update asset paths to point to theme's shared assets
|
||||
# Theme assets stay in themes/{theme_id}/assets/ - they're shared
|
||||
style_assets = self._build_style_assets_mapping(theme_id, style_params)
|
||||
|
||||
# Update style_params to use theme asset paths (not job-specific)
|
||||
style_params = self._update_asset_paths_in_style(theme_id, style_params)
|
||||
|
||||
# Upload modified style_params.json to job's style folder
|
||||
job_style_path = f"uploads/{job_id}/style/style_params.json"
|
||||
self.storage.upload_json(job_style_path, style_params)
|
||||
|
||||
logger.info(f"Prepared job {job_id} style from theme {theme_id}")
|
||||
|
||||
return job_style_path, style_assets
|
||||
|
||||
def _build_style_assets_mapping(
|
||||
self, theme_id: str, style_params: Dict[str, Any]
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
Build the style_assets mapping for a theme.
|
||||
|
||||
Maps asset keys to their GCS paths in the theme folder.
|
||||
"""
|
||||
style_assets = {}
|
||||
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
|
||||
|
||||
# Check each potential asset type
|
||||
asset_checks = [
|
||||
("intro_background", "intro", "background_image"),
|
||||
("karaoke_background", "karaoke", "background_image"),
|
||||
("end_background", "end", "background_image"),
|
||||
("font", "intro", "font"), # Font is typically shared across sections
|
||||
("cdg_instrumental_background", "cdg", "instrumental_background"),
|
||||
("cdg_title_background", "cdg", "title_screen_background"),
|
||||
("cdg_outro_background", "cdg", "outro_background"),
|
||||
]
|
||||
|
||||
for asset_key, section, field in asset_checks:
|
||||
if section in style_params and field in style_params[section]:
|
||||
value = style_params[section][field]
|
||||
if value and isinstance(value, str):
|
||||
# Check if it's a theme asset path or already has the full path
|
||||
if value.startswith(f"{THEMES_PREFIX}/"):
|
||||
# Already a full theme path
|
||||
style_assets[asset_key] = value
|
||||
elif not value.startswith("/") and not value.startswith("gs://"):
|
||||
# Relative path - prepend theme assets prefix
|
||||
asset_path = f"{theme_assets_prefix}/{os.path.basename(value)}"
|
||||
if self.storage.file_exists(asset_path):
|
||||
style_assets[asset_key] = asset_path
|
||||
|
||||
return style_assets
|
||||
|
||||
def _update_asset_paths_in_style(
|
||||
self, theme_id: str, style_params: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Update asset paths in style_params to use full GCS theme paths.
|
||||
|
||||
This ensures the style_loader can find assets when processing.
|
||||
"""
|
||||
result = copy.deepcopy(style_params)
|
||||
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
|
||||
|
||||
# Fields that contain asset paths
|
||||
path_fields = {
|
||||
"intro": ["background_image", "font"],
|
||||
"end": ["background_image", "font"],
|
||||
"karaoke": ["background_image", "font_path"],
|
||||
"cdg": [
|
||||
"font_path",
|
||||
"instrumental_background",
|
||||
"title_screen_background",
|
||||
"outro_background",
|
||||
],
|
||||
}
|
||||
|
||||
for section, fields in path_fields.items():
|
||||
if section not in result:
|
||||
continue
|
||||
for field in fields:
|
||||
if field not in result[section]:
|
||||
continue
|
||||
value = result[section][field]
|
||||
if value and isinstance(value, str):
|
||||
# Skip if already a full path or URL
|
||||
if value.startswith(f"{THEMES_PREFIX}/") or value.startswith("gs://"):
|
||||
continue
|
||||
# Skip absolute local paths (shouldn't happen but be safe)
|
||||
if value.startswith("/"):
|
||||
continue
|
||||
# Update to full theme asset path
|
||||
result[section][field] = f"{theme_assets_prefix}/{os.path.basename(value)}"
|
||||
|
||||
return result
|
||||
|
||||
def theme_exists(self, theme_id: str) -> bool:
|
||||
"""
|
||||
Check if a theme exists.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
True if theme exists, False otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
return any(t.id == theme_id for t in metadata.themes)
|
||||
|
||||
def get_default_theme_id(self) -> Optional[str]:
|
||||
"""
|
||||
Get the ID of the default theme.
|
||||
|
||||
Returns:
|
||||
Default theme ID if one exists, None otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
default_theme = next((t for t in metadata.themes if t.is_default), None)
|
||||
return default_theme.id if default_theme else None
|
||||
|
||||
def invalidate_cache(self) -> None:
|
||||
"""Force invalidation of the metadata cache."""
|
||||
self._metadata_cache = None
|
||||
self._cache_time = None
|
||||
logger.info("Theme metadata cache invalidated")
|
||||
|
||||
|
||||
# Singleton instance with thread-safe initialization
|
||||
_theme_service: Optional[ThemeService] = None
|
||||
_theme_service_lock = threading.Lock()
|
||||
|
||||
|
||||
def get_theme_service() -> ThemeService:
|
||||
"""Get or create the singleton ThemeService instance (thread-safe)."""
|
||||
global _theme_service
|
||||
if _theme_service is None:
|
||||
with _theme_service_lock:
|
||||
# Double-check after acquiring lock
|
||||
if _theme_service is None:
|
||||
_theme_service = ThemeService()
|
||||
return _theme_service
|
||||
@@ -330,7 +330,7 @@ class TestReviewEndpoints:
|
||||
|
||||
class TestRenderVideoWorker:
|
||||
"""Test render video worker endpoint."""
|
||||
|
||||
|
||||
def test_render_video_worker_endpoint_exists(self, client, auth_headers):
|
||||
"""Test that render-video worker endpoint exists."""
|
||||
# Create a job
|
||||
@@ -340,9 +340,9 @@ class TestRenderVideoWorker:
|
||||
json={"url": "https://youtube.com/watch?v=render-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
|
||||
# Test render-video worker endpoint exists
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
|
||||
@@ -798,6 +798,254 @@ class TestRemoteDownloadPath:
|
||||
assert result.filepath == "/tmp/test.flac"
|
||||
|
||||
|
||||
class TestAudioSearchThemeSupport:
|
||||
"""Test theme support in audio search requests.
|
||||
|
||||
These tests verify that theme_id and color_overrides are properly:
|
||||
1. Accepted in AudioSearchRequest model
|
||||
2. Passed through to JobCreate
|
||||
3. Result in correct CDG/TXT defaults when theme is set
|
||||
4. Theme style is PREPARED (copied to job folder) when theme_id is set
|
||||
"""
|
||||
|
||||
def test_audio_search_request_accepts_theme_id(self):
|
||||
"""Test AudioSearchRequest model has theme_id field."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad"
|
||||
)
|
||||
|
||||
assert request.theme_id == "nomad"
|
||||
|
||||
def test_audio_search_request_accepts_color_overrides(self):
|
||||
"""Test AudioSearchRequest model has color_overrides field."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
color_overrides={
|
||||
"artist_color": "#ff0000",
|
||||
"title_color": "#00ff00",
|
||||
}
|
||||
)
|
||||
|
||||
assert request.color_overrides["artist_color"] == "#ff0000"
|
||||
assert request.color_overrides["title_color"] == "#00ff00"
|
||||
|
||||
def test_audio_search_request_theme_defaults_cdg_txt(self):
|
||||
"""Test that when theme_id is set, CDG/TXT defaults to enabled.
|
||||
|
||||
This is the key behavior: selecting a theme should automatically
|
||||
enable CDG and TXT output formats.
|
||||
"""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
|
||||
|
||||
# When theme_id is set, enable_cdg/enable_txt should default to True
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad"
|
||||
# enable_cdg and enable_txt are None (not specified)
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
assert resolved_cdg is True, "CDG should be enabled by default when theme is set"
|
||||
assert resolved_txt is True, "TXT should be enabled by default when theme is set"
|
||||
|
||||
def test_audio_search_request_no_theme_no_cdg_txt(self):
|
||||
"""Test that without theme_id, CDG/TXT defaults to disabled."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
|
||||
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
# No theme_id, no enable_cdg, no enable_txt
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
assert resolved_cdg is False, "CDG should be disabled by default without theme"
|
||||
assert resolved_txt is False, "TXT should be disabled by default without theme"
|
||||
|
||||
def test_explicit_cdg_txt_overrides_theme_default(self):
|
||||
"""Test that explicit enable_cdg/enable_txt values override theme defaults."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
|
||||
|
||||
# Theme set (would default to True), but explicitly disabled
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
enable_cdg=False,
|
||||
enable_txt=False,
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
assert resolved_cdg is False, "Explicit False should override theme default"
|
||||
assert resolved_txt is False, "Explicit False should override theme default"
|
||||
|
||||
def test_explicit_cdg_txt_enables_without_theme(self):
|
||||
"""Test that explicit True enables CDG/TXT even without theme."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
|
||||
|
||||
# No theme (would default to False), but explicitly enabled
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
enable_cdg=True,
|
||||
enable_txt=True,
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
assert resolved_cdg is True, "Explicit True should enable CDG without theme"
|
||||
assert resolved_txt is True, "Explicit True should enable TXT without theme"
|
||||
|
||||
def test_job_create_receives_theme_from_audio_search(self):
|
||||
"""Test that JobCreate model can receive theme_id and color_overrides."""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
color_overrides={
|
||||
"artist_color": "#ff0000",
|
||||
"sung_lyrics_color": "#00ff00"
|
||||
},
|
||||
enable_cdg=True,
|
||||
enable_txt=True,
|
||||
)
|
||||
|
||||
assert job_create.theme_id == "nomad"
|
||||
assert job_create.color_overrides["artist_color"] == "#ff0000"
|
||||
assert job_create.color_overrides["sung_lyrics_color"] == "#00ff00"
|
||||
assert job_create.enable_cdg is True
|
||||
assert job_create.enable_txt is True
|
||||
|
||||
def test_job_model_stores_theme_configuration(self):
|
||||
"""Test Job model stores theme configuration correctly."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.utcnow(),
|
||||
updated_at=datetime.utcnow(),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
color_overrides={
|
||||
"artist_color": "#ff0000"
|
||||
},
|
||||
enable_cdg=True,
|
||||
enable_txt=True,
|
||||
)
|
||||
|
||||
assert job.theme_id == "nomad"
|
||||
assert job.color_overrides["artist_color"] == "#ff0000"
|
||||
assert job.enable_cdg is True
|
||||
assert job.enable_txt is True
|
||||
|
||||
|
||||
class TestAudioSearchThemePreparation:
|
||||
"""Test that audio search endpoint properly prepares theme styles.
|
||||
|
||||
CRITICAL: This test class was added to catch the bug where audio_search.py
|
||||
accepted theme_id but never called prepare_job_style(), resulting in jobs
|
||||
with theme_id set but no style_params_gcs_path or style_assets.
|
||||
|
||||
The symptom: Preview videos show black background instead of themed background.
|
||||
Job ID ffb0b8fa in production demonstrated this issue.
|
||||
"""
|
||||
|
||||
def test_audio_search_with_theme_calls_prepare_job_style(self):
|
||||
"""Test that creating a job via audio search with theme_id prepares the style.
|
||||
|
||||
CRITICAL: This test verifies that when a job is created via the audio-search
|
||||
endpoint with a theme_id (and no custom style files), the code calls
|
||||
prepare_job_style() to set:
|
||||
1. style_params_gcs_path (pointing to the copied style_params.json)
|
||||
2. style_assets (populated with asset mappings)
|
||||
|
||||
Without this, LyricsTranscriber won't have access to the theme's styles
|
||||
and preview videos will have black backgrounds instead of themed ones.
|
||||
"""
|
||||
# Import the endpoint module to access internal functions
|
||||
from backend.api.routes import audio_search as audio_search_module
|
||||
|
||||
# Check if there's an import of prepare_job_style from theme_service
|
||||
source_code_path = audio_search_module.__file__
|
||||
with open(source_code_path, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
# Check for either:
|
||||
# 1. Import of _prepare_theme_for_job from file_upload
|
||||
# 2. Import of prepare_job_style from theme_service
|
||||
# 3. Inline call to theme_service.prepare_job_style
|
||||
has_theme_prep_import = (
|
||||
'_prepare_theme_for_job' in source_code or
|
||||
'prepare_job_style' in source_code
|
||||
)
|
||||
|
||||
assert has_theme_prep_import, (
|
||||
"audio_search.py does not import or call prepare_job_style() or _prepare_theme_for_job(). "
|
||||
"When theme_id is provided without custom style files, the endpoint MUST call "
|
||||
"prepare_job_style() to copy the theme's style_params.json to the job folder. "
|
||||
"Without this, LyricsTranscriber won't have style configuration and preview "
|
||||
"videos will have black backgrounds instead of themed ones."
|
||||
)
|
||||
|
||||
def test_audio_search_code_handles_theme_preparation(self):
|
||||
"""Verify audio_search.py has theme preparation logic similar to file_upload.py.
|
||||
|
||||
This test compares audio_search.py with file_upload.py to ensure they have
|
||||
similar theme preparation patterns. file_upload.py correctly prepares themes,
|
||||
audio_search.py should do the same.
|
||||
"""
|
||||
from backend.api.routes import audio_search as audio_search_module
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
# Read source code of both modules
|
||||
with open(audio_search_module.__file__, 'r') as f:
|
||||
audio_search_code = f.read()
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
file_upload_code = f.read()
|
||||
|
||||
# file_upload.py has this pattern for theme preparation:
|
||||
# if body.theme_id and not has_style_params_upload:
|
||||
# style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(...)
|
||||
assert '_prepare_theme_for_job' in file_upload_code, \
|
||||
"file_upload.py should have _prepare_theme_for_job function"
|
||||
|
||||
# audio_search.py should have similar pattern
|
||||
# Look for either the function or a similar conditional check
|
||||
# STRICT CHECK: Look for actual theme preparation logic being CALLED, not just mentioned
|
||||
has_theme_preparation_call = (
|
||||
# Check for import or call of theme preparation function
|
||||
'_prepare_theme_for_job(' in audio_search_code or
|
||||
'prepare_job_style(' in audio_search_code
|
||||
)
|
||||
|
||||
assert has_theme_preparation_call, (
|
||||
"audio_search.py does not CALL theme preparation logic. "
|
||||
"It accepts theme_id but never prepares the theme style like file_upload.py does. "
|
||||
"Add a call to _prepare_theme_for_job() when theme_id is set and no style files are uploaded."
|
||||
)
|
||||
|
||||
|
||||
class TestFlacfetchIntegration:
|
||||
"""
|
||||
Integration tests that verify flacfetch imports work correctly.
|
||||
|
||||
@@ -1123,9 +1123,9 @@ class TestFinaliseOnlyModels:
|
||||
]
|
||||
)
|
||||
|
||||
# CDG/TXT disabled by default (requires style config)
|
||||
assert request.enable_cdg is False
|
||||
assert request.enable_txt is False
|
||||
# CDG/TXT default to None (server resolves based on theme_id)
|
||||
assert request.enable_cdg is None
|
||||
assert request.enable_txt is None
|
||||
assert request.brand_prefix is None
|
||||
assert request.keep_brand_code is None
|
||||
assert request.enable_youtube_upload is False
|
||||
@@ -1275,9 +1275,9 @@ class TestCreateJobFromUrlRequest:
|
||||
assert request.url == "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
|
||||
assert request.artist is None
|
||||
assert request.title is None
|
||||
# CDG/TXT disabled by default (requires style config)
|
||||
assert request.enable_cdg is False
|
||||
assert request.enable_txt is False
|
||||
# CDG/TXT default to None (server resolves based on theme_id)
|
||||
assert request.enable_cdg is None
|
||||
assert request.enable_txt is None
|
||||
|
||||
def test_create_with_artist_and_title(self):
|
||||
"""Test creating request with URL, artist, and title."""
|
||||
|
||||
@@ -0,0 +1,516 @@
|
||||
"""
|
||||
Tests for the theme service and API endpoints.
|
||||
|
||||
Tests theme listing, retrieval, color override application, and job style preparation.
|
||||
"""
|
||||
import json
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, Mock
|
||||
from datetime import datetime
|
||||
|
||||
from backend.models.theme import (
|
||||
ThemeSummary,
|
||||
ThemeDetail,
|
||||
ThemeMetadata,
|
||||
ThemeRegistry,
|
||||
ColorOverrides,
|
||||
hex_to_rgba,
|
||||
rgba_to_hex,
|
||||
)
|
||||
from backend.services.theme_service import ThemeService, get_theme_service
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Color Conversion Tests
|
||||
# =============================================================================
|
||||
|
||||
class TestColorConversion:
|
||||
"""Tests for hex/rgba color conversion utilities."""
|
||||
|
||||
def test_hex_to_rgba_basic(self):
|
||||
"""Test basic hex to RGBA conversion."""
|
||||
result = hex_to_rgba("#7070F7")
|
||||
assert result == "112, 112, 247, 255"
|
||||
|
||||
def test_hex_to_rgba_with_custom_alpha(self):
|
||||
"""Test hex to RGBA with custom alpha."""
|
||||
result = hex_to_rgba("#FF0000", alpha=128)
|
||||
assert result == "255, 0, 0, 128"
|
||||
|
||||
def test_hex_to_rgba_lowercase(self):
|
||||
"""Test hex with lowercase letters."""
|
||||
result = hex_to_rgba("#abcdef")
|
||||
assert result == "171, 205, 239, 255"
|
||||
|
||||
def test_hex_to_rgba_without_hash(self):
|
||||
"""Test hex without leading hash."""
|
||||
result = hex_to_rgba("FFFFFF")
|
||||
assert result == "255, 255, 255, 255"
|
||||
|
||||
def test_rgba_to_hex_basic(self):
|
||||
"""Test basic RGBA to hex conversion."""
|
||||
result = rgba_to_hex("112, 112, 247, 255")
|
||||
assert result == "#7070f7"
|
||||
|
||||
def test_rgba_to_hex_ignores_alpha(self):
|
||||
"""Test that alpha is ignored in conversion."""
|
||||
result = rgba_to_hex("255, 0, 0, 128")
|
||||
assert result == "#ff0000"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# ColorOverrides Model Tests
|
||||
# =============================================================================
|
||||
|
||||
class TestColorOverrides:
|
||||
"""Tests for ColorOverrides model."""
|
||||
|
||||
def test_has_overrides_empty(self):
|
||||
"""Test has_overrides with no overrides set."""
|
||||
overrides = ColorOverrides()
|
||||
assert overrides.has_overrides() is False
|
||||
|
||||
def test_has_overrides_with_artist_color(self):
|
||||
"""Test has_overrides with artist_color set."""
|
||||
overrides = ColorOverrides(artist_color="#FF0000")
|
||||
assert overrides.has_overrides() is True
|
||||
|
||||
def test_has_overrides_with_all_colors(self):
|
||||
"""Test has_overrides with all colors set."""
|
||||
overrides = ColorOverrides(
|
||||
artist_color="#FF0000",
|
||||
title_color="#00FF00",
|
||||
sung_lyrics_color="#0000FF",
|
||||
unsung_lyrics_color="#FFFF00",
|
||||
)
|
||||
assert overrides.has_overrides() is True
|
||||
|
||||
def test_to_dict_excludes_none(self):
|
||||
"""Test to_dict excludes None values."""
|
||||
overrides = ColorOverrides(artist_color="#FF0000")
|
||||
result = overrides.to_dict()
|
||||
assert result == {"artist_color": "#FF0000"}
|
||||
assert "title_color" not in result
|
||||
|
||||
def test_color_validation_valid(self):
|
||||
"""Test color validation accepts valid hex colors."""
|
||||
overrides = ColorOverrides(artist_color="#AbCdEf")
|
||||
assert overrides.artist_color == "#AbCdEf"
|
||||
|
||||
def test_color_validation_invalid(self):
|
||||
"""Test color validation rejects invalid colors."""
|
||||
with pytest.raises(ValueError):
|
||||
ColorOverrides(artist_color="red")
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
ColorOverrides(artist_color="#GGG")
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# ThemeService Tests
|
||||
# =============================================================================
|
||||
|
||||
class TestThemeService:
|
||||
"""Tests for ThemeService."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage(self):
|
||||
"""Create a mock storage service."""
|
||||
storage = MagicMock()
|
||||
return storage
|
||||
|
||||
@pytest.fixture
|
||||
def sample_metadata(self):
|
||||
"""Sample theme metadata for testing."""
|
||||
return {
|
||||
"version": 1,
|
||||
"themes": [
|
||||
{
|
||||
"id": "nomad",
|
||||
"name": "Nomad Karaoke",
|
||||
"description": "Golden artist text, professional look",
|
||||
"is_default": True,
|
||||
},
|
||||
{
|
||||
"id": "default",
|
||||
"name": "Default",
|
||||
"description": "Clean white text on black background",
|
||||
"is_default": False,
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
@pytest.fixture
|
||||
def sample_style_params(self):
|
||||
"""Sample style params for testing."""
|
||||
return {
|
||||
"intro": {
|
||||
"artist_color": "#ffdf6b",
|
||||
"title_color": "#ffffff",
|
||||
"background_image": "intro_bg.png",
|
||||
"font": "Montserrat-Bold.ttf",
|
||||
},
|
||||
"end": {
|
||||
"artist_color": "#ffdf6b",
|
||||
"title_color": "#ffffff",
|
||||
"background_image": "end_bg.png",
|
||||
"font": "Montserrat-Bold.ttf",
|
||||
},
|
||||
"karaoke": {
|
||||
"primary_color": "112, 112, 247, 255",
|
||||
"secondary_color": "255, 255, 255, 255",
|
||||
"background_image": "karaoke_bg.png",
|
||||
"font_path": "Montserrat-Bold.ttf",
|
||||
},
|
||||
"cdg": {
|
||||
"active_fill": "#7070F7",
|
||||
"inactive_fill": "#FFFFFF",
|
||||
"font_path": "Montserrat-Bold.ttf",
|
||||
},
|
||||
}
|
||||
|
||||
def test_list_themes_returns_summaries(self, mock_storage, sample_metadata):
|
||||
"""Test list_themes returns ThemeSummary objects."""
|
||||
mock_storage.download_json.return_value = sample_metadata
|
||||
mock_storage.file_exists.return_value = True
|
||||
mock_storage.generate_signed_url.return_value = "https://signed-url.com/preview.png"
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
themes = service.list_themes()
|
||||
|
||||
assert len(themes) == 2
|
||||
assert themes[0].id == "nomad"
|
||||
assert themes[0].name == "Nomad Karaoke"
|
||||
assert themes[0].is_default is True
|
||||
assert themes[0].preview_url is not None
|
||||
|
||||
def test_list_themes_handles_missing_preview(self, mock_storage, sample_metadata):
|
||||
"""Test list_themes handles missing preview images gracefully."""
|
||||
mock_storage.download_json.return_value = sample_metadata
|
||||
mock_storage.file_exists.return_value = False
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
themes = service.list_themes()
|
||||
|
||||
assert len(themes) == 2
|
||||
assert themes[0].preview_url is None
|
||||
|
||||
def test_get_theme_returns_detail(self, mock_storage, sample_metadata, sample_style_params):
|
||||
"""Test get_theme returns ThemeDetail with style params."""
|
||||
mock_storage.download_json.side_effect = [sample_metadata, sample_style_params]
|
||||
mock_storage.file_exists.return_value = True
|
||||
mock_storage.generate_signed_url.return_value = "https://signed-url.com/preview.png"
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
theme = service.get_theme("nomad")
|
||||
|
||||
assert theme is not None
|
||||
assert theme.id == "nomad"
|
||||
assert theme.name == "Nomad Karaoke"
|
||||
assert "intro" in theme.style_params
|
||||
assert theme.style_params["intro"]["artist_color"] == "#ffdf6b"
|
||||
|
||||
def test_get_theme_returns_none_for_unknown(self, mock_storage, sample_metadata):
|
||||
"""Test get_theme returns None for unknown theme ID."""
|
||||
mock_storage.download_json.return_value = sample_metadata
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
theme = service.get_theme("unknown-theme")
|
||||
|
||||
assert theme is None
|
||||
|
||||
def test_theme_exists_true(self, mock_storage, sample_metadata):
|
||||
"""Test theme_exists returns True for existing theme."""
|
||||
mock_storage.download_json.return_value = sample_metadata
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
assert service.theme_exists("nomad") is True
|
||||
|
||||
def test_theme_exists_false(self, mock_storage, sample_metadata):
|
||||
"""Test theme_exists returns False for unknown theme."""
|
||||
mock_storage.download_json.return_value = sample_metadata
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
assert service.theme_exists("unknown") is False
|
||||
|
||||
def test_get_default_theme_id(self, mock_storage, sample_metadata):
|
||||
"""Test get_default_theme_id returns the default theme."""
|
||||
mock_storage.download_json.return_value = sample_metadata
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
assert service.get_default_theme_id() == "nomad"
|
||||
|
||||
def test_get_default_theme_id_no_default(self, mock_storage):
|
||||
"""Test get_default_theme_id returns None when no default."""
|
||||
mock_storage.download_json.return_value = {
|
||||
"version": 1,
|
||||
"themes": [{"id": "test", "name": "Test", "description": "Test", "is_default": False}],
|
||||
}
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
assert service.get_default_theme_id() is None
|
||||
|
||||
def test_apply_color_overrides_no_changes(self, sample_style_params):
|
||||
"""Test apply_color_overrides with empty overrides."""
|
||||
service = ThemeService()
|
||||
overrides = ColorOverrides()
|
||||
|
||||
result = service.apply_color_overrides(sample_style_params, overrides)
|
||||
|
||||
# Should return original (no deep copy needed when no changes)
|
||||
assert result == sample_style_params
|
||||
|
||||
def test_apply_color_overrides_artist_color(self, sample_style_params):
|
||||
"""Test apply_color_overrides applies artist_color."""
|
||||
service = ThemeService()
|
||||
overrides = ColorOverrides(artist_color="#FF0000")
|
||||
|
||||
result = service.apply_color_overrides(sample_style_params, overrides)
|
||||
|
||||
assert result["intro"]["artist_color"] == "#FF0000"
|
||||
assert result["end"]["artist_color"] == "#FF0000"
|
||||
assert result["cdg"]["artist_color"] == "#FF0000"
|
||||
# Original should be unchanged
|
||||
assert sample_style_params["intro"]["artist_color"] == "#ffdf6b"
|
||||
|
||||
def test_apply_color_overrides_sung_lyrics_color(self, sample_style_params):
|
||||
"""Test apply_color_overrides applies sung_lyrics_color with conversion."""
|
||||
service = ThemeService()
|
||||
overrides = ColorOverrides(sung_lyrics_color="#FF0000")
|
||||
|
||||
result = service.apply_color_overrides(sample_style_params, overrides)
|
||||
|
||||
# Karaoke uses RGBA format
|
||||
assert result["karaoke"]["primary_color"] == "255, 0, 0, 255"
|
||||
# CDG uses hex
|
||||
assert result["cdg"]["active_fill"] == "#FF0000"
|
||||
|
||||
def test_apply_color_overrides_unsung_lyrics_color(self, sample_style_params):
|
||||
"""Test apply_color_overrides applies unsung_lyrics_color."""
|
||||
service = ThemeService()
|
||||
overrides = ColorOverrides(unsung_lyrics_color="#00FF00")
|
||||
|
||||
result = service.apply_color_overrides(sample_style_params, overrides)
|
||||
|
||||
assert result["karaoke"]["secondary_color"] == "0, 255, 0, 255"
|
||||
assert result["cdg"]["inactive_fill"] == "#00FF00"
|
||||
|
||||
def test_metadata_cache(self, mock_storage, sample_metadata):
|
||||
"""Test metadata caching works."""
|
||||
mock_storage.download_json.return_value = sample_metadata
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
|
||||
# First call loads from GCS
|
||||
service.list_themes()
|
||||
assert mock_storage.download_json.call_count == 1
|
||||
|
||||
# Second call uses cache
|
||||
service.list_themes()
|
||||
assert mock_storage.download_json.call_count == 1
|
||||
|
||||
def test_invalidate_cache(self, mock_storage, sample_metadata):
|
||||
"""Test cache invalidation."""
|
||||
mock_storage.download_json.return_value = sample_metadata
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
|
||||
# Load metadata
|
||||
service.list_themes()
|
||||
assert mock_storage.download_json.call_count == 1
|
||||
|
||||
# Invalidate cache
|
||||
service.invalidate_cache()
|
||||
|
||||
# Next call should reload
|
||||
service.list_themes()
|
||||
assert mock_storage.download_json.call_count == 2
|
||||
|
||||
def test_prepare_job_style(self, mock_storage, sample_metadata, sample_style_params):
|
||||
"""Test prepare_job_style creates job style from theme."""
|
||||
mock_storage.download_json.side_effect = [sample_metadata, sample_style_params]
|
||||
mock_storage.file_exists.return_value = True
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
style_path, style_assets = service.prepare_job_style("job123", "nomad")
|
||||
|
||||
# Should upload modified style_params.json
|
||||
mock_storage.upload_json.assert_called_once()
|
||||
assert style_path == "uploads/job123/style/style_params.json"
|
||||
|
||||
def test_prepare_job_style_with_overrides(self, mock_storage, sample_metadata, sample_style_params):
|
||||
"""Test prepare_job_style applies color overrides."""
|
||||
# prepare_job_style calls get_theme_style_params which directly downloads style_params.json
|
||||
# No metadata lookup happens unless we've cached it previously
|
||||
mock_storage.download_json.return_value = sample_style_params
|
||||
mock_storage.file_exists.return_value = True
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
overrides = ColorOverrides(artist_color="#FF0000")
|
||||
|
||||
style_path, style_assets = service.prepare_job_style("job123", "nomad", overrides)
|
||||
|
||||
# Check that upload_json was called with modified style params
|
||||
call_args = mock_storage.upload_json.call_args
|
||||
uploaded_path = call_args[0][0] # First positional arg is the path
|
||||
uploaded_style = call_args[0][1] # Second positional arg is the data
|
||||
|
||||
assert uploaded_path == "uploads/job123/style/style_params.json"
|
||||
# Check that artist_color override was applied
|
||||
assert uploaded_style["intro"]["artist_color"] == "#FF0000"
|
||||
assert uploaded_style["end"]["artist_color"] == "#FF0000"
|
||||
|
||||
def test_prepare_job_style_unknown_theme_raises(self, mock_storage, sample_metadata):
|
||||
"""Test prepare_job_style raises for unknown theme."""
|
||||
# First download_json returns metadata (no unknown theme)
|
||||
# Second download_json for style_params.json should fail
|
||||
def download_json_side_effect(path):
|
||||
if "_metadata.json" in path:
|
||||
return sample_metadata
|
||||
# Unknown theme path - raise exception
|
||||
raise Exception(f"File not found: {path}")
|
||||
|
||||
mock_storage.download_json.side_effect = download_json_side_effect
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
|
||||
with pytest.raises(ValueError, match="Theme not found"):
|
||||
service.prepare_job_style("job123", "unknown-theme")
|
||||
|
||||
def test_get_youtube_description(self, mock_storage, sample_metadata):
|
||||
"""Test get_youtube_description returns template text."""
|
||||
mock_storage.download_json.return_value = sample_metadata
|
||||
mock_storage.file_exists.return_value = True
|
||||
|
||||
# Mock the download_file to write content to temp file
|
||||
def mock_download(gcs_path, local_path):
|
||||
with open(local_path, "w") as f:
|
||||
f.write("Thank you for watching!")
|
||||
|
||||
mock_storage.download_file.side_effect = mock_download
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
desc = service.get_youtube_description("nomad")
|
||||
|
||||
assert desc == "Thank you for watching!"
|
||||
|
||||
def test_get_youtube_description_not_found(self, mock_storage, sample_metadata):
|
||||
"""Test get_youtube_description returns None when no template."""
|
||||
mock_storage.download_json.return_value = sample_metadata
|
||||
mock_storage.file_exists.return_value = False
|
||||
|
||||
service = ThemeService(storage=mock_storage)
|
||||
desc = service.get_youtube_description("nomad")
|
||||
|
||||
assert desc is None
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# API Endpoint Tests
|
||||
# =============================================================================
|
||||
|
||||
class TestThemeAPI:
|
||||
"""Tests for theme API endpoints."""
|
||||
|
||||
@pytest.fixture
|
||||
def sample_metadata(self):
|
||||
"""Sample theme metadata."""
|
||||
return {
|
||||
"version": 1,
|
||||
"themes": [
|
||||
{
|
||||
"id": "nomad",
|
||||
"name": "Nomad Karaoke",
|
||||
"description": "Golden artist text",
|
||||
"is_default": True,
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
def test_list_themes_endpoint(self, test_client, sample_metadata):
|
||||
"""Test GET /api/themes returns theme list."""
|
||||
with patch("backend.api.routes.themes.get_theme_service") as mock_get_service:
|
||||
mock_service = MagicMock()
|
||||
mock_service.list_themes.return_value = [
|
||||
ThemeSummary(
|
||||
id="nomad",
|
||||
name="Nomad Karaoke",
|
||||
description="Golden artist text",
|
||||
preview_url="https://example.com/preview.png",
|
||||
is_default=True,
|
||||
)
|
||||
]
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
response = test_client.get("/api/themes")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert len(data["themes"]) == 1
|
||||
assert data["themes"][0]["id"] == "nomad"
|
||||
assert data["themes"][0]["is_default"] is True
|
||||
|
||||
def test_get_theme_endpoint(self, test_client):
|
||||
"""Test GET /api/themes/{theme_id} returns theme detail."""
|
||||
with patch("backend.api.routes.themes.get_theme_service") as mock_get_service:
|
||||
mock_service = MagicMock()
|
||||
mock_service.get_theme.return_value = ThemeDetail(
|
||||
id="nomad",
|
||||
name="Nomad Karaoke",
|
||||
description="Golden artist text",
|
||||
is_default=True,
|
||||
style_params={"intro": {"artist_color": "#ffdf6b"}},
|
||||
)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
response = test_client.get("/api/themes/nomad")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["theme"]["id"] == "nomad"
|
||||
assert "style_params" in data["theme"]
|
||||
|
||||
def test_get_theme_not_found(self, test_client):
|
||||
"""Test GET /api/themes/{theme_id} returns 404 for unknown theme."""
|
||||
with patch("backend.api.routes.themes.get_theme_service") as mock_get_service:
|
||||
mock_service = MagicMock()
|
||||
mock_service.get_theme.return_value = None
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
response = test_client.get("/api/themes/unknown")
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_get_theme_preview_endpoint(self, test_client):
|
||||
"""Test GET /api/themes/{theme_id}/preview returns preview URL."""
|
||||
with patch("backend.api.routes.themes.get_theme_service") as mock_get_service:
|
||||
mock_service = MagicMock()
|
||||
mock_service.theme_exists.return_value = True
|
||||
mock_service.get_theme.return_value = ThemeDetail(
|
||||
id="nomad",
|
||||
name="Nomad Karaoke",
|
||||
description="Golden artist text",
|
||||
preview_url="https://example.com/preview.png",
|
||||
is_default=True,
|
||||
)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
response = test_client.get("/api/themes/nomad/preview")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "preview_url" in data
|
||||
|
||||
def test_get_youtube_description_endpoint(self, test_client):
|
||||
"""Test GET /api/themes/{theme_id}/youtube-description returns description."""
|
||||
with patch("backend.api.routes.themes.get_theme_service") as mock_get_service:
|
||||
mock_service = MagicMock()
|
||||
mock_service.theme_exists.return_value = True
|
||||
mock_service.get_youtube_description.return_value = "Thank you for watching!"
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
response = test_client.get("/api/themes/nomad/youtube-description")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["description"] == "Thank you for watching!"
|
||||
+183
@@ -0,0 +1,183 @@
|
||||
# API Reference
|
||||
|
||||
Base URL: `https://api.nomadkaraoke.com` (production) or `http://localhost:8000` (local)
|
||||
|
||||
## Authentication
|
||||
|
||||
All endpoints except `/health` and `/api/themes` require authentication.
|
||||
|
||||
```bash
|
||||
curl -H "Authorization: Bearer YOUR_TOKEN" ...
|
||||
```
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Health
|
||||
|
||||
```http
|
||||
GET /health
|
||||
```
|
||||
|
||||
No auth required. Returns service status.
|
||||
|
||||
### Jobs
|
||||
|
||||
#### Create Job with Upload
|
||||
|
||||
```http
|
||||
POST /api/jobs/upload
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
file: <audio file>
|
||||
artist: "Artist Name"
|
||||
title: "Song Title"
|
||||
```
|
||||
|
||||
Returns job ID. Triggers async processing.
|
||||
|
||||
#### Get Job Status
|
||||
|
||||
```http
|
||||
GET /api/jobs/{job_id}
|
||||
```
|
||||
|
||||
Response includes:
|
||||
- `status` - Current job state
|
||||
- `file_urls` - Signed URLs for outputs (when ready)
|
||||
- `state_data` - Worker progress details
|
||||
- `timeline` - Processing history
|
||||
|
||||
#### List Jobs
|
||||
|
||||
```http
|
||||
GET /api/jobs
|
||||
GET /api/jobs?status=complete
|
||||
GET /api/jobs?limit=10&offset=0
|
||||
```
|
||||
|
||||
#### Delete Job
|
||||
|
||||
```http
|
||||
DELETE /api/jobs/{job_id}
|
||||
```
|
||||
|
||||
### Review
|
||||
|
||||
#### Get Correction Data
|
||||
|
||||
```http
|
||||
GET /api/review/{job_id}/correction-data
|
||||
```
|
||||
|
||||
Returns lyrics correction data for the review UI.
|
||||
|
||||
#### Complete Review
|
||||
|
||||
```http
|
||||
POST /api/review/{job_id}/complete
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"corrections": [...],
|
||||
"corrected_segments": [...]
|
||||
}
|
||||
```
|
||||
|
||||
Saves corrections and triggers video rendering.
|
||||
|
||||
#### Generate Preview
|
||||
|
||||
```http
|
||||
POST /api/review/{job_id}/preview-video
|
||||
```
|
||||
|
||||
Generates preview video during review.
|
||||
|
||||
#### Stream Audio
|
||||
|
||||
```http
|
||||
GET /api/review/{job_id}/audio/{stem_type}
|
||||
```
|
||||
|
||||
Streams audio for review playback.
|
||||
|
||||
### Instrumental Selection
|
||||
|
||||
```http
|
||||
POST /api/jobs/{job_id}/select-instrumental
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"selection": "clean" // or "with_backing"
|
||||
}
|
||||
```
|
||||
|
||||
### Audio Search
|
||||
|
||||
```http
|
||||
GET /api/audio-search?q=song+name
|
||||
```
|
||||
|
||||
Search for songs (flacfetch integration).
|
||||
|
||||
### Themes
|
||||
|
||||
```http
|
||||
GET /api/themes
|
||||
```
|
||||
|
||||
No auth required. Returns available video themes.
|
||||
|
||||
### Internal (Admin Only)
|
||||
|
||||
These endpoints are used by workers and require admin tokens.
|
||||
|
||||
```http
|
||||
POST /api/internal/jobs/{job_id}/trigger-audio
|
||||
POST /api/internal/jobs/{job_id}/trigger-lyrics
|
||||
POST /api/internal/jobs/{job_id}/trigger-screens
|
||||
POST /api/internal/jobs/{job_id}/trigger-render-video
|
||||
POST /api/internal/jobs/{job_id}/trigger-video
|
||||
```
|
||||
|
||||
## Job States
|
||||
|
||||
| State | Description |
|
||||
|-------|-------------|
|
||||
| `pending` | Created, not started |
|
||||
| `downloading` | Processing input |
|
||||
| `separating_stage1` | Audio separation (1/2) |
|
||||
| `separating_stage2` | Audio separation (2/2) |
|
||||
| `transcribing` | Lyrics transcription |
|
||||
| `generating_screens` | Title/end screens |
|
||||
| `awaiting_review` | Waiting for human review |
|
||||
| `in_review` | Human reviewing |
|
||||
| `review_complete` | Review submitted |
|
||||
| `rendering_video` | Generating karaoke video |
|
||||
| `awaiting_instrumental_selection` | Waiting for selection |
|
||||
| `instrumental_selected` | Selection made |
|
||||
| `generating_video` | Final encoding |
|
||||
| `complete` | Done |
|
||||
| `failed` | Error |
|
||||
|
||||
## Error Responses
|
||||
|
||||
```json
|
||||
{
|
||||
"detail": "Error message"
|
||||
}
|
||||
```
|
||||
|
||||
Common status codes:
|
||||
- `400` - Bad request
|
||||
- `401` - Unauthorized
|
||||
- `404` - Job not found
|
||||
- `500` - Server error
|
||||
|
||||
## Rate Limits
|
||||
|
||||
No rate limits currently implemented.
|
||||
|
||||
## Webhooks
|
||||
|
||||
Not yet implemented. Jobs must be polled for status.
|
||||
@@ -0,0 +1,162 @@
|
||||
# Architecture
|
||||
|
||||
## System Overview
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Frontend (Cloudflare Pages) │
|
||||
│ Next.js + TypeScript │
|
||||
│ https://gen.nomadkaraoke.com │
|
||||
└───────────────────────────┬─────────────────────────────────────┘
|
||||
│ HTTPS
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Backend API (Cloud Run) │
|
||||
│ FastAPI + Python 3.12 │
|
||||
│ https://api.nomadkaraoke.com │
|
||||
│ │
|
||||
│ Workers: │
|
||||
│ • Audio Worker - Stem separation via Modal API │
|
||||
│ • Lyrics Worker - Transcription via AudioShake │
|
||||
│ • Screens Worker - Title/end screen generation │
|
||||
│ • Render Worker - Karaoke video with lyrics │
|
||||
│ • Video Worker - Final assembly and encoding │
|
||||
└───────────────────────────┬─────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────┼─────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────────┐
|
||||
│ Firestore│ │ GCS │ │Secret Manager│
|
||||
│ (jobs) │ │ (files) │ │ (API keys) │
|
||||
└──────────┘ └──────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
## Processing Pipeline
|
||||
|
||||
```text
|
||||
1. UPLOAD
|
||||
POST /api/jobs/upload
|
||||
• Save audio to GCS
|
||||
• Create job in Firestore
|
||||
• Trigger parallel workers
|
||||
|
||||
2. PARALLEL PROCESSING
|
||||
┌─────────────────┐ ┌─────────────────┐
|
||||
│ Audio Worker │ │ Lyrics Worker │
|
||||
│ Modal API │ │ AudioShake │
|
||||
│ 2-stage sep │ │ Auto-correct │
|
||||
└────────┬────────┘ └────────┬────────┘
|
||||
└──────────┬───────────┘
|
||||
▼
|
||||
3. SCREENS GENERATION
|
||||
• Title screen (5s video)
|
||||
• End screen (5s video)
|
||||
▼
|
||||
4. HUMAN REVIEW (AWAITING_REVIEW)
|
||||
• React UI for lyrics correction
|
||||
• Preview video generation
|
||||
• Submit corrections
|
||||
▼
|
||||
5. VIDEO RENDERING
|
||||
• Merge corrections with original data
|
||||
• Generate karaoke video with lyrics
|
||||
▼
|
||||
6. INSTRUMENTAL SELECTION
|
||||
• User chooses: clean or with_backing
|
||||
▼
|
||||
7. FINAL VIDEO
|
||||
• Remux with selected instrumental
|
||||
• Concatenate: title + karaoke + end
|
||||
• Encode to 4 formats
|
||||
▼
|
||||
8. COMPLETE
|
||||
• User downloads via signed URLs
|
||||
```
|
||||
|
||||
## Job States
|
||||
|
||||
| State | Description | Human Action |
|
||||
|-------|-------------|--------------|
|
||||
| `pending` | Job created | - |
|
||||
| `downloading` | Processing audio | - |
|
||||
| `separating_stage1` | Audio separation stage 1 | - |
|
||||
| `separating_stage2` | Audio separation stage 2 | - |
|
||||
| `transcribing` | Lyrics transcription | - |
|
||||
| `generating_screens` | Creating title/end screens | - |
|
||||
| `awaiting_review` | **Waiting for human** | Review lyrics |
|
||||
| `in_review` | Human actively reviewing | Edit lyrics |
|
||||
| `review_complete` | Review submitted | - |
|
||||
| `rendering_video` | Generating karaoke video | - |
|
||||
| `awaiting_instrumental_selection` | **Waiting for human** | Choose audio |
|
||||
| `instrumental_selected` | Selection made | - |
|
||||
| `generating_video` | Final encoding | - |
|
||||
| `complete` | All done | Download files |
|
||||
| `failed` | Error occurred | - |
|
||||
|
||||
## GCS File Structure
|
||||
|
||||
```text
|
||||
jobs/{job_id}/
|
||||
├── input/
|
||||
│ └── original.flac # Uploaded audio
|
||||
├── stems/
|
||||
│ ├── vocals.flac
|
||||
│ ├── instrumental_clean.flac
|
||||
│ ├── instrumental_with_backing.flac
|
||||
│ └── ...
|
||||
├── lyrics/
|
||||
│ ├── corrections.json # Auto-corrected
|
||||
│ ├── corrections_updated.json # After human review
|
||||
│ └── karaoke.ass
|
||||
├── screens/
|
||||
│ ├── title.mov
|
||||
│ └── end.mov
|
||||
├── videos/
|
||||
│ └── with_vocals.mkv # Karaoke video
|
||||
└── finals/
|
||||
├── lossless_4k.mp4
|
||||
├── lossless_4k.mkv
|
||||
├── lossy_4k.mp4
|
||||
└── lossy_720p.mp4
|
||||
```
|
||||
|
||||
## LyricsTranscriber Integration
|
||||
|
||||
**Key Design Decision**: We use LyricsTranscriber as a **library**, not a server.
|
||||
|
||||
```text
|
||||
CLI Mode (Original): Cloud Mode (Our Approach):
|
||||
──────────────────── ──────────────────────────
|
||||
LyricsTranscriber LyricsTranscriber
|
||||
├── transcribe() ├── transcribe() ✅
|
||||
├── correct() ├── correct() ✅
|
||||
├── ReviewServer.start() │
|
||||
│ └── BLOCKS (waits) │ (Skip server, save JSON)
|
||||
│ │ ... ASYNC GAP ...
|
||||
│ │ (Human reviews via React)
|
||||
└── OutputGenerator │
|
||||
└── OutputGenerator ✅
|
||||
(called after review)
|
||||
```
|
||||
|
||||
**Use**: `LyricsTranscriber`, `CorrectionResult`, `OutputGenerator`, `OutputConfig`
|
||||
**Don't use**: `ReviewServer`, blocking waits
|
||||
|
||||
## External Services
|
||||
|
||||
| Service | Purpose | Required |
|
||||
|---------|---------|----------|
|
||||
| Modal | GPU audio separation | Yes |
|
||||
| AudioShake | Lyrics transcription | Yes |
|
||||
| Genius | Reference lyrics | Yes |
|
||||
| YouTube API | Video upload | Optional |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Backend**: FastAPI, Python 3.12, Cloud Run
|
||||
- **Frontend**: Next.js, TypeScript, Cloudflare Pages
|
||||
- **Database**: Firestore
|
||||
- **Storage**: Google Cloud Storage
|
||||
- **Secrets**: Google Secret Manager
|
||||
- **IaC**: Pulumi
|
||||
- **CI/CD**: GitHub Actions
|
||||
@@ -0,0 +1,216 @@
|
||||
# Development Guide
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.12
|
||||
- Node.js 20+
|
||||
- Docker (for GCS emulator)
|
||||
- Google Cloud SDK with Firestore emulator
|
||||
|
||||
```bash
|
||||
# Install Firestore emulator
|
||||
gcloud components install cloud-firestore-emulator
|
||||
```
|
||||
|
||||
### Backend Setup
|
||||
|
||||
```bash
|
||||
# Create and activate venv
|
||||
python3.12 -m venv venv
|
||||
source venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
poetry install
|
||||
|
||||
# Copy environment template
|
||||
cp .env.example .env
|
||||
# Edit .env with required API keys
|
||||
```
|
||||
|
||||
### Frontend Setup
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
cp .env.example .env.local
|
||||
# Edit .env.local with KARAOKE_ACCESS_TOKEN
|
||||
```
|
||||
|
||||
## Running Locally
|
||||
|
||||
### Backend with Emulators
|
||||
|
||||
```bash
|
||||
# Start emulators and backend
|
||||
./scripts/run-backend-local.sh --with-emulators
|
||||
|
||||
# Backend runs at http://localhost:8000
|
||||
# API docs at http://localhost:8000/docs
|
||||
```
|
||||
|
||||
### Frontend
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm run dev
|
||||
# Runs at http://localhost:3000
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Backend Tests
|
||||
|
||||
```bash
|
||||
# Run all tests (required before commit)
|
||||
make test 2>&1 | tail -n 500
|
||||
|
||||
# This runs in order:
|
||||
# 1. Unit tests with coverage
|
||||
# 2. Backend unit tests
|
||||
# 3. E2E integration tests with emulators
|
||||
```
|
||||
|
||||
**Important**: Set 10 minute timeout. Tests can take a while with emulators.
|
||||
|
||||
### Frontend Tests
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm run test:all 2>&1 | tail -n 200
|
||||
|
||||
# Individual commands:
|
||||
npm run test:unit # Jest unit tests
|
||||
npm run test:e2e # Playwright E2E tests
|
||||
```
|
||||
|
||||
### Test Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────┐
|
||||
│ Unit Tests (Mocked) │ Fast, isolated
|
||||
│ ~62 tests, <1s │
|
||||
├─────────────────────────────────┤
|
||||
│ Emulator Integration Tests │ Local Firestore/GCS
|
||||
│ ~11 tests, ~2s │
|
||||
├─────────────────────────────────┤
|
||||
│ E2E Tests │ Full API + Frontend
|
||||
│ Requires deployed backend │
|
||||
└─────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Automatic (CI/CD)
|
||||
|
||||
Push to `main` triggers GitHub Actions:
|
||||
1. Run tests
|
||||
2. Build Docker image
|
||||
3. Deploy to Cloud Run
|
||||
4. Deploy frontend to Cloudflare Pages
|
||||
|
||||
### Manual Backend Deploy
|
||||
|
||||
```bash
|
||||
# Preview infrastructure changes
|
||||
cd infrastructure
|
||||
pulumi preview
|
||||
|
||||
# Apply changes
|
||||
pulumi up
|
||||
```
|
||||
|
||||
### Manual Frontend Deploy
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm run build
|
||||
npx wrangler pages deploy out
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Backend (.env)
|
||||
|
||||
```bash
|
||||
# Required
|
||||
AUDIOSHAKE_API_TOKEN=...
|
||||
GENIUS_ACCESS_TOKEN=...
|
||||
AUDIO_SEPARATOR_API_URL=... # Modal API endpoint
|
||||
|
||||
# GCP (auto-configured in Cloud Run)
|
||||
GOOGLE_CLOUD_PROJECT=...
|
||||
GCS_BUCKET_NAME=...
|
||||
|
||||
# Auth
|
||||
ADMIN_TOKENS=token1,token2
|
||||
```
|
||||
|
||||
### Frontend (.env.local)
|
||||
|
||||
```bash
|
||||
NEXT_PUBLIC_API_URL=http://localhost:8000
|
||||
KARAOKE_ACCESS_TOKEN=... # For E2E tests
|
||||
```
|
||||
|
||||
## Key Scripts
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `make test` | Run all backend tests |
|
||||
| `./scripts/run-backend-local.sh` | Start backend with emulators |
|
||||
| `./scripts/start-emulators.sh` | Start emulators only |
|
||||
| `./scripts/stop-emulators.sh` | Stop emulators |
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
karaoke-gen/
|
||||
├── backend/
|
||||
│ ├── api/routes/ # FastAPI endpoints
|
||||
│ ├── models/ # Pydantic models
|
||||
│ ├── services/ # Business logic
|
||||
│ ├── workers/ # Background workers
|
||||
│ └── tests/ # Backend tests
|
||||
├── frontend/
|
||||
│ ├── src/
|
||||
│ │ ├── app/ # Next.js pages
|
||||
│ │ ├── components/ # React components
|
||||
│ │ └── lib/ # Utilities
|
||||
│ └── tests/ # Frontend tests
|
||||
├── infrastructure/ # Pulumi IaC
|
||||
├── karaoke_gen/ # CLI package
|
||||
└── scripts/ # Dev scripts
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Emulators Won't Start
|
||||
|
||||
```bash
|
||||
# Check if already running
|
||||
lsof -i :8080 # Firestore
|
||||
lsof -i :4443 # GCS
|
||||
|
||||
# Stop them
|
||||
./scripts/stop-emulators.sh
|
||||
```
|
||||
|
||||
### Tests Fail with Auth Errors
|
||||
|
||||
Ensure `ADMIN_TOKENS` is set in environment or `.env` file.
|
||||
|
||||
### Docker Issues
|
||||
|
||||
```bash
|
||||
# Ensure Docker is running
|
||||
docker info
|
||||
|
||||
# For GCS emulator
|
||||
docker pull fsouza/fake-gcs-server
|
||||
```
|
||||
|
||||
### Coverage Below Target
|
||||
|
||||
Run `./scripts/run-tests.sh --coverage` and check `htmlcov/index.html` to see uncovered lines.
|
||||
@@ -0,0 +1,164 @@
|
||||
# Lessons Learned
|
||||
|
||||
Key insights for future AI agents working on this codebase.
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
### LyricsTranscriber: Library, Not Server
|
||||
|
||||
**Problem**: LyricsTranscriber's `ReviewServer` blocks waiting for human input - incompatible with async cloud architecture.
|
||||
|
||||
**Solution**: Use LyricsTranscriber as a library:
|
||||
- Call `transcribe()` and `correct()` for processing
|
||||
- Save `corrections.json` to GCS
|
||||
- Skip `ReviewServer` entirely
|
||||
- Call `OutputGenerator` only after human review completes
|
||||
|
||||
**What to use**: `LyricsTranscriber`, `CorrectionResult`, `OutputGenerator`, `OutputConfig`, `CorrectionOperations`
|
||||
|
||||
**What NOT to use**: `ReviewServer`, `server.start()`, blocking waits
|
||||
|
||||
### Video Generation Timing
|
||||
|
||||
**Wrong**: Generate video during lyrics worker, then review
|
||||
**Right**: Lyrics worker saves JSON → Human reviews → Render worker generates video
|
||||
|
||||
Video must include human-corrected lyrics, so generation happens AFTER review.
|
||||
|
||||
### Parallel Processing
|
||||
|
||||
Audio separation and lyrics transcription run in parallel:
|
||||
- Both can fail independently
|
||||
- Progress tracked separately in `state_data`
|
||||
- Screens generation waits for both to complete
|
||||
|
||||
## Common Gotchas
|
||||
|
||||
### Pydantic Model Fields
|
||||
|
||||
**Problem**: Setting a field that doesn't exist in the Pydantic model silently does nothing.
|
||||
|
||||
```python
|
||||
# BAD - fails silently if field doesn't exist
|
||||
job.input_media_gcs_path = "path/to/file"
|
||||
|
||||
# GOOD - explicit field in model
|
||||
class Job(BaseModel):
|
||||
input_media_gcs_path: Optional[str] = None
|
||||
```
|
||||
|
||||
**Lesson**: Always verify Pydantic model has the field before setting it.
|
||||
|
||||
### Styles Configuration
|
||||
|
||||
ASS subtitle generator requires ALL style fields present and properly typed:
|
||||
|
||||
```python
|
||||
required_fields = [
|
||||
"font", "font_path", "ass_name", # ass_name often missed
|
||||
"primary_color", "secondary_color", "outline_color", "back_color",
|
||||
"bold", "italic", "underline", "strike_out",
|
||||
"scale_x", "scale_y", "spacing", "angle",
|
||||
"border_style", "outline", "shadow",
|
||||
"margin_l", "margin_r", "margin_v", "encoding"
|
||||
]
|
||||
|
||||
# Critical: font_path must be string, not None
|
||||
"font_path": "" # Correct
|
||||
"font_path": None # Causes ASS writer to fail
|
||||
```
|
||||
|
||||
### Frontend Data Format
|
||||
|
||||
The React review UI sends partial correction data:
|
||||
|
||||
```python
|
||||
# Frontend sends:
|
||||
{"corrections": [...], "corrected_segments": [...]}
|
||||
|
||||
# Backend must merge with original corrections.json
|
||||
```
|
||||
|
||||
### Firestore Consistency
|
||||
|
||||
**Problem**: Update job, trigger worker immediately, worker reads stale data.
|
||||
|
||||
**Solution**: Verify updates propagated before triggering workers:
|
||||
```python
|
||||
# Update job
|
||||
await job_manager.update_job(job_id, updates)
|
||||
|
||||
# Verify update visible
|
||||
job = await job_manager.get_job(job_id)
|
||||
if job.input_media_gcs_path:
|
||||
await trigger_worker()
|
||||
```
|
||||
|
||||
## Testing Insights
|
||||
|
||||
### Emulator Tests Catch Real Bugs
|
||||
|
||||
Unit tests with mocks didn't catch the `input_media_gcs_path` bug. Emulator integration tests did because they use real Firestore behavior.
|
||||
|
||||
**Lesson**: Run `make test` which includes emulator tests before every commit.
|
||||
|
||||
### Test Isolation
|
||||
|
||||
Emulator tests need their own `conftest.py` that does NOT mock `google.cloud.*`. Mocks from unit tests can leak into integration tests.
|
||||
|
||||
## Deployment Notes
|
||||
|
||||
### Cloud Run Timeouts
|
||||
|
||||
Default 5-minute timeout may not be enough for video encoding. Consider:
|
||||
- Cloud Run Jobs for long-running tasks
|
||||
- Breaking encoding into smaller chunks
|
||||
- Increasing timeout for specific operations
|
||||
|
||||
### Secret Manager Access
|
||||
|
||||
Cloud Run service account needs `Secret Manager Secret Accessor` role. This is managed via Pulumi in `infrastructure/`.
|
||||
|
||||
### CI/CD: GitHub Actions
|
||||
|
||||
Migrated from Cloud Build to GitHub Actions for:
|
||||
- Simpler configuration
|
||||
- Better visibility
|
||||
- Faster feedback
|
||||
|
||||
### Self-Hosted Runner on GCP
|
||||
|
||||
GitHub-hosted runners have ~14GB disk, which caused failures during Docker builds. Solution: self-hosted runner on GCP with 200GB SSD.
|
||||
|
||||
**Key learnings:**
|
||||
|
||||
1. **GPG in startup scripts**: Always use `gpg --batch` for non-interactive execution
|
||||
2. **setup-python on self-hosted**: Needs Python in tool cache at `$RUNNER/_work/_tool/Python/<version>/x64/` with `x64.complete` marker
|
||||
3. **Debian vs Ubuntu**: PPAs don't work on Debian; use upstream repos (Temurin, NodeSource)
|
||||
4. **pyenv for Python**: More reliable than source builds or distro packages for specific versions
|
||||
|
||||
See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
|
||||
|
||||
## Performance Observations
|
||||
|
||||
| Operation | Duration |
|
||||
|-----------|----------|
|
||||
| Audio separation | 5-8 min |
|
||||
| Lyrics transcription | 2-3 min |
|
||||
| Auto-correction | 30 sec |
|
||||
| Screens generation | 30 sec |
|
||||
| Video rendering | 10-15 min |
|
||||
| Final encoding | 5-10 min |
|
||||
| **Total** | 30-50 min |
|
||||
|
||||
Human review time varies (5-15 min typical).
|
||||
|
||||
## What We'd Do Differently
|
||||
|
||||
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
|
||||
|
||||
2. **Use emulator tests from day one** - Faster feedback than deploying to test
|
||||
|
||||
3. **Design for async human review upfront** - Avoided rearchitecting after discovering ReviewServer blocks
|
||||
|
||||
4. **Keep docs minimal and current** - Less documentation, but always accurate
|
||||
@@ -0,0 +1,318 @@
|
||||
# Mobile UX Best Practices
|
||||
|
||||
This document outlines the mobile-first UX principles and implementation patterns used in this project to ensure a great experience across all screen sizes.
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. Mobile-First Responsive Design
|
||||
|
||||
Design for the smallest screen first, then progressively enhance for larger screens.
|
||||
|
||||
```css
|
||||
/* Base styles for mobile */
|
||||
.element {
|
||||
width: 100%;
|
||||
padding: 12px;
|
||||
}
|
||||
|
||||
/* Enhance for larger screens */
|
||||
@media (min-width: 640px) {
|
||||
.element {
|
||||
width: 280px;
|
||||
padding: 16px;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. No Horizontal Overflow
|
||||
|
||||
The most critical mobile UX issue is horizontal scrolling. Always ensure content fits within the viewport width.
|
||||
|
||||
**Testing for overflow:**
|
||||
```typescript
|
||||
async function hasNoHorizontalOverflow(page: Page): Promise<boolean> {
|
||||
return await page.evaluate(() => {
|
||||
const docWidth = document.documentElement.scrollWidth;
|
||||
const viewportWidth = window.innerWidth;
|
||||
return docWidth <= viewportWidth + 2; // Small tolerance for rounding
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
**Common causes of horizontal overflow:**
|
||||
- Fixed widths (e.g., `width: 340px`)
|
||||
- Horizontal flex layouts without wrapping
|
||||
- Images without `max-width: 100%`
|
||||
- Tables without horizontal scroll containers
|
||||
|
||||
### 3. Touch Target Sizing
|
||||
|
||||
Interactive elements must be easily tappable. Follow these minimum sizes:
|
||||
|
||||
| Platform | Minimum Size | Recommended |
|
||||
|----------|--------------|-------------|
|
||||
| Apple iOS | 44x44px | 48x48px |
|
||||
| Material Design | 48x48px | 48x48px |
|
||||
| Web (general) | 32x32px | 44x44px |
|
||||
|
||||
**Implementation:**
|
||||
```tsx
|
||||
<Button className="min-h-[44px] px-3">
|
||||
<Icon className="w-4 h-4" />
|
||||
<span className="hidden sm:inline">Label</span>
|
||||
</Button>
|
||||
```
|
||||
|
||||
### 4. Progressive Disclosure
|
||||
|
||||
Show less content on mobile, with clear paths to access more:
|
||||
|
||||
- **Icon-only buttons on mobile, labels on desktop:**
|
||||
```tsx
|
||||
<Icon className="w-4 h-4 sm:mr-2" />
|
||||
<span className="hidden sm:inline">Action Label</span>
|
||||
```
|
||||
|
||||
- **Collapsed navigation on mobile**
|
||||
- **Truncated text with expansion options**
|
||||
|
||||
## Layout Patterns
|
||||
|
||||
### Responsive Grid
|
||||
|
||||
Use responsive grid columns that stack on mobile:
|
||||
|
||||
```tsx
|
||||
// Bad - always 2 columns
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
|
||||
// Good - stack on mobile, 2 columns on larger screens
|
||||
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 sm:gap-4">
|
||||
```
|
||||
|
||||
### Flexible Containers
|
||||
|
||||
Avoid fixed widths. Use percentage-based or responsive widths:
|
||||
|
||||
```css
|
||||
/* Bad */
|
||||
.sidebar {
|
||||
width: 340px;
|
||||
}
|
||||
|
||||
/* Good */
|
||||
.sidebar {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
@media (min-width: 768px) {
|
||||
.sidebar {
|
||||
width: 340px;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Flex Wrap for Toolbars
|
||||
|
||||
Allow toolbar items to wrap on smaller screens:
|
||||
|
||||
```tsx
|
||||
<div className="flex flex-wrap gap-2 items-center">
|
||||
{/* Items will wrap naturally */}
|
||||
</div>
|
||||
```
|
||||
|
||||
## Typography
|
||||
|
||||
### Readable Font Sizes
|
||||
|
||||
Minimum font sizes for readability without zooming:
|
||||
|
||||
| Element | Minimum Size | Recommended |
|
||||
|---------|--------------|-------------|
|
||||
| Body text | 14px | 16px |
|
||||
| Labels | 12px | 14px |
|
||||
| Small text/badges | 11px | 12px |
|
||||
|
||||
### Responsive Headings
|
||||
|
||||
```tsx
|
||||
<h1 className="text-base sm:text-xl font-bold truncate">
|
||||
Title Text
|
||||
</h1>
|
||||
```
|
||||
|
||||
## Form Design
|
||||
|
||||
### Input Fields on Mobile
|
||||
|
||||
- Stack form fields vertically on mobile
|
||||
- Use full-width inputs for easier tapping
|
||||
- Ensure proper input type for mobile keyboard optimization
|
||||
|
||||
```tsx
|
||||
<input
|
||||
type="email" // Shows email keyboard
|
||||
inputMode="numeric" // Shows number pad
|
||||
className="w-full min-h-[44px]"
|
||||
/>
|
||||
```
|
||||
|
||||
### Labels and Placeholders
|
||||
|
||||
- Labels should be visible (not just placeholders)
|
||||
- Adequate spacing between label and input
|
||||
- Clear validation feedback
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Automated Testing with Playwright
|
||||
|
||||
Test across multiple viewport sizes representing real devices:
|
||||
|
||||
```typescript
|
||||
const MOBILE_VIEWPORTS = {
|
||||
'iPhone SE': { width: 375, height: 667 },
|
||||
'iPhone 14': { width: 390, height: 844 },
|
||||
'Pixel 7': { width: 412, height: 915 },
|
||||
'iPad Mini': { width: 768, height: 1024 },
|
||||
};
|
||||
|
||||
for (const [name, viewport] of Object.entries(MOBILE_VIEWPORTS)) {
|
||||
test(`page works on ${name}`, async ({ page }) => {
|
||||
await page.setViewportSize(viewport);
|
||||
await page.goto('/');
|
||||
|
||||
// Test for horizontal overflow
|
||||
const hasOverflow = await page.evaluate(() =>
|
||||
document.documentElement.scrollWidth > window.innerWidth + 2
|
||||
);
|
||||
expect(hasOverflow).toBe(false);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Visual Regression Testing
|
||||
|
||||
Capture screenshots at different viewports to catch visual regressions:
|
||||
|
||||
```typescript
|
||||
await expect(page).toHaveScreenshot(`homepage-${deviceName}.png`, {
|
||||
fullPage: true,
|
||||
maxDiffPixelRatio: 0.1,
|
||||
});
|
||||
```
|
||||
|
||||
### Touch Target Validation
|
||||
|
||||
```typescript
|
||||
const buttons = page.locator('button, a, [role="button"]');
|
||||
for (const button of await buttons.all()) {
|
||||
const box = await button.boundingBox();
|
||||
if (box) {
|
||||
expect(box.width).toBeGreaterThanOrEqual(32);
|
||||
expect(box.height).toBeGreaterThanOrEqual(32);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Framework-Specific Guidelines
|
||||
|
||||
### Tailwind CSS (Next.js Frontend)
|
||||
|
||||
Use responsive prefixes consistently:
|
||||
|
||||
```plaintext
|
||||
sm: 640px - Small tablets and up
|
||||
md: 768px - Tablets and up
|
||||
lg: 1024px - Laptops and up
|
||||
xl: 1280px - Desktops and up
|
||||
```
|
||||
|
||||
**Common patterns:**
|
||||
```tsx
|
||||
// Stack on mobile, side-by-side on larger
|
||||
className="flex flex-col sm:flex-row"
|
||||
|
||||
// Hide on mobile, show on larger
|
||||
className="hidden sm:block"
|
||||
|
||||
// Reduce padding on mobile
|
||||
className="p-3 sm:p-4"
|
||||
|
||||
// Smaller font on mobile
|
||||
className="text-xs sm:text-sm"
|
||||
```
|
||||
|
||||
### Material UI (Lyrics Review UI)
|
||||
|
||||
Use the `useMediaQuery` hook:
|
||||
|
||||
```tsx
|
||||
const theme = useTheme();
|
||||
const isMobile = useMediaQuery(theme.breakpoints.down('md'));
|
||||
|
||||
return (
|
||||
<Box sx={{
|
||||
flexDirection: isMobile ? 'column' : 'row',
|
||||
width: isMobile ? '100%' : '280px',
|
||||
}}>
|
||||
```
|
||||
|
||||
### Vanilla CSS (Instrumental Review)
|
||||
|
||||
Use CSS media queries:
|
||||
|
||||
```css
|
||||
.element {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
@media (min-width: 768px) {
|
||||
.element {
|
||||
width: 340px;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Checklist for New Features
|
||||
|
||||
Before merging any UI changes, verify:
|
||||
|
||||
- [ ] No horizontal overflow on any mobile viewport (320px - 480px)
|
||||
- [ ] All interactive elements are at least 44x44px
|
||||
- [ ] Text is readable without zooming (minimum 11px)
|
||||
- [ ] Form inputs stack vertically on mobile
|
||||
- [ ] Navigation is accessible on mobile
|
||||
- [ ] Images scale properly with `max-width: 100%`
|
||||
- [ ] Modals/dialogs fit within mobile viewport
|
||||
- [ ] Buttons show icons-only on very small screens
|
||||
- [ ] Playwright mobile tests pass
|
||||
|
||||
## Common Fixes
|
||||
|
||||
### Fix: Horizontal Overflow
|
||||
|
||||
1. **Check for fixed widths** - Replace with responsive values
|
||||
2. **Add flex-wrap** - Allow content to wrap
|
||||
3. **Add overflow-x: hidden** - On containers (use sparingly)
|
||||
4. **Check images** - Add `max-width: 100%`
|
||||
|
||||
### Fix: Touch Target Too Small
|
||||
|
||||
1. **Add minimum height** - `min-h-[44px]`
|
||||
2. **Add padding** - `p-3` or `px-4 py-3`
|
||||
3. **Use icon-only on mobile** - Hide text labels
|
||||
|
||||
### Fix: Text Too Small
|
||||
|
||||
1. **Use responsive font sizes** - `text-xs sm:text-sm`
|
||||
2. **Set minimum body font size** - At least 14px
|
||||
3. **Increase line height** - For better readability
|
||||
|
||||
## Resources
|
||||
|
||||
- [Apple Human Interface Guidelines - Touch Targets](https://developer.apple.com/design/human-interface-guidelines/accessibility)
|
||||
- [Material Design - Touch Targets](https://m3.material.io/foundations/accessible-design/accessibility-basics)
|
||||
- [Tailwind CSS Responsive Design](https://tailwindcss.com/docs/responsive-design)
|
||||
- [Playwright Testing](https://playwright.dev/docs/test-mobile)
|
||||
+34
-177
@@ -1,193 +1,50 @@
|
||||
# Karaoke-Gen Documentation
|
||||
|
||||
This directory contains all documentation for the karaoke-gen project, organized by purpose.
|
||||
> **Status**: Production-ready. First full E2E flow completed 2025-12-28.
|
||||
|
||||
## 📋 Current Status (Updated: 2025-12-08)
|
||||
## What Works
|
||||
|
||||
**Project:** Cloud-hosted karaoke generation with async human review
|
||||
**Backend:** 🔄 Phase 1.3 (85%) - Implementing review architecture
|
||||
**Frontend:** ⏸️ Not started (waiting for backend review flow)
|
||||
**Deployment:** ✅ Cloud Run with custom domain (api.nomadkaraoke.com)
|
||||
**Testing:** 🔄 Local emulator testing in progress
|
||||
- **Web App**: https://gen.nomadkaraoke.com - Upload audio, review lyrics, download videos
|
||||
- **Backend API**: https://api.nomadkaraoke.com - FastAPI on Cloud Run
|
||||
- **CLI Tools**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
|
||||
### Key Discovery
|
||||
## Current State
|
||||
|
||||
We identified that the `LyricsTranscriber` library's `ReviewServer` **blocks** waiting for human input, which doesn't work in our async cloud architecture. The solution is to:
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| Audio upload & separation | Working |
|
||||
| Lyrics transcription | Working |
|
||||
| Human lyrics review | Working |
|
||||
| Preview video generation | Working |
|
||||
| Instrumental selection | Working |
|
||||
| Multi-format encoding | Working |
|
||||
| Token-based auth | Working |
|
||||
| Payment flow | Not started |
|
||||
| CI/CD self-hosted runner | Working (GCP) |
|
||||
|
||||
1. Use LyricsTranscriber for transcription + auto-correction
|
||||
2. Save corrections to GCS for async human review
|
||||
3. Use `OutputGenerator` to render video **after** review completes
|
||||
## Known Issues
|
||||
|
||||
See `00-current-plan/CURRENT-STATUS.md` for full details.
|
||||
- CDG format generation requires additional style configuration
|
||||
- Long audio files (>10 min) may timeout on some workers
|
||||
|
||||
---
|
||||
## Quick Links
|
||||
|
||||
## 📂 Documentation Structure
|
||||
| Doc | Purpose |
|
||||
|-----|---------|
|
||||
| [ARCHITECTURE.md](ARCHITECTURE.md) | System design, data flow, integrations |
|
||||
| [DEVELOPMENT.md](DEVELOPMENT.md) | Local setup, testing, deployment |
|
||||
| [API.md](API.md) | Backend endpoint reference |
|
||||
| [LESSONS-LEARNED.md](LESSONS-LEARNED.md) | Key insights for future agents |
|
||||
| [archive/](archive/) | Historical documentation |
|
||||
|
||||
### `00-current-plan/` - **START HERE**
|
||||
Active migration plan and current status.
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| **`CURRENT-STATUS.md`** | Where we are now, what's working, what's next |
|
||||
| **`WHATS-NEXT.md`** | Detailed plan for review architecture implementation |
|
||||
| **`WORKER-IMPLEMENTATION-PLAN.md`** | Worker responsibilities and implementation details |
|
||||
| `MASTER-PLAN.md` | Overall migration strategy |
|
||||
| `CLOUD-MIGRATION-REQUIREMENTS.md` | Architectural analysis from CLI workflow |
|
||||
|
||||
### `01-reference/` - Reference Documentation
|
||||
Core system documentation.
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| **`ARCHITECTURE.md`** | Cloud + CLI architecture with diagrams |
|
||||
| `KARAOKE-GEN-CLI-WORKFLOW.md` | CLI workflow breakdown (8 stages) |
|
||||
| `API-MANUAL-TESTING.md` | How to test backend API with curl |
|
||||
| `AUTHENTICATION.md` | Token-based auth system |
|
||||
| `CLOUD-RUN-ARCHITECTURE.md` | Cloud Run concepts |
|
||||
|
||||
### `02-implementation-history/` - Session Logs
|
||||
Historical implementation records.
|
||||
|
||||
### `03-deployment/` - Deployment Guides
|
||||
How to deploy and operate the system.
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| **`EMULATOR-TESTING.md`** | Local testing with Firestore/GCS emulators |
|
||||
| **`SETUP-VENV.md`** | Python venv setup |
|
||||
| **`LOCAL-VALIDATION.md`** | Validate before deploying |
|
||||
| **`OBSERVABILITY-GUIDE.md`** | Debug production issues |
|
||||
|
||||
### `04-testing/` - Testing Guides
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| **`TESTING-GUIDE.md`** | Complete testing guide |
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### I want to understand the current state:
|
||||
1. Read `00-current-plan/CURRENT-STATUS.md` - where we are now
|
||||
2. Read `00-current-plan/WHATS-NEXT.md` - what needs to be done
|
||||
3. Read `01-reference/ARCHITECTURE.md` - system architecture
|
||||
|
||||
### I want to work on the backend:
|
||||
1. Read `03-deployment/EMULATOR-TESTING.md` - local testing setup
|
||||
2. Read `00-current-plan/WORKER-IMPLEMENTATION-PLAN.md` - worker details
|
||||
3. Run `./scripts/run-backend-local.sh --with-emulators`
|
||||
|
||||
### I want to test:
|
||||
```bash
|
||||
# Start backend with emulators
|
||||
./scripts/run-backend-local.sh --with-emulators
|
||||
|
||||
# Upload test file
|
||||
curl -X POST http://localhost:8000/api/jobs/upload \
|
||||
-F "file=@tests/data/waterloo10sec.flac" \
|
||||
-F "artist=ABBA" \
|
||||
-F "title=Waterloo"
|
||||
|
||||
# Check status
|
||||
curl http://localhost:8000/api/jobs/{job_id}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Current Phase: Review Architecture
|
||||
|
||||
**Goal:** Implement async human review workflow
|
||||
|
||||
**What Works:**
|
||||
- ✅ File upload → GCS
|
||||
- ✅ Audio separation via Modal API
|
||||
- ✅ Lyrics transcription via AudioShake
|
||||
- ✅ Auto-correction via LyricsTranscriber
|
||||
- ✅ Title/end screen generation
|
||||
- ✅ Instrumental selection endpoint
|
||||
|
||||
**What's Being Implemented:**
|
||||
- 🔄 Review API endpoints (`GET/POST /api/jobs/{id}/review`)
|
||||
- 🔄 Render Video Worker (uses `OutputGenerator` after review)
|
||||
- 🔄 Updated state machine with `RENDERING_VIDEO` state
|
||||
|
||||
**The Key Insight:**
|
||||
```
|
||||
WRONG: Lyrics Worker → Generate Video → Review → ???
|
||||
RIGHT: Lyrics Worker → Save JSON → Human Review → Render Video → Final Assembly
|
||||
```
|
||||
|
||||
Video generation happens AFTER human review, not during the lyrics worker.
|
||||
|
||||
---
|
||||
|
||||
## 📝 Key Technical Decisions
|
||||
|
||||
### Why Not Use LyricsTranscriber's ReviewServer?
|
||||
|
||||
The `ReviewServer` class (in `lyrics_transcriber/review/server.py`):
|
||||
- Starts a local web server on port 8000
|
||||
- Opens a browser window
|
||||
- **BLOCKS** the thread until human submits corrections
|
||||
- Designed for local CLI operation
|
||||
|
||||
This doesn't work in Cloud Run where:
|
||||
- We can't open browsers
|
||||
- Workers must be non-blocking
|
||||
- Human review happens asynchronously via React frontend
|
||||
|
||||
**Solution:** Use the data structures and `OutputGenerator` directly, skip the server.
|
||||
|
||||
### Why Separate Video Rendering?
|
||||
## Workflow Summary
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Lyrics Worker │ ──▶ │ Human Reviews │ ──▶ │ Render Worker │
|
||||
│ │ │ (React UI) │ │ │
|
||||
│ • Transcribe │ │ • Corrects │ │ • OutputGenerator│
|
||||
│ • Auto-correct │ │ • Saves JSON │ │ • with_vocals.mkv│
|
||||
│ • Save JSON │ │ │ │ │
|
||||
│ • NO VIDEO │ │ │ │ │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
Upload Audio → Parallel Processing → Human Review → Video Generation → Download
|
||||
│ │
|
||||
(Modal + AudioShake) (LyricsTranscriber UI)
|
||||
```
|
||||
|
||||
The video must include the human-corrected lyrics, so it can only be generated AFTER review.
|
||||
## For AI Agents
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Job State Flow
|
||||
|
||||
```
|
||||
PENDING → DOWNLOADING → [parallel audio+lyrics] → GENERATING_SCREENS
|
||||
↓
|
||||
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE
|
||||
↓
|
||||
RENDERING_VIDEO (NEW)
|
||||
↓
|
||||
AWAITING_INSTRUMENTAL_SELECTION → INSTRUMENTAL_SELECTED
|
||||
↓
|
||||
GENERATING_VIDEO → COMPLETE
|
||||
```
|
||||
|
||||
Two human interaction points:
|
||||
1. **AWAITING_REVIEW** - User corrects lyrics
|
||||
2. **AWAITING_INSTRUMENTAL_SELECTION** - User chooses audio track
|
||||
|
||||
---
|
||||
|
||||
## 📞 Getting Help
|
||||
|
||||
1. Check `00-current-plan/CURRENT-STATUS.md` first
|
||||
2. Look for specific guides in `03-deployment/`
|
||||
3. Review session history in `02-implementation-history/`
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| Emulator not starting | Check ports 8080, 4443 are free |
|
||||
| "Missing lyrics video" | Review hasn't happened yet - expected behavior |
|
||||
| Workers not progressing | Check `state_data` in Firestore |
|
||||
| FFmpeg errors | Ensure FFmpeg is in PATH |
|
||||
Start with `CLAUDE.md` in the repo root for essential rules, then reference these docs as needed. Keep this README current when project state changes.
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,80 @@
|
||||
# Self-Hosted GitHub Actions Runner on GCP
|
||||
|
||||
**Date**: 2025-12-28
|
||||
**Status**: Deployed and operational
|
||||
|
||||
## Problem
|
||||
|
||||
GitHub-hosted runners have ~14GB disk space. CI jobs were failing with "No space left on device" during:
|
||||
- Docker builds (multi-stage, large dependencies)
|
||||
- Emulator tests (Docker + Java + Firestore/GCS emulators)
|
||||
- Package builds with poetry
|
||||
|
||||
Workarounds like `jlumbroso/free-disk-space` action helped but weren't reliable.
|
||||
|
||||
## Solution
|
||||
|
||||
Deployed a self-hosted runner on GCP with 200GB SSD via Pulumi IaC.
|
||||
|
||||
### Infrastructure Added
|
||||
|
||||
| Resource | Purpose |
|
||||
|----------|---------|
|
||||
| `github-runner` VM | e2-standard-4, 200GB SSD, Debian 12 |
|
||||
| `github-runner` service account | Minimal permissions for runner |
|
||||
| `github-runner-pat` secret | PAT for runner registration |
|
||||
| IAM bindings | Artifact Registry reader, Secret Manager accessor, logging |
|
||||
|
||||
### Jobs Moved to Self-Hosted
|
||||
|
||||
- `package-build-test`
|
||||
- `backend-unit-tests`
|
||||
- `backend-emulator-tests`
|
||||
- `deploy-backend`
|
||||
|
||||
These use `runs-on: [self-hosted, linux, gcp]` with labels.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
### 1. GPG tty errors in startup script
|
||||
|
||||
**Problem**: `gpg --dearmor` failed with "cannot open /dev/tty" in non-interactive startup.
|
||||
|
||||
**Fix**: Add `--batch` flag to all GPG commands.
|
||||
|
||||
### 2. setup-python can't find Python 3.13 on Debian 12
|
||||
|
||||
**Problem**: `actions/setup-python@v5` couldn't find pre-built Python 3.13 for Debian 12 on self-hosted runners.
|
||||
|
||||
**Fix**:
|
||||
1. Install Python 3.13 via pyenv on the runner
|
||||
2. Copy to tool cache: `_work/_tool/Python/3.13.0/x64/`
|
||||
3. Create marker file: `x64.complete`
|
||||
|
||||
### 3. Java 21 not in Debian repos
|
||||
|
||||
**Problem**: `openjdk-21-jdk` not available in Debian 12 stable.
|
||||
|
||||
**Fix**: Use Temurin (Adoptium) repository directly.
|
||||
|
||||
## Key Files
|
||||
|
||||
- `infrastructure/__main__.py` - Runner VM and IAM setup
|
||||
- `infrastructure/README.md` - Setup instructions
|
||||
- `.github/workflows/ci.yml` - Jobs using self-hosted runner
|
||||
|
||||
## Lessons for Future
|
||||
|
||||
1. **Self-hosted runners need explicit tool setup** - Unlike GitHub-hosted, tools aren't pre-installed in expected locations
|
||||
2. **setup-python tool cache location**: `$RUNNER_DIR/_work/_tool/Python/<version>/x64/`
|
||||
3. **GPG in scripts**: Always use `--batch` for non-interactive execution
|
||||
4. **Debian vs Ubuntu**: Many GitHub Actions assume Ubuntu; PPAs don't work on Debian
|
||||
|
||||
## Cost
|
||||
|
||||
~$50/month for e2-standard-4 running 24/7 (covered by GCP free credits).
|
||||
|
||||
## PRs
|
||||
|
||||
- #95: Initial self-hosted runner setup
|
||||
- #97: Make startup script reproducible (pyenv, tool cache, GPG fixes)
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,197 @@
|
||||
# GitHub Actions Docker Builds - Cost Optimization
|
||||
|
||||
**Date:** 2025-12-28
|
||||
**Problem:** Cloud Build costing $100+/month with slow builds
|
||||
**Solution:** Move Docker builds to GitHub Actions with registry caching
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the migration from Google Cloud Build to GitHub Actions for Docker image builds. This change eliminates Cloud Build costs while improving build speeds through better caching.
|
||||
|
||||
## Cost Comparison
|
||||
|
||||
| Approach | Monthly Cost | Build Speed | Caching |
|
||||
|----------|-------------|-------------|---------|
|
||||
| **Cloud Build (N1_HIGHCPU_32)** | ~$100+ | 5-15 min | Limited |
|
||||
| **GitHub Actions + Registry Cache** | **$0** | 2-5 min | Excellent |
|
||||
|
||||
### Why Cloud Build Was Expensive
|
||||
|
||||
1. **N1_HIGHCPU_32 pricing**: ~$0.064/build-minute (21x more than default)
|
||||
2. **No persistent layer cache**: Each build re-downloaded dependencies
|
||||
3. **Cold start overhead**: Spinning up build VMs added latency
|
||||
4. **Large dependency tree**: torch, transformers, etc. = ~500 packages
|
||||
|
||||
### Why GitHub Actions Is Free
|
||||
|
||||
- **Public repositories**: Unlimited build minutes on GitHub-hosted runners
|
||||
- **Registry cache**: Layers cached in Artifact Registry, persist between builds
|
||||
- **BuildKit mode=max**: Caches ALL intermediate layers, not just final
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ NEW BUILD PIPELINE │
|
||||
├─────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ GitHub Actions (Free) │
|
||||
│ ├── docker/setup-buildx-action (BuildKit) │
|
||||
│ ├── google-github-actions/auth (Workload Identity) │
|
||||
│ └── docker/build-push-action │
|
||||
│ ├── cache-from: type=registry,ref=...karaoke-backend:cache │
|
||||
│ └── cache-to: type=registry,ref=...karaoke-backend:cache,mode=max│
|
||||
│ │
|
||||
│ Artifact Registry │
|
||||
│ ├── karaoke-backend:latest (production image) │
|
||||
│ ├── karaoke-backend:cache (build cache - ALL layers) │
|
||||
│ ├── karaoke-backend-base:latest (base image with deps) │
|
||||
│ └── karaoke-backend-base:cache (base cache) │
|
||||
│ │
|
||||
│ Build Times (Expected): │
|
||||
│ ├── First build (cold): ~8-10 min │
|
||||
│ ├── Code-only changes: ~2-3 min (cache hit) │
|
||||
│ └── Dependency changes: ~6-8 min (partial cache hit) │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
### Two-Stage Build Pattern
|
||||
|
||||
We maintain the existing two-stage build pattern for efficiency:
|
||||
|
||||
1. **Base Image** (`karaoke-backend-base`)
|
||||
- Contains Python 3.11, ffmpeg, system dependencies
|
||||
- All pip dependencies pre-installed
|
||||
- Only rebuilt when `poetry.lock` or `Dockerfile.base` changes
|
||||
- Hash-based change detection via image labels
|
||||
|
||||
2. **App Image** (`karaoke-backend`)
|
||||
- Built FROM the base image
|
||||
- Contains only application code
|
||||
- Fast rebuilds (~2 min) for code-only changes
|
||||
|
||||
### Registry Cache
|
||||
|
||||
The key optimization is using Artifact Registry as a persistent layer cache:
|
||||
|
||||
```yaml
|
||||
- uses: docker/build-push-action@v6
|
||||
with:
|
||||
cache-from: type=registry,ref=.../karaoke-backend:cache
|
||||
cache-to: type=registry,ref=.../karaoke-backend:cache,mode=max
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- `mode=max` caches ALL layers, not just the final image
|
||||
- Cache persists between CI runs (unlike GitHub Actions cache which has 10GB limit)
|
||||
- Layers only re-built when inputs change
|
||||
|
||||
---
|
||||
|
||||
## Files Changed
|
||||
|
||||
### New Files
|
||||
|
||||
- **`.dockerignore`** - Reduces build context from ~1GB to ~10MB
|
||||
|
||||
### Modified Files
|
||||
|
||||
- **`.github/workflows/ci.yml`** - `deploy-backend` job rewritten to use:
|
||||
- `docker/setup-buildx-action@v3`
|
||||
- `docker/build-push-action@v6`
|
||||
- Registry caching to Artifact Registry
|
||||
|
||||
### Kept (as fallback)
|
||||
|
||||
- **`cloudbuild.yaml`** - Still works for manual triggers
|
||||
- **`cloudbuild-base.yaml`** - Still works for manual base image builds
|
||||
|
||||
---
|
||||
|
||||
## Manual Builds (Fallback)
|
||||
|
||||
If GitHub Actions fails, you can still use Cloud Build:
|
||||
|
||||
```bash
|
||||
# Build base image (only if dependencies changed)
|
||||
gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
|
||||
|
||||
# Build and deploy app image
|
||||
gcloud builds submit --config=cloudbuild.yaml --project=nomadkaraoke
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Check Build Cache Usage
|
||||
|
||||
View cache images in Artifact Registry:
|
||||
|
||||
```bash
|
||||
gcloud artifacts docker images list \
|
||||
us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo \
|
||||
--include-tags
|
||||
```
|
||||
|
||||
You should see `*:cache` tags for both base and app images.
|
||||
|
||||
### Compare Build Times
|
||||
|
||||
- **GitHub Actions**: Check workflow run times in Actions tab
|
||||
- **Previous Cloud Build**: ~5-15 min per build
|
||||
- **Expected with cache**: ~2-3 min for code changes
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Cache Miss (Slow Build)
|
||||
|
||||
If builds are slow despite no dependency changes:
|
||||
|
||||
1. Check if cache image exists:
|
||||
```bash
|
||||
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:cache
|
||||
```
|
||||
|
||||
2. Force cache rebuild by pushing to `main` with a small change
|
||||
|
||||
### Disk Space Issues
|
||||
|
||||
GitHub Actions runners have limited disk. If base image builds fail:
|
||||
|
||||
1. The workflow includes `jlumbroso/free-disk-space` action
|
||||
2. It removes Android SDK, Haskell, .NET, etc. to free ~20GB
|
||||
|
||||
### Authentication Issues
|
||||
|
||||
If `docker push` fails with permission errors:
|
||||
|
||||
1. Check Workload Identity is configured correctly
|
||||
2. Verify service account has `roles/artifactregistry.writer`
|
||||
|
||||
---
|
||||
|
||||
## Future Improvements
|
||||
|
||||
1. **Parallel base/app builds**: If base is unchanged, start app build immediately
|
||||
2. **GitHub Actions cache fallback**: Use GHA cache when registry is slow
|
||||
3. **Build metrics**: Track build times and cache hit rates
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Docker Build with GitHub Actions](https://docs.docker.com/build/ci/github-actions/)
|
||||
- [Registry Cache Backend](https://docs.docker.com/build/cache/backends/registry/)
|
||||
- [Cache Management](https://docs.docker.com/build/ci/github-actions/cache/)
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,233 @@
|
||||
# Job Status Display Improvement Plan
|
||||
|
||||
**Status:** Planned
|
||||
**Created:** 2025-12-27
|
||||
**Priority:** HIGH - User experience improvement
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Users see generic status messages like "Downloading..." that persist for too long, even after the download is complete and other processing stages are happening. This makes it hard to understand:
|
||||
1. What stage the job is actually at
|
||||
2. How much progress has been made
|
||||
3. How much longer the job might take
|
||||
|
||||
## Current Implementation
|
||||
|
||||
### Backend Status System
|
||||
The backend has 28 distinct job statuses organized into 8 processing stages:
|
||||
|
||||
```
|
||||
Stage 1: Setup (PENDING)
|
||||
Stage 2: Audio Search (SEARCHING_AUDIO, AWAITING_AUDIO_SELECTION, DOWNLOADING_AUDIO)
|
||||
Stage 3: Download (DOWNLOADING)
|
||||
Stage 4: Parallel Processing
|
||||
- Audio: SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE
|
||||
- Lyrics: TRANSCRIBING → CORRECTING → LYRICS_COMPLETE
|
||||
Stage 5: Screen Generation (GENERATING_SCREENS, APPLYING_PADDING)
|
||||
Stage 6: Lyrics Review (AWAITING_REVIEW, IN_REVIEW, REVIEW_COMPLETE) ⚠️ BLOCKING
|
||||
Stage 7: Video Rendering (RENDERING_VIDEO)
|
||||
Stage 8: Instrumental Selection (AWAITING_INSTRUMENTAL_SELECTION) ⚠️ BLOCKING
|
||||
Stage 9: Final Processing (INSTRUMENTAL_SELECTED, GENERATING_VIDEO, ENCODING, PACKAGING)
|
||||
Stage 10: Distribution (UPLOADING, NOTIFYING, COMPLETE)
|
||||
```
|
||||
|
||||
### Key Findings
|
||||
|
||||
1. **Progress field exists but unused**: The `Job` model has a `progress: number` field (0-100) that's returned by the API but NOT displayed in the frontend
|
||||
2. **Parallel processing complicates status**: Audio and lyrics workers run in parallel, updating `state_data.audio_progress` and `state_data.lyrics_progress` independently
|
||||
3. **Status labels are static**: Frontend shows "Downloading..." but doesn't reflect that parallel workers are running
|
||||
4. **10-second polling**: Frontend polls every 10 seconds, so status updates are not real-time
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
### 1. Step-Based Progress Display
|
||||
|
||||
Show a `[X/10] Status...` format that gives users a clear sense of overall progress:
|
||||
|
||||
```
|
||||
[1/10] Setting up...
|
||||
[2/10] Searching for audio...
|
||||
[3/10] Downloading audio...
|
||||
[4/10] Processing audio & lyrics... ← Parallel stage
|
||||
[5/10] Generating screens...
|
||||
[6/10] Needs review ← Blocking
|
||||
[7/10] Rendering video...
|
||||
[8/10] Select instrumental ← Blocking
|
||||
[9/10] Encoding final video...
|
||||
[10/10] Uploading...
|
||||
```
|
||||
|
||||
### 2. Status-to-Step Mapping
|
||||
|
||||
```typescript
|
||||
const STATUS_STEP_MAP: Record<string, { step: number; total: number; label: string }> = {
|
||||
// Step 1: Setup
|
||||
pending: { step: 1, total: 10, label: "Setting up" },
|
||||
|
||||
// Step 2: Audio Search (optional)
|
||||
searching_audio: { step: 2, total: 10, label: "Searching for audio" },
|
||||
awaiting_audio_selection: { step: 2, total: 10, label: "Select audio source" },
|
||||
|
||||
// Step 3: Download
|
||||
downloading_audio: { step: 3, total: 10, label: "Downloading audio" },
|
||||
downloading: { step: 3, total: 10, label: "Downloading" },
|
||||
|
||||
// Step 4: Parallel Processing (Audio + Lyrics)
|
||||
separating_stage1: { step: 4, total: 10, label: "Separating audio (1/2)" },
|
||||
separating_stage2: { step: 4, total: 10, label: "Separating audio (2/2)" },
|
||||
audio_complete: { step: 4, total: 10, label: "Audio ready" },
|
||||
transcribing: { step: 4, total: 10, label: "Transcribing lyrics" },
|
||||
correcting: { step: 4, total: 10, label: "Correcting lyrics" },
|
||||
lyrics_complete: { step: 4, total: 10, label: "Lyrics ready" },
|
||||
|
||||
// Step 5: Screen Generation
|
||||
generating_screens: { step: 5, total: 10, label: "Generating screens" },
|
||||
applying_padding: { step: 5, total: 10, label: "Syncing countdown" },
|
||||
|
||||
// Step 6: Review (BLOCKING)
|
||||
awaiting_review: { step: 6, total: 10, label: "Needs review" },
|
||||
in_review: { step: 6, total: 10, label: "In review" },
|
||||
|
||||
// Step 7: Video Rendering
|
||||
review_complete: { step: 7, total: 10, label: "Review complete" },
|
||||
rendering_video: { step: 7, total: 10, label: "Rendering video" },
|
||||
|
||||
// Step 8: Instrumental Selection (BLOCKING)
|
||||
awaiting_instrumental_selection: { step: 8, total: 10, label: "Select instrumental" },
|
||||
|
||||
// Step 9: Final Encoding
|
||||
instrumental_selected: { step: 9, total: 10, label: "Instrumental selected" },
|
||||
generating_video: { step: 9, total: 10, label: "Generating final video" },
|
||||
encoding: { step: 9, total: 10, label: "Encoding video" },
|
||||
packaging: { step: 9, total: 10, label: "Packaging files" },
|
||||
|
||||
// Step 10: Distribution
|
||||
uploading: { step: 10, total: 10, label: "Uploading" },
|
||||
notifying: { step: 10, total: 10, label: "Sending notifications" },
|
||||
complete: { step: 10, total: 10, label: "Complete" },
|
||||
|
||||
// Terminal states
|
||||
failed: { step: 0, total: 10, label: "Failed" },
|
||||
cancelled: { step: 0, total: 10, label: "Cancelled" },
|
||||
prep_complete: { step: 10, total: 10, label: "Prep complete" },
|
||||
};
|
||||
```
|
||||
|
||||
### 3. Enhanced Parallel Processing Display
|
||||
|
||||
When audio and lyrics are processing in parallel, show both activities:
|
||||
|
||||
```typescript
|
||||
// Check state_data for parallel progress
|
||||
if (job.state_data?.audio_progress && job.state_data?.lyrics_progress) {
|
||||
const audioStatus = job.state_data.audio_progress.stage;
|
||||
const lyricsStatus = job.state_data.lyrics_progress.stage;
|
||||
|
||||
// Show combined status like "Audio: separating, Lyrics: transcribing"
|
||||
if (audioStatus !== 'audio_complete' || lyricsStatus !== 'lyrics_complete') {
|
||||
return {
|
||||
step: 4,
|
||||
total: 10,
|
||||
label: buildParallelLabel(audioStatus, lyricsStatus)
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Visual Progress Bar (Optional Enhancement)
|
||||
|
||||
Add a thin progress bar below the status text:
|
||||
|
||||
```tsx
|
||||
<div className="w-full h-1 bg-slate-700 rounded mt-1">
|
||||
<div
|
||||
className="h-full bg-blue-500 rounded transition-all"
|
||||
style={{ width: `${(step / total) * 100}%` }}
|
||||
/>
|
||||
</div>
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Core Changes (MVP)
|
||||
|
||||
1. **Create status utility** (`frontend/lib/job-status.ts`):
|
||||
- Define `STATUS_STEP_MAP` constant
|
||||
- Create `getJobStep(job: Job)` function
|
||||
- Create `formatStepStatus(step, total, label)` function
|
||||
|
||||
2. **Update JobCard component**:
|
||||
- Import status utility
|
||||
- Replace static `StatusText` with step-based format
|
||||
- Keep existing color coding from `statusConfig`
|
||||
|
||||
3. **Add unit tests** (`frontend/__tests__/job-status.test.ts`):
|
||||
- Test all status mappings
|
||||
- Test parallel processing detection
|
||||
- Test edge cases (unknown status, null values)
|
||||
|
||||
### Phase 2: Enhanced Parallel Processing (Optional)
|
||||
|
||||
1. **Add parallel progress detection**:
|
||||
- Check `state_data.audio_progress` and `state_data.lyrics_progress`
|
||||
- Build combined label showing both activities
|
||||
|
||||
2. **Add visual progress bar**:
|
||||
- Thin progress indicator based on step number
|
||||
- Smooth transitions between steps
|
||||
|
||||
### Phase 3: Backend Improvements (Future)
|
||||
|
||||
1. **Add estimated time remaining**:
|
||||
- Track average duration per stage from historical data
|
||||
- Return ETA in API response
|
||||
|
||||
2. **WebSocket for real-time updates**:
|
||||
- Replace polling with push notifications
|
||||
- Smoother progress updates during long operations
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `frontend/lib/job-status.ts` | NEW: Status utility functions |
|
||||
| `frontend/components/job/JobCard.tsx` | Update status display |
|
||||
| `frontend/__tests__/job-status.test.ts` | NEW: Unit tests |
|
||||
| `frontend/lib/api.ts` | Add `state_data` types (if needed) |
|
||||
|
||||
## Testing Plan
|
||||
|
||||
1. **Unit tests**:
|
||||
- All status mappings return correct step/label
|
||||
- Unknown statuses handled gracefully
|
||||
- Parallel progress detection works
|
||||
|
||||
2. **Integration tests** (E2E):
|
||||
- Verify status displays correctly at each stage
|
||||
- Verify step numbers increment as job progresses
|
||||
|
||||
3. **Manual testing**:
|
||||
- Submit a job and observe status progression
|
||||
- Verify parallel processing shows meaningful status
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] All statuses show `[X/10] Label` format
|
||||
- [ ] Step numbers progress logically (never decrease except on retry)
|
||||
- [ ] Parallel processing shows combined activity
|
||||
- [ ] Blocking statuses clearly indicated for user action
|
||||
- [ ] All existing tests pass
|
||||
- [ ] New unit tests have >80% coverage
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
The changes are frontend-only and purely cosmetic. If issues arise:
|
||||
1. Revert the JobCard changes to use the old `statusConfig` approach
|
||||
2. The backend API remains unchanged
|
||||
|
||||
## Notes
|
||||
|
||||
- The 10-step system is a simplification of the 28 backend statuses for user comprehension
|
||||
- Parallel processing (step 4) groups multiple backend statuses into one user-facing step
|
||||
- Blocking steps (6, 8) require user action and should be visually distinct
|
||||
- The `progress` field from backend could be used for sub-step progress bars in the future
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loaded 100 of 161 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user