Compare commits

..
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.5 2614a2c14c Address CodeRabbit review feedback
- Add thread-safe singleton initialization for ThemeService using double-checked locking
- Make BRANDING_DIR configurable via environment variable in upload-theme.sh
- Fix trap command quoting (use single quotes for deferred expansion)
- Add validation for required branding files before upload

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 17:51:43 -05:00
Andrew BeveridgeandClaude Opus 4.5 80115b7e34 Add theme system for pre-made video styles
- Backend: Add theme models, service, and API routes for theme selection
- Backend: Update job creation to accept theme_id and color_overrides
- Backend: Enable CDG/TXT by default when theme is selected
- Frontend: Add ThemeSelector and ColorOverridesPanel components
- Frontend: Integrate theme selection into JobSubmission form
- CLI: Add --theme argument for remote CLI
- Upload Nomad theme to GCS at themes/nomad/

Themes are stored in GCS and include style params, assets, YouTube
description templates, and preview images. Users can select a theme
and optionally override colors (artist, title, sung/unsung lyrics).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 15:17:48 -05:00
146 changed files with 2404 additions and 6687 deletions

No files matched your search

-74
View File
@@ -1,74 +0,0 @@
---
description: Periodic documentation maintenance and organization check
allowed-tools: Read, Glob, Grep, Edit, Write, Bash
---
# Documentation Maintenance Check
You are performing periodic maintenance on the project documentation.
## Checks to Perform
### 1. Structure Compliance
Verify docs/ follows the correct structure:
```
docs/
├── README.md # Current status + navigation
├── ARCHITECTURE.md # System design
├── DEVELOPMENT.md # Dev setup, testing, deployment
├── API.md # Backend API reference
├── LESSONS-LEARNED.md # Accumulated wisdom
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
```
Report any files that don't belong or are misplaced.
### 2. Freshness Check
- Read `docs/README.md` - is the status section current?
- Check recent git commits - do they suggest docs should be updated?
- Look for stale information that contradicts current code
### 3. Archive Organization
- Verify all archive files have YYYY-MM-DD prefix
- Check for any docs that should be archived (completed work, old plans)
- Remove truly obsolete content (empty files, duplicate info)
### 4. Cross-Reference Check
- Verify CLAUDE.md points to current doc locations
- Check that docs reference each other correctly
- Ensure no broken internal links
### 5. Content Quality
- Flag any docs over 500 lines (may need splitting)
- Identify duplicate information across docs
- Note sections that seem outdated
## Actions to Take
1. **Fix structural issues** - Move/rename misplaced files
2. **Update stale content** - Refresh outdated information
3. **Archive completed work** - Move old docs to archive/
4. **Report findings** - Summarize what was found and fixed
## Output Format
Provide a maintenance report:
```
## Documentation Maintenance Report - YYYY-MM-DD
### Issues Found
- [list of issues]
### Actions Taken
- [list of fixes made]
### Recommendations
- [suggestions for future attention]
### Status
- [ ] Structure: OK/Fixed
- [ ] Freshness: OK/Updated
- [ ] Archive: OK/Organized
- [ ] Cross-refs: OK/Fixed
- [ ] Quality: OK/Flagged
```
-43
View File
@@ -1,43 +0,0 @@
---
description: Review current session for documentation updates before merging PR
allowed-tools: Read, Glob, Grep, Edit, Write
---
# Pre-Merge Documentation Review
You are reviewing this chat session to identify documentation updates needed before merging the current PR.
## Your Task
1. **Analyze the current session** - What work was done? What was learned? What changed?
2. **Check if docs need updates:**
- Does `docs/README.md` status section need updating?
- Should anything be added to `docs/LESSONS-LEARNED.md`?
- Does `docs/ARCHITECTURE.md` need changes (if architecture changed)?
- Does `docs/DEVELOPMENT.md` need changes (if dev workflow changed)?
- Does `docs/API.md` need changes (if API changed)?
3. **For significant completed work:**
- Create `docs/archive/YYYY-MM-DD-topic.md` summarizing the work
- Use today's date and a descriptive topic name
4. **Update docs/README.md** with:
- Current status if it changed
- Any new known issues discovered
- Link to new archive doc if created
## Guidelines
- Keep updates concise and factual
- Focus on information useful for future AI agents
- Don't duplicate information that's in the code
- Archive docs should capture "what was done and why" not implementation details
## Output
After reviewing, either:
- Make the necessary documentation updates
- Report "No documentation updates needed" with brief explanation why
Remember: Good documentation helps the next agent continue where you left off.
-83
View File
@@ -1,83 +0,0 @@
# .dockerignore - Reduce Docker build context size
# This is critical for fast builds as it reduces upload time and cache invalidation
# Git
.git/
.gitignore
.gitmodules
# Node.js / Frontend (not needed for backend image)
node_modules/
frontend/
frontend-react/
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# Python virtual environments (use Docker's Python instead)
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
.venv/
venv/
backend/venv/
ENV/
env/
*.egg-info/
.eggs/
dist/
build/
*.egg
# IDE
.idea/
.vscode/
*.swp
*.swo
*~
.DS_Store
# Testing (not needed in production image)
.pytest_cache/
.coverage
htmlcov/
.tox/
.nox/
coverage.xml
*.cover
tests/
backend/tests/
# Documentation (not needed in Docker image)
docs/
*.md
!README.md
# Infrastructure (not needed for Docker build)
infrastructure/
# CI/CD configs
.github/
# Cloud Build configs (keep for reference but not in image)
cloudbuild*.yaml
# Local development files
lyrics_transcriber_local/
*.local
.env
.env.*
*.bak
*.tmp
Thumbs.db
# Large test/sample files
tests/data/
*.mp3
*.wav
*.mp4
*.mkv
*.flac
+140 -354
View File
@@ -17,120 +17,12 @@ concurrency:
jobs:
# ============================================
# PATH FILTERING - Determine what changed
# Only needed for PRs (main branch skips tests)
# ============================================
changes:
name: "Detect Changes"
runs-on: ubuntu-latest
if: github.ref != 'refs/heads/main'
outputs:
frontend: ${{ steps.filter.outputs.frontend }}
backend: ${{ steps.filter.outputs.backend }}
package: ${{ steps.filter.outputs.package }}
infra: ${{ steps.filter.outputs.infra }}
deps: ${{ steps.filter.outputs.deps }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
frontend:
- 'frontend/**'
backend:
- 'backend/**'
package:
- 'karaoke_gen/**'
- 'lyrics_transcriber_temp/**'
- 'tests/**'
infra:
- 'infrastructure/**'
deps:
- 'poetry.lock'
- name: Summary
run: |
echo "### Change Detection Results" >> $GITHUB_STEP_SUMMARY
echo "| Area | Changed |" >> $GITHUB_STEP_SUMMARY
echo "|------|---------|" >> $GITHUB_STEP_SUMMARY
echo "| Frontend | ${{ steps.filter.outputs.frontend }} |" >> $GITHUB_STEP_SUMMARY
echo "| Backend | ${{ steps.filter.outputs.backend }} |" >> $GITHUB_STEP_SUMMARY
echo "| Package | ${{ steps.filter.outputs.package }} |" >> $GITHUB_STEP_SUMMARY
echo "| Infrastructure | ${{ steps.filter.outputs.infra }} |" >> $GITHUB_STEP_SUMMARY
echo "| Dependencies | ${{ steps.filter.outputs.deps }} |" >> $GITHUB_STEP_SUMMARY
# ============================================
# FRONTEND TESTS (PRs only - main skips tests)
# ============================================
frontend-unit-tests:
name: "Frontend - Unit Tests"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
steps:
- uses: actions/checkout@v4
- name: Setup Node.js with caching
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Run unit tests with coverage
working-directory: frontend
run: npm run test:ci
- name: Upload coverage report
if: always()
uses: actions/upload-artifact@v4
with:
name: frontend-coverage
path: frontend/coverage/
retention-days: 7
frontend-build:
name: "Frontend - Build"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
steps:
- uses: actions/checkout@v4
- name: Setup Node.js with caching
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Build frontend
working-directory: frontend
env:
NEXT_PUBLIC_API_URL: https://api.nomadkaraoke.com
run: npm run build
# ============================================
# PACKAGE TESTS (PRs only - main skips tests)
# PACKAGE TESTS (karaoke-gen Python CLI)
# ============================================
package-unit-tests:
name: "Package - Unit Tests (70% coverage)"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- uses: actions/checkout@v4
with:
@@ -147,14 +39,6 @@ jobs:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-package-${{ runner.os }}-py3.13-
- name: Install dependencies
run: poetry install --with dev
@@ -163,12 +47,6 @@ jobs:
sudo apt-get update
sudo apt-get install -y ffmpeg
- name: Cache spaCy model
uses: actions/cache@v4
with:
path: ~/.cache/spacy
key: spacy-en_core_web_sm-${{ runner.os }}
- name: Install spaCy language models
run: poetry run python -m spacy download en_core_web_sm
@@ -187,8 +65,6 @@ jobs:
package-integration-tests:
name: "Package - Integration Tests"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- uses: actions/checkout@v4
with:
@@ -205,14 +81,6 @@ jobs:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-package-${{ runner.os }}-py3.13-
- name: Install dependencies
run: poetry install --with dev
@@ -221,27 +89,16 @@ jobs:
sudo apt-get update
sudo apt-get install -y ffmpeg
- name: Cache spaCy model
uses: actions/cache@v4
with:
path: ~/.cache/spacy
key: spacy-en_core_web_sm-${{ runner.os }}
- name: Install spaCy language models
run: poetry run python -m spacy download en_core_web_sm
- name: Run integration tests
run: poetry run pytest tests/integration/ -v
package-build-test:
name: "Package - Build & Install (Test)"
runs-on: [self-hosted, linux, gcp]
needs: [changes, package-unit-tests, package-integration-tests]
if: >-
github.ref != 'refs/heads/main' && always() &&
(needs.package-unit-tests.result == 'success' || needs.package-unit-tests.result == 'skipped') &&
(needs.package-integration-tests.result == 'success' || needs.package-integration-tests.result == 'skipped') &&
(needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
package-build:
name: "Package - Build & Install"
runs-on: ubuntu-latest
needs: [package-unit-tests, package-integration-tests]
steps:
- uses: actions/checkout@v4
with:
@@ -261,20 +118,36 @@ jobs:
- name: Build package
run: poetry build
- name: Free disk space
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
haskell: true
dotnet: false
large-packages: false
docker-images: false
swap-storage: false
- name: Test package installation
run: |
pip install dist/*.whl
python -m karaoke_gen.utils.gen_cli --help
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: package-dist
path: dist/
retention-days: 1
# ============================================
# BACKEND TESTS (PRs only - main skips tests)
# BACKEND TESTS (Cloud Run API service)
# ============================================
backend-unit-tests:
name: "Backend - Unit Tests"
runs-on: [self-hosted, linux, gcp]
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -309,10 +182,15 @@ jobs:
backend-emulator-tests:
name: "Backend - Emulator Integration Tests"
runs-on: [self-hosted, linux, gcp]
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
runs-on: ubuntu-latest
steps:
- name: Free disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /usr/local/share/boost
df -h
- name: Checkout code
uses: actions/checkout@v4
with:
@@ -335,6 +213,14 @@ jobs:
distribution: "temurin"
java-version: "21"
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Install Firestore emulator
run: gcloud components install beta cloud-firestore-emulator --quiet
- name: Install dependencies
run: poetry install
@@ -357,8 +243,6 @@ jobs:
backend-lint:
name: "Backend - Code Quality"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true')
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -376,14 +260,6 @@ jobs:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-lint-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-lint-${{ runner.os }}-py3.13-
- name: Install dependencies
run: poetry install
@@ -392,79 +268,10 @@ jobs:
poetry run python -m py_compile backend/main.py backend/config.py
echo "✅ Basic syntax validation passed"
# ============================================
# CI GATE - Required status check for PRs
# Only runs on PRs, not on main
# ============================================
ci-gate:
name: "CI Gate"
runs-on: ubuntu-latest
needs: [frontend-unit-tests, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
if: github.ref != 'refs/heads/main' && always()
steps:
- name: Check results
run: |
results=(
"${{ needs.frontend-unit-tests.result }}"
"${{ needs.frontend-build.result }}"
"${{ needs.package-unit-tests.result }}"
"${{ needs.package-integration-tests.result }}"
"${{ needs.backend-unit-tests.result }}"
"${{ needs.backend-emulator-tests.result }}"
"${{ needs.backend-lint.result }}"
)
for result in "${results[@]}"; do
if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
echo "❌ CI failed"
exit 1
fi
done
echo "✅ CI passed"
# ============================================
# DEPLOYMENTS (main branch only)
# Tests already passed on PR, so we skip them here
# ============================================
package-build:
name: "Package - Build"
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install Poetry
uses: snok/install-poetry@v1
with:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-package-${{ runner.os }}-py3.13-
- name: Build package
run: poetry build
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: package-dist
path: dist/
retention-days: 1
package-publish:
name: "Deploy - Publish to PyPI"
runs-on: ubuntu-latest
@@ -556,16 +363,17 @@ jobs:
name: "Deploy - Frontend (GitHub Pages)"
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Node.js with caching
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Enable Yarn (for lyrics frontend)
run: corepack enable
@@ -582,14 +390,6 @@ jobs:
run: npm run build
# ===== Build Lyrics Review Frontend (Vite) =====
- name: Cache Yarn dependencies (lyrics frontend)
uses: actions/cache@v4
with:
path: lyrics_transcriber_temp/lyrics_transcriber/frontend/node_modules
key: yarn-lyrics-${{ runner.os }}-${{ hashFiles('lyrics_transcriber_temp/lyrics_transcriber/frontend/yarn.lock') }}
restore-keys: |
yarn-lyrics-${{ runner.os }}-
- name: Install lyrics frontend dependencies
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
run: yarn install --frozen-lockfile
@@ -626,28 +426,20 @@ jobs:
deploy-backend:
name: "Deploy - Backend (Cloud Run)"
runs-on: [self-hosted, linux, gcp]
runs-on: ubuntu-latest
needs: [backend-unit-tests, backend-emulator-tests, backend-lint]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
REPOSITORY: karaoke-repo
BASE_IMAGE_NAME: karaoke-backend-base
APP_IMAGE_NAME: karaoke-backend
steps:
# ============================================
# Check if deployment is needed FIRST
# ============================================
- name: Checkout code
uses: actions/checkout@v4
with:
submodules: false
fetch-depth: 0 # Need full history for tag check
- name: Set up Python 3.13
- name: Set up Python (for version extraction)
uses: actions/setup-python@v5
with:
python-version: "3.13"
@@ -661,7 +453,7 @@ jobs:
VERSION=$(poetry version --short)
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "📋 Current version: $VERSION"
if git ls-remote --tags origin | grep -q "refs/tags/v$VERSION$"; then
echo "needs_deploy=false" >> $GITHUB_OUTPUT
echo "⏭️ Tag v$VERSION already exists - skipping backend deployment"
@@ -671,8 +463,6 @@ jobs:
echo "✨ Tag v$VERSION does not exist - proceeding with deployment"
fi
# No disk cleanup needed - self-hosted runner has 200GB SSD
- name: Authenticate to Google Cloud (Workload Identity)
if: steps.version_check.outputs.needs_deploy == 'true'
id: auth
@@ -687,37 +477,21 @@ jobs:
with:
version: "latest"
- name: Configure Docker for Artifact Registry
if: steps.version_check.outputs.needs_deploy == 'true'
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
# ============================================
# Docker Build with BuildKit + Registry Cache
# This replaces expensive Cloud Build with free GitHub Actions
# Registry cache persists between builds for fast rebuilds
# ============================================
- name: Set up Docker Buildx
if: steps.version_check.outputs.needs_deploy == 'true'
uses: docker/setup-buildx-action@v3
- name: Calculate base image hash
if: steps.version_check.outputs.needs_deploy == 'true'
id: base-hash
run: |
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
echo "hash=$CURRENT_HASH" >> $GITHUB_OUTPUT
echo "📋 Base content hash: $CURRENT_HASH"
- name: Check if base image exists and is current
- name: Check if base image needs rebuild
if: steps.version_check.outputs.needs_deploy == 'true'
id: check-base
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
echo "🔍 Checking if base image needs rebuild..."
# Try to get the existing base image's hash label
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
echo "Current combined hash (poetry.lock + Dockerfile.base): $CURRENT_HASH"
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
BASE_IMAGE="us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest"
EXISTING_HASH=""
if docker pull $BASE_IMAGE 2>/dev/null; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
@@ -726,10 +500,10 @@ jobs:
fi
echo "Existing base image hash: $EXISTING_HASH"
else
echo "⚠️ No existing base image found - will build"
echo "⚠️ No existing base image found"
fi
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
if [ "$CURRENT_HASH" != "$EXISTING_HASH" ]; then
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
else
@@ -737,73 +511,88 @@ jobs:
echo "needs_rebuild=false" >> $GITHUB_OUTPUT
fi
- name: Build and push base image (if needed)
- name: Rebuild base image (if needed)
if: steps.version_check.outputs.needs_deploy == 'true' && steps.check-base.outputs.needs_rebuild == 'true'
uses: docker/build-push-action@v6
with:
context: .
file: backend/Dockerfile.base
push: true
tags: |
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:${{ github.sha }}
# Registry cache for ALL layers (mode=max) - persists between builds
# ignore-error=true allows build to continue if cache doesn't exist yet
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,ignore-error=true
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,mode=max,ignore-error=true
build-args: |
BUILD_DATE=${{ github.run_id }}
BUILD_ID=${{ github.sha }}
BASE_CONTENT_HASH=${{ steps.base-hash.outputs.hash }}
# Provenance disabled for smaller image and faster push
provenance: false
run: |
echo "🏗️ Rebuilding base image with updated dependencies..."
- name: Build and push app image
if: steps.version_check.outputs.needs_deploy == 'true'
uses: docker/build-push-action@v6
with:
context: .
file: backend/Dockerfile
push: true
tags: |
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:latest
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:v${{ steps.version_check.outputs.version }}
# Registry cache for app layers
# ignore-error=true allows build to continue if cache doesn't exist yet
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,ignore-error=true
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,mode=max,ignore-error=true
provenance: false
BASE_BUILD_ID=$(gcloud builds submit \
--config=cloudbuild-base.yaml \
--project=nomadkaraoke \
--format='value(id)' \
--async \
.)
# ============================================
# Deploy to Cloud Run
# ============================================
echo "Base Build ID: $BASE_BUILD_ID"
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BASE_BUILD_ID?project=nomadkaraoke"
- name: Deploy to Cloud Run
echo "Waiting for base image build to complete..."
while true; do
BUILD_STATUS=$(gcloud builds describe $BASE_BUILD_ID --project=nomadkaraoke --format='value(status)')
echo "Base build status: $BUILD_STATUS"
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
echo "✅ Base image build completed successfully"
break
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
echo "❌ Base image build failed with status: $BUILD_STATUS"
echo ""
echo "==================== BUILD LOGS ===================="
gcloud logging read "resource.type=build AND resource.labels.build_id=$BASE_BUILD_ID" \
--project=nomadkaraoke \
--format="value(textPayload)" \
--limit=200 \
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
echo "===================================================="
gcloud builds log $BASE_BUILD_ID --project=nomadkaraoke 2>&1 || true
exit 1
fi
sleep 30
done
- name: Trigger Cloud Build
if: steps.version_check.outputs.needs_deploy == 'true'
run: |
VERSION="${{ steps.version_check.outputs.version }}"
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
echo "🚀 Triggering Cloud Build for deployment..."
echo "Commit: ${{ github.sha }}"
echo "Version: ${{ steps.version_check.outputs.version }}"
echo "🚀 Deploying to Cloud Run..."
echo "Image: $IMAGE"
echo "Version: $VERSION"
BUILD_ID=$(gcloud builds submit \
--config=cloudbuild.yaml \
--project=nomadkaraoke \
--format='value(id)' \
--async \
.)
gcloud run deploy karaoke-backend \
--image $IMAGE \
--region us-central1 \
--platform managed \
--allow-unauthenticated \
--memory 16Gi \
--cpu 8 \
--cpu-boost \
--execution-environment gen2 \
--timeout 1800 \
--concurrency 1 \
--max-instances 50 \
--min-instances 0 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
echo "Build ID: $BUILD_ID"
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=nomadkaraoke"
echo "Waiting for build to complete..."
while true; do
BUILD_STATUS=$(gcloud builds describe $BUILD_ID --project=nomadkaraoke --format='value(status)')
echo "Build status: $BUILD_STATUS"
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
echo "✅ Build completed successfully"
break
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
echo "❌ Build failed with status: $BUILD_STATUS"
echo ""
echo "==================== BUILD LOGS ===================="
gcloud logging read "resource.type=build AND resource.labels.build_id=$BUILD_ID" \
--project=nomadkaraoke \
--format="value(textPayload)" \
--limit=200 \
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
echo "===================================================="
echo ""
gcloud builds log $BUILD_ID --project=nomadkaraoke 2>&1 || true
exit 1
fi
sleep 30
done
- name: Verify deployment and version
if: steps.version_check.outputs.needs_deploy == 'true'
@@ -815,22 +604,22 @@ jobs:
echo "Service URL: $SERVICE_URL"
echo "Expected version: $EXPECTED_VERSION"
# Wait for new revision to be ready
echo "Waiting for new revision to be ready..."
sleep 15
# Basic health check
curl -f "$SERVICE_URL/api/health" || exit 1
echo "✅ Service is healthy"
# Version verification with retries (Cloud Run may take time to route to new revision)
MAX_RETRIES=6
RETRY_DELAY=10
for i in $(seq 1 $MAX_RETRIES); do
echo "Checking deployed version (attempt $i/$MAX_RETRIES)..."
DEPLOYED_VERSION=$(curl -sf "$SERVICE_URL/api/health/detailed" | jq -r '.version // empty')
if [ "$DEPLOYED_VERSION" = "$EXPECTED_VERSION" ]; then
echo "✅ Version verified: $DEPLOYED_VERSION"
break
@@ -854,10 +643,7 @@ jobs:
echo "Commit: ${{ github.sha }}"
echo "Branch: ${{ github.ref_name }}"
echo "Run: ${{ github.run_number }}"
echo ""
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
- name: Skipped deployment summary
if: steps.version_check.outputs.needs_deploy == 'false'
run: |
-6
View File
@@ -12,9 +12,6 @@ tests/outputs
/input
/output/
# Cloud Infra
.emulator-logs
# Next.js / React Frontend (./frontend)
node_modules/
frontend/node_modules/
@@ -227,6 +224,3 @@ cython_debug/
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
flacfetch
# Test output directories
Rick Astley*/
File renamed without changes.
+108 -74
View File
@@ -1,97 +1,131 @@
# Karaoke-Gen AI Assistant Guidelines
# Project Guidelines for AI Assistants
## Project Overview
This document contains guidelines and rules for AI assistants working on this codebase.
**Karaoke video generation platform** - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
## Core Principles
- **Production**: <https://gen.nomadkaraoke.com> (frontend), <https://api.nomadkaraoke.com> (backend)
- **CLI**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
- **Repo**: <https://github.com/nomadkaraoke/karaoke-gen>
**Always be testing.** Proactively evaluate, read, run, fix, and improve tests whenever you touch any part of the system—frontend, backend, infrastructure. Every code change should be thoroughly tested. Leave everything better tested than you found it.
## Quick Reference
**Always be documenting.** Continuously read, write, improve, update, and organize markdown documentation. Reference existing docs and keep them accurate as the codebase evolves.
| What | Where |
|------|-------|
| Current status | `docs/README.md` |
| Architecture | `docs/ARCHITECTURE.md` |
| Dev setup & testing | `docs/DEVELOPMENT.md` |
| API reference | `docs/API.md` |
| Past learnings | `docs/LESSONS-LEARNED.md` |
**Build for maintainability.** Keep code SOLID, modular, and clean. Favor simplicity and clarity. Future developers (and AI agents) will thank you.
## Tech Stack
## Branching Workflow
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
- **Frontend**: Next.js on Cloudflare Pages
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
**Never commit directly to `main`.** Always work on a feature branch.
## Essential Rules
Create descriptive branch names like `feature/add-discord-notifications`, `fix/audio-sync-issue`, or `refactor/cleanup-api-routes`.
### Git Workflow
- **Never commit directly to `main`** - always use feature branches
- **Use git worktrees** for all changes:
### Using Git Worktrees
**For non-trivial changes** (new features, multi-file edits, refactors):
- Create a git worktree in the parent folder before making any file changes:
```bash
git worktree add -b feature/xyz ../karaoke-gen-feat-xyz main
git worktree add ../karaoke-gen-feat-xyz feature/xyz
```
- Create PR with summary, changes, testing info
- Merge only after CI passes
- This keeps the main directory clean on `main` and enables parallel work on multiple features
- Name the worktree folder descriptively, e.g., `../karaoke-gen-fix-audio-sync`
### Testing (Required Before Commit)
**For trivial changes** (typos, single-line fixes):
- Creating a branch in the current directory is acceptable
- Use `git checkout -b fix/typo-xyz` before making changes
**Cleanup:** When done with a worktree, remove it with:
```bash
# Backend (10 min timeout, limit output)
make test 2>&1 | tail -n 500
git worktree remove ../karaoke-gen-feat-xyz
```
# Frontend
### Creating a Pull Request
Once all changes are complete, tests pass, and commits are pushed, create a PR with:
1. **Summary** - Brief overview of the changes
2. **Changes Made** - List of specific modifications
3. **Testing** - How changes were tested (unit tests, manual testing)
4. **Related Issues** - Reference any GitHub issues (e.g., "Fixes #123")
Only merge after review and CI checks pass.
## Infrastructure Changes
All Google Cloud infrastructure (IAM, Cloud Storage, Cloud Run, Secret Manager, Artifact Registry, etc.) must be managed via **Pulumi IaC** in the `infrastructure/` folder.
### Rules
- **Reading/querying** with `gcloud` CLI is fine for investigation and debugging
- **Temporary changes** via `gcloud` CLI or Console are acceptable for urgent fixes or experimentation, but **must be cleaned up** and properly codified in the IaC afterward
- **Permanent changes** must go through Pulumi — never leave `gcloud` or Console changes as the final state
### Workflow
1. Make changes in `infrastructure/`
2. Run `pulumi preview` and verify the output
3. Run `pulumi up` to apply
### Authentication Issues
If you encounter any authentication issues with `gcloud` or `pulumi` (e.g., expired tokens, permission errors), **stop and notify the user** so they can run the appropriate auth commands in their external terminal:
```bash
gcloud auth login
gcloud auth application-default login
```
Do not attempt to work around auth issues or make assumptions about credentials.
## Version Management
When committing and pushing changes, always bump the patch version in `pyproject.toml`.
The version is located at `tool.poetry.version` in `pyproject.toml`. For example, if the current version is `0.70.3`, bump it to `0.70.4`.
This ensures every commit pushed to the repository has a unique version number for tracking and deployment purposes.
## Testing Requirements
**Before committing and pushing ANY changes, you MUST run the full test suite.**
### Backend Test Command
```bash
make test 2>&1 | tail -n 500
```
**Important for AI agents:**
- Set a **10 minute timeout** when running tests (they can take a while with emulators)
- Always pipe output through `tail -n 500` to limit output and preserve context window
- The tail output will show test results and any failures at the end
This runs all backend tests in order and fails fast if any test fails:
1. Unit tests (karaoke_gen package) with coverage check
2. Backend unit tests
3. E2E integration tests with emulators (auto-starts/stops emulators)
### Frontend Test Command
When working in the `frontend/` directory, run:
```bash
cd frontend && npm run test:all 2>&1 | tail -n 200
```
### Version Bumping
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
- Skip for docs-only changes
This runs:
1. **Unit tests** (Jest) - Component and utility tests
2. **E2E tests** (Playwright) - Browser-based integration tests
### Infrastructure
- All GCP changes via Pulumi in `infrastructure/`
- `gcloud` CLI for reading/debugging only
- Stop and notify user on auth issues
**Note:** E2E tests require `KARAOKE_ACCESS_TOKEN` environment variable to be set in `frontend/.env.local`.
## Documentation Maintenance
Individual test commands:
- `npm run test:unit` - Run unit tests only
- `npm run test:e2e` - Run E2E tests only
- `npm run test:ci` - Run unit tests with coverage
### Structure
```text
docs/
├── README.md # Current status + navigation (UPDATE THIS)
├── ARCHITECTURE.md # System design
├── DEVELOPMENT.md # Dev setup, testing, deployment
├── API.md # Backend API reference
├── LESSONS-LEARNED.md # Key insights for future agents
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
```
### Pre-commit Checklist
### Rules
1. **Before merging PRs**: Run `/docs-review` to check if docs need updating
2. **Periodically**: Run `/docs-maintain` to verify docs are organized
3. **For significant work**: Create `docs/archive/YYYY-MM-DD-topic.md`
4. **Add learnings**: Update `docs/LESSONS-LEARNED.md` with insights
5. **Keep status current**: Update `docs/README.md` when project state changes
- [ ] `make test` passes (for backend changes)
- [ ] `cd frontend && npm run test:all` passes (for frontend changes)
- [ ] Version in `pyproject.toml` has been bumped
- [ ] `coderabbit review --plain` has been run to get comprehensive analysis and suggestions for cleaner, more maintainable code. Apply the feedback to improve accessibility, structure, and best practices.
### What Goes Where
- **README.md**: Current status, known issues, quick links
- **ARCHITECTURE.md**: System design, data flow, tech decisions
- **DEVELOPMENT.md**: Setup, testing, deployment, CI/CD
- **API.md**: Endpoints, request/response formats
- **LESSONS-LEARNED.md**: What worked, what didn't, gotchas
- **archive/**: Completed work, session summaries, old plans
**Do not commit or push code that fails tests. Fix any test failures before proceeding.**
## Related Projects
### flacfetch
Location: `/Users/andrew/Projects/flacfetch`
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
## Pre-Commit Checklist
- [ ] Tests pass (`make test` / `npm run test:all`)
- [ ] Version bumped (if code changed)
- [ ] `/docs-review` run (check for doc updates)
- [ ] PR created with clear description
+2 -72
View File
@@ -71,40 +71,8 @@ export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
#### Option 2: Local Whisper (No Cloud Required)
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
```bash
# Install with local Whisper support
pip install "karaoke-gen[local-whisper]"
# Optional: Configure model size (tiny, base, small, medium, large)
export WHISPER_MODEL_SIZE="medium"
# Optional: Force specific device (cpu, cuda, mps)
export WHISPER_DEVICE="cpu"
```
**Model Size Guide:**
| Model | VRAM | Speed | Quality |
|-------|------|-------|---------|
| tiny | ~1GB | Fast | Lower |
| base | ~1GB | Fast | Basic |
| small | ~2GB | Medium | Good |
| medium | ~5GB | Slower | Better |
| large | ~10GB | Slowest | Best |
**CPU-Only Installation** (no GPU required):
```bash
# Pre-install CPU-only PyTorch first
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install "karaoke-gen[local-whisper]"
```
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
#### Option 3: Whisper via RunPod
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
#### Option 2: Whisper via RunPod
Open-source alternative using OpenAI's Whisper model on RunPod infrastructure.
```bash
export RUNPOD_API_KEY="your_runpod_key"
@@ -606,44 +574,6 @@ If the output video has quality problems:
- Check available codecs: `ffmpeg -codecs`
- For 4K output, ensure sufficient disk space (10GB+ per track)
### Local Whisper Issues
#### GPU Out of Memory
If you get CUDA out of memory errors:
```bash
# Use a smaller model
export WHISPER_MODEL_SIZE="small" # or "tiny"
# Or force CPU mode
export WHISPER_DEVICE="cpu"
```
#### Slow Transcription on CPU
CPU transcription is significantly slower than GPU. For faster processing:
- Use a smaller model (`tiny` or `base`)
- Consider using cloud transcription (AudioShake or RunPod)
- On Apple Silicon, the `small` model offers good speed/quality balance
#### Model Download Issues
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
- Check your internet connection
- Set a custom cache directory: `export WHISPER_CACHE_DIR="/path/with/space"`
- Models are cached in `~/.cache/whisper/` by default
#### whisper-timestamped Not Found
If you get "whisper-timestamped is not installed":
```bash
pip install "karaoke-gen[local-whisper]"
# Or install directly:
pip install whisper-timestamped
```
#### Disabling Local Whisper
If you want to disable local Whisper (e.g., to force cloud transcription):
```bash
export ENABLE_LOCAL_WHISPER="false"
```
---
## 🧪 Development
+7 -68
View File
@@ -85,13 +85,9 @@ class AudioSearchRequest(BaseModel):
# Auto-download mode
auto_download: bool = Field(False, description="Automatically select best result and download")
# Theme configuration
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
# Processing options (CDG/TXT require style config, disabled by default)
enable_cdg: bool = Field(False, description="Generate CDG+MP3 package (requires style config)")
enable_txt: bool = Field(False, description="Generate TXT+MP3 package (requires style config)")
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
@@ -184,35 +180,6 @@ class AudioSelectResponse(BaseModel):
selected_provider: str
def _resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool],
enable_txt: Optional[bool]
) -> Tuple[bool, bool]:
"""
Resolve CDG/TXT settings based on theme and explicit settings.
When a theme is selected, CDG and TXT are enabled by default.
Explicit True/False values always override the default.
Args:
theme_id: Theme identifier (if any)
enable_cdg: Explicit CDG setting (None means use default)
enable_txt: Explicit TXT setting (None means use default)
Returns:
Tuple of (resolved_enable_cdg, resolved_enable_txt)
"""
# Default based on whether theme is set
default_enabled = theme_id is not None
# Explicit values override defaults, None uses default
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
resolved_txt = enable_txt if enable_txt is not None else default_enabled
return resolved_cdg, resolved_txt
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
"""Extract metadata from request for job tracking."""
headers = dict(request.headers)
@@ -494,20 +461,13 @@ async def search_audio(
# Extract request metadata
request_metadata = extract_request_metadata(request, created_from="audio_search")
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Create job
job_create = JobCreate(
artist=body.artist,
title=body.title,
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
enable_cdg=body.enable_cdg,
enable_txt=body.enable_txt,
brand_prefix=effective_brand_prefix,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=effective_youtube_description,
@@ -537,28 +497,7 @@ async def search_audio(
'audio_search_title': body.title,
'auto_download': body.auto_download,
})
# If theme is set and no custom style files are being uploaded, prepare theme style now
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
# can access the style configuration for preview videos
if body.theme_id and not body.style_files:
from backend.api.routes.file_upload import _prepare_theme_for_job
try:
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
job_id, body.theme_id, body.color_overrides
)
theme_update = {
'style_params_gcs_path': style_params_path,
'style_assets': theme_style_assets,
}
if youtube_desc and not effective_youtube_description:
theme_update['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, theme_update)
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
except Exception as e:
logger.warning(f"Failed to prepare theme '{body.theme_id}' for job {job_id}: {e}")
# Continue without theme - job can still be processed with defaults
# Handle style file uploads if provided
style_upload_urls: List[StyleUploadUrl] = []
style_assets = {}
-5
View File
@@ -61,11 +61,6 @@ class JobManager:
discord_webhook_url=job_create.discord_webhook_url,
dropbox_path=job_create.dropbox_path,
gdrive_folder_id=job_create.gdrive_folder_id,
# Theme configuration
theme_id=job_create.theme_id,
color_overrides=job_create.color_overrides,
style_params_gcs_path=job_create.style_params_gcs_path,
style_assets=job_create.style_assets,
# Two-phase workflow (Batch 6)
prep_only=job_create.prep_only,
finalise_only=job_create.finalise_only,
@@ -330,7 +330,7 @@ class TestReviewEndpoints:
class TestRenderVideoWorker:
"""Test render video worker endpoint."""
def test_render_video_worker_endpoint_exists(self, client, auth_headers):
"""Test that render-video worker endpoint exists."""
# Create a job
@@ -340,9 +340,9 @@ class TestRenderVideoWorker:
json={"url": "https://youtube.com/watch?v=render-test"}
)
job_id = create_resp.json()["job_id"]
time.sleep(0.2)
# Test render-video worker endpoint exists
response = client.post(
"/api/internal/workers/render-video",
-248
View File
@@ -798,254 +798,6 @@ class TestRemoteDownloadPath:
assert result.filepath == "/tmp/test.flac"
class TestAudioSearchThemeSupport:
"""Test theme support in audio search requests.
These tests verify that theme_id and color_overrides are properly:
1. Accepted in AudioSearchRequest model
2. Passed through to JobCreate
3. Result in correct CDG/TXT defaults when theme is set
4. Theme style is PREPARED (copied to job folder) when theme_id is set
"""
def test_audio_search_request_accepts_theme_id(self):
"""Test AudioSearchRequest model has theme_id field."""
from backend.api.routes.audio_search import AudioSearchRequest
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
theme_id="nomad"
)
assert request.theme_id == "nomad"
def test_audio_search_request_accepts_color_overrides(self):
"""Test AudioSearchRequest model has color_overrides field."""
from backend.api.routes.audio_search import AudioSearchRequest
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
theme_id="nomad",
color_overrides={
"artist_color": "#ff0000",
"title_color": "#00ff00",
}
)
assert request.color_overrides["artist_color"] == "#ff0000"
assert request.color_overrides["title_color"] == "#00ff00"
def test_audio_search_request_theme_defaults_cdg_txt(self):
"""Test that when theme_id is set, CDG/TXT defaults to enabled.
This is the key behavior: selecting a theme should automatically
enable CDG and TXT output formats.
"""
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
# When theme_id is set, enable_cdg/enable_txt should default to True
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
theme_id="nomad"
# enable_cdg and enable_txt are None (not specified)
)
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
assert resolved_cdg is True, "CDG should be enabled by default when theme is set"
assert resolved_txt is True, "TXT should be enabled by default when theme is set"
def test_audio_search_request_no_theme_no_cdg_txt(self):
"""Test that without theme_id, CDG/TXT defaults to disabled."""
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song"
# No theme_id, no enable_cdg, no enable_txt
)
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
assert resolved_cdg is False, "CDG should be disabled by default without theme"
assert resolved_txt is False, "TXT should be disabled by default without theme"
def test_explicit_cdg_txt_overrides_theme_default(self):
"""Test that explicit enable_cdg/enable_txt values override theme defaults."""
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
# Theme set (would default to True), but explicitly disabled
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
theme_id="nomad",
enable_cdg=False,
enable_txt=False,
)
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
assert resolved_cdg is False, "Explicit False should override theme default"
assert resolved_txt is False, "Explicit False should override theme default"
def test_explicit_cdg_txt_enables_without_theme(self):
"""Test that explicit True enables CDG/TXT even without theme."""
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
# No theme (would default to False), but explicitly enabled
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
enable_cdg=True,
enable_txt=True,
)
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
assert resolved_cdg is True, "Explicit True should enable CDG without theme"
assert resolved_txt is True, "Explicit True should enable TXT without theme"
def test_job_create_receives_theme_from_audio_search(self):
"""Test that JobCreate model can receive theme_id and color_overrides."""
job_create = JobCreate(
artist="Test Artist",
title="Test Song",
theme_id="nomad",
color_overrides={
"artist_color": "#ff0000",
"sung_lyrics_color": "#00ff00"
},
enable_cdg=True,
enable_txt=True,
)
assert job_create.theme_id == "nomad"
assert job_create.color_overrides["artist_color"] == "#ff0000"
assert job_create.color_overrides["sung_lyrics_color"] == "#00ff00"
assert job_create.enable_cdg is True
assert job_create.enable_txt is True
def test_job_model_stores_theme_configuration(self):
"""Test Job model stores theme configuration correctly."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.utcnow(),
updated_at=datetime.utcnow(),
artist="Test Artist",
title="Test Song",
theme_id="nomad",
color_overrides={
"artist_color": "#ff0000"
},
enable_cdg=True,
enable_txt=True,
)
assert job.theme_id == "nomad"
assert job.color_overrides["artist_color"] == "#ff0000"
assert job.enable_cdg is True
assert job.enable_txt is True
class TestAudioSearchThemePreparation:
"""Test that audio search endpoint properly prepares theme styles.
CRITICAL: This test class was added to catch the bug where audio_search.py
accepted theme_id but never called prepare_job_style(), resulting in jobs
with theme_id set but no style_params_gcs_path or style_assets.
The symptom: Preview videos show black background instead of themed background.
Job ID ffb0b8fa in production demonstrated this issue.
"""
def test_audio_search_with_theme_calls_prepare_job_style(self):
"""Test that creating a job via audio search with theme_id prepares the style.
CRITICAL: This test verifies that when a job is created via the audio-search
endpoint with a theme_id (and no custom style files), the code calls
prepare_job_style() to set:
1. style_params_gcs_path (pointing to the copied style_params.json)
2. style_assets (populated with asset mappings)
Without this, LyricsTranscriber won't have access to the theme's styles
and preview videos will have black backgrounds instead of themed ones.
"""
# Import the endpoint module to access internal functions
from backend.api.routes import audio_search as audio_search_module
# Check if there's an import of prepare_job_style from theme_service
source_code_path = audio_search_module.__file__
with open(source_code_path, 'r') as f:
source_code = f.read()
# Check for either:
# 1. Import of _prepare_theme_for_job from file_upload
# 2. Import of prepare_job_style from theme_service
# 3. Inline call to theme_service.prepare_job_style
has_theme_prep_import = (
'_prepare_theme_for_job' in source_code or
'prepare_job_style' in source_code
)
assert has_theme_prep_import, (
"audio_search.py does not import or call prepare_job_style() or _prepare_theme_for_job(). "
"When theme_id is provided without custom style files, the endpoint MUST call "
"prepare_job_style() to copy the theme's style_params.json to the job folder. "
"Without this, LyricsTranscriber won't have style configuration and preview "
"videos will have black backgrounds instead of themed ones."
)
def test_audio_search_code_handles_theme_preparation(self):
"""Verify audio_search.py has theme preparation logic similar to file_upload.py.
This test compares audio_search.py with file_upload.py to ensure they have
similar theme preparation patterns. file_upload.py correctly prepares themes,
audio_search.py should do the same.
"""
from backend.api.routes import audio_search as audio_search_module
from backend.api.routes import file_upload as file_upload_module
# Read source code of both modules
with open(audio_search_module.__file__, 'r') as f:
audio_search_code = f.read()
with open(file_upload_module.__file__, 'r') as f:
file_upload_code = f.read()
# file_upload.py has this pattern for theme preparation:
# if body.theme_id and not has_style_params_upload:
# style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(...)
assert '_prepare_theme_for_job' in file_upload_code, \
"file_upload.py should have _prepare_theme_for_job function"
# audio_search.py should have similar pattern
# Look for either the function or a similar conditional check
# STRICT CHECK: Look for actual theme preparation logic being CALLED, not just mentioned
has_theme_preparation_call = (
# Check for import or call of theme preparation function
'_prepare_theme_for_job(' in audio_search_code or
'prepare_job_style(' in audio_search_code
)
assert has_theme_preparation_call, (
"audio_search.py does not CALL theme preparation logic. "
"It accepts theme_id but never prepares the theme style like file_upload.py does. "
"Add a call to _prepare_theme_for_job() when theme_id is set and no style files are uploaded."
)
class TestFlacfetchIntegration:
"""
Integration tests that verify flacfetch imports work correctly.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
-183
View File
@@ -1,183 +0,0 @@
# API Reference
Base URL: `https://api.nomadkaraoke.com` (production) or `http://localhost:8000` (local)
## Authentication
All endpoints except `/health` and `/api/themes` require authentication.
```bash
curl -H "Authorization: Bearer YOUR_TOKEN" ...
```
## Endpoints
### Health
```http
GET /health
```
No auth required. Returns service status.
### Jobs
#### Create Job with Upload
```http
POST /api/jobs/upload
Content-Type: multipart/form-data
file: <audio file>
artist: "Artist Name"
title: "Song Title"
```
Returns job ID. Triggers async processing.
#### Get Job Status
```http
GET /api/jobs/{job_id}
```
Response includes:
- `status` - Current job state
- `file_urls` - Signed URLs for outputs (when ready)
- `state_data` - Worker progress details
- `timeline` - Processing history
#### List Jobs
```http
GET /api/jobs
GET /api/jobs?status=complete
GET /api/jobs?limit=10&offset=0
```
#### Delete Job
```http
DELETE /api/jobs/{job_id}
```
### Review
#### Get Correction Data
```http
GET /api/review/{job_id}/correction-data
```
Returns lyrics correction data for the review UI.
#### Complete Review
```http
POST /api/review/{job_id}/complete
Content-Type: application/json
{
"corrections": [...],
"corrected_segments": [...]
}
```
Saves corrections and triggers video rendering.
#### Generate Preview
```http
POST /api/review/{job_id}/preview-video
```
Generates preview video during review.
#### Stream Audio
```http
GET /api/review/{job_id}/audio/{stem_type}
```
Streams audio for review playback.
### Instrumental Selection
```http
POST /api/jobs/{job_id}/select-instrumental
Content-Type: application/json
{
"selection": "clean" // or "with_backing"
}
```
### Audio Search
```http
GET /api/audio-search?q=song+name
```
Search for songs (flacfetch integration).
### Themes
```http
GET /api/themes
```
No auth required. Returns available video themes.
### Internal (Admin Only)
These endpoints are used by workers and require admin tokens.
```http
POST /api/internal/jobs/{job_id}/trigger-audio
POST /api/internal/jobs/{job_id}/trigger-lyrics
POST /api/internal/jobs/{job_id}/trigger-screens
POST /api/internal/jobs/{job_id}/trigger-render-video
POST /api/internal/jobs/{job_id}/trigger-video
```
## Job States
| State | Description |
|-------|-------------|
| `pending` | Created, not started |
| `downloading` | Processing input |
| `separating_stage1` | Audio separation (1/2) |
| `separating_stage2` | Audio separation (2/2) |
| `transcribing` | Lyrics transcription |
| `generating_screens` | Title/end screens |
| `awaiting_review` | Waiting for human review |
| `in_review` | Human reviewing |
| `review_complete` | Review submitted |
| `rendering_video` | Generating karaoke video |
| `awaiting_instrumental_selection` | Waiting for selection |
| `instrumental_selected` | Selection made |
| `generating_video` | Final encoding |
| `complete` | Done |
| `failed` | Error |
## Error Responses
```json
{
"detail": "Error message"
}
```
Common status codes:
- `400` - Bad request
- `401` - Unauthorized
- `404` - Job not found
- `500` - Server error
## Rate Limits
No rate limits currently implemented.
## Webhooks
Not yet implemented. Jobs must be polled for status.
-162
View File
@@ -1,162 +0,0 @@
# Architecture
## System Overview
```text
┌─────────────────────────────────────────────────────────────────┐
│ Frontend (Cloudflare Pages) │
│ Next.js + TypeScript │
│ https://gen.nomadkaraoke.com │
└───────────────────────────┬─────────────────────────────────────┘
│ HTTPS
▼
┌─────────────────────────────────────────────────────────────────┐
│ Backend API (Cloud Run) │
│ FastAPI + Python 3.12 │
│ https://api.nomadkaraoke.com │
│ │
│ Workers: │
│ • Audio Worker - Stem separation via Modal API │
│ • Lyrics Worker - Transcription via AudioShake │
│ • Screens Worker - Title/end screen generation │
│ • Render Worker - Karaoke video with lyrics │
│ • Video Worker - Final assembly and encoding │
└───────────────────────────┬─────────────────────────────────────┘
│
┌─────────────┼─────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────────┐
│ Firestore│ │ GCS │ │Secret Manager│
│ (jobs) │ │ (files) │ │ (API keys) │
└──────────┘ └──────────┘ └──────────────┘
```
## Processing Pipeline
```text
1. UPLOAD
POST /api/jobs/upload
• Save audio to GCS
• Create job in Firestore
• Trigger parallel workers
2. PARALLEL PROCESSING
┌─────────────────┐ ┌─────────────────┐
│ Audio Worker │ │ Lyrics Worker │
│ Modal API │ │ AudioShake │
│ 2-stage sep │ │ Auto-correct │
└────────┬────────┘ └────────┬────────┘
└──────────┬───────────┘
▼
3. SCREENS GENERATION
• Title screen (5s video)
• End screen (5s video)
▼
4. HUMAN REVIEW (AWAITING_REVIEW)
• React UI for lyrics correction
• Preview video generation
• Submit corrections
▼
5. VIDEO RENDERING
• Merge corrections with original data
• Generate karaoke video with lyrics
▼
6. INSTRUMENTAL SELECTION
• User chooses: clean or with_backing
▼
7. FINAL VIDEO
• Remux with selected instrumental
• Concatenate: title + karaoke + end
• Encode to 4 formats
▼
8. COMPLETE
• User downloads via signed URLs
```
## Job States
| State | Description | Human Action |
|-------|-------------|--------------|
| `pending` | Job created | - |
| `downloading` | Processing audio | - |
| `separating_stage1` | Audio separation stage 1 | - |
| `separating_stage2` | Audio separation stage 2 | - |
| `transcribing` | Lyrics transcription | - |
| `generating_screens` | Creating title/end screens | - |
| `awaiting_review` | **Waiting for human** | Review lyrics |
| `in_review` | Human actively reviewing | Edit lyrics |
| `review_complete` | Review submitted | - |
| `rendering_video` | Generating karaoke video | - |
| `awaiting_instrumental_selection` | **Waiting for human** | Choose audio |
| `instrumental_selected` | Selection made | - |
| `generating_video` | Final encoding | - |
| `complete` | All done | Download files |
| `failed` | Error occurred | - |
## GCS File Structure
```text
jobs/{job_id}/
├── input/
│ └── original.flac # Uploaded audio
├── stems/
│ ├── vocals.flac
│ ├── instrumental_clean.flac
│ ├── instrumental_with_backing.flac
│ └── ...
├── lyrics/
│ ├── corrections.json # Auto-corrected
│ ├── corrections_updated.json # After human review
│ └── karaoke.ass
├── screens/
│ ├── title.mov
│ └── end.mov
├── videos/
│ └── with_vocals.mkv # Karaoke video
└── finals/
├── lossless_4k.mp4
├── lossless_4k.mkv
├── lossy_4k.mp4
└── lossy_720p.mp4
```
## LyricsTranscriber Integration
**Key Design Decision**: We use LyricsTranscriber as a **library**, not a server.
```text
CLI Mode (Original): Cloud Mode (Our Approach):
──────────────────── ──────────────────────────
LyricsTranscriber LyricsTranscriber
├── transcribe() ├── transcribe() ✅
├── correct() ├── correct() ✅
├── ReviewServer.start() │
│ └── BLOCKS (waits) │ (Skip server, save JSON)
│ │ ... ASYNC GAP ...
│ │ (Human reviews via React)
└── OutputGenerator │
└── OutputGenerator ✅
(called after review)
```
**Use**: `LyricsTranscriber`, `CorrectionResult`, `OutputGenerator`, `OutputConfig`
**Don't use**: `ReviewServer`, blocking waits
## External Services
| Service | Purpose | Required |
|---------|---------|----------|
| Modal | GPU audio separation | Yes |
| AudioShake | Lyrics transcription | Yes |
| Genius | Reference lyrics | Yes |
| YouTube API | Video upload | Optional |
## Tech Stack
- **Backend**: FastAPI, Python 3.12, Cloud Run
- **Frontend**: Next.js, TypeScript, Cloudflare Pages
- **Database**: Firestore
- **Storage**: Google Cloud Storage
- **Secrets**: Google Secret Manager
- **IaC**: Pulumi
- **CI/CD**: GitHub Actions
-216
View File
@@ -1,216 +0,0 @@
# Development Guide
## Quick Start
### Prerequisites
- Python 3.12
- Node.js 20+
- Docker (for GCS emulator)
- Google Cloud SDK with Firestore emulator
```bash
# Install Firestore emulator
gcloud components install cloud-firestore-emulator
```
### Backend Setup
```bash
# Create and activate venv
python3.12 -m venv venv
source venv/bin/activate
# Install dependencies
poetry install
# Copy environment template
cp .env.example .env
# Edit .env with required API keys
```
### Frontend Setup
```bash
cd frontend
npm install
cp .env.example .env.local
# Edit .env.local with KARAOKE_ACCESS_TOKEN
```
## Running Locally
### Backend with Emulators
```bash
# Start emulators and backend
./scripts/run-backend-local.sh --with-emulators
# Backend runs at http://localhost:8000
# API docs at http://localhost:8000/docs
```
### Frontend
```bash
cd frontend
npm run dev
# Runs at http://localhost:3000
```
## Testing
### Backend Tests
```bash
# Run all tests (required before commit)
make test 2>&1 | tail -n 500
# This runs in order:
# 1. Unit tests with coverage
# 2. Backend unit tests
# 3. E2E integration tests with emulators
```
**Important**: Set 10 minute timeout. Tests can take a while with emulators.
### Frontend Tests
```bash
cd frontend
npm run test:all 2>&1 | tail -n 200
# Individual commands:
npm run test:unit # Jest unit tests
npm run test:e2e # Playwright E2E tests
```
### Test Architecture
```
┌─────────────────────────────────┐
│ Unit Tests (Mocked) │ Fast, isolated
│ ~62 tests, <1s │
├─────────────────────────────────┤
│ Emulator Integration Tests │ Local Firestore/GCS
│ ~11 tests, ~2s │
├─────────────────────────────────┤
│ E2E Tests │ Full API + Frontend
│ Requires deployed backend │
└─────────────────────────────────┘
```
## Deployment
### Automatic (CI/CD)
Push to `main` triggers GitHub Actions:
1. Run tests
2. Build Docker image
3. Deploy to Cloud Run
4. Deploy frontend to Cloudflare Pages
### Manual Backend Deploy
```bash
# Preview infrastructure changes
cd infrastructure
pulumi preview
# Apply changes
pulumi up
```
### Manual Frontend Deploy
```bash
cd frontend
npm run build
npx wrangler pages deploy out
```
## Environment Variables
### Backend (.env)
```bash
# Required
AUDIOSHAKE_API_TOKEN=...
GENIUS_ACCESS_TOKEN=...
AUDIO_SEPARATOR_API_URL=... # Modal API endpoint
# GCP (auto-configured in Cloud Run)
GOOGLE_CLOUD_PROJECT=...
GCS_BUCKET_NAME=...
# Auth
ADMIN_TOKENS=token1,token2
```
### Frontend (.env.local)
```bash
NEXT_PUBLIC_API_URL=http://localhost:8000
KARAOKE_ACCESS_TOKEN=... # For E2E tests
```
## Key Scripts
| Script | Purpose |
|--------|---------|
| `make test` | Run all backend tests |
| `./scripts/run-backend-local.sh` | Start backend with emulators |
| `./scripts/start-emulators.sh` | Start emulators only |
| `./scripts/stop-emulators.sh` | Stop emulators |
## Project Structure
```
karaoke-gen/
├── backend/
│ ├── api/routes/ # FastAPI endpoints
│ ├── models/ # Pydantic models
│ ├── services/ # Business logic
│ ├── workers/ # Background workers
│ └── tests/ # Backend tests
├── frontend/
│ ├── src/
│ │ ├── app/ # Next.js pages
│ │ ├── components/ # React components
│ │ └── lib/ # Utilities
│ └── tests/ # Frontend tests
├── infrastructure/ # Pulumi IaC
├── karaoke_gen/ # CLI package
└── scripts/ # Dev scripts
```
## Troubleshooting
### Emulators Won't Start
```bash
# Check if already running
lsof -i :8080 # Firestore
lsof -i :4443 # GCS
# Stop them
./scripts/stop-emulators.sh
```
### Tests Fail with Auth Errors
Ensure `ADMIN_TOKENS` is set in environment or `.env` file.
### Docker Issues
```bash
# Ensure Docker is running
docker info
# For GCS emulator
docker pull fsouza/fake-gcs-server
```
### Coverage Below Target
Run `./scripts/run-tests.sh --coverage` and check `htmlcov/index.html` to see uncovered lines.
-151
View File
@@ -1,151 +0,0 @@
# Lessons Learned
Key insights for future AI agents working on this codebase.
## Architecture Decisions
### LyricsTranscriber: Library, Not Server
**Problem**: LyricsTranscriber's `ReviewServer` blocks waiting for human input - incompatible with async cloud architecture.
**Solution**: Use LyricsTranscriber as a library:
- Call `transcribe()` and `correct()` for processing
- Save `corrections.json` to GCS
- Skip `ReviewServer` entirely
- Call `OutputGenerator` only after human review completes
**What to use**: `LyricsTranscriber`, `CorrectionResult`, `OutputGenerator`, `OutputConfig`, `CorrectionOperations`
**What NOT to use**: `ReviewServer`, `server.start()`, blocking waits
### Video Generation Timing
**Wrong**: Generate video during lyrics worker, then review
**Right**: Lyrics worker saves JSON → Human reviews → Render worker generates video
Video must include human-corrected lyrics, so generation happens AFTER review.
### Parallel Processing
Audio separation and lyrics transcription run in parallel:
- Both can fail independently
- Progress tracked separately in `state_data`
- Screens generation waits for both to complete
## Common Gotchas
### Pydantic Model Fields
**Problem**: Setting a field that doesn't exist in the Pydantic model silently does nothing.
```python
# BAD - fails silently if field doesn't exist
job.input_media_gcs_path = "path/to/file"
# GOOD - explicit field in model
class Job(BaseModel):
input_media_gcs_path: Optional[str] = None
```
**Lesson**: Always verify Pydantic model has the field before setting it.
### Styles Configuration
ASS subtitle generator requires ALL style fields present and properly typed:
```python
required_fields = [
"font", "font_path", "ass_name", # ass_name often missed
"primary_color", "secondary_color", "outline_color", "back_color",
"bold", "italic", "underline", "strike_out",
"scale_x", "scale_y", "spacing", "angle",
"border_style", "outline", "shadow",
"margin_l", "margin_r", "margin_v", "encoding"
]
# Critical: font_path must be string, not None
"font_path": "" # Correct
"font_path": None # Causes ASS writer to fail
```
### Frontend Data Format
The React review UI sends partial correction data:
```python
# Frontend sends:
{"corrections": [...], "corrected_segments": [...]}
# Backend must merge with original corrections.json
```
### Firestore Consistency
**Problem**: Update job, trigger worker immediately, worker reads stale data.
**Solution**: Verify updates propagated before triggering workers:
```python
# Update job
await job_manager.update_job(job_id, updates)
# Verify update visible
job = await job_manager.get_job(job_id)
if job.input_media_gcs_path:
await trigger_worker()
```
## Testing Insights
### Emulator Tests Catch Real Bugs
Unit tests with mocks didn't catch the `input_media_gcs_path` bug. Emulator integration tests did because they use real Firestore behavior.
**Lesson**: Run `make test` which includes emulator tests before every commit.
### Test Isolation
Emulator tests need their own `conftest.py` that does NOT mock `google.cloud.*`. Mocks from unit tests can leak into integration tests.
## Deployment Notes
### Cloud Run Timeouts
Default 5-minute timeout may not be enough for video encoding. Consider:
- Cloud Run Jobs for long-running tasks
- Breaking encoding into smaller chunks
- Increasing timeout for specific operations
### Secret Manager Access
Cloud Run service account needs `Secret Manager Secret Accessor` role. This is managed via Pulumi in `infrastructure/`.
### CI/CD: GitHub Actions
Migrated from Cloud Build to GitHub Actions for:
- Simpler configuration
- Better visibility
- Faster feedback
## Performance Observations
| Operation | Duration |
|-----------|----------|
| Audio separation | 5-8 min |
| Lyrics transcription | 2-3 min |
| Auto-correction | 30 sec |
| Screens generation | 30 sec |
| Video rendering | 10-15 min |
| Final encoding | 5-10 min |
| **Total** | 30-50 min |
Human review time varies (5-15 min typical).
## What We'd Do Differently
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
2. **Use emulator tests from day one** - Faster feedback than deploying to test
3. **Design for async human review upfront** - Avoided rearchitecting after discovering ReviewServer blocks
4. **Keep docs minimal and current** - Less documentation, but always accurate
-318
View File
@@ -1,318 +0,0 @@
# Mobile UX Best Practices
This document outlines the mobile-first UX principles and implementation patterns used in this project to ensure a great experience across all screen sizes.
## Core Principles
### 1. Mobile-First Responsive Design
Design for the smallest screen first, then progressively enhance for larger screens.
```css
/* Base styles for mobile */
.element {
width: 100%;
padding: 12px;
}
/* Enhance for larger screens */
@media (min-width: 640px) {
.element {
width: 280px;
padding: 16px;
}
}
```
### 2. No Horizontal Overflow
The most critical mobile UX issue is horizontal scrolling. Always ensure content fits within the viewport width.
**Testing for overflow:**
```typescript
async function hasNoHorizontalOverflow(page: Page): Promise<boolean> {
return await page.evaluate(() => {
const docWidth = document.documentElement.scrollWidth;
const viewportWidth = window.innerWidth;
return docWidth <= viewportWidth + 2; // Small tolerance for rounding
});
}
```
**Common causes of horizontal overflow:**
- Fixed widths (e.g., `width: 340px`)
- Horizontal flex layouts without wrapping
- Images without `max-width: 100%`
- Tables without horizontal scroll containers
### 3. Touch Target Sizing
Interactive elements must be easily tappable. Follow these minimum sizes:
| Platform | Minimum Size | Recommended |
|----------|--------------|-------------|
| Apple iOS | 44x44px | 48x48px |
| Material Design | 48x48px | 48x48px |
| Web (general) | 32x32px | 44x44px |
**Implementation:**
```tsx
<Button className="min-h-[44px] px-3">
<Icon className="w-4 h-4" />
<span className="hidden sm:inline">Label</span>
</Button>
```
### 4. Progressive Disclosure
Show less content on mobile, with clear paths to access more:
- **Icon-only buttons on mobile, labels on desktop:**
```tsx
<Icon className="w-4 h-4 sm:mr-2" />
<span className="hidden sm:inline">Action Label</span>
```
- **Collapsed navigation on mobile**
- **Truncated text with expansion options**
## Layout Patterns
### Responsive Grid
Use responsive grid columns that stack on mobile:
```tsx
// Bad - always 2 columns
<div className="grid grid-cols-2 gap-4">
// Good - stack on mobile, 2 columns on larger screens
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 sm:gap-4">
```
### Flexible Containers
Avoid fixed widths. Use percentage-based or responsive widths:
```css
/* Bad */
.sidebar {
width: 340px;
}
/* Good */
.sidebar {
width: 100%;
}
@media (min-width: 768px) {
.sidebar {
width: 340px;
}
}
```
### Flex Wrap for Toolbars
Allow toolbar items to wrap on smaller screens:
```tsx
<div className="flex flex-wrap gap-2 items-center">
{/* Items will wrap naturally */}
</div>
```
## Typography
### Readable Font Sizes
Minimum font sizes for readability without zooming:
| Element | Minimum Size | Recommended |
|---------|--------------|-------------|
| Body text | 14px | 16px |
| Labels | 12px | 14px |
| Small text/badges | 11px | 12px |
### Responsive Headings
```tsx
<h1 className="text-base sm:text-xl font-bold truncate">
Title Text
</h1>
```
## Form Design
### Input Fields on Mobile
- Stack form fields vertically on mobile
- Use full-width inputs for easier tapping
- Ensure proper input type for mobile keyboard optimization
```tsx
<input
type="email" // Shows email keyboard
inputMode="numeric" // Shows number pad
className="w-full min-h-[44px]"
/>
```
### Labels and Placeholders
- Labels should be visible (not just placeholders)
- Adequate spacing between label and input
- Clear validation feedback
## Testing Strategy
### Automated Testing with Playwright
Test across multiple viewport sizes representing real devices:
```typescript
const MOBILE_VIEWPORTS = {
'iPhone SE': { width: 375, height: 667 },
'iPhone 14': { width: 390, height: 844 },
'Pixel 7': { width: 412, height: 915 },
'iPad Mini': { width: 768, height: 1024 },
};
for (const [name, viewport] of Object.entries(MOBILE_VIEWPORTS)) {
test(`page works on ${name}`, async ({ page }) => {
await page.setViewportSize(viewport);
await page.goto('/');
// Test for horizontal overflow
const hasOverflow = await page.evaluate(() =>
document.documentElement.scrollWidth > window.innerWidth + 2
);
expect(hasOverflow).toBe(false);
});
}
```
### Visual Regression Testing
Capture screenshots at different viewports to catch visual regressions:
```typescript
await expect(page).toHaveScreenshot(`homepage-${deviceName}.png`, {
fullPage: true,
maxDiffPixelRatio: 0.1,
});
```
### Touch Target Validation
```typescript
const buttons = page.locator('button, a, [role="button"]');
for (const button of await buttons.all()) {
const box = await button.boundingBox();
if (box) {
expect(box.width).toBeGreaterThanOrEqual(32);
expect(box.height).toBeGreaterThanOrEqual(32);
}
}
```
## Framework-Specific Guidelines
### Tailwind CSS (Next.js Frontend)
Use responsive prefixes consistently:
```plaintext
sm: 640px - Small tablets and up
md: 768px - Tablets and up
lg: 1024px - Laptops and up
xl: 1280px - Desktops and up
```
**Common patterns:**
```tsx
// Stack on mobile, side-by-side on larger
className="flex flex-col sm:flex-row"
// Hide on mobile, show on larger
className="hidden sm:block"
// Reduce padding on mobile
className="p-3 sm:p-4"
// Smaller font on mobile
className="text-xs sm:text-sm"
```
### Material UI (Lyrics Review UI)
Use the `useMediaQuery` hook:
```tsx
const theme = useTheme();
const isMobile = useMediaQuery(theme.breakpoints.down('md'));
return (
<Box sx={{
flexDirection: isMobile ? 'column' : 'row',
width: isMobile ? '100%' : '280px',
}}>
```
### Vanilla CSS (Instrumental Review)
Use CSS media queries:
```css
.element {
width: 100%;
}
@media (min-width: 768px) {
.element {
width: 340px;
}
}
```
## Checklist for New Features
Before merging any UI changes, verify:
- [ ] No horizontal overflow on any mobile viewport (320px - 480px)
- [ ] All interactive elements are at least 44x44px
- [ ] Text is readable without zooming (minimum 11px)
- [ ] Form inputs stack vertically on mobile
- [ ] Navigation is accessible on mobile
- [ ] Images scale properly with `max-width: 100%`
- [ ] Modals/dialogs fit within mobile viewport
- [ ] Buttons show icons-only on very small screens
- [ ] Playwright mobile tests pass
## Common Fixes
### Fix: Horizontal Overflow
1. **Check for fixed widths** - Replace with responsive values
2. **Add flex-wrap** - Allow content to wrap
3. **Add overflow-x: hidden** - On containers (use sparingly)
4. **Check images** - Add `max-width: 100%`
### Fix: Touch Target Too Small
1. **Add minimum height** - `min-h-[44px]`
2. **Add padding** - `p-3` or `px-4 py-3`
3. **Use icon-only on mobile** - Hide text labels
### Fix: Text Too Small
1. **Use responsive font sizes** - `text-xs sm:text-sm`
2. **Set minimum body font size** - At least 14px
3. **Increase line height** - For better readability
## Resources
- [Apple Human Interface Guidelines - Touch Targets](https://developer.apple.com/design/human-interface-guidelines/accessibility)
- [Material Design - Touch Targets](https://m3.material.io/foundations/accessible-design/accessibility-basics)
- [Tailwind CSS Responsive Design](https://tailwindcss.com/docs/responsive-design)
- [Playwright Testing](https://playwright.dev/docs/test-mobile)
File renamed without changes.
File renamed without changes.
+178 -34
View File
@@ -1,49 +1,193 @@
# Karaoke-Gen Documentation
> **Status**: Production-ready. First full E2E flow completed 2025-12-28.
This directory contains all documentation for the karaoke-gen project, organized by purpose.
## What Works
## 📋 Current Status (Updated: 2025-12-08)
- **Web App**: https://gen.nomadkaraoke.com - Upload audio, review lyrics, download videos
- **Backend API**: https://api.nomadkaraoke.com - FastAPI on Cloud Run
- **CLI Tools**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
**Project:** Cloud-hosted karaoke generation with async human review
**Backend:** 🔄 Phase 1.3 (85%) - Implementing review architecture
**Frontend:** ⏸️ Not started (waiting for backend review flow)
**Deployment:** ✅ Cloud Run with custom domain (api.nomadkaraoke.com)
**Testing:** 🔄 Local emulator testing in progress
## Current State
### Key Discovery
| Component | Status |
|-----------|--------|
| Audio upload & separation | Working |
| Lyrics transcription | Working |
| Human lyrics review | Working |
| Preview video generation | Working |
| Instrumental selection | Working |
| Multi-format encoding | Working |
| Token-based auth | Working |
| Payment flow | Not started |
We identified that the `LyricsTranscriber` library's `ReviewServer` **blocks** waiting for human input, which doesn't work in our async cloud architecture. The solution is to:
## Known Issues
1. Use LyricsTranscriber for transcription + auto-correction
2. Save corrections to GCS for async human review
3. Use `OutputGenerator` to render video **after** review completes
- CDG format generation requires additional style configuration
- Long audio files (>10 min) may timeout on some workers
See `00-current-plan/CURRENT-STATUS.md` for full details.
## Quick Links
---
| Doc | Purpose |
|-----|---------|
| [ARCHITECTURE.md](ARCHITECTURE.md) | System design, data flow, integrations |
| [DEVELOPMENT.md](DEVELOPMENT.md) | Local setup, testing, deployment |
| [API.md](API.md) | Backend endpoint reference |
| [LESSONS-LEARNED.md](LESSONS-LEARNED.md) | Key insights for future agents |
| [archive/](archive/) | Historical documentation |
## 📂 Documentation Structure
## Workflow Summary
### `00-current-plan/` - **START HERE**
Active migration plan and current status.
```
Upload Audio → Parallel Processing → Human Review → Video Generation → Download
│ │
(Modal + AudioShake) (LyricsTranscriber UI)
| File | Purpose |
|------|---------|
| **`CURRENT-STATUS.md`** | Where we are now, what's working, what's next |
| **`WHATS-NEXT.md`** | Detailed plan for review architecture implementation |
| **`WORKER-IMPLEMENTATION-PLAN.md`** | Worker responsibilities and implementation details |
| `MASTER-PLAN.md` | Overall migration strategy |
| `CLOUD-MIGRATION-REQUIREMENTS.md` | Architectural analysis from CLI workflow |
### `01-reference/` - Reference Documentation
Core system documentation.
| File | Purpose |
|------|---------|
| **`ARCHITECTURE.md`** | Cloud + CLI architecture with diagrams |
| `KARAOKE-GEN-CLI-WORKFLOW.md` | CLI workflow breakdown (8 stages) |
| `API-MANUAL-TESTING.md` | How to test backend API with curl |
| `AUTHENTICATION.md` | Token-based auth system |
| `CLOUD-RUN-ARCHITECTURE.md` | Cloud Run concepts |
### `02-implementation-history/` - Session Logs
Historical implementation records.
### `03-deployment/` - Deployment Guides
How to deploy and operate the system.
| File | Purpose |
|------|---------|
| **`EMULATOR-TESTING.md`** | Local testing with Firestore/GCS emulators |
| **`SETUP-VENV.md`** | Python venv setup |
| **`LOCAL-VALIDATION.md`** | Validate before deploying |
| **`OBSERVABILITY-GUIDE.md`** | Debug production issues |
### `04-testing/` - Testing Guides
| File | Purpose |
|------|---------|
| **`TESTING-GUIDE.md`** | Complete testing guide |
---
## 🚀 Quick Start
### I want to understand the current state:
1. Read `00-current-plan/CURRENT-STATUS.md` - where we are now
2. Read `00-current-plan/WHATS-NEXT.md` - what needs to be done
3. Read `01-reference/ARCHITECTURE.md` - system architecture
### I want to work on the backend:
1. Read `03-deployment/EMULATOR-TESTING.md` - local testing setup
2. Read `00-current-plan/WORKER-IMPLEMENTATION-PLAN.md` - worker details
3. Run `./scripts/run-backend-local.sh --with-emulators`
### I want to test:
```bash
# Start backend with emulators
./scripts/run-backend-local.sh --with-emulators
# Upload test file
curl -X POST http://localhost:8000/api/jobs/upload \
-F "file=@tests/data/waterloo10sec.flac" \
-F "artist=ABBA" \
-F "title=Waterloo"
# Check status
curl http://localhost:8000/api/jobs/{job_id}
```
## For AI Agents
---
Start with `CLAUDE.md` in the repo root for essential rules, then reference these docs as needed. Keep this README current when project state changes.
## 🎯 Current Phase: Review Architecture
**Goal:** Implement async human review workflow
**What Works:**
- ✅ File upload → GCS
- ✅ Audio separation via Modal API
- ✅ Lyrics transcription via AudioShake
- ✅ Auto-correction via LyricsTranscriber
- ✅ Title/end screen generation
- ✅ Instrumental selection endpoint
**What's Being Implemented:**
- 🔄 Review API endpoints (`GET/POST /api/jobs/{id}/review`)
- 🔄 Render Video Worker (uses `OutputGenerator` after review)
- 🔄 Updated state machine with `RENDERING_VIDEO` state
**The Key Insight:**
```
WRONG: Lyrics Worker → Generate Video → Review → ???
RIGHT: Lyrics Worker → Save JSON → Human Review → Render Video → Final Assembly
```
Video generation happens AFTER human review, not during the lyrics worker.
---
## 📝 Key Technical Decisions
### Why Not Use LyricsTranscriber's ReviewServer?
The `ReviewServer` class (in `lyrics_transcriber/review/server.py`):
- Starts a local web server on port 8000
- Opens a browser window
- **BLOCKS** the thread until human submits corrections
- Designed for local CLI operation
This doesn't work in Cloud Run where:
- We can't open browsers
- Workers must be non-blocking
- Human review happens asynchronously via React frontend
**Solution:** Use the data structures and `OutputGenerator` directly, skip the server.
### Why Separate Video Rendering?
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Lyrics Worker │ ──▶ │ Human Reviews │ ──▶ │ Render Worker │
│ │ │ (React UI) │ │ │
│ • Transcribe │ │ • Corrects │ │ • OutputGenerator│
│ • Auto-correct │ │ • Saves JSON │ │ • with_vocals.mkv│
│ • Save JSON │ │ │ │ │
│ • NO VIDEO │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
```
The video must include the human-corrected lyrics, so it can only be generated AFTER review.
---
## 🔄 Job State Flow
```
PENDING → DOWNLOADING → [parallel audio+lyrics] → GENERATING_SCREENS
↓
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE
↓
RENDERING_VIDEO (NEW)
↓
AWAITING_INSTRUMENTAL_SELECTION → INSTRUMENTAL_SELECTED
↓
GENERATING_VIDEO → COMPLETE
```
Two human interaction points:
1. **AWAITING_REVIEW** - User corrects lyrics
2. **AWAITING_INSTRUMENTAL_SELECTION** - User chooses audio track
---
## 📞 Getting Help
1. Check `00-current-plan/CURRENT-STATUS.md` first
2. Look for specific guides in `03-deployment/`
3. Review session history in `02-implementation-history/`
### Common Issues
| Issue | Solution |
|-------|----------|
| Emulator not starting | Check ports 8080, 4443 are free |
| "Missing lyrics video" | Review hasn't happened yet - expected behavior |
| Workers not progressing | Check `state_data` in Firestore |
| FFmpeg errors | Ensure FFmpeg is in PATH |
File renamed without changes.
File renamed without changes.
@@ -1,197 +0,0 @@
# GitHub Actions Docker Builds - Cost Optimization
**Date:** 2025-12-28
**Problem:** Cloud Build costing $100+/month with slow builds
**Solution:** Move Docker builds to GitHub Actions with registry caching
---
## Overview
This document describes the migration from Google Cloud Build to GitHub Actions for Docker image builds. This change eliminates Cloud Build costs while improving build speeds through better caching.
## Cost Comparison
| Approach | Monthly Cost | Build Speed | Caching |
|----------|-------------|-------------|---------|
| **Cloud Build (N1_HIGHCPU_32)** | ~$100+ | 5-15 min | Limited |
| **GitHub Actions + Registry Cache** | **$0** | 2-5 min | Excellent |
### Why Cloud Build Was Expensive
1. **N1_HIGHCPU_32 pricing**: ~$0.064/build-minute (21x more than default)
2. **No persistent layer cache**: Each build re-downloaded dependencies
3. **Cold start overhead**: Spinning up build VMs added latency
4. **Large dependency tree**: torch, transformers, etc. = ~500 packages
### Why GitHub Actions Is Free
- **Public repositories**: Unlimited build minutes on GitHub-hosted runners
- **Registry cache**: Layers cached in Artifact Registry, persist between builds
- **BuildKit mode=max**: Caches ALL intermediate layers, not just final
---
## Architecture
```
┌─────────────────────────────────────────────────────────────────────────┐
│ NEW BUILD PIPELINE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ GitHub Actions (Free) │
│ ├── docker/setup-buildx-action (BuildKit) │
│ ├── google-github-actions/auth (Workload Identity) │
│ └── docker/build-push-action │
│ ├── cache-from: type=registry,ref=...karaoke-backend:cache │
│ └── cache-to: type=registry,ref=...karaoke-backend:cache,mode=max│
│ │
│ Artifact Registry │
│ ├── karaoke-backend:latest (production image) │
│ ├── karaoke-backend:cache (build cache - ALL layers) │
│ ├── karaoke-backend-base:latest (base image with deps) │
│ └── karaoke-backend-base:cache (base cache) │
│ │
│ Build Times (Expected): │
│ ├── First build (cold): ~8-10 min │
│ ├── Code-only changes: ~2-3 min (cache hit) │
│ └── Dependency changes: ~6-8 min (partial cache hit) │
│ │
└─────────────────────────────────────────────────────────────────────────┘
```
---
## How It Works
### Two-Stage Build Pattern
We maintain the existing two-stage build pattern for efficiency:
1. **Base Image** (`karaoke-backend-base`)
- Contains Python 3.11, ffmpeg, system dependencies
- All pip dependencies pre-installed
- Only rebuilt when `poetry.lock` or `Dockerfile.base` changes
- Hash-based change detection via image labels
2. **App Image** (`karaoke-backend`)
- Built FROM the base image
- Contains only application code
- Fast rebuilds (~2 min) for code-only changes
### Registry Cache
The key optimization is using Artifact Registry as a persistent layer cache:
```yaml
- uses: docker/build-push-action@v6
with:
cache-from: type=registry,ref=.../karaoke-backend:cache
cache-to: type=registry,ref=.../karaoke-backend:cache,mode=max
```
**Benefits:**
- `mode=max` caches ALL layers, not just the final image
- Cache persists between CI runs (unlike GitHub Actions cache which has 10GB limit)
- Layers only re-built when inputs change
---
## Files Changed
### New Files
- **`.dockerignore`** - Reduces build context from ~1GB to ~10MB
### Modified Files
- **`.github/workflows/ci.yml`** - `deploy-backend` job rewritten to use:
- `docker/setup-buildx-action@v3`
- `docker/build-push-action@v6`
- Registry caching to Artifact Registry
### Kept (as fallback)
- **`cloudbuild.yaml`** - Still works for manual triggers
- **`cloudbuild-base.yaml`** - Still works for manual base image builds
---
## Manual Builds (Fallback)
If GitHub Actions fails, you can still use Cloud Build:
```bash
# Build base image (only if dependencies changed)
gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
# Build and deploy app image
gcloud builds submit --config=cloudbuild.yaml --project=nomadkaraoke
```
---
## Monitoring
### Check Build Cache Usage
View cache images in Artifact Registry:
```bash
gcloud artifacts docker images list \
us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo \
--include-tags
```
You should see `*:cache` tags for both base and app images.
### Compare Build Times
- **GitHub Actions**: Check workflow run times in Actions tab
- **Previous Cloud Build**: ~5-15 min per build
- **Expected with cache**: ~2-3 min for code changes
---
## Troubleshooting
### Cache Miss (Slow Build)
If builds are slow despite no dependency changes:
1. Check if cache image exists:
```bash
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:cache
```
2. Force cache rebuild by pushing to `main` with a small change
### Disk Space Issues
GitHub Actions runners have limited disk. If base image builds fail:
1. The workflow includes `jlumbroso/free-disk-space` action
2. It removes Android SDK, Haskell, .NET, etc. to free ~20GB
### Authentication Issues
If `docker push` fails with permission errors:
1. Check Workload Identity is configured correctly
2. Verify service account has `roles/artifactregistry.writer`
---
## Future Improvements
1. **Parallel base/app builds**: If base is unchanged, start app build immediately
2. **GitHub Actions cache fallback**: Use GHA cache when registry is slow
3. **Build metrics**: Track build times and cache hit rates
---
## References
- [Docker Build with GitHub Actions](https://docs.docker.com/build/ci/github-actions/)
- [Registry Cache Backend](https://docs.docker.com/build/cache/backends/registry/)
- [Cache Management](https://docs.docker.com/build/ci/github-actions/cache/)
@@ -1,233 +0,0 @@
# Job Status Display Improvement Plan
**Status:** Planned
**Created:** 2025-12-27
**Priority:** HIGH - User experience improvement
## Problem Statement
Users see generic status messages like "Downloading..." that persist for too long, even after the download is complete and other processing stages are happening. This makes it hard to understand:
1. What stage the job is actually at
2. How much progress has been made
3. How much longer the job might take
## Current Implementation
### Backend Status System
The backend has 28 distinct job statuses organized into 8 processing stages:
```
Stage 1: Setup (PENDING)
Stage 2: Audio Search (SEARCHING_AUDIO, AWAITING_AUDIO_SELECTION, DOWNLOADING_AUDIO)
Stage 3: Download (DOWNLOADING)
Stage 4: Parallel Processing
- Audio: SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE
- Lyrics: TRANSCRIBING → CORRECTING → LYRICS_COMPLETE
Stage 5: Screen Generation (GENERATING_SCREENS, APPLYING_PADDING)
Stage 6: Lyrics Review (AWAITING_REVIEW, IN_REVIEW, REVIEW_COMPLETE) ⚠️ BLOCKING
Stage 7: Video Rendering (RENDERING_VIDEO)
Stage 8: Instrumental Selection (AWAITING_INSTRUMENTAL_SELECTION) ⚠️ BLOCKING
Stage 9: Final Processing (INSTRUMENTAL_SELECTED, GENERATING_VIDEO, ENCODING, PACKAGING)
Stage 10: Distribution (UPLOADING, NOTIFYING, COMPLETE)
```
### Key Findings
1. **Progress field exists but unused**: The `Job` model has a `progress: number` field (0-100) that's returned by the API but NOT displayed in the frontend
2. **Parallel processing complicates status**: Audio and lyrics workers run in parallel, updating `state_data.audio_progress` and `state_data.lyrics_progress` independently
3. **Status labels are static**: Frontend shows "Downloading..." but doesn't reflect that parallel workers are running
4. **10-second polling**: Frontend polls every 10 seconds, so status updates are not real-time
## Proposed Solution
### 1. Step-Based Progress Display
Show a `[X/10] Status...` format that gives users a clear sense of overall progress:
```
[1/10] Setting up...
[2/10] Searching for audio...
[3/10] Downloading audio...
[4/10] Processing audio & lyrics... ← Parallel stage
[5/10] Generating screens...
[6/10] Needs review ← Blocking
[7/10] Rendering video...
[8/10] Select instrumental ← Blocking
[9/10] Encoding final video...
[10/10] Uploading...
```
### 2. Status-to-Step Mapping
```typescript
const STATUS_STEP_MAP: Record<string, { step: number; total: number; label: string }> = {
// Step 1: Setup
pending: { step: 1, total: 10, label: "Setting up" },
// Step 2: Audio Search (optional)
searching_audio: { step: 2, total: 10, label: "Searching for audio" },
awaiting_audio_selection: { step: 2, total: 10, label: "Select audio source" },
// Step 3: Download
downloading_audio: { step: 3, total: 10, label: "Downloading audio" },
downloading: { step: 3, total: 10, label: "Downloading" },
// Step 4: Parallel Processing (Audio + Lyrics)
separating_stage1: { step: 4, total: 10, label: "Separating audio (1/2)" },
separating_stage2: { step: 4, total: 10, label: "Separating audio (2/2)" },
audio_complete: { step: 4, total: 10, label: "Audio ready" },
transcribing: { step: 4, total: 10, label: "Transcribing lyrics" },
correcting: { step: 4, total: 10, label: "Correcting lyrics" },
lyrics_complete: { step: 4, total: 10, label: "Lyrics ready" },
// Step 5: Screen Generation
generating_screens: { step: 5, total: 10, label: "Generating screens" },
applying_padding: { step: 5, total: 10, label: "Syncing countdown" },
// Step 6: Review (BLOCKING)
awaiting_review: { step: 6, total: 10, label: "Needs review" },
in_review: { step: 6, total: 10, label: "In review" },
// Step 7: Video Rendering
review_complete: { step: 7, total: 10, label: "Review complete" },
rendering_video: { step: 7, total: 10, label: "Rendering video" },
// Step 8: Instrumental Selection (BLOCKING)
awaiting_instrumental_selection: { step: 8, total: 10, label: "Select instrumental" },
// Step 9: Final Encoding
instrumental_selected: { step: 9, total: 10, label: "Instrumental selected" },
generating_video: { step: 9, total: 10, label: "Generating final video" },
encoding: { step: 9, total: 10, label: "Encoding video" },
packaging: { step: 9, total: 10, label: "Packaging files" },
// Step 10: Distribution
uploading: { step: 10, total: 10, label: "Uploading" },
notifying: { step: 10, total: 10, label: "Sending notifications" },
complete: { step: 10, total: 10, label: "Complete" },
// Terminal states
failed: { step: 0, total: 10, label: "Failed" },
cancelled: { step: 0, total: 10, label: "Cancelled" },
prep_complete: { step: 10, total: 10, label: "Prep complete" },
};
```
### 3. Enhanced Parallel Processing Display
When audio and lyrics are processing in parallel, show both activities:
```typescript
// Check state_data for parallel progress
if (job.state_data?.audio_progress && job.state_data?.lyrics_progress) {
const audioStatus = job.state_data.audio_progress.stage;
const lyricsStatus = job.state_data.lyrics_progress.stage;
// Show combined status like "Audio: separating, Lyrics: transcribing"
if (audioStatus !== 'audio_complete' || lyricsStatus !== 'lyrics_complete') {
return {
step: 4,
total: 10,
label: buildParallelLabel(audioStatus, lyricsStatus)
};
}
}
```
### 4. Visual Progress Bar (Optional Enhancement)
Add a thin progress bar below the status text:
```tsx
<div className="w-full h-1 bg-slate-700 rounded mt-1">
<div
className="h-full bg-blue-500 rounded transition-all"
style={{ width: `${(step / total) * 100}%` }}
/>
</div>
```
## Implementation Steps
### Phase 1: Core Changes (MVP)
1. **Create status utility** (`frontend/lib/job-status.ts`):
- Define `STATUS_STEP_MAP` constant
- Create `getJobStep(job: Job)` function
- Create `formatStepStatus(step, total, label)` function
2. **Update JobCard component**:
- Import status utility
- Replace static `StatusText` with step-based format
- Keep existing color coding from `statusConfig`
3. **Add unit tests** (`frontend/__tests__/job-status.test.ts`):
- Test all status mappings
- Test parallel processing detection
- Test edge cases (unknown status, null values)
### Phase 2: Enhanced Parallel Processing (Optional)
1. **Add parallel progress detection**:
- Check `state_data.audio_progress` and `state_data.lyrics_progress`
- Build combined label showing both activities
2. **Add visual progress bar**:
- Thin progress indicator based on step number
- Smooth transitions between steps
### Phase 3: Backend Improvements (Future)
1. **Add estimated time remaining**:
- Track average duration per stage from historical data
- Return ETA in API response
2. **WebSocket for real-time updates**:
- Replace polling with push notifications
- Smoother progress updates during long operations
## Files to Modify
| File | Changes |
|------|---------|
| `frontend/lib/job-status.ts` | NEW: Status utility functions |
| `frontend/components/job/JobCard.tsx` | Update status display |
| `frontend/__tests__/job-status.test.ts` | NEW: Unit tests |
| `frontend/lib/api.ts` | Add `state_data` types (if needed) |
## Testing Plan
1. **Unit tests**:
- All status mappings return correct step/label
- Unknown statuses handled gracefully
- Parallel progress detection works
2. **Integration tests** (E2E):
- Verify status displays correctly at each stage
- Verify step numbers increment as job progresses
3. **Manual testing**:
- Submit a job and observe status progression
- Verify parallel processing shows meaningful status
## Success Criteria
- [ ] All statuses show `[X/10] Label` format
- [ ] Step numbers progress logically (never decrease except on retry)
- [ ] Parallel processing shows combined activity
- [ ] Blocking statuses clearly indicated for user action
- [ ] All existing tests pass
- [ ] New unit tests have >80% coverage
## Rollback Plan
The changes are frontend-only and purely cosmetic. If issues arise:
1. Revert the JobCard changes to use the old `statusConfig` approach
2. The backend API remains unchanged
## Notes
- The 10-step system is a simplification of the 28 backend statuses for user comprehension
- Parallel processing (step 4) groups multiple backend statuses into one user-facing step
- Blocking steps (6, 8) require user action and should be visually distinct
- The `progress` field from backend could be used for sub-step progress bars in the future
@@ -1,485 +0,0 @@
# Job Debugging Guide
This guide explains how to access debug info, observability, metrics, logs, and data about a given job ID in the karaoke-gen cloud backend.
**Last Updated:** 2025-12-28
---
## Prerequisites
### Service Account Authentication
For automated access, set the service account impersonation variable:
```bash
export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=claude-automation@nomadkaraoke.iam.gserviceaccount.com
```
This service account has permissions to:
- Read Firestore documents
- Read Cloud Storage files
- Query Cloud Logging
- Describe Cloud Run services
- List Cloud Tasks queues
### Required Tools
- `gcloud` CLI (authenticated)
- Python 3.10+ with `google-cloud-firestore` installed
- Access to the `nomadkaraoke` GCP project
---
## Quick Reference
| Data Type | Location | Access Method |
|-----------|----------|---------------|
| Job state & metadata | Firestore `jobs/{job_id}` | Python script |
| Processing logs | Cloud Logging | `gcloud logging read` |
| Audio/video files | GCS bucket | `gcloud storage` |
| Service health | Cloud Run | `gcloud run services describe` |
| Task queues | Cloud Tasks | `gcloud tasks queues list` |
---
## 1. Fetching Job Data from Firestore
Firestore stores all job state including status, timeline, worker logs, and file URLs.
### Using Python (Recommended)
A helper script is included at `scripts/get_job.py`. You can also create your own:
```python
#!/usr/bin/env python3
"""Fetch and display job data from Firestore."""
import sys
import json
from google.cloud import firestore
def get_job(job_id: str):
db = firestore.Client(project='nomadkaraoke')
doc = db.collection('jobs').document(job_id).get()
if doc.exists:
data = doc.to_dict()
print(json.dumps(data, indent=2, default=str))
else:
print(f"Job {job_id} not found")
sys.exit(1)
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python get_job.py <job_id>")
sys.exit(1)
get_job(sys.argv[1])
```
Run it:
```bash
python get_job.py ffb0b8fa
```
### Example Output
```json
{
"status": "in_review",
"artist": "piri",
"title": "dog",
"created_at": "2025-12-28T01:55:23.914454",
"audio_search_artist": "piri",
"audio_search_title": "dog",
"timeline": [
{
"timestamp": "2025-12-28T01:55:23.995388",
"message": "Searching for: piri - dog",
"progress": 5,
"status": "searching_audio"
},
{
"timestamp": "2025-12-28T01:55:24.249594",
"message": "Found 20 audio sources. Waiting for selection.",
"progress": 10,
"status": "awaiting_audio_selection"
}
// ... more timeline entries
],
"worker_logs": [
{
"level": "INFO",
"message": "=== AUDIO WORKER STARTED ===",
"timestamp": "2025-12-28T01:55:37.559644Z",
"worker": "audio"
}
// ... more worker logs
]
}
```
### Key Fields to Check
| Field | Description |
|-------|-------------|
| `status` | Current job status (e.g., `in_review`, `complete`, `failed`) |
| `error_message` | Error description if job failed |
| `timeline` | Ordered list of status transitions with timestamps |
| `worker_logs` | Logs captured from worker processes |
| `file_urls` | Signed URLs for generated files |
| `state_data` | Worker-specific progress data |
---
## 2. Viewing Cloud Run Logs
Logs are stored in Cloud Logging and can be queried with `gcloud logging read`.
### Search Logs for a Job ID
```bash
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND "ffb0b8fa"' \
--project=nomadkaraoke \
--limit=50 \
--format='table(timestamp,severity,jsonPayload.message,jsonPayload.logger)'
```
### Filter by Severity (Errors Only)
```bash
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND "ffb0b8fa" AND severity>=ERROR' \
--project=nomadkaraoke \
--limit=20 \
--format=json
```
### Filter by Time Window
```bash
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND "ffb0b8fa" AND timestamp>="2025-12-28T01:55:00Z" AND timestamp<="2025-12-28T02:05:00Z"' \
--project=nomadkaraoke \
--limit=100 \
--order=asc \
--format='table(timestamp,severity,jsonPayload.message)'
```
### Search for Exceptions
```bash
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND (jsonPayload.message:"Exception" OR jsonPayload.message:"Traceback")' \
--project=nomadkaraoke \
--limit=10 \
--format='table(timestamp,severity,jsonPayload.message)'
```
### View HTTP Request Logs
```bash
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND httpRequest.requestUrl:"ffb0b8fa"' \
--project=nomadkaraoke \
--limit=20 \
--format='table(timestamp,httpRequest.status,httpRequest.requestMethod,httpRequest.requestUrl)'
```
---
## 3. Accessing Job Files in Cloud Storage
Files are stored in `gs://karaoke-gen-storage-nomadkaraoke/jobs/{job_id}/`.
### List All Files for a Job
```bash
gcloud storage ls -r gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/ --project=nomadkaraoke
```
Example output:
```
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/:
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/lyrics/:
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/lyrics/corrections.json
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/lyrics/karaoke.lrc
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/lyrics/piri - dog (Lyrics Genius).txt
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/lyrics/uncorrected.txt
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/previews/:
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/previews/86b528e04907.mp4
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/screens/:
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/screens/end.jpg
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/screens/end.mov
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/screens/title.jpg
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/screens/title.mov
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/stems/:
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/stems/backing_vocals.flac
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/stems/instrumental_clean.flac
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/stems/instrumental_with_backing.flac
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/stems/lead_vocals.flac
gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/stems/vocals_clean.flac
```
### View File Content (Small Files)
```bash
gcloud storage cat gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/lyrics/corrections.json --project=nomadkaraoke | head -50
```
### Get File Metadata
```bash
gcloud storage objects describe gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/stems/instrumental_clean.flac --project=nomadkaraoke
```
Example output:
```yaml
bucket: karaoke-gen-storage-nomadkaraoke
content_type: application/octet-stream
crc32c_hash: /uXURg==
creation_time: 2025-12-28T01:57:44+0000
size: 18115881
storage_class: STANDARD
```
### Download a File
```bash
gcloud storage cp gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/stems/instrumental_clean.flac ./instrumental.flac --project=nomadkaraoke
```
### GCS Folder Structure
```
jobs/{job_id}/
├── lyrics/
│ ├── corrections.json # Lyrics with timing corrections
│ ├── karaoke.lrc # LRC format lyrics
│ ├── uncorrected.txt # Raw transcription
│ └── *.txt # Reference lyrics from sources
├── previews/
│ └── {hash}.mp4 # Preview video for lyrics review
├── screens/
│ ├── title.mov/.jpg/.png # Title screen
│ └── end.mov/.jpg/.png # End screen
├── stems/
│ ├── instrumental_clean.flac
│ ├── instrumental_with_backing.flac
│ ├── vocals_clean.flac
│ ├── lead_vocals.flac
│ └── backing_vocals.flac
├── videos/
│ └── with_vocals.mkv # Karaoke video with lyrics overlay
└── finals/
├── lossless_4k.mp4
├── lossless_4k.mkv
├── lossy_4k.mp4
└── lossy_720p.mp4
```
---
## 4. Cloud Run Service Status
### Check Service Health
```bash
gcloud run services describe karaoke-backend \
--project=nomadkaraoke \
--region=us-central1 \
--format='yaml(status.conditions,status.latestReadyRevisionName,status.url)'
```
Example output:
```yaml
status:
conditions:
- lastTransitionTime: '2025-12-28T01:29:43.465349Z'
status: 'True'
type: Ready
- lastTransitionTime: '2025-12-28T01:29:42.192338Z'
status: 'True'
type: ConfigurationsReady
- lastTransitionTime: '2025-12-28T01:29:43.430737Z'
status: 'True'
type: RoutesReady
latestReadyRevisionName: karaoke-backend-00203-c7q
url: https://karaoke-backend-ipzqd2k4yq-uc.a.run.app
```
### List Recent Revisions
```bash
gcloud run revisions list \
--service=karaoke-backend \
--project=nomadkaraoke \
--region=us-central1 \
--limit=5 \
--format='table(name,status.conditions.type,status.conditions.status,createTime)'
```
### View Revision Details
```bash
gcloud run revisions describe karaoke-backend-00203-c7q \
--project=nomadkaraoke \
--region=us-central1 \
--format='yaml(spec.containerConcurrency,spec.containers[0].resources)'
```
---
## 5. Cloud Tasks Queues
The backend uses Cloud Tasks for async worker processing.
### List All Queues
```bash
gcloud tasks queues list --project=nomadkaraoke --location=us-central1
```
Example output:
```
QUEUE_NAME STATE MAX_NUM_OF_TASKS MAX_RATE (/sec) MAX_ATTEMPTS
audio-worker-queue RUNNING 50 10.0 3
lyrics-worker-queue RUNNING 50 10.0 3
render-worker-queue RUNNING 20 5.0 2
screens-worker-queue RUNNING 100 50.0 3
video-worker-queue RUNNING 10 3.0 2
```
### View Queue Details
```bash
gcloud tasks queues describe audio-worker-queue \
--project=nomadkaraoke \
--location=us-central1
```
### List Tasks in a Queue
```bash
gcloud tasks list \
--queue=audio-worker-queue \
--project=nomadkaraoke \
--location=us-central1 \
--limit=10
```
---
## 6. Common Debugging Scenarios
### Scenario: Job Stuck in Status
1. **Check job state:**
```bash
python get_job.py ffb0b8fa | jq '.status, .error_message, .timeline[-3:]'
```
2. **Check for errors in logs:**
```bash
gcloud logging read 'resource.labels.service_name="karaoke-backend" AND "ffb0b8fa" AND severity>=ERROR' --project=nomadkaraoke --limit=5 --format=json
```
3. **Check if worker task is queued:**
```bash
gcloud tasks list --queue=audio-worker-queue --project=nomadkaraoke --location=us-central1
```
### Scenario: Job Failed
1. **Get error message from Firestore:**
```bash
python get_job.py ffb0b8fa | jq '.error_message, .error_details'
```
2. **Search for exception stack trace:**
```bash
gcloud logging read 'resource.labels.service_name="karaoke-backend" AND "ffb0b8fa" AND jsonPayload.message:"Traceback"' --project=nomadkaraoke --limit=3 --format='value(jsonPayload.message)'
```
### Scenario: Missing Output Files
1. **List all generated files:**
```bash
gcloud storage ls -r gs://karaoke-gen-storage-nomadkaraoke/jobs/ffb0b8fa/ --project=nomadkaraoke
```
2. **Check worker logs for upload failures:**
```bash
gcloud logging read 'resource.labels.service_name="karaoke-backend" AND "ffb0b8fa" AND (jsonPayload.message:"upload" OR jsonPayload.message:"GCS")' --project=nomadkaraoke --limit=20 --format='table(timestamp,jsonPayload.message)'
```
### Scenario: Audio Separation Slow/Failing
1. **Check audio worker logs:**
```bash
gcloud logging read 'resource.labels.service_name="karaoke-backend" AND "ffb0b8fa" AND jsonPayload.worker:"audio"' --project=nomadkaraoke --limit=30 --format='table(timestamp,jsonPayload.message)'
```
2. **Check Modal API status in worker logs:**
```bash
python get_job.py ffb0b8fa | jq '.worker_logs | map(select(.worker == "audio"))'
```
---
## 7. Useful Aliases
Add these to your shell profile for quick access:
```bash
# Job debugging aliases (update paths as needed)
alias kj='python scripts/get_job.py'
alias klogs='gcloud logging read "resource.labels.service_name=\"karaoke-backend\"" --project=nomadkaraoke --limit=50 --format="table(timestamp,severity,jsonPayload.message)"'
alias kfiles='gcloud storage ls -r gs://karaoke-gen-storage-nomadkaraoke/jobs/'
alias kservice='gcloud run services describe karaoke-backend --project=nomadkaraoke --region=us-central1'
alias kqueues='gcloud tasks queues list --project=nomadkaraoke --location=us-central1'
# Usage:
# kj ffb0b8fa # Get job data
# klogs | grep ffb0b8fa # Search logs
# kfiles ffb0b8fa/ # List job files
```
---
## 8. API Access (Alternative)
The backend API also provides job information if you have an access token.
### Get Job via API
```bash
export TOKEN="your-access-token"
curl -s -H "Authorization: Bearer $TOKEN" \
"https://api.nomadkaraoke.com/api/jobs/ffb0b8fa" | jq
```
### List Jobs
```bash
curl -s -H "Authorization: Bearer $TOKEN" \
"https://api.nomadkaraoke.com/api/jobs?limit=10" | jq
```
---
## Related Documentation
- [Architecture Overview](./ARCHITECTURE.md) - System architecture and data flow
- [Frontend Guide](./FRONTEND-GUIDE.md) - Web UI documentation
- [Backend Feature Parity](../00-current-plan/BACKEND-FEATURE-PARITY-PLAN.md) - Feature status
- [Scalable Architecture Plan](../00-current-plan/SCALABLE-ARCHITECTURE-PLAN.md) - Cloud Tasks and scaling
Loaded 100 of 146 files, more files were not shown because too many files have changed in this diff. Show more