mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 21:17:56 +09:00
Compare commits
No files matched your search
@@ -0,0 +1,74 @@
|
||||
---
|
||||
description: Periodic documentation maintenance and organization check
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write, Bash
|
||||
---
|
||||
|
||||
# Documentation Maintenance Check
|
||||
|
||||
You are performing periodic maintenance on the project documentation.
|
||||
|
||||
## Checks to Perform
|
||||
|
||||
### 1. Structure Compliance
|
||||
Verify docs/ follows the correct structure:
|
||||
```
|
||||
docs/
|
||||
├── README.md # Current status + navigation
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Accumulated wisdom
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
Report any files that don't belong or are misplaced.
|
||||
|
||||
### 2. Freshness Check
|
||||
- Read `docs/README.md` - is the status section current?
|
||||
- Check recent git commits - do they suggest docs should be updated?
|
||||
- Look for stale information that contradicts current code
|
||||
|
||||
### 3. Archive Organization
|
||||
- Verify all archive files have YYYY-MM-DD prefix
|
||||
- Check for any docs that should be archived (completed work, old plans)
|
||||
- Remove truly obsolete content (empty files, duplicate info)
|
||||
|
||||
### 4. Cross-Reference Check
|
||||
- Verify CLAUDE.md points to current doc locations
|
||||
- Check that docs reference each other correctly
|
||||
- Ensure no broken internal links
|
||||
|
||||
### 5. Content Quality
|
||||
- Flag any docs over 500 lines (may need splitting)
|
||||
- Identify duplicate information across docs
|
||||
- Note sections that seem outdated
|
||||
|
||||
## Actions to Take
|
||||
|
||||
1. **Fix structural issues** - Move/rename misplaced files
|
||||
2. **Update stale content** - Refresh outdated information
|
||||
3. **Archive completed work** - Move old docs to archive/
|
||||
4. **Report findings** - Summarize what was found and fixed
|
||||
|
||||
## Output Format
|
||||
|
||||
Provide a maintenance report:
|
||||
```
|
||||
## Documentation Maintenance Report - YYYY-MM-DD
|
||||
|
||||
### Issues Found
|
||||
- [list of issues]
|
||||
|
||||
### Actions Taken
|
||||
- [list of fixes made]
|
||||
|
||||
### Recommendations
|
||||
- [suggestions for future attention]
|
||||
|
||||
### Status
|
||||
- [ ] Structure: OK/Fixed
|
||||
- [ ] Freshness: OK/Updated
|
||||
- [ ] Archive: OK/Organized
|
||||
- [ ] Cross-refs: OK/Fixed
|
||||
- [ ] Quality: OK/Flagged
|
||||
```
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
description: Review current session for documentation updates before merging PR
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write
|
||||
---
|
||||
|
||||
# Pre-Merge Documentation Review
|
||||
|
||||
You are reviewing this chat session to identify documentation updates needed before merging the current PR.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. **Analyze the current session** - What work was done? What was learned? What changed?
|
||||
|
||||
2. **Check if docs need updates:**
|
||||
- Does `docs/README.md` status section need updating?
|
||||
- Should anything be added to `docs/LESSONS-LEARNED.md`?
|
||||
- Does `docs/ARCHITECTURE.md` need changes (if architecture changed)?
|
||||
- Does `docs/DEVELOPMENT.md` need changes (if dev workflow changed)?
|
||||
- Does `docs/API.md` need changes (if API changed)?
|
||||
|
||||
3. **For significant completed work:**
|
||||
- Create `docs/archive/YYYY-MM-DD-topic.md` summarizing the work
|
||||
- Use today's date and a descriptive topic name
|
||||
|
||||
4. **Update docs/README.md** with:
|
||||
- Current status if it changed
|
||||
- Any new known issues discovered
|
||||
- Link to new archive doc if created
|
||||
|
||||
## Guidelines
|
||||
|
||||
- Keep updates concise and factual
|
||||
- Focus on information useful for future AI agents
|
||||
- Don't duplicate information that's in the code
|
||||
- Archive docs should capture "what was done and why" not implementation details
|
||||
|
||||
## Output
|
||||
|
||||
After reviewing, either:
|
||||
- Make the necessary documentation updates
|
||||
- Report "No documentation updates needed" with brief explanation why
|
||||
|
||||
Remember: Good documentation helps the next agent continue where you left off.
|
||||
@@ -0,0 +1,83 @@
|
||||
# .dockerignore - Reduce Docker build context size
|
||||
# This is critical for fast builds as it reduces upload time and cache invalidation
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
.gitmodules
|
||||
|
||||
# Node.js / Frontend (not needed for backend image)
|
||||
node_modules/
|
||||
frontend/
|
||||
frontend-react/
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Python virtual environments (use Docker's Python instead)
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
backend/venv/
|
||||
ENV/
|
||||
env/
|
||||
*.egg-info/
|
||||
.eggs/
|
||||
dist/
|
||||
build/
|
||||
*.egg
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
|
||||
# Testing (not needed in production image)
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
coverage.xml
|
||||
*.cover
|
||||
tests/
|
||||
backend/tests/
|
||||
|
||||
# Documentation (not needed in Docker image)
|
||||
docs/
|
||||
*.md
|
||||
!README.md
|
||||
|
||||
# Infrastructure (not needed for Docker build)
|
||||
infrastructure/
|
||||
|
||||
# CI/CD configs
|
||||
.github/
|
||||
|
||||
# Cloud Build configs (keep for reference but not in image)
|
||||
cloudbuild*.yaml
|
||||
|
||||
# Local development files
|
||||
lyrics_transcriber_local/
|
||||
*.local
|
||||
.env
|
||||
.env.*
|
||||
*.bak
|
||||
*.tmp
|
||||
Thumbs.db
|
||||
|
||||
# Large test/sample files
|
||||
tests/data/
|
||||
*.mp3
|
||||
*.wav
|
||||
*.mp4
|
||||
*.mkv
|
||||
*.flac
|
||||
+248
-118
@@ -18,11 +18,13 @@ concurrency:
|
||||
jobs:
|
||||
# ============================================
|
||||
# PATH FILTERING - Determine what changed
|
||||
# Only needed for PRs (main branch skips tests)
|
||||
# ============================================
|
||||
|
||||
changes:
|
||||
name: "Detect Changes"
|
||||
runs-on: ubuntu-latest
|
||||
if: github.ref != 'refs/heads/main'
|
||||
outputs:
|
||||
frontend: ${{ steps.filter.outputs.frontend }}
|
||||
backend: ${{ steps.filter.outputs.backend }}
|
||||
@@ -61,14 +63,14 @@ jobs:
|
||||
echo "| Dependencies | ${{ steps.filter.outputs.deps }} |" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
# ============================================
|
||||
# FRONTEND TESTS
|
||||
# FRONTEND TESTS (PRs only - main skips tests)
|
||||
# ============================================
|
||||
|
||||
frontend-unit-tests:
|
||||
name: "Frontend - Unit Tests"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: ${{ needs.changes.outputs.frontend == 'true' || github.ref == 'refs/heads/main' }}
|
||||
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
@@ -95,15 +97,40 @@ jobs:
|
||||
path: frontend/coverage/
|
||||
retention-days: 7
|
||||
|
||||
frontend-build:
|
||||
name: "Frontend - Build"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js with caching
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Build frontend
|
||||
working-directory: frontend
|
||||
env:
|
||||
NEXT_PUBLIC_API_URL: https://api.nomadkaraoke.com
|
||||
run: npm run build
|
||||
|
||||
# ============================================
|
||||
# PACKAGE TESTS (karaoke-gen Python CLI)
|
||||
# PACKAGE TESTS (PRs only - main skips tests)
|
||||
# ============================================
|
||||
|
||||
package-unit-tests:
|
||||
name: "Package - Unit Tests (70% coverage)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: ${{ needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true' || github.ref == 'refs/heads/main' }}
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -161,7 +188,7 @@ jobs:
|
||||
name: "Package - Integration Tests"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: ${{ needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true' || github.ref == 'refs/heads/main' }}
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -206,11 +233,15 @@ jobs:
|
||||
- name: Run integration tests
|
||||
run: poetry run pytest tests/integration/ -v
|
||||
|
||||
package-build:
|
||||
name: "Package - Build & Install"
|
||||
package-build-test:
|
||||
name: "Package - Build & Install (Test)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [changes, package-unit-tests, package-integration-tests]
|
||||
if: ${{ always() && (needs.package-unit-tests.result == 'success' || needs.package-unit-tests.result == 'skipped') && (needs.package-integration-tests.result == 'success' || needs.package-integration-tests.result == 'skipped') && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true' || github.ref == 'refs/heads/main') }}
|
||||
if: >-
|
||||
github.ref != 'refs/heads/main' && always() &&
|
||||
(needs.package-unit-tests.result == 'success' || needs.package-unit-tests.result == 'skipped') &&
|
||||
(needs.package-integration-tests.result == 'success' || needs.package-integration-tests.result == 'skipped') &&
|
||||
(needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -254,23 +285,23 @@ jobs:
|
||||
pip install dist/*.whl
|
||||
python -m karaoke_gen.utils.gen_cli --help
|
||||
|
||||
- name: Upload build artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
retention-days: 1
|
||||
|
||||
# ============================================
|
||||
# BACKEND TESTS (Cloud Run API service)
|
||||
# BACKEND TESTS (PRs only - main skips tests)
|
||||
# ============================================
|
||||
|
||||
backend-unit-tests:
|
||||
name: "Backend - Unit Tests"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: ${{ needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true' || github.ref == 'refs/heads/main' }}
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Free disk space
|
||||
run: |
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo rm -rf /opt/ghc
|
||||
sudo rm -rf /usr/local/share/boost
|
||||
df -h
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -314,14 +345,18 @@ jobs:
|
||||
name: "Backend - Emulator Integration Tests"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: ${{ needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true' || github.ref == 'refs/heads/main' }}
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Free disk space
|
||||
run: |
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo rm -rf /opt/ghc
|
||||
sudo rm -rf /usr/local/share/boost
|
||||
df -h
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: false
|
||||
android: true
|
||||
haskell: true
|
||||
dotnet: true
|
||||
large-packages: false
|
||||
docker-images: false
|
||||
swap-storage: false
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
@@ -384,7 +419,7 @@ jobs:
|
||||
name: "Backend - Code Quality"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: ${{ needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || github.ref == 'refs/heads/main' }}
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true')
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
@@ -419,9 +454,78 @@ jobs:
|
||||
echo "✅ Basic syntax validation passed"
|
||||
|
||||
# ============================================
|
||||
# DEPLOYMENTS (main branch only)
|
||||
# CI GATE - Required status check for PRs
|
||||
# Only runs on PRs, not on main
|
||||
# ============================================
|
||||
|
||||
ci-gate:
|
||||
name: "CI Gate"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [frontend-unit-tests, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
if: github.ref != 'refs/heads/main' && always()
|
||||
steps:
|
||||
- name: Check results
|
||||
run: |
|
||||
results=(
|
||||
"${{ needs.frontend-unit-tests.result }}"
|
||||
"${{ needs.frontend-build.result }}"
|
||||
"${{ needs.package-unit-tests.result }}"
|
||||
"${{ needs.package-integration-tests.result }}"
|
||||
"${{ needs.backend-unit-tests.result }}"
|
||||
"${{ needs.backend-emulator-tests.result }}"
|
||||
"${{ needs.backend-lint.result }}"
|
||||
)
|
||||
for result in "${results[@]}"; do
|
||||
if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
|
||||
echo "❌ CI failed"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "✅ CI passed"
|
||||
|
||||
# ============================================
|
||||
# DEPLOYMENTS (main branch only)
|
||||
# Tests already passed on PR, so we skip them here
|
||||
# ============================================
|
||||
|
||||
package-build:
|
||||
name: "Package - Build"
|
||||
runs-on: ubuntu-latest
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
with:
|
||||
virtualenvs-create: true
|
||||
virtualenvs-in-project: true
|
||||
|
||||
- name: Cache Poetry dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .venv
|
||||
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
|
||||
restore-keys: |
|
||||
venv-package-${{ runner.os }}-py3.13-
|
||||
|
||||
- name: Build package
|
||||
run: poetry build
|
||||
|
||||
- name: Upload build artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
retention-days: 1
|
||||
|
||||
package-publish:
|
||||
name: "Deploy - Publish to PyPI"
|
||||
runs-on: ubuntu-latest
|
||||
@@ -512,11 +616,7 @@ jobs:
|
||||
deploy-frontend:
|
||||
name: "Deploy - Frontend (GitHub Pages)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [frontend-unit-tests]
|
||||
if: ${{ always() && github.ref == 'refs/heads/main' && github.event_name == 'push' && (needs.frontend-unit-tests.result == 'success' || needs.frontend-unit-tests.result == 'skipped') }}
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
@@ -588,12 +688,21 @@ jobs:
|
||||
deploy-backend:
|
||||
name: "Deploy - Backend (Cloud Run)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
if: ${{ always() && github.ref == 'refs/heads/main' && github.event_name == 'push' && (needs.backend-unit-tests.result == 'success' || needs.backend-unit-tests.result == 'skipped') && (needs.backend-emulator-tests.result == 'success' || needs.backend-emulator-tests.result == 'skipped') && (needs.backend-lint.result == 'success' || needs.backend-lint.result == 'skipped') }}
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
REPOSITORY: karaoke-repo
|
||||
BASE_IMAGE_NAME: karaoke-backend-base
|
||||
APP_IMAGE_NAME: karaoke-backend
|
||||
steps:
|
||||
# ============================================
|
||||
# Check if deployment is needed FIRST
|
||||
# This avoids wasting 3+ mins on disk cleanup when skipping
|
||||
# ============================================
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -624,6 +733,22 @@ jobs:
|
||||
echo "✨ Tag v$VERSION does not exist - proceeding with deployment"
|
||||
fi
|
||||
|
||||
# ============================================
|
||||
# Free disk space only if deploying
|
||||
# This prevents "No space left on device" errors during Docker build
|
||||
# ============================================
|
||||
- name: Free disk space (aggressive)
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: true
|
||||
android: true
|
||||
haskell: true
|
||||
dotnet: true
|
||||
large-packages: true
|
||||
docker-images: true
|
||||
swap-storage: true
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: auth
|
||||
@@ -638,21 +763,37 @@ jobs:
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Check if base image needs rebuild
|
||||
- name: Configure Docker for Artifact Registry
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
# ============================================
|
||||
# Docker Build with BuildKit + Registry Cache
|
||||
# This replaces expensive Cloud Build with free GitHub Actions
|
||||
# Registry cache persists between builds for fast rebuilds
|
||||
# ============================================
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Calculate base image hash
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: base-hash
|
||||
run: |
|
||||
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
|
||||
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
|
||||
echo "hash=$CURRENT_HASH" >> $GITHUB_OUTPUT
|
||||
echo "📋 Base content hash: $CURRENT_HASH"
|
||||
|
||||
- name: Check if base image exists and is current
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: check-base
|
||||
run: |
|
||||
echo "🔍 Checking if base image needs rebuild..."
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
|
||||
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
|
||||
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
|
||||
echo "Current combined hash (poetry.lock + Dockerfile.base): $CURRENT_HASH"
|
||||
|
||||
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
|
||||
|
||||
BASE_IMAGE="us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest"
|
||||
# Try to get the existing base image's hash label
|
||||
EXISTING_HASH=""
|
||||
|
||||
if docker pull $BASE_IMAGE 2>/dev/null; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
@@ -661,10 +802,10 @@ jobs:
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
else
|
||||
echo "⚠️ No existing base image found"
|
||||
echo "⚠️ No existing base image found - will build"
|
||||
fi
|
||||
|
||||
if [ "$CURRENT_HASH" != "$EXISTING_HASH" ]; then
|
||||
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
|
||||
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
|
||||
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
@@ -672,88 +813,73 @@ jobs:
|
||||
echo "needs_rebuild=false" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Rebuild base image (if needed)
|
||||
- name: Build and push base image (if needed)
|
||||
if: steps.version_check.outputs.needs_deploy == 'true' && steps.check-base.outputs.needs_rebuild == 'true'
|
||||
run: |
|
||||
echo "🏗️ Rebuilding base image with updated dependencies..."
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: backend/Dockerfile.base
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:${{ github.sha }}
|
||||
# Registry cache for ALL layers (mode=max) - persists between builds
|
||||
# ignore-error=true allows build to continue if cache doesn't exist yet
|
||||
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,ignore-error=true
|
||||
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,mode=max,ignore-error=true
|
||||
build-args: |
|
||||
BUILD_DATE=${{ github.run_id }}
|
||||
BUILD_ID=${{ github.sha }}
|
||||
BASE_CONTENT_HASH=${{ steps.base-hash.outputs.hash }}
|
||||
# Provenance disabled for smaller image and faster push
|
||||
provenance: false
|
||||
|
||||
BASE_BUILD_ID=$(gcloud builds submit \
|
||||
--config=cloudbuild-base.yaml \
|
||||
--project=nomadkaraoke \
|
||||
--format='value(id)' \
|
||||
--async \
|
||||
.)
|
||||
- name: Build and push app image
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: backend/Dockerfile
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:latest
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}
|
||||
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:v${{ steps.version_check.outputs.version }}
|
||||
# Registry cache for app layers
|
||||
# ignore-error=true allows build to continue if cache doesn't exist yet
|
||||
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,ignore-error=true
|
||||
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,mode=max,ignore-error=true
|
||||
provenance: false
|
||||
|
||||
echo "Base Build ID: $BASE_BUILD_ID"
|
||||
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BASE_BUILD_ID?project=nomadkaraoke"
|
||||
# ============================================
|
||||
# Deploy to Cloud Run
|
||||
# ============================================
|
||||
|
||||
echo "Waiting for base image build to complete..."
|
||||
while true; do
|
||||
BUILD_STATUS=$(gcloud builds describe $BASE_BUILD_ID --project=nomadkaraoke --format='value(status)')
|
||||
echo "Base build status: $BUILD_STATUS"
|
||||
|
||||
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
|
||||
echo "✅ Base image build completed successfully"
|
||||
break
|
||||
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
|
||||
echo "❌ Base image build failed with status: $BUILD_STATUS"
|
||||
echo ""
|
||||
echo "==================== BUILD LOGS ===================="
|
||||
gcloud logging read "resource.type=build AND resource.labels.build_id=$BASE_BUILD_ID" \
|
||||
--project=nomadkaraoke \
|
||||
--format="value(textPayload)" \
|
||||
--limit=200 \
|
||||
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
|
||||
echo "===================================================="
|
||||
gcloud builds log $BASE_BUILD_ID --project=nomadkaraoke 2>&1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 30
|
||||
done
|
||||
|
||||
- name: Trigger Cloud Build
|
||||
- name: Deploy to Cloud Run
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: |
|
||||
echo "🚀 Triggering Cloud Build for deployment..."
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Version: ${{ steps.version_check.outputs.version }}"
|
||||
VERSION="${{ steps.version_check.outputs.version }}"
|
||||
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
|
||||
|
||||
BUILD_ID=$(gcloud builds submit \
|
||||
--config=cloudbuild.yaml \
|
||||
--project=nomadkaraoke \
|
||||
--format='value(id)' \
|
||||
--async \
|
||||
.)
|
||||
echo "🚀 Deploying to Cloud Run..."
|
||||
echo "Image: $IMAGE"
|
||||
echo "Version: $VERSION"
|
||||
|
||||
echo "Build ID: $BUILD_ID"
|
||||
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=nomadkaraoke"
|
||||
|
||||
echo "Waiting for build to complete..."
|
||||
while true; do
|
||||
BUILD_STATUS=$(gcloud builds describe $BUILD_ID --project=nomadkaraoke --format='value(status)')
|
||||
echo "Build status: $BUILD_STATUS"
|
||||
|
||||
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
|
||||
echo "✅ Build completed successfully"
|
||||
break
|
||||
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
|
||||
echo "❌ Build failed with status: $BUILD_STATUS"
|
||||
echo ""
|
||||
echo "==================== BUILD LOGS ===================="
|
||||
gcloud logging read "resource.type=build AND resource.labels.build_id=$BUILD_ID" \
|
||||
--project=nomadkaraoke \
|
||||
--format="value(textPayload)" \
|
||||
--limit=200 \
|
||||
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
|
||||
echo "===================================================="
|
||||
echo ""
|
||||
gcloud builds log $BUILD_ID --project=nomadkaraoke 2>&1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 30
|
||||
done
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image $IMAGE \
|
||||
--region us-central1 \
|
||||
--platform managed \
|
||||
--allow-unauthenticated \
|
||||
--memory 16Gi \
|
||||
--cpu 8 \
|
||||
--cpu-boost \
|
||||
--execution-environment gen2 \
|
||||
--timeout 1800 \
|
||||
--concurrency 1 \
|
||||
--max-instances 50 \
|
||||
--min-instances 0 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
|
||||
|
||||
- name: Verify deployment and version
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
@@ -804,6 +930,10 @@ jobs:
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Branch: ${{ github.ref_name }}"
|
||||
echo "Run: ${{ github.run_number }}"
|
||||
echo ""
|
||||
echo "📊 Cost savings: This build used FREE GitHub Actions instead of Cloud Build"
|
||||
echo " Previously: ~\$0.064/min with N1_HIGHCPU_32"
|
||||
echo " Now: \$0 (public repo) with registry-cached builds"
|
||||
|
||||
- name: Skipped deployment summary
|
||||
if: steps.version_check.outputs.needs_deploy == 'false'
|
||||
|
||||
@@ -227,3 +227,6 @@ cython_debug/
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
flacfetch
|
||||
|
||||
# Test output directories
|
||||
Rick Astley*/
|
||||
@@ -1,131 +1,97 @@
|
||||
# Project Guidelines for AI Assistants
|
||||
# Karaoke-Gen AI Assistant Guidelines
|
||||
|
||||
This document contains guidelines and rules for AI assistants working on this codebase.
|
||||
## Project Overview
|
||||
|
||||
## Core Principles
|
||||
**Karaoke video generation platform** - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
|
||||
|
||||
**Always be testing.** Proactively evaluate, read, run, fix, and improve tests whenever you touch any part of the system—frontend, backend, infrastructure. Every code change should be thoroughly tested. Leave everything better tested than you found it.
|
||||
- **Production**: <https://gen.nomadkaraoke.com> (frontend), <https://api.nomadkaraoke.com> (backend)
|
||||
- **CLI**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
- **Repo**: <https://github.com/nomadkaraoke/karaoke-gen>
|
||||
|
||||
**Always be documenting.** Continuously read, write, improve, update, and organize markdown documentation. Reference existing docs and keep them accurate as the codebase evolves.
|
||||
## Quick Reference
|
||||
|
||||
**Build for maintainability.** Keep code SOLID, modular, and clean. Favor simplicity and clarity. Future developers (and AI agents) will thank you.
|
||||
| What | Where |
|
||||
|------|-------|
|
||||
| Current status | `docs/README.md` |
|
||||
| Architecture | `docs/ARCHITECTURE.md` |
|
||||
| Dev setup & testing | `docs/DEVELOPMENT.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
|
||||
## Branching Workflow
|
||||
## Tech Stack
|
||||
|
||||
**Never commit directly to `main`.** Always work on a feature branch.
|
||||
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
|
||||
- **Frontend**: Next.js on Cloudflare Pages
|
||||
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
|
||||
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
|
||||
|
||||
Create descriptive branch names like `feature/add-discord-notifications`, `fix/audio-sync-issue`, or `refactor/cleanup-api-routes`.
|
||||
## Essential Rules
|
||||
|
||||
### Using Git Worktrees
|
||||
|
||||
**For non-trivial changes** (new features, multi-file edits, refactors):
|
||||
- Create a git worktree in the parent folder before making any file changes:
|
||||
### Git Workflow
|
||||
- **Never commit directly to `main`** - always use feature branches
|
||||
- **Use git worktrees** for all changes:
|
||||
```bash
|
||||
git worktree add ../karaoke-gen-feat-xyz feature/xyz
|
||||
git worktree add -b feature/xyz ../karaoke-gen-feat-xyz main
|
||||
```
|
||||
- This keeps the main directory clean on `main` and enables parallel work on multiple features
|
||||
- Name the worktree folder descriptively, e.g., `../karaoke-gen-fix-audio-sync`
|
||||
|
||||
**For trivial changes** (typos, single-line fixes):
|
||||
- Creating a branch in the current directory is acceptable
|
||||
- Use `git checkout -b fix/typo-xyz` before making changes
|
||||
|
||||
**Cleanup:** When done with a worktree, remove it with:
|
||||
```bash
|
||||
git worktree remove ../karaoke-gen-feat-xyz
|
||||
```
|
||||
|
||||
### Creating a Pull Request
|
||||
|
||||
Once all changes are complete, tests pass, and commits are pushed, create a PR with:
|
||||
|
||||
1. **Summary** - Brief overview of the changes
|
||||
2. **Changes Made** - List of specific modifications
|
||||
3. **Testing** - How changes were tested (unit tests, manual testing)
|
||||
4. **Related Issues** - Reference any GitHub issues (e.g., "Fixes #123")
|
||||
|
||||
Only merge after review and CI checks pass.
|
||||
|
||||
## Infrastructure Changes
|
||||
|
||||
All Google Cloud infrastructure (IAM, Cloud Storage, Cloud Run, Secret Manager, Artifact Registry, etc.) must be managed via **Pulumi IaC** in the `infrastructure/` folder.
|
||||
|
||||
### Rules
|
||||
|
||||
- **Reading/querying** with `gcloud` CLI is fine for investigation and debugging
|
||||
- **Temporary changes** via `gcloud` CLI or Console are acceptable for urgent fixes or experimentation, but **must be cleaned up** and properly codified in the IaC afterward
|
||||
- **Permanent changes** must go through Pulumi — never leave `gcloud` or Console changes as the final state
|
||||
|
||||
### Workflow
|
||||
|
||||
1. Make changes in `infrastructure/`
|
||||
2. Run `pulumi preview` and verify the output
|
||||
3. Run `pulumi up` to apply
|
||||
|
||||
### Authentication Issues
|
||||
|
||||
If you encounter any authentication issues with `gcloud` or `pulumi` (e.g., expired tokens, permission errors), **stop and notify the user** so they can run the appropriate auth commands in their external terminal:
|
||||
|
||||
```bash
|
||||
gcloud auth login
|
||||
gcloud auth application-default login
|
||||
```
|
||||
|
||||
Do not attempt to work around auth issues or make assumptions about credentials.
|
||||
|
||||
## Version Management
|
||||
|
||||
When committing and pushing changes, always bump the patch version in `pyproject.toml`.
|
||||
|
||||
The version is located at `tool.poetry.version` in `pyproject.toml`. For example, if the current version is `0.70.3`, bump it to `0.70.4`.
|
||||
|
||||
This ensures every commit pushed to the repository has a unique version number for tracking and deployment purposes.
|
||||
|
||||
## Testing Requirements
|
||||
|
||||
**Before committing and pushing ANY changes, you MUST run the full test suite.**
|
||||
|
||||
### Backend Test Command
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
### Testing (Required Before Commit)
|
||||
```bash
|
||||
# Backend (10 min timeout, limit output)
|
||||
make test 2>&1 | tail -n 500
|
||||
```
|
||||
|
||||
**Important for AI agents:**
|
||||
- Set a **10 minute timeout** when running tests (they can take a while with emulators)
|
||||
- Always pipe output through `tail -n 500` to limit output and preserve context window
|
||||
- The tail output will show test results and any failures at the end
|
||||
|
||||
This runs all backend tests in order and fails fast if any test fails:
|
||||
1. Unit tests (karaoke_gen package) with coverage check
|
||||
2. Backend unit tests
|
||||
3. E2E integration tests with emulators (auto-starts/stops emulators)
|
||||
|
||||
### Frontend Test Command
|
||||
|
||||
When working in the `frontend/` directory, run:
|
||||
|
||||
```bash
|
||||
# Frontend
|
||||
cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
```
|
||||
|
||||
This runs:
|
||||
1. **Unit tests** (Jest) - Component and utility tests
|
||||
2. **E2E tests** (Playwright) - Browser-based integration tests
|
||||
### Version Bumping
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
|
||||
- Skip for docs-only changes
|
||||
|
||||
**Note:** E2E tests require `KARAOKE_ACCESS_TOKEN` environment variable to be set in `frontend/.env.local`.
|
||||
### Infrastructure
|
||||
- All GCP changes via Pulumi in `infrastructure/`
|
||||
- `gcloud` CLI for reading/debugging only
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
Individual test commands:
|
||||
- `npm run test:unit` - Run unit tests only
|
||||
- `npm run test:e2e` - Run E2E tests only
|
||||
- `npm run test:ci` - Run unit tests with coverage
|
||||
## Documentation Maintenance
|
||||
|
||||
### Pre-commit Checklist
|
||||
### Structure
|
||||
```text
|
||||
docs/
|
||||
├── README.md # Current status + navigation (UPDATE THIS)
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Key insights for future agents
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
- [ ] `make test` passes (for backend changes)
|
||||
- [ ] `cd frontend && npm run test:all` passes (for frontend changes)
|
||||
- [ ] Version in `pyproject.toml` has been bumped
|
||||
- [ ] `coderabbit review --plain` has been run to get comprehensive analysis and suggestions for cleaner, more maintainable code. Apply the feedback to improve accessibility, structure, and best practices.
|
||||
### Rules
|
||||
1. **Before merging PRs**: Run `/docs-review` to check if docs need updating
|
||||
2. **Periodically**: Run `/docs-maintain` to verify docs are organized
|
||||
3. **For significant work**: Create `docs/archive/YYYY-MM-DD-topic.md`
|
||||
4. **Add learnings**: Update `docs/LESSONS-LEARNED.md` with insights
|
||||
5. **Keep status current**: Update `docs/README.md` when project state changes
|
||||
|
||||
**Do not commit or push code that fails tests. Fix any test failures before proceeding.**
|
||||
### What Goes Where
|
||||
- **README.md**: Current status, known issues, quick links
|
||||
- **ARCHITECTURE.md**: System design, data flow, tech decisions
|
||||
- **DEVELOPMENT.md**: Setup, testing, deployment, CI/CD
|
||||
- **API.md**: Endpoints, request/response formats
|
||||
- **LESSONS-LEARNED.md**: What worked, what didn't, gotchas
|
||||
- **archive/**: Completed work, session summaries, old plans
|
||||
|
||||
## Related Projects
|
||||
|
||||
### flacfetch
|
||||
Location: `/Users/andrew/Projects/flacfetch`
|
||||
|
||||
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
|
||||
|
||||
## Pre-Commit Checklist
|
||||
|
||||
- [ ] Tests pass (`make test` / `npm run test:all`)
|
||||
- [ ] Version bumped (if code changed)
|
||||
- [ ] `/docs-review` run (check for doc updates)
|
||||
- [ ] PR created with clear description
|
||||
@@ -71,8 +71,40 @@ export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
|
||||
|
||||
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
|
||||
|
||||
#### Option 2: Whisper via RunPod
|
||||
Open-source alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
#### Option 2: Local Whisper (No Cloud Required)
|
||||
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
|
||||
|
||||
```bash
|
||||
# Install with local Whisper support
|
||||
pip install karaoke-gen[local-whisper]
|
||||
|
||||
# Optional: Configure model size (tiny, base, small, medium, large)
|
||||
export WHISPER_MODEL_SIZE="medium"
|
||||
|
||||
# Optional: Force specific device (cpu, cuda, mps)
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
**Model Size Guide:**
|
||||
| Model | VRAM | Speed | Quality |
|
||||
|-------|------|-------|---------|
|
||||
| tiny | ~1GB | Fast | Lower |
|
||||
| base | ~1GB | Fast | Basic |
|
||||
| small | ~2GB | Medium | Good |
|
||||
| medium | ~5GB | Slower | Better |
|
||||
| large | ~10GB | Slowest | Best |
|
||||
|
||||
**CPU-Only Installation** (no GPU required):
|
||||
```bash
|
||||
# Pre-install CPU-only PyTorch first
|
||||
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install karaoke-gen[local-whisper]
|
||||
```
|
||||
|
||||
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
|
||||
|
||||
#### Option 3: Whisper via RunPod
|
||||
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
|
||||
```bash
|
||||
export RUNPOD_API_KEY="your_runpod_key"
|
||||
@@ -574,6 +606,44 @@ If the output video has quality problems:
|
||||
- Check available codecs: `ffmpeg -codecs`
|
||||
- For 4K output, ensure sufficient disk space (10GB+ per track)
|
||||
|
||||
### Local Whisper Issues
|
||||
|
||||
#### GPU Out of Memory
|
||||
If you get CUDA out of memory errors:
|
||||
```bash
|
||||
# Use a smaller model
|
||||
export WHISPER_MODEL_SIZE="small" # or "tiny"
|
||||
|
||||
# Or force CPU mode
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
#### Slow Transcription on CPU
|
||||
CPU transcription is significantly slower than GPU. For faster processing:
|
||||
- Use a smaller model (`tiny` or `base`)
|
||||
- Consider using cloud transcription (AudioShake or RunPod)
|
||||
- On Apple Silicon, the `small` model offers good speed/quality balance
|
||||
|
||||
#### Model Download Issues
|
||||
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
|
||||
- Check your internet connection
|
||||
- Set a custom cache directory: `export WHISPER_CACHE_DIR="/path/with/space"`
|
||||
- Models are cached in `~/.cache/whisper/` by default
|
||||
|
||||
#### whisper-timestamped Not Found
|
||||
If you get "whisper-timestamped is not installed":
|
||||
```bash
|
||||
pip install karaoke-gen[local-whisper]
|
||||
# Or install directly:
|
||||
pip install whisper-timestamped
|
||||
```
|
||||
|
||||
#### Disabling Local Whisper
|
||||
If you want to disable local Whisper (e.g., to force cloud transcription):
|
||||
```bash
|
||||
export ENABLE_LOCAL_WHISPER="false"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Development
|
||||
|
||||
+183
@@ -0,0 +1,183 @@
|
||||
# API Reference
|
||||
|
||||
Base URL: `https://api.nomadkaraoke.com` (production) or `http://localhost:8000` (local)
|
||||
|
||||
## Authentication
|
||||
|
||||
All endpoints except `/health` and `/api/themes` require authentication.
|
||||
|
||||
```bash
|
||||
curl -H "Authorization: Bearer YOUR_TOKEN" ...
|
||||
```
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Health
|
||||
|
||||
```http
|
||||
GET /health
|
||||
```
|
||||
|
||||
No auth required. Returns service status.
|
||||
|
||||
### Jobs
|
||||
|
||||
#### Create Job with Upload
|
||||
|
||||
```http
|
||||
POST /api/jobs/upload
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
file: <audio file>
|
||||
artist: "Artist Name"
|
||||
title: "Song Title"
|
||||
```
|
||||
|
||||
Returns job ID. Triggers async processing.
|
||||
|
||||
#### Get Job Status
|
||||
|
||||
```http
|
||||
GET /api/jobs/{job_id}
|
||||
```
|
||||
|
||||
Response includes:
|
||||
- `status` - Current job state
|
||||
- `file_urls` - Signed URLs for outputs (when ready)
|
||||
- `state_data` - Worker progress details
|
||||
- `timeline` - Processing history
|
||||
|
||||
#### List Jobs
|
||||
|
||||
```http
|
||||
GET /api/jobs
|
||||
GET /api/jobs?status=complete
|
||||
GET /api/jobs?limit=10&offset=0
|
||||
```
|
||||
|
||||
#### Delete Job
|
||||
|
||||
```http
|
||||
DELETE /api/jobs/{job_id}
|
||||
```
|
||||
|
||||
### Review
|
||||
|
||||
#### Get Correction Data
|
||||
|
||||
```http
|
||||
GET /api/review/{job_id}/correction-data
|
||||
```
|
||||
|
||||
Returns lyrics correction data for the review UI.
|
||||
|
||||
#### Complete Review
|
||||
|
||||
```http
|
||||
POST /api/review/{job_id}/complete
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"corrections": [...],
|
||||
"corrected_segments": [...]
|
||||
}
|
||||
```
|
||||
|
||||
Saves corrections and triggers video rendering.
|
||||
|
||||
#### Generate Preview
|
||||
|
||||
```http
|
||||
POST /api/review/{job_id}/preview-video
|
||||
```
|
||||
|
||||
Generates preview video during review.
|
||||
|
||||
#### Stream Audio
|
||||
|
||||
```http
|
||||
GET /api/review/{job_id}/audio/{stem_type}
|
||||
```
|
||||
|
||||
Streams audio for review playback.
|
||||
|
||||
### Instrumental Selection
|
||||
|
||||
```http
|
||||
POST /api/jobs/{job_id}/select-instrumental
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"selection": "clean" // or "with_backing"
|
||||
}
|
||||
```
|
||||
|
||||
### Audio Search
|
||||
|
||||
```http
|
||||
GET /api/audio-search?q=song+name
|
||||
```
|
||||
|
||||
Search for songs (flacfetch integration).
|
||||
|
||||
### Themes
|
||||
|
||||
```http
|
||||
GET /api/themes
|
||||
```
|
||||
|
||||
No auth required. Returns available video themes.
|
||||
|
||||
### Internal (Admin Only)
|
||||
|
||||
These endpoints are used by workers and require admin tokens.
|
||||
|
||||
```http
|
||||
POST /api/internal/jobs/{job_id}/trigger-audio
|
||||
POST /api/internal/jobs/{job_id}/trigger-lyrics
|
||||
POST /api/internal/jobs/{job_id}/trigger-screens
|
||||
POST /api/internal/jobs/{job_id}/trigger-render-video
|
||||
POST /api/internal/jobs/{job_id}/trigger-video
|
||||
```
|
||||
|
||||
## Job States
|
||||
|
||||
| State | Description |
|
||||
|-------|-------------|
|
||||
| `pending` | Created, not started |
|
||||
| `downloading` | Processing input |
|
||||
| `separating_stage1` | Audio separation (1/2) |
|
||||
| `separating_stage2` | Audio separation (2/2) |
|
||||
| `transcribing` | Lyrics transcription |
|
||||
| `generating_screens` | Title/end screens |
|
||||
| `awaiting_review` | Waiting for human review |
|
||||
| `in_review` | Human reviewing |
|
||||
| `review_complete` | Review submitted |
|
||||
| `rendering_video` | Generating karaoke video |
|
||||
| `awaiting_instrumental_selection` | Waiting for selection |
|
||||
| `instrumental_selected` | Selection made |
|
||||
| `generating_video` | Final encoding |
|
||||
| `complete` | Done |
|
||||
| `failed` | Error |
|
||||
|
||||
## Error Responses
|
||||
|
||||
```json
|
||||
{
|
||||
"detail": "Error message"
|
||||
}
|
||||
```
|
||||
|
||||
Common status codes:
|
||||
- `400` - Bad request
|
||||
- `401` - Unauthorized
|
||||
- `404` - Job not found
|
||||
- `500` - Server error
|
||||
|
||||
## Rate Limits
|
||||
|
||||
No rate limits currently implemented.
|
||||
|
||||
## Webhooks
|
||||
|
||||
Not yet implemented. Jobs must be polled for status.
|
||||
@@ -0,0 +1,162 @@
|
||||
# Architecture
|
||||
|
||||
## System Overview
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Frontend (Cloudflare Pages) │
|
||||
│ Next.js + TypeScript │
|
||||
│ https://gen.nomadkaraoke.com │
|
||||
└───────────────────────────┬─────────────────────────────────────┘
|
||||
│ HTTPS
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Backend API (Cloud Run) │
|
||||
│ FastAPI + Python 3.12 │
|
||||
│ https://api.nomadkaraoke.com │
|
||||
│ │
|
||||
│ Workers: │
|
||||
│ • Audio Worker - Stem separation via Modal API │
|
||||
│ • Lyrics Worker - Transcription via AudioShake │
|
||||
│ • Screens Worker - Title/end screen generation │
|
||||
│ • Render Worker - Karaoke video with lyrics │
|
||||
│ • Video Worker - Final assembly and encoding │
|
||||
└───────────────────────────┬─────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────┼─────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────────┐
|
||||
│ Firestore│ │ GCS │ │Secret Manager│
|
||||
│ (jobs) │ │ (files) │ │ (API keys) │
|
||||
└──────────┘ └──────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
## Processing Pipeline
|
||||
|
||||
```text
|
||||
1. UPLOAD
|
||||
POST /api/jobs/upload
|
||||
• Save audio to GCS
|
||||
• Create job in Firestore
|
||||
• Trigger parallel workers
|
||||
|
||||
2. PARALLEL PROCESSING
|
||||
┌─────────────────┐ ┌─────────────────┐
|
||||
│ Audio Worker │ │ Lyrics Worker │
|
||||
│ Modal API │ │ AudioShake │
|
||||
│ 2-stage sep │ │ Auto-correct │
|
||||
└────────┬────────┘ └────────┬────────┘
|
||||
└──────────┬───────────┘
|
||||
▼
|
||||
3. SCREENS GENERATION
|
||||
• Title screen (5s video)
|
||||
• End screen (5s video)
|
||||
▼
|
||||
4. HUMAN REVIEW (AWAITING_REVIEW)
|
||||
• React UI for lyrics correction
|
||||
• Preview video generation
|
||||
• Submit corrections
|
||||
▼
|
||||
5. VIDEO RENDERING
|
||||
• Merge corrections with original data
|
||||
• Generate karaoke video with lyrics
|
||||
▼
|
||||
6. INSTRUMENTAL SELECTION
|
||||
• User chooses: clean or with_backing
|
||||
▼
|
||||
7. FINAL VIDEO
|
||||
• Remux with selected instrumental
|
||||
• Concatenate: title + karaoke + end
|
||||
• Encode to 4 formats
|
||||
▼
|
||||
8. COMPLETE
|
||||
• User downloads via signed URLs
|
||||
```
|
||||
|
||||
## Job States
|
||||
|
||||
| State | Description | Human Action |
|
||||
|-------|-------------|--------------|
|
||||
| `pending` | Job created | - |
|
||||
| `downloading` | Processing audio | - |
|
||||
| `separating_stage1` | Audio separation stage 1 | - |
|
||||
| `separating_stage2` | Audio separation stage 2 | - |
|
||||
| `transcribing` | Lyrics transcription | - |
|
||||
| `generating_screens` | Creating title/end screens | - |
|
||||
| `awaiting_review` | **Waiting for human** | Review lyrics |
|
||||
| `in_review` | Human actively reviewing | Edit lyrics |
|
||||
| `review_complete` | Review submitted | - |
|
||||
| `rendering_video` | Generating karaoke video | - |
|
||||
| `awaiting_instrumental_selection` | **Waiting for human** | Choose audio |
|
||||
| `instrumental_selected` | Selection made | - |
|
||||
| `generating_video` | Final encoding | - |
|
||||
| `complete` | All done | Download files |
|
||||
| `failed` | Error occurred | - |
|
||||
|
||||
## GCS File Structure
|
||||
|
||||
```text
|
||||
jobs/{job_id}/
|
||||
├── input/
|
||||
│ └── original.flac # Uploaded audio
|
||||
├── stems/
|
||||
│ ├── vocals.flac
|
||||
│ ├── instrumental_clean.flac
|
||||
│ ├── instrumental_with_backing.flac
|
||||
│ └── ...
|
||||
├── lyrics/
|
||||
│ ├── corrections.json # Auto-corrected
|
||||
│ ├── corrections_updated.json # After human review
|
||||
│ └── karaoke.ass
|
||||
├── screens/
|
||||
│ ├── title.mov
|
||||
│ └── end.mov
|
||||
├── videos/
|
||||
│ └── with_vocals.mkv # Karaoke video
|
||||
└── finals/
|
||||
├── lossless_4k.mp4
|
||||
├── lossless_4k.mkv
|
||||
├── lossy_4k.mp4
|
||||
└── lossy_720p.mp4
|
||||
```
|
||||
|
||||
## LyricsTranscriber Integration
|
||||
|
||||
**Key Design Decision**: We use LyricsTranscriber as a **library**, not a server.
|
||||
|
||||
```text
|
||||
CLI Mode (Original): Cloud Mode (Our Approach):
|
||||
──────────────────── ──────────────────────────
|
||||
LyricsTranscriber LyricsTranscriber
|
||||
├── transcribe() ├── transcribe() ✅
|
||||
├── correct() ├── correct() ✅
|
||||
├── ReviewServer.start() │
|
||||
│ └── BLOCKS (waits) │ (Skip server, save JSON)
|
||||
│ │ ... ASYNC GAP ...
|
||||
│ │ (Human reviews via React)
|
||||
└── OutputGenerator │
|
||||
└── OutputGenerator ✅
|
||||
(called after review)
|
||||
```
|
||||
|
||||
**Use**: `LyricsTranscriber`, `CorrectionResult`, `OutputGenerator`, `OutputConfig`
|
||||
**Don't use**: `ReviewServer`, blocking waits
|
||||
|
||||
## External Services
|
||||
|
||||
| Service | Purpose | Required |
|
||||
|---------|---------|----------|
|
||||
| Modal | GPU audio separation | Yes |
|
||||
| AudioShake | Lyrics transcription | Yes |
|
||||
| Genius | Reference lyrics | Yes |
|
||||
| YouTube API | Video upload | Optional |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Backend**: FastAPI, Python 3.12, Cloud Run
|
||||
- **Frontend**: Next.js, TypeScript, Cloudflare Pages
|
||||
- **Database**: Firestore
|
||||
- **Storage**: Google Cloud Storage
|
||||
- **Secrets**: Google Secret Manager
|
||||
- **IaC**: Pulumi
|
||||
- **CI/CD**: GitHub Actions
|
||||
@@ -0,0 +1,216 @@
|
||||
# Development Guide
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.12
|
||||
- Node.js 20+
|
||||
- Docker (for GCS emulator)
|
||||
- Google Cloud SDK with Firestore emulator
|
||||
|
||||
```bash
|
||||
# Install Firestore emulator
|
||||
gcloud components install cloud-firestore-emulator
|
||||
```
|
||||
|
||||
### Backend Setup
|
||||
|
||||
```bash
|
||||
# Create and activate venv
|
||||
python3.12 -m venv venv
|
||||
source venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
poetry install
|
||||
|
||||
# Copy environment template
|
||||
cp .env.example .env
|
||||
# Edit .env with required API keys
|
||||
```
|
||||
|
||||
### Frontend Setup
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
cp .env.example .env.local
|
||||
# Edit .env.local with KARAOKE_ACCESS_TOKEN
|
||||
```
|
||||
|
||||
## Running Locally
|
||||
|
||||
### Backend with Emulators
|
||||
|
||||
```bash
|
||||
# Start emulators and backend
|
||||
./scripts/run-backend-local.sh --with-emulators
|
||||
|
||||
# Backend runs at http://localhost:8000
|
||||
# API docs at http://localhost:8000/docs
|
||||
```
|
||||
|
||||
### Frontend
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm run dev
|
||||
# Runs at http://localhost:3000
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Backend Tests
|
||||
|
||||
```bash
|
||||
# Run all tests (required before commit)
|
||||
make test 2>&1 | tail -n 500
|
||||
|
||||
# This runs in order:
|
||||
# 1. Unit tests with coverage
|
||||
# 2. Backend unit tests
|
||||
# 3. E2E integration tests with emulators
|
||||
```
|
||||
|
||||
**Important**: Set 10 minute timeout. Tests can take a while with emulators.
|
||||
|
||||
### Frontend Tests
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm run test:all 2>&1 | tail -n 200
|
||||
|
||||
# Individual commands:
|
||||
npm run test:unit # Jest unit tests
|
||||
npm run test:e2e # Playwright E2E tests
|
||||
```
|
||||
|
||||
### Test Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────┐
|
||||
│ Unit Tests (Mocked) │ Fast, isolated
|
||||
│ ~62 tests, <1s │
|
||||
├─────────────────────────────────┤
|
||||
│ Emulator Integration Tests │ Local Firestore/GCS
|
||||
│ ~11 tests, ~2s │
|
||||
├─────────────────────────────────┤
|
||||
│ E2E Tests │ Full API + Frontend
|
||||
│ Requires deployed backend │
|
||||
└─────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Automatic (CI/CD)
|
||||
|
||||
Push to `main` triggers GitHub Actions:
|
||||
1. Run tests
|
||||
2. Build Docker image
|
||||
3. Deploy to Cloud Run
|
||||
4. Deploy frontend to Cloudflare Pages
|
||||
|
||||
### Manual Backend Deploy
|
||||
|
||||
```bash
|
||||
# Preview infrastructure changes
|
||||
cd infrastructure
|
||||
pulumi preview
|
||||
|
||||
# Apply changes
|
||||
pulumi up
|
||||
```
|
||||
|
||||
### Manual Frontend Deploy
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm run build
|
||||
npx wrangler pages deploy out
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Backend (.env)
|
||||
|
||||
```bash
|
||||
# Required
|
||||
AUDIOSHAKE_API_TOKEN=...
|
||||
GENIUS_ACCESS_TOKEN=...
|
||||
AUDIO_SEPARATOR_API_URL=... # Modal API endpoint
|
||||
|
||||
# GCP (auto-configured in Cloud Run)
|
||||
GOOGLE_CLOUD_PROJECT=...
|
||||
GCS_BUCKET_NAME=...
|
||||
|
||||
# Auth
|
||||
ADMIN_TOKENS=token1,token2
|
||||
```
|
||||
|
||||
### Frontend (.env.local)
|
||||
|
||||
```bash
|
||||
NEXT_PUBLIC_API_URL=http://localhost:8000
|
||||
KARAOKE_ACCESS_TOKEN=... # For E2E tests
|
||||
```
|
||||
|
||||
## Key Scripts
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `make test` | Run all backend tests |
|
||||
| `./scripts/run-backend-local.sh` | Start backend with emulators |
|
||||
| `./scripts/start-emulators.sh` | Start emulators only |
|
||||
| `./scripts/stop-emulators.sh` | Stop emulators |
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
karaoke-gen/
|
||||
├── backend/
|
||||
│ ├── api/routes/ # FastAPI endpoints
|
||||
│ ├── models/ # Pydantic models
|
||||
│ ├── services/ # Business logic
|
||||
│ ├── workers/ # Background workers
|
||||
│ └── tests/ # Backend tests
|
||||
├── frontend/
|
||||
│ ├── src/
|
||||
│ │ ├── app/ # Next.js pages
|
||||
│ │ ├── components/ # React components
|
||||
│ │ └── lib/ # Utilities
|
||||
│ └── tests/ # Frontend tests
|
||||
├── infrastructure/ # Pulumi IaC
|
||||
├── karaoke_gen/ # CLI package
|
||||
└── scripts/ # Dev scripts
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Emulators Won't Start
|
||||
|
||||
```bash
|
||||
# Check if already running
|
||||
lsof -i :8080 # Firestore
|
||||
lsof -i :4443 # GCS
|
||||
|
||||
# Stop them
|
||||
./scripts/stop-emulators.sh
|
||||
```
|
||||
|
||||
### Tests Fail with Auth Errors
|
||||
|
||||
Ensure `ADMIN_TOKENS` is set in environment or `.env` file.
|
||||
|
||||
### Docker Issues
|
||||
|
||||
```bash
|
||||
# Ensure Docker is running
|
||||
docker info
|
||||
|
||||
# For GCS emulator
|
||||
docker pull fsouza/fake-gcs-server
|
||||
```
|
||||
|
||||
### Coverage Below Target
|
||||
|
||||
Run `./scripts/run-tests.sh --coverage` and check `htmlcov/index.html` to see uncovered lines.
|
||||
@@ -0,0 +1,151 @@
|
||||
# Lessons Learned
|
||||
|
||||
Key insights for future AI agents working on this codebase.
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
### LyricsTranscriber: Library, Not Server
|
||||
|
||||
**Problem**: LyricsTranscriber's `ReviewServer` blocks waiting for human input - incompatible with async cloud architecture.
|
||||
|
||||
**Solution**: Use LyricsTranscriber as a library:
|
||||
- Call `transcribe()` and `correct()` for processing
|
||||
- Save `corrections.json` to GCS
|
||||
- Skip `ReviewServer` entirely
|
||||
- Call `OutputGenerator` only after human review completes
|
||||
|
||||
**What to use**: `LyricsTranscriber`, `CorrectionResult`, `OutputGenerator`, `OutputConfig`, `CorrectionOperations`
|
||||
|
||||
**What NOT to use**: `ReviewServer`, `server.start()`, blocking waits
|
||||
|
||||
### Video Generation Timing
|
||||
|
||||
**Wrong**: Generate video during lyrics worker, then review
|
||||
**Right**: Lyrics worker saves JSON → Human reviews → Render worker generates video
|
||||
|
||||
Video must include human-corrected lyrics, so generation happens AFTER review.
|
||||
|
||||
### Parallel Processing
|
||||
|
||||
Audio separation and lyrics transcription run in parallel:
|
||||
- Both can fail independently
|
||||
- Progress tracked separately in `state_data`
|
||||
- Screens generation waits for both to complete
|
||||
|
||||
## Common Gotchas
|
||||
|
||||
### Pydantic Model Fields
|
||||
|
||||
**Problem**: Setting a field that doesn't exist in the Pydantic model silently does nothing.
|
||||
|
||||
```python
|
||||
# BAD - fails silently if field doesn't exist
|
||||
job.input_media_gcs_path = "path/to/file"
|
||||
|
||||
# GOOD - explicit field in model
|
||||
class Job(BaseModel):
|
||||
input_media_gcs_path: Optional[str] = None
|
||||
```
|
||||
|
||||
**Lesson**: Always verify Pydantic model has the field before setting it.
|
||||
|
||||
### Styles Configuration
|
||||
|
||||
ASS subtitle generator requires ALL style fields present and properly typed:
|
||||
|
||||
```python
|
||||
required_fields = [
|
||||
"font", "font_path", "ass_name", # ass_name often missed
|
||||
"primary_color", "secondary_color", "outline_color", "back_color",
|
||||
"bold", "italic", "underline", "strike_out",
|
||||
"scale_x", "scale_y", "spacing", "angle",
|
||||
"border_style", "outline", "shadow",
|
||||
"margin_l", "margin_r", "margin_v", "encoding"
|
||||
]
|
||||
|
||||
# Critical: font_path must be string, not None
|
||||
"font_path": "" # Correct
|
||||
"font_path": None # Causes ASS writer to fail
|
||||
```
|
||||
|
||||
### Frontend Data Format
|
||||
|
||||
The React review UI sends partial correction data:
|
||||
|
||||
```python
|
||||
# Frontend sends:
|
||||
{"corrections": [...], "corrected_segments": [...]}
|
||||
|
||||
# Backend must merge with original corrections.json
|
||||
```
|
||||
|
||||
### Firestore Consistency
|
||||
|
||||
**Problem**: Update job, trigger worker immediately, worker reads stale data.
|
||||
|
||||
**Solution**: Verify updates propagated before triggering workers:
|
||||
```python
|
||||
# Update job
|
||||
await job_manager.update_job(job_id, updates)
|
||||
|
||||
# Verify update visible
|
||||
job = await job_manager.get_job(job_id)
|
||||
if job.input_media_gcs_path:
|
||||
await trigger_worker()
|
||||
```
|
||||
|
||||
## Testing Insights
|
||||
|
||||
### Emulator Tests Catch Real Bugs
|
||||
|
||||
Unit tests with mocks didn't catch the `input_media_gcs_path` bug. Emulator integration tests did because they use real Firestore behavior.
|
||||
|
||||
**Lesson**: Run `make test` which includes emulator tests before every commit.
|
||||
|
||||
### Test Isolation
|
||||
|
||||
Emulator tests need their own `conftest.py` that does NOT mock `google.cloud.*`. Mocks from unit tests can leak into integration tests.
|
||||
|
||||
## Deployment Notes
|
||||
|
||||
### Cloud Run Timeouts
|
||||
|
||||
Default 5-minute timeout may not be enough for video encoding. Consider:
|
||||
- Cloud Run Jobs for long-running tasks
|
||||
- Breaking encoding into smaller chunks
|
||||
- Increasing timeout for specific operations
|
||||
|
||||
### Secret Manager Access
|
||||
|
||||
Cloud Run service account needs `Secret Manager Secret Accessor` role. This is managed via Pulumi in `infrastructure/`.
|
||||
|
||||
### CI/CD: GitHub Actions
|
||||
|
||||
Migrated from Cloud Build to GitHub Actions for:
|
||||
- Simpler configuration
|
||||
- Better visibility
|
||||
- Faster feedback
|
||||
|
||||
## Performance Observations
|
||||
|
||||
| Operation | Duration |
|
||||
|-----------|----------|
|
||||
| Audio separation | 5-8 min |
|
||||
| Lyrics transcription | 2-3 min |
|
||||
| Auto-correction | 30 sec |
|
||||
| Screens generation | 30 sec |
|
||||
| Video rendering | 10-15 min |
|
||||
| Final encoding | 5-10 min |
|
||||
| **Total** | 30-50 min |
|
||||
|
||||
Human review time varies (5-15 min typical).
|
||||
|
||||
## What We'd Do Differently
|
||||
|
||||
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
|
||||
|
||||
2. **Use emulator tests from day one** - Faster feedback than deploying to test
|
||||
|
||||
3. **Design for async human review upfront** - Avoided rearchitecting after discovering ReviewServer blocks
|
||||
|
||||
4. **Keep docs minimal and current** - Less documentation, but always accurate
|
||||
+33
-177
@@ -1,193 +1,49 @@
|
||||
# Karaoke-Gen Documentation
|
||||
|
||||
This directory contains all documentation for the karaoke-gen project, organized by purpose.
|
||||
> **Status**: Production-ready. First full E2E flow completed 2025-12-28.
|
||||
|
||||
## 📋 Current Status (Updated: 2025-12-08)
|
||||
## What Works
|
||||
|
||||
**Project:** Cloud-hosted karaoke generation with async human review
|
||||
**Backend:** 🔄 Phase 1.3 (85%) - Implementing review architecture
|
||||
**Frontend:** ⏸️ Not started (waiting for backend review flow)
|
||||
**Deployment:** ✅ Cloud Run with custom domain (api.nomadkaraoke.com)
|
||||
**Testing:** 🔄 Local emulator testing in progress
|
||||
- **Web App**: https://gen.nomadkaraoke.com - Upload audio, review lyrics, download videos
|
||||
- **Backend API**: https://api.nomadkaraoke.com - FastAPI on Cloud Run
|
||||
- **CLI Tools**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
|
||||
### Key Discovery
|
||||
## Current State
|
||||
|
||||
We identified that the `LyricsTranscriber` library's `ReviewServer` **blocks** waiting for human input, which doesn't work in our async cloud architecture. The solution is to:
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| Audio upload & separation | Working |
|
||||
| Lyrics transcription | Working |
|
||||
| Human lyrics review | Working |
|
||||
| Preview video generation | Working |
|
||||
| Instrumental selection | Working |
|
||||
| Multi-format encoding | Working |
|
||||
| Token-based auth | Working |
|
||||
| Payment flow | Not started |
|
||||
|
||||
1. Use LyricsTranscriber for transcription + auto-correction
|
||||
2. Save corrections to GCS for async human review
|
||||
3. Use `OutputGenerator` to render video **after** review completes
|
||||
## Known Issues
|
||||
|
||||
See `00-current-plan/CURRENT-STATUS.md` for full details.
|
||||
- CDG format generation requires additional style configuration
|
||||
- Long audio files (>10 min) may timeout on some workers
|
||||
|
||||
---
|
||||
## Quick Links
|
||||
|
||||
## 📂 Documentation Structure
|
||||
| Doc | Purpose |
|
||||
|-----|---------|
|
||||
| [ARCHITECTURE.md](ARCHITECTURE.md) | System design, data flow, integrations |
|
||||
| [DEVELOPMENT.md](DEVELOPMENT.md) | Local setup, testing, deployment |
|
||||
| [API.md](API.md) | Backend endpoint reference |
|
||||
| [LESSONS-LEARNED.md](LESSONS-LEARNED.md) | Key insights for future agents |
|
||||
| [archive/](archive/) | Historical documentation |
|
||||
|
||||
### `00-current-plan/` - **START HERE**
|
||||
Active migration plan and current status.
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| **`CURRENT-STATUS.md`** | Where we are now, what's working, what's next |
|
||||
| **`WHATS-NEXT.md`** | Detailed plan for review architecture implementation |
|
||||
| **`WORKER-IMPLEMENTATION-PLAN.md`** | Worker responsibilities and implementation details |
|
||||
| `MASTER-PLAN.md` | Overall migration strategy |
|
||||
| `CLOUD-MIGRATION-REQUIREMENTS.md` | Architectural analysis from CLI workflow |
|
||||
|
||||
### `01-reference/` - Reference Documentation
|
||||
Core system documentation.
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| **`ARCHITECTURE.md`** | Cloud + CLI architecture with diagrams |
|
||||
| `KARAOKE-GEN-CLI-WORKFLOW.md` | CLI workflow breakdown (8 stages) |
|
||||
| `API-MANUAL-TESTING.md` | How to test backend API with curl |
|
||||
| `AUTHENTICATION.md` | Token-based auth system |
|
||||
| `CLOUD-RUN-ARCHITECTURE.md` | Cloud Run concepts |
|
||||
|
||||
### `02-implementation-history/` - Session Logs
|
||||
Historical implementation records.
|
||||
|
||||
### `03-deployment/` - Deployment Guides
|
||||
How to deploy and operate the system.
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| **`EMULATOR-TESTING.md`** | Local testing with Firestore/GCS emulators |
|
||||
| **`SETUP-VENV.md`** | Python venv setup |
|
||||
| **`LOCAL-VALIDATION.md`** | Validate before deploying |
|
||||
| **`OBSERVABILITY-GUIDE.md`** | Debug production issues |
|
||||
|
||||
### `04-testing/` - Testing Guides
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| **`TESTING-GUIDE.md`** | Complete testing guide |
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### I want to understand the current state:
|
||||
1. Read `00-current-plan/CURRENT-STATUS.md` - where we are now
|
||||
2. Read `00-current-plan/WHATS-NEXT.md` - what needs to be done
|
||||
3. Read `01-reference/ARCHITECTURE.md` - system architecture
|
||||
|
||||
### I want to work on the backend:
|
||||
1. Read `03-deployment/EMULATOR-TESTING.md` - local testing setup
|
||||
2. Read `00-current-plan/WORKER-IMPLEMENTATION-PLAN.md` - worker details
|
||||
3. Run `./scripts/run-backend-local.sh --with-emulators`
|
||||
|
||||
### I want to test:
|
||||
```bash
|
||||
# Start backend with emulators
|
||||
./scripts/run-backend-local.sh --with-emulators
|
||||
|
||||
# Upload test file
|
||||
curl -X POST http://localhost:8000/api/jobs/upload \
|
||||
-F "file=@tests/data/waterloo10sec.flac" \
|
||||
-F "artist=ABBA" \
|
||||
-F "title=Waterloo"
|
||||
|
||||
# Check status
|
||||
curl http://localhost:8000/api/jobs/{job_id}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Current Phase: Review Architecture
|
||||
|
||||
**Goal:** Implement async human review workflow
|
||||
|
||||
**What Works:**
|
||||
- ✅ File upload → GCS
|
||||
- ✅ Audio separation via Modal API
|
||||
- ✅ Lyrics transcription via AudioShake
|
||||
- ✅ Auto-correction via LyricsTranscriber
|
||||
- ✅ Title/end screen generation
|
||||
- ✅ Instrumental selection endpoint
|
||||
|
||||
**What's Being Implemented:**
|
||||
- 🔄 Review API endpoints (`GET/POST /api/jobs/{id}/review`)
|
||||
- 🔄 Render Video Worker (uses `OutputGenerator` after review)
|
||||
- 🔄 Updated state machine with `RENDERING_VIDEO` state
|
||||
|
||||
**The Key Insight:**
|
||||
```
|
||||
WRONG: Lyrics Worker → Generate Video → Review → ???
|
||||
RIGHT: Lyrics Worker → Save JSON → Human Review → Render Video → Final Assembly
|
||||
```
|
||||
|
||||
Video generation happens AFTER human review, not during the lyrics worker.
|
||||
|
||||
---
|
||||
|
||||
## 📝 Key Technical Decisions
|
||||
|
||||
### Why Not Use LyricsTranscriber's ReviewServer?
|
||||
|
||||
The `ReviewServer` class (in `lyrics_transcriber/review/server.py`):
|
||||
- Starts a local web server on port 8000
|
||||
- Opens a browser window
|
||||
- **BLOCKS** the thread until human submits corrections
|
||||
- Designed for local CLI operation
|
||||
|
||||
This doesn't work in Cloud Run where:
|
||||
- We can't open browsers
|
||||
- Workers must be non-blocking
|
||||
- Human review happens asynchronously via React frontend
|
||||
|
||||
**Solution:** Use the data structures and `OutputGenerator` directly, skip the server.
|
||||
|
||||
### Why Separate Video Rendering?
|
||||
## Workflow Summary
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Lyrics Worker │ ──▶ │ Human Reviews │ ──▶ │ Render Worker │
|
||||
│ │ │ (React UI) │ │ │
|
||||
│ • Transcribe │ │ • Corrects │ │ • OutputGenerator│
|
||||
│ • Auto-correct │ │ • Saves JSON │ │ • with_vocals.mkv│
|
||||
│ • Save JSON │ │ │ │ │
|
||||
│ • NO VIDEO │ │ │ │ │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
Upload Audio → Parallel Processing → Human Review → Video Generation → Download
|
||||
│ │
|
||||
(Modal + AudioShake) (LyricsTranscriber UI)
|
||||
```
|
||||
|
||||
The video must include the human-corrected lyrics, so it can only be generated AFTER review.
|
||||
## For AI Agents
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Job State Flow
|
||||
|
||||
```
|
||||
PENDING → DOWNLOADING → [parallel audio+lyrics] → GENERATING_SCREENS
|
||||
↓
|
||||
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE
|
||||
↓
|
||||
RENDERING_VIDEO (NEW)
|
||||
↓
|
||||
AWAITING_INSTRUMENTAL_SELECTION → INSTRUMENTAL_SELECTED
|
||||
↓
|
||||
GENERATING_VIDEO → COMPLETE
|
||||
```
|
||||
|
||||
Two human interaction points:
|
||||
1. **AWAITING_REVIEW** - User corrects lyrics
|
||||
2. **AWAITING_INSTRUMENTAL_SELECTION** - User chooses audio track
|
||||
|
||||
---
|
||||
|
||||
## 📞 Getting Help
|
||||
|
||||
1. Check `00-current-plan/CURRENT-STATUS.md` first
|
||||
2. Look for specific guides in `03-deployment/`
|
||||
3. Review session history in `02-implementation-history/`
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| Emulator not starting | Check ports 8080, 4443 are free |
|
||||
| "Missing lyrics video" | Review hasn't happened yet - expected behavior |
|
||||
| Workers not progressing | Check `state_data` in Firestore |
|
||||
| FFmpeg errors | Ensure FFmpeg is in PATH |
|
||||
Start with `CLAUDE.md` in the repo root for essential rules, then reference these docs as needed. Keep this README current when project state changes.
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,197 @@
|
||||
# GitHub Actions Docker Builds - Cost Optimization
|
||||
|
||||
**Date:** 2025-12-28
|
||||
**Problem:** Cloud Build costing $100+/month with slow builds
|
||||
**Solution:** Move Docker builds to GitHub Actions with registry caching
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the migration from Google Cloud Build to GitHub Actions for Docker image builds. This change eliminates Cloud Build costs while improving build speeds through better caching.
|
||||
|
||||
## Cost Comparison
|
||||
|
||||
| Approach | Monthly Cost | Build Speed | Caching |
|
||||
|----------|-------------|-------------|---------|
|
||||
| **Cloud Build (N1_HIGHCPU_32)** | ~$100+ | 5-15 min | Limited |
|
||||
| **GitHub Actions + Registry Cache** | **$0** | 2-5 min | Excellent |
|
||||
|
||||
### Why Cloud Build Was Expensive
|
||||
|
||||
1. **N1_HIGHCPU_32 pricing**: ~$0.064/build-minute (21x more than default)
|
||||
2. **No persistent layer cache**: Each build re-downloaded dependencies
|
||||
3. **Cold start overhead**: Spinning up build VMs added latency
|
||||
4. **Large dependency tree**: torch, transformers, etc. = ~500 packages
|
||||
|
||||
### Why GitHub Actions Is Free
|
||||
|
||||
- **Public repositories**: Unlimited build minutes on GitHub-hosted runners
|
||||
- **Registry cache**: Layers cached in Artifact Registry, persist between builds
|
||||
- **BuildKit mode=max**: Caches ALL intermediate layers, not just final
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ NEW BUILD PIPELINE │
|
||||
├─────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ GitHub Actions (Free) │
|
||||
│ ├── docker/setup-buildx-action (BuildKit) │
|
||||
│ ├── google-github-actions/auth (Workload Identity) │
|
||||
│ └── docker/build-push-action │
|
||||
│ ├── cache-from: type=registry,ref=...karaoke-backend:cache │
|
||||
│ └── cache-to: type=registry,ref=...karaoke-backend:cache,mode=max│
|
||||
│ │
|
||||
│ Artifact Registry │
|
||||
│ ├── karaoke-backend:latest (production image) │
|
||||
│ ├── karaoke-backend:cache (build cache - ALL layers) │
|
||||
│ ├── karaoke-backend-base:latest (base image with deps) │
|
||||
│ └── karaoke-backend-base:cache (base cache) │
|
||||
│ │
|
||||
│ Build Times (Expected): │
|
||||
│ ├── First build (cold): ~8-10 min │
|
||||
│ ├── Code-only changes: ~2-3 min (cache hit) │
|
||||
│ └── Dependency changes: ~6-8 min (partial cache hit) │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
### Two-Stage Build Pattern
|
||||
|
||||
We maintain the existing two-stage build pattern for efficiency:
|
||||
|
||||
1. **Base Image** (`karaoke-backend-base`)
|
||||
- Contains Python 3.11, ffmpeg, system dependencies
|
||||
- All pip dependencies pre-installed
|
||||
- Only rebuilt when `poetry.lock` or `Dockerfile.base` changes
|
||||
- Hash-based change detection via image labels
|
||||
|
||||
2. **App Image** (`karaoke-backend`)
|
||||
- Built FROM the base image
|
||||
- Contains only application code
|
||||
- Fast rebuilds (~2 min) for code-only changes
|
||||
|
||||
### Registry Cache
|
||||
|
||||
The key optimization is using Artifact Registry as a persistent layer cache:
|
||||
|
||||
```yaml
|
||||
- uses: docker/build-push-action@v6
|
||||
with:
|
||||
cache-from: type=registry,ref=.../karaoke-backend:cache
|
||||
cache-to: type=registry,ref=.../karaoke-backend:cache,mode=max
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- `mode=max` caches ALL layers, not just the final image
|
||||
- Cache persists between CI runs (unlike GitHub Actions cache which has 10GB limit)
|
||||
- Layers only re-built when inputs change
|
||||
|
||||
---
|
||||
|
||||
## Files Changed
|
||||
|
||||
### New Files
|
||||
|
||||
- **`.dockerignore`** - Reduces build context from ~1GB to ~10MB
|
||||
|
||||
### Modified Files
|
||||
|
||||
- **`.github/workflows/ci.yml`** - `deploy-backend` job rewritten to use:
|
||||
- `docker/setup-buildx-action@v3`
|
||||
- `docker/build-push-action@v6`
|
||||
- Registry caching to Artifact Registry
|
||||
|
||||
### Kept (as fallback)
|
||||
|
||||
- **`cloudbuild.yaml`** - Still works for manual triggers
|
||||
- **`cloudbuild-base.yaml`** - Still works for manual base image builds
|
||||
|
||||
---
|
||||
|
||||
## Manual Builds (Fallback)
|
||||
|
||||
If GitHub Actions fails, you can still use Cloud Build:
|
||||
|
||||
```bash
|
||||
# Build base image (only if dependencies changed)
|
||||
gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
|
||||
|
||||
# Build and deploy app image
|
||||
gcloud builds submit --config=cloudbuild.yaml --project=nomadkaraoke
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Check Build Cache Usage
|
||||
|
||||
View cache images in Artifact Registry:
|
||||
|
||||
```bash
|
||||
gcloud artifacts docker images list \
|
||||
us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo \
|
||||
--include-tags
|
||||
```
|
||||
|
||||
You should see `*:cache` tags for both base and app images.
|
||||
|
||||
### Compare Build Times
|
||||
|
||||
- **GitHub Actions**: Check workflow run times in Actions tab
|
||||
- **Previous Cloud Build**: ~5-15 min per build
|
||||
- **Expected with cache**: ~2-3 min for code changes
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Cache Miss (Slow Build)
|
||||
|
||||
If builds are slow despite no dependency changes:
|
||||
|
||||
1. Check if cache image exists:
|
||||
```bash
|
||||
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:cache
|
||||
```
|
||||
|
||||
2. Force cache rebuild by pushing to `main` with a small change
|
||||
|
||||
### Disk Space Issues
|
||||
|
||||
GitHub Actions runners have limited disk. If base image builds fail:
|
||||
|
||||
1. The workflow includes `jlumbroso/free-disk-space` action
|
||||
2. It removes Android SDK, Haskell, .NET, etc. to free ~20GB
|
||||
|
||||
### Authentication Issues
|
||||
|
||||
If `docker push` fails with permission errors:
|
||||
|
||||
1. Check Workload Identity is configured correctly
|
||||
2. Verify service account has `roles/artifactregistry.writer`
|
||||
|
||||
---
|
||||
|
||||
## Future Improvements
|
||||
|
||||
1. **Parallel base/app builds**: If base is unchanged, start app build immediately
|
||||
2. **GitHub Actions cache fallback**: Use GHA cache when registry is slow
|
||||
3. **Build metrics**: Track build times and cache hit rates
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Docker Build with GitHub Actions](https://docs.docker.com/build/ci/github-actions/)
|
||||
- [Registry Cache Backend](https://docs.docker.com/build/cache/backends/registry/)
|
||||
- [Cache Management](https://docs.docker.com/build/ci/github-actions/cache/)
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loaded 100 of 128 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user