Compare commits

...
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.5 d2c8c24eee fix: make runner startup script reproducible
Fixes issues discovered during initial runner setup:

- Use `gpg --batch` for all GPG commands to avoid tty errors
- Install Python 3.13 via pyenv (not broken PPA/source build)
- Set up Python 3.13 in tool cache for setup-python action
- Use Temurin directly for Java 21 (skip broken Debian repos)

These changes ensure the runner VM can be recreated from scratch
and work correctly with GitHub Actions setup-python.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 12:03:10 -05:00
Andrew Beveridge ☄️ 0b57b8f610 Merge pull request #95 from nomadkaraoke/feature/self-hosted-runner
feat: add self-hosted GitHub Actions runner on GCP
2025-12-28 11:51:59 -05:00
Andrew BeveridgeandClaude Opus 4.5 050ca6a441 fix: add setup-python and install-poetry to self-hosted runner jobs
Addresses CodeRabbit review feedback - the self-hosted runner jobs
need proper Python and Poetry setup via actions rather than relying
on pre-installed versions. This ensures Python 3.13 compatibility.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 11:49:09 -05:00
Andrew BeveridgeandClaude Opus 4.5 2725352d89 feat: add self-hosted GitHub Actions runner on GCP
Adds a dedicated GCP VM (e2-standard-4, 200GB SSD) to run GitHub Actions
jobs that were hitting disk space limits on GitHub-hosted runners (14GB).

Infrastructure changes:
- New service account `github-runner` with minimal permissions
- Secret Manager secret for GitHub PAT (runner registration)
- Compute instance with startup script that installs Docker, Python 3.13,
  Node.js 20, Java 21, Poetry, FFmpeg, and registers as a GitHub runner

Workflow changes:
- `package-build-test` now uses self-hosted runner
- `backend-unit-tests` now uses self-hosted runner
- `backend-emulator-tests` now uses self-hosted runner
- `deploy-backend` now uses self-hosted runner
- Removed disk cleanup steps (no longer needed with 200GB)

Setup requires adding GitHub PAT to Secret Manager after `pulumi up`:
  echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 11:18:50 -05:00
Andrew Beveridge ☄️ 762cdf03c9 Merge pull request #94 from nomadkaraoke/fix/pip-install-quotes-91
docs: Add quotes to pip install commands for cross-shell compatibility
2025-12-28 11:11:47 -05:00
Andrew BeveridgeandClaude Opus 4.5 7df3a21525 docs: Add quotes to pip install commands for cross-shell compatibility
Wrap package names with extras notation in double quotes to prevent
zsh and other shells from interpreting square brackets as glob patterns.

Fixes #91

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 11:10:09 -05:00
Andrew Beveridge ☄️ f72f772f96 Merge pull request #88 from nomadkaraoke/feature/mobile-ux-improvements
feat: improve mobile responsiveness across all UIs
2025-12-28 03:13:58 -05:00
Andrew BeveridgeandClaude Opus 4.5 7a30407fdb fix: Address CodeRabbit review feedback
- Add language identifier to markdown code block in docs
- Increase Undo/Redo button touch targets from 32px to 44px

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 03:08:53 -05:00
Andrew BeveridgeandClaude Opus 4.5 e0e35f5335 feat: improve mobile responsiveness across all UIs
- Main Frontend:
  - Stack Artist/Title inputs vertically on mobile (< 640px)
  - Improve tab button touch targets with min-height 44px
  - Make header responsive with icon-only buttons on small screens
  - Hide button text labels on mobile, show icons only

- Lyrics Review UI (Material UI):
  - Make handler panel width responsive (100% on mobile)
  - Fix button toolbar to wrap on mobile
  - Adjust heights for mobile viewports

- Instrumental Selection UI (Static HTML):
  - Add comprehensive mobile media queries
  - Stack bottom section vertically on mobile
  - Make selection panel full-width on mobile
  - Increase touch targets to 44px minimum
  - Allow scrolling on mobile viewports

- Testing:
  - Add mobile device projects to Playwright config (Pixel 7, iPhone 14)
  - Create comprehensive mobile responsiveness test suite
  - Test no horizontal overflow, touch targets, form usability

- Documentation:
  - Add MOBILE-UX-BEST-PRACTICES.md with patterns and guidelines

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 03:08:05 -05:00
Andrew Beveridge ☄️ 79be51c95d Merge pull request #89 from nomadkaraoke/refactor/documentation-restructure
docs: Restructure documentation for LLM agent clarity
2025-12-28 03:00:33 -05:00
Andrew BeveridgeandClaude Opus 4.5 a3f0c08f99 fix: Address markdown linting issues from CodeRabbit review
- Wrap bare URLs in angle brackets in CLAUDE.md (MD034)
- Add language specifiers to fenced code blocks (MD040):
  - `text` for directory structures and ASCII diagrams
  - `http` for HTTP request examples

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 02:58:22 -05:00
Andrew BeveridgeandClaude Opus 4.5 986a5e499f docs: Restructure documentation for LLM agent clarity
## Summary
Complete documentation reorganization to make it more effective for AI agents working on this codebase.

## Changes Made
- Consolidated 88 docs into 5 active files + archive
- New structure: README.md, ARCHITECTURE.md, DEVELOPMENT.md, API.md, LESSONS-LEARNED.md
- Added Claude Code slash commands: /docs-review (pre-merge), /docs-maintain (periodic)
- Updated CLAUDE.md with documentation maintenance rules
- Moved all historical docs to docs/archive/ with YYYY-MM-DD prefixes

## New Documentation Structure
```
docs/
├── README.md          # Current status + navigation
├── ARCHITECTURE.md    # System design
├── DEVELOPMENT.md     # Dev setup, testing, deployment
├── API.md             # Backend API reference
├── LESSONS-LEARNED.md # Key insights for future agents
└── archive/           # 93 historical docs with date prefixes
```

## Testing
- All 57 backend tests pass
- No code changes, docs-only reorganization

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 02:43:02 -05:00
Andrew Beveridge ☄️ b16f57c267 Merge pull request #86 from nomadkaraoke/fix/consolidate-instrumental-ui
Consolidate instrumental review UI to single source
2025-12-28 01:44:06 -05:00
Andrew Beveridge ☄️ 22a5b119ec Merge pull request #87 from nomadkaraoke/fix/ci-skip-disk-cleanup
ci: skip disk cleanup when backend deployment not needed
2025-12-28 01:43:15 -05:00
Andrew BeveridgeandClaude Opus 4.5 5600df6783 ci: skip disk cleanup when backend deployment not needed
Move version check before the "Free disk space" step and add a
conditional so it only runs when deployment is actually needed.
This saves ~3+ minutes on pushes to main when the version tag
already exists.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 01:42:11 -05:00
Andrew BeveridgeandClaude Opus 4.5 9bed9839b8 Fix seekTo undefined audio variable and use absolute logo URL
- Add missing 'const audio = document.getElementById' in seekTo()
- Use absolute URL for logo with onerror fallback for CLI mode

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 01:35:32 -05:00
Andrew Beveridge ede3c5eca0 Merge main into fix/consolidate-instrumental-ui 2025-12-28 01:29:30 -05:00
Andrew BeveridgeandClaude Opus 4.5 5ecc4bb1ae Consolidate instrumental review UI to single source
Remove duplicate instrumental UI from frontend/public/ and apply PR #85
fixes to the canonical source in karaoke_gen/instrumental_review/static/.

This ensures both CLI and cloud versions use the same code:
- CLI: Served by FastAPI InstrumentalReviewServer
- Cloud: CI copies from karaoke_gen/ to GitHub Pages

Fixes applied to canonical source:
- Use duration_seconds instead of duration from waveform API
- Add NaN guards to formatTime(), seekTo(), and click/drag handlers
- Replace microphone emoji with Nomad Karaoke logo

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 01:26:38 -05:00
Andrew Beveridge ☄️ 10cb868481 Merge pull request #80 from nomadkaraoke/feature/local-whisper-transcriber
Add local Whisper transcription support
2025-12-28 01:24:18 -05:00
Andrew Beveridge 02a64faecc Updated deps 2025-12-28 01:07:25 -05:00
Andrew BeveridgeandClaude Opus 4.5 ba4ec72040 Merge main into feature/local-whisper-transcriber
Resolved version conflict by keeping 0.76.24 from feature branch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 01:07:06 -05:00
Andrew Beveridge ☄️ ae2846b90b Merge pull request #85 from nomadkaraoke/fix/instrumental-ui-duration-nan
Fix instrumental UI NaN duration causing seek to fail
2025-12-28 01:05:27 -05:00
Andrew BeveridgeandClaude Opus 4.5 8c0eab7d7b Increase logo height to 40px
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 01:04:13 -05:00
Andrew BeveridgeandClaude Opus 4.5 0f51f595a3 Replace microphone emoji with Nomad Karaoke logo in instrumental UI
- Added logo-img CSS class for proper image sizing
- Updated header to use the SVG logo instead of emoji

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 01:02:18 -05:00
Andrew BeveridgeandClaude Opus 4.5 98bb30114e Fix instrumental UI NaN duration causing seek to fail
The waveform API returns `duration_seconds` but the frontend was reading
`waveformData.duration` which is undefined, causing duration to be NaN.

Fixes:
- Read `duration_seconds` from API response instead of `duration`
- Add guard in formatTime() to return '0:00' for NaN/Infinity
- Add guard in seekTo() to prevent setting audio.currentTime to NaN
- Add guards in waveform click handlers to prevent NaN calculations

This fixes the 'NaN:NaN' display and the TypeError when clicking to seek.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 01:01:26 -05:00
Andrew Beveridge ☄️ e88110a25c Merge pull request #84 from nomadkaraoke/fix/instrumental-ui-auth
Fix instrumental UI authentication token not being passed in URL
2025-12-28 00:48:34 -05:00
Andrew Beveridge 12a62db7f8 Remove test files and add to gitignore 2025-12-28 00:48:13 -05:00
Andrew Beveridge ☄️ c4a0cb2a63 Merge pull request #83 from nomadkaraoke/fix/docker-cache-ignore-error
Fix Docker registry cache error on first build
2025-12-28 00:47:45 -05:00
Andrew BeveridgeandClaude Opus 4.5 b0a16a90a4 Fix OutputPaths attribute access (dataclass not dict)
OutputPaths is a dataclass, not a dictionary. Changed .get() calls
to proper attribute access (.video, .ass).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 00:46:54 -05:00
Andrew BeveridgeandClaude Opus 4.5 2f6cd7a745 Fix Docker registry cache error and optimize CI workflow
Changes:
1. Add ignore-error=true to cache-from/cache-to so builds continue
   gracefully when cache doesn't exist (first build)

2. Add aggressive disk space cleanup at start of deploy-backend using
   jlumbroso/free-disk-space action (fixes "No space left on device")

3. Skip tests on main branch - tests only run on PRs now
   - PRs run full test suite with path filtering
   - Main branch only runs build/deploy jobs
   - CI Gate only required for PRs, not main

This significantly speeds up deployments on main since tests
already passed on the PR before merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 00:46:02 -05:00
Andrew BeveridgeandClaude Opus 4.5 1da1a17235 Fix instrumental UI authentication token not being passed in URL
The instrumental review UI was getting 401 errors because the frontend
JobCard component was looking for the token in job.state_data.instrumental_token
when it's actually stored at job.instrumental_token (same pattern as review_token).

Changes:
- Add instrumental_token field to Job interface in api.ts
- Fix JobCard.tsx getInstrumentalUrl() to use job.instrumental_token
- Update test to use correct token location

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 00:40:44 -05:00
Andrew Beveridge daa6807b39 Remove accidentally committed test audio files 2025-12-28 00:36:34 -05:00
Andrew BeveridgeandClaude Opus 4.5 06c8587e99 Fix missing lyrics_results argument in generate_outputs call
The OutputGenerator.generate_outputs() method requires a lyrics_results
argument. Since we're at the video rendering stage after loading
corrections from JSON, we pass an empty dict as the lyrics were already
written during the transcription phase.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 00:35:24 -05:00
Andrew Beveridge 81c7d1cc7f Merge main into feature/local-whisper-transcriber 2025-12-28 00:31:56 -05:00
Andrew BeveridgeandClaude Opus 4.5 b2c8608e0c Use default styles when no output_styles_json provided
When rendering video or CDG without a styles file, the OutputGenerator
now uses sensible default karaoke styles instead of raising an error.
This allows end-to-end karaoke generation to work without requiring a
styles JSON file.

Changes:
- Add _get_default_styles() method with embedded default karaoke styles
- Modify init to use defaults when styles file is missing/None
- Update test to verify default styles are loaded correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 00:29:18 -05:00
Andrew Beveridge ☄️ d6f22f5494 Merge pull request #82 from nomadkaraoke/feature/github-actions-docker-builds
Replace Cloud Build with GitHub Actions for Docker builds
2025-12-28 00:25:11 -05:00
Andrew BeveridgeandClaude Opus 4.5 ea1c659828 Replace Cloud Build with GitHub Actions for Docker builds
Cost optimization: Migrate from expensive Cloud Build ($100+/month) to
free GitHub Actions with registry caching.

Changes:
- Add .dockerignore to reduce build context (~1GB → ~10MB)
- Rewrite deploy-backend job to use docker/build-push-action
- Use Artifact Registry as persistent layer cache (mode=max)
- Keep cloudbuild*.yaml as fallback for manual builds
- Add documentation for new build process

Benefits:
- $0 monthly cost (public repo, free GitHub Actions)
- Faster builds with registry-cached layers
- Simpler deployment pipeline (all in one workflow)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 00:22:39 -05:00
Andrew BeveridgeandClaude Opus 4.5 8bd8f3343f Fix pip extras for local-whisper installation
Poetry extras require the optional dependency to be in the main
[tool.poetry.dependencies] section with `optional = true`, not just
in a separate group. The group syntax is only for Poetry's own
`poetry install --with` command.

Now `pip install karaoke-gen[local-whisper]` correctly installs
whisper-timestamped.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 00:19:19 -05:00
Andrew BeveridgeandClaude Opus 4.5 ce07080fc7 Add disk space cleanup to backend-unit-tests CI job
Prevents transient "No space left on device" failures on GitHub Actions
runners by freeing up disk space before running tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 00:13:13 -05:00
Andrew BeveridgeandClaude Opus 4.5 eceab9f422 Fix LocalWhisper integration in lyrics_processor.py
- Add LocalWhisper check to _check_transcription_providers()
- Update error message to include LocalWhisper as option 3
- Pass runpod_api_key, whisper_runpod_id, and enable_local_whisper
  to TranscriberConfig

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 00:04:31 -05:00
Andrew Beveridge ☄️ e4223634d1 Merge pull request #81 from nomadkaraoke/fix/remove-instrumental-selector-export
Fix stale exports in job/index.ts and add frontend build CI check
2025-12-27 23:52:42 -05:00
Andrew BeveridgeandClaude Opus 4.5 6586ea3584 Update poetry.lock for local-whisper extras
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 23:52:23 -05:00
Andrew BeveridgeandClaude Opus 4.5 b7c1850a3b Fix stale exports in job/index.ts and add frontend build CI check
The previous PR removed InstrumentalSelector.tsx but didn't update the
barrel export file, breaking the production build.

Changes:
- Remove stale InstrumentalSelector and StatusBadge exports from index.ts
- Add frontend-build job to CI workflow to catch build failures on PRs
- Add frontend-build to ci-gate and deploy-frontend dependencies

This ensures future PRs will fail CI if they break the frontend build.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 23:51:42 -05:00
Andrew BeveridgeandClaude Opus 4.5 e4dbd7393e Add local Whisper transcription support
Implements LocalWhisperTranscriber for running Whisper inference locally
using the whisper-timestamped library, addressing issue #68.

Changes:
- Add LocalWhisperTranscriber class with lazy model loading
- Support auto-detection of device (CPU/CUDA/MPS) with fallback
- Add configuration via environment variables (WHISPER_MODEL_SIZE,
  WHISPER_DEVICE, WHISPER_CACHE_DIR, WHISPER_LANGUAGE)
- Integrate into controller with priority 3 (fallback after cloud services)
- Add whisper-timestamped as optional dependency [local-whisper]
- Add comprehensive unit tests
- Update README with setup instructions and troubleshooting guide

Installation:
  pip install karaoke-gen[local-whisper]

For CPU-only systems:
  pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
  pip install karaoke-gen[local-whisper]

Fixes #68

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 23:40:34 -05:00
Andrew Beveridge ☄️ 7736a4a8f7 Merge pull request #79 from nomadkaraoke/feature/instrumental-review-ui
Replace primitive instrumental selector with full-featured review UI
2025-12-27 23:36:45 -05:00
Andrew BeveridgeandClaude Opus 4.5 156b83ace1 Replace primitive instrumental selector with full-featured review UI
The web frontend now opens the same rich instrumental review UI that the
CLI uses. This provides waveform visualization, audio playback for
different stems, mute region selection, and custom instrumental creation.

Changes:
- Add instrumental review static HTML to frontend/public/instrumental/
- Update JobCard to open /instrumental/ page instead of showing dialog
- Remove InstrumentalSelector.tsx primitive dialog component
- Remove orphaned instrumental-review components (never used)
- Update tests to reflect link-based navigation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 23:34:20 -05:00
Andrew Beveridge ☄️ 8978eb6d96 Merge pull request #77 from nomadkaraoke/fix/flacfetch-youtube-url
Update flacfetch to 0.15.1 (fixes YouTube URL download)
2025-12-27 23:10:12 -05:00
Andrew Beveridge ☄️ c4143947ca Merge pull request #78 from nomadkaraoke/feature/mandatory-worktrees
Make git worktrees mandatory for all changes
2025-12-27 23:09:07 -05:00
Andrew BeveridgeandClaude Opus 4.5 8a28c95786 Simplify flacfetch workflow: no PR required
flacfetch changes can be committed directly to main once tests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 23:07:36 -05:00
Andrew BeveridgeandClaude Opus 4.5 48acb6c4e1 Add flacfetch as a related project with workflow docs
Documents that flacfetch is a dependency we own, located at
/Users/andrew/Projects/flacfetch, and provides the workflow for
making changes to it and updating karaoke-gen's dependency.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 23:07:05 -05:00
Andrew BeveridgeandClaude Opus 4.5 da60495144 Clarify version bump rule applies only to code changes
Documentation-only changes (CLAUDE.md, README.md, etc.) don't require
version bumps since they don't affect deployed code.

Also reverts the unnecessary version bump from the previous commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 23:04:39 -05:00
Andrew BeveridgeandClaude Opus 4.5 e282f40758 Make git worktrees mandatory for all changes
Since all changes must go through a PR, worktrees are now required for
all changes (not just non-trivial ones). This keeps the main directory
clean on main and enables parallel work.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 23:03:54 -05:00
Andrew BeveridgeandClaude Opus 4.5 aaeda8679a Update flacfetch to 0.15.1 (fixes YouTube URL download)
Fixes the "Release.__init__() missing 1 required positional argument: 'quality'"
error that occurred when downloading from YouTube URLs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 22:59:31 -05:00
Andrew Beveridge ☄️ 6da6656c15 Merge pull request #76 from nomadkaraoke/fix/remove-deployment-tracking
Remove deployment tracking to fix PR banner
2025-12-27 22:50:44 -05:00
Andrew BeveridgeandClaude Opus 4.5 d81591ca99 Remove deployment tracking to fix PR banner
Remove the `environment` block from the deploy-frontend job to prevent
GitHub from creating deployment entries. This fixes the "This branch
has not been deployed" banner that appears on all PRs.

The deployment to GitHub Pages still works via actions/deploy-pages@v4,
but without tracked environment deployments.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 22:49:44 -05:00
Andrew Beveridge ☄️ 0e66ac84ec Merge pull request #74 from nomadkaraoke/fix/youtube-url-support
Fix YouTube URL support in karaoke-gen CLI
2025-12-27 22:48:12 -05:00
Andrew BeveridgeandClaude Opus 4.5 d8e3542850 Address CodeRabbit lint feedback
- Remove f-string prefix from strings without placeholders (F541)
- Add 'from e' to ImportError handler to preserve stack trace (B904)
- Use raw string for regex pattern in pytest match (RUF043)
- Remove unused result variables (F841)
- Remove redundant KaraokePrep import in fixture (F811)
- Rename unused mock parameters with underscore prefix (ARG002)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 22:36:22 -05:00
Andrew Beveridge ☄️ 8b66e97459 Merge pull request #75 from nomadkaraoke/fix/progress-display-during-parallel-processing
Fix progress display to show step 4 when workers are active during download
2025-12-27 22:34:53 -05:00
Andrew BeveridgeandClaude Opus 4.5 5c71d58a8d Fix progress display to show step 4 when workers are active during download
When the backend status is "downloading" but parallel workers (audio/lyrics)
are actively running, the frontend now correctly displays step 4 progress
instead of appearing stuck at step 3 "Downloading".

This improves UX by showing detailed worker progress (e.g., "Audio 1/2 + Transcribing")
when both workers are active, making the job appear responsive.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 22:26:26 -05:00
Andrew Beveridge 8dc0a96734 Added CI gate check 2025-12-27 22:24:52 -05:00
Andrew BeveridgeandClaude Opus 4.5 78d7eedeee Add tests for YouTube URL support
Add comprehensive test coverage for the new download_from_url()
functionality:

- FlacFetchAudioFetcher.download_from_url() tests:
  - YouTube URL download success
  - youtu.be short URL support
  - Embed URL support
  - Filename generation from artist/title
  - Filename fallback to video ID
  - Error handling for download failures
  - Output directory creation

- RemoteFlacFetchAudioFetcher.download_from_url() tests:
  - Verifies local flacfetch is used (not remote API)
  - Video ID extraction
  - Error handling

- KaraokePrep URL handling tests:
  - process() with YouTube URL
  - process() with URL but no artist/title
  - process() with youtu.be short URL
  - _is_url() helper method edge cases

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 22:18:17 -05:00
Andrew Beveridge 6fa81c1b69 Updated dependencies 2025-12-27 22:10:59 -05:00
Andrew BeveridgeandClaude Opus 4.5 706e6f7527 Fix YouTube URL support in karaoke-gen CLI
Restore YouTube URL input functionality that was broken during
flacfetch integration. Now users can pass a YouTube URL as the
first argument to karaoke-gen again.

Changes:
- Add download_from_url() abstract method to AudioFetcher base class
- Implement download_from_url() in FlacFetchAudioFetcher using
  flacfetch's download_by_id() method for direct YouTube downloads
- Implement download_from_url() in RemoteFlacFetchAudioFetcher
  using local flacfetch for YouTube (no remote API needed)
- Update KaraokePrep.process() to detect URLs and route them to
  the new download_from_url() method instead of flacfetch search
- Add _is_url() helper method for URL detection
- Add _use_url_download flag to differentiate URL downloads from
  artist+title searches

Fixes #67

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 22:08:19 -05:00
Andrew Beveridge ☄️ 8bbb1351f8 Merge pull request #73 from nomadkaraoke/feature/ci-improvements
ci: Add path filtering, caching, and frontend tests to GHA workflow
2025-12-27 21:50:35 -05:00
Andrew BeveridgeandClaude Opus 4.5 18e97dc0a0 ci: Use poetry.lock exclusively for deps filter
Remove pyproject.toml from the deps filter since poetry.lock is
already used as the source of truth for dependency changes elsewhere
(e.g., base image rebuild triggers). Version bumps alone shouldn't
trigger full test suites.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 21:48:50 -05:00
Andrew BeveridgeandClaude Opus 4.5 642c80164c ci: Add path filtering, caching, and frontend tests to GHA workflow
- Add `changes` job using dorny/paths-filter@v3 to detect what changed
- Add frontend-unit-tests job (Jest with coverage)
- Make all test jobs conditional based on path changes:
  - Frontend tests: run only if frontend/** changed
  - Backend tests: run if backend/** or package or deps changed
  - Package tests: run if karaoke_gen/** or tests/** or deps changed
- Add Poetry dependency caching using actions/cache@v4:
  - Cache .venv directory keyed on poetry.lock hash
  - Shared cache between package, backend, and lint jobs
- Add npm caching using actions/setup-node's built-in cache feature
- Add Yarn caching for lyrics frontend in deploy-frontend job
- Add spaCy model caching
- Jobs always run on main branch to ensure full validation before deploy
- Add summary table to changes job showing what areas changed

This significantly reduces CI time for focused PRs (e.g., frontend-only
changes skip all backend tests) while ensuring comprehensive testing
before merging to main.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 21:40:56 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 46eca212fe Fix: Prepare theme styles when creating job via audio search (#72)
* Fix: Prepare theme styles when creating job via audio search

When creating a job via the audio search endpoint with a theme_id,
the theme styles were not being prepared. This resulted in jobs that
had theme_id set but no style_params_gcs_path or style_assets, causing
preview videos to have black backgrounds instead of themed backgrounds.

The fix adds a call to _prepare_theme_for_job() in audio_search.py
when theme_id is provided and no custom style files are being uploaded.
This matches the behavior of file_upload.py.

Also adds regression tests to catch this issue:
- test_audio_search_with_theme_calls_prepare_job_style
- test_audio_search_code_handles_theme_preparation

Fixes issue observed in production job ffb0b8fa.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Remove unused variable from test (CodeRabbit feedback)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 21:38:05 -05:00
Andrew BeveridgeandClaude Opus 4.5 8d8e64ccc0 docs: Add job debugging guide with tested gcloud commands
Add comprehensive documentation for debugging jobs in production:
- Firestore job data access via Python script
- Cloud Logging queries for job-specific logs
- GCS file listing and inspection
- Cloud Run service health checks
- Cloud Tasks queue monitoring
- Common debugging scenarios

Also adds scripts/get_job.py helper for quick job lookups.

All commands tested against real job ID ffb0b8fa.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 21:26:10 -05:00
Andrew Beveridge ☄️ 50f895e88a Merge pull request #71 from nomadkaraoke/feature/improve-job-status-display
feat: Improve job status display with step-based progress
2025-12-27 21:24:44 -05:00
Andrew BeveridgeandClaude Opus 4.5 7218b4d176 feat: Add step-based job status display with progress indicator
- Create job-status.ts utility that maps 28+ backend statuses to a 10-step user-friendly system
- Display status as "[X/10] Status..." format for better progress visibility
- Add visual progress bar that matches status color
- Show "Action needed" badge for blocking states (review, instrumental selection)
- Handle parallel processing (audio + lyrics) with combined status display
- Add comprehensive unit tests (40+ test cases) for all status mappings
- Update JobCard to use new StatusIndicator and ProgressBar components

This improves user experience by providing clearer feedback about job progress
instead of showing static labels like "Downloading..." for extended periods.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 21:17:44 -05:00
Andrew BeveridgeandClaude Opus 4.5 c052fec326 Fix E2E test API request timeouts
Increase timeout for Playwright API requests from default 60s to 120s
to handle slower API responses during job polling.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 20:54:56 -05:00
Andrew BeveridgeandClaude Opus 4.5 70eca20c7a Consolidate production E2E tests into single comprehensive flow
Merged production-full-journey.spec.ts and production-theme-system.spec.ts
into production-e2e.spec.ts. The combined test:

- Uses cached song (piri/dog) for fast flacfetch responses
- Tests theme selection as part of the core journey
- Validates theme_id is correctly set on jobs
- Follows the complete flow: search -> audio -> review -> instrumental -> complete

This eliminates redundancy and ensures theme testing is integrated with
the main E2E flow rather than being a separate concern.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 20:35:03 -05:00
Andrew BeveridgeandClaude Opus 4.5 0e6a0b43ce Update production E2E tests to use real cached song
Use "piri" / "dog" for all production E2E tests instead of fake
test data. This leverages flacfetch's new caching system for fast
search responses without repeated API calls to trackers.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 20:31:50 -05:00
Andrew BeveridgeandClaude Opus 4.5 3cbfbd14d1 Fix: Pass theme configuration from JobCreate to Job
The create_job() function was not passing theme_id, color_overrides,
style_params_gcs_path, or style_assets from JobCreate to the Job
constructor. This caused jobs created via audio search to have null
theme_id even when a theme was specified.

Also enables video recording for all Playwright tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 20:17:42 -05:00
Andrew Beveridge ☄️ 2f4265e9d0 Fix: Add theme support to audio search endpoint (#70)
Fixes bug where themes selected in the frontend were not being passed through when creating jobs via the Search tab. Audio search jobs now correctly receive theme_id and color_overrides, and CDG/TXT outputs are automatically enabled when a theme is selected.

- Add theme_id and color_overrides fields to AudioSearchRequest
- Add _resolve_cdg_txt_defaults() helper for smart CDG/TXT defaults
- Add comprehensive backend tests for theme support
- Add E2E Playwright tests for production theme system with full UI flow
2025-12-27 19:15:28 -05:00
Andrew Beveridge ☄️ 4a74a2cc3a Add theme system for pre-made video styles (#69)
Adds a comprehensive theme system for karaoke video styles:

- Theme selection and customizable color overrides for karaoke jobs
- Public Themes API for listing, previewing, and fetching theme details
- Theme selector and color-overrides UI in job submission flows
- CLI option to specify a theme when preparing/submitting jobs
- Thread-safe singleton initialization for ThemeService

🤖 Generated with Claude Code
2025-12-27 17:55:47 -05:00
160 changed files with 9190 additions and 2457 deletions

No files matched your search

+74
View File
@@ -0,0 +1,74 @@
---
description: Periodic documentation maintenance and organization check
allowed-tools: Read, Glob, Grep, Edit, Write, Bash
---
# Documentation Maintenance Check
You are performing periodic maintenance on the project documentation.
## Checks to Perform
### 1. Structure Compliance
Verify docs/ follows the correct structure:
```
docs/
├── README.md # Current status + navigation
├── ARCHITECTURE.md # System design
├── DEVELOPMENT.md # Dev setup, testing, deployment
├── API.md # Backend API reference
├── LESSONS-LEARNED.md # Accumulated wisdom
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
```
Report any files that don't belong or are misplaced.
### 2. Freshness Check
- Read `docs/README.md` - is the status section current?
- Check recent git commits - do they suggest docs should be updated?
- Look for stale information that contradicts current code
### 3. Archive Organization
- Verify all archive files have YYYY-MM-DD prefix
- Check for any docs that should be archived (completed work, old plans)
- Remove truly obsolete content (empty files, duplicate info)
### 4. Cross-Reference Check
- Verify CLAUDE.md points to current doc locations
- Check that docs reference each other correctly
- Ensure no broken internal links
### 5. Content Quality
- Flag any docs over 500 lines (may need splitting)
- Identify duplicate information across docs
- Note sections that seem outdated
## Actions to Take
1. **Fix structural issues** - Move/rename misplaced files
2. **Update stale content** - Refresh outdated information
3. **Archive completed work** - Move old docs to archive/
4. **Report findings** - Summarize what was found and fixed
## Output Format
Provide a maintenance report:
```
## Documentation Maintenance Report - YYYY-MM-DD
### Issues Found
- [list of issues]
### Actions Taken
- [list of fixes made]
### Recommendations
- [suggestions for future attention]
### Status
- [ ] Structure: OK/Fixed
- [ ] Freshness: OK/Updated
- [ ] Archive: OK/Organized
- [ ] Cross-refs: OK/Fixed
- [ ] Quality: OK/Flagged
```
+43
View File
@@ -0,0 +1,43 @@
---
description: Review current session for documentation updates before merging PR
allowed-tools: Read, Glob, Grep, Edit, Write
---
# Pre-Merge Documentation Review
You are reviewing this chat session to identify documentation updates needed before merging the current PR.
## Your Task
1. **Analyze the current session** - What work was done? What was learned? What changed?
2. **Check if docs need updates:**
- Does `docs/README.md` status section need updating?
- Should anything be added to `docs/LESSONS-LEARNED.md`?
- Does `docs/ARCHITECTURE.md` need changes (if architecture changed)?
- Does `docs/DEVELOPMENT.md` need changes (if dev workflow changed)?
- Does `docs/API.md` need changes (if API changed)?
3. **For significant completed work:**
- Create `docs/archive/YYYY-MM-DD-topic.md` summarizing the work
- Use today's date and a descriptive topic name
4. **Update docs/README.md** with:
- Current status if it changed
- Any new known issues discovered
- Link to new archive doc if created
## Guidelines
- Keep updates concise and factual
- Focus on information useful for future AI agents
- Don't duplicate information that's in the code
- Archive docs should capture "what was done and why" not implementation details
## Output
After reviewing, either:
- Make the necessary documentation updates
- Report "No documentation updates needed" with brief explanation why
Remember: Good documentation helps the next agent continue where you left off.
+83
View File
@@ -0,0 +1,83 @@
# .dockerignore - Reduce Docker build context size
# This is critical for fast builds as it reduces upload time and cache invalidation
# Git
.git/
.gitignore
.gitmodules
# Node.js / Frontend (not needed for backend image)
node_modules/
frontend/
frontend-react/
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# Python virtual environments (use Docker's Python instead)
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
.venv/
venv/
backend/venv/
ENV/
env/
*.egg-info/
.eggs/
dist/
build/
*.egg
# IDE
.idea/
.vscode/
*.swp
*.swo
*~
.DS_Store
# Testing (not needed in production image)
.pytest_cache/
.coverage
htmlcov/
.tox/
.nox/
coverage.xml
*.cover
tests/
backend/tests/
# Documentation (not needed in Docker image)
docs/
*.md
!README.md
# Infrastructure (not needed for Docker build)
infrastructure/
# CI/CD configs
.github/
# Cloud Build configs (keep for reference but not in image)
cloudbuild*.yaml
# Local development files
lyrics_transcriber_local/
*.local
.env
.env.*
*.bak
*.tmp
Thumbs.db
# Large test/sample files
tests/data/
*.mp3
*.wav
*.mp4
*.mkv
*.flac
+355 -141
View File
@@ -17,12 +17,120 @@ concurrency:
jobs:
# ============================================
# PACKAGE TESTS (karaoke-gen Python CLI)
# PATH FILTERING - Determine what changed
# Only needed for PRs (main branch skips tests)
# ============================================
changes:
name: "Detect Changes"
runs-on: ubuntu-latest
if: github.ref != 'refs/heads/main'
outputs:
frontend: ${{ steps.filter.outputs.frontend }}
backend: ${{ steps.filter.outputs.backend }}
package: ${{ steps.filter.outputs.package }}
infra: ${{ steps.filter.outputs.infra }}
deps: ${{ steps.filter.outputs.deps }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
frontend:
- 'frontend/**'
backend:
- 'backend/**'
package:
- 'karaoke_gen/**'
- 'lyrics_transcriber_temp/**'
- 'tests/**'
infra:
- 'infrastructure/**'
deps:
- 'poetry.lock'
- name: Summary
run: |
echo "### Change Detection Results" >> $GITHUB_STEP_SUMMARY
echo "| Area | Changed |" >> $GITHUB_STEP_SUMMARY
echo "|------|---------|" >> $GITHUB_STEP_SUMMARY
echo "| Frontend | ${{ steps.filter.outputs.frontend }} |" >> $GITHUB_STEP_SUMMARY
echo "| Backend | ${{ steps.filter.outputs.backend }} |" >> $GITHUB_STEP_SUMMARY
echo "| Package | ${{ steps.filter.outputs.package }} |" >> $GITHUB_STEP_SUMMARY
echo "| Infrastructure | ${{ steps.filter.outputs.infra }} |" >> $GITHUB_STEP_SUMMARY
echo "| Dependencies | ${{ steps.filter.outputs.deps }} |" >> $GITHUB_STEP_SUMMARY
# ============================================
# FRONTEND TESTS (PRs only - main skips tests)
# ============================================
frontend-unit-tests:
name: "Frontend - Unit Tests"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
steps:
- uses: actions/checkout@v4
- name: Setup Node.js with caching
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Run unit tests with coverage
working-directory: frontend
run: npm run test:ci
- name: Upload coverage report
if: always()
uses: actions/upload-artifact@v4
with:
name: frontend-coverage
path: frontend/coverage/
retention-days: 7
frontend-build:
name: "Frontend - Build"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
steps:
- uses: actions/checkout@v4
- name: Setup Node.js with caching
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Build frontend
working-directory: frontend
env:
NEXT_PUBLIC_API_URL: https://api.nomadkaraoke.com
run: npm run build
# ============================================
# PACKAGE TESTS (PRs only - main skips tests)
# ============================================
package-unit-tests:
name: "Package - Unit Tests (70% coverage)"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- uses: actions/checkout@v4
with:
@@ -39,6 +147,14 @@ jobs:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-package-${{ runner.os }}-py3.13-
- name: Install dependencies
run: poetry install --with dev
@@ -47,6 +163,12 @@ jobs:
sudo apt-get update
sudo apt-get install -y ffmpeg
- name: Cache spaCy model
uses: actions/cache@v4
with:
path: ~/.cache/spacy
key: spacy-en_core_web_sm-${{ runner.os }}
- name: Install spaCy language models
run: poetry run python -m spacy download en_core_web_sm
@@ -65,6 +187,8 @@ jobs:
package-integration-tests:
name: "Package - Integration Tests"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- uses: actions/checkout@v4
with:
@@ -81,6 +205,14 @@ jobs:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-package-${{ runner.os }}-py3.13-
- name: Install dependencies
run: poetry install --with dev
@@ -89,16 +221,27 @@ jobs:
sudo apt-get update
sudo apt-get install -y ffmpeg
- name: Cache spaCy model
uses: actions/cache@v4
with:
path: ~/.cache/spacy
key: spacy-en_core_web_sm-${{ runner.os }}
- name: Install spaCy language models
run: poetry run python -m spacy download en_core_web_sm
- name: Run integration tests
run: poetry run pytest tests/integration/ -v
package-build:
name: "Package - Build & Install"
runs-on: ubuntu-latest
needs: [package-unit-tests, package-integration-tests]
package-build-test:
name: "Package - Build & Install (Test)"
runs-on: [self-hosted, linux, gcp]
needs: [changes, package-unit-tests, package-integration-tests]
if: >-
github.ref != 'refs/heads/main' && always() &&
(needs.package-unit-tests.result == 'success' || needs.package-unit-tests.result == 'skipped') &&
(needs.package-integration-tests.result == 'success' || needs.package-integration-tests.result == 'skipped') &&
(needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- uses: actions/checkout@v4
with:
@@ -118,36 +261,20 @@ jobs:
- name: Build package
run: poetry build
- name: Free disk space
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
haskell: true
dotnet: false
large-packages: false
docker-images: false
swap-storage: false
- name: Test package installation
run: |
pip install dist/*.whl
python -m karaoke_gen.utils.gen_cli --help
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: package-dist
path: dist/
retention-days: 1
# ============================================
# BACKEND TESTS (Cloud Run API service)
# BACKEND TESTS (PRs only - main skips tests)
# ============================================
backend-unit-tests:
name: "Backend - Unit Tests"
runs-on: ubuntu-latest
runs-on: [self-hosted, linux, gcp]
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -182,15 +309,10 @@ jobs:
backend-emulator-tests:
name: "Backend - Emulator Integration Tests"
runs-on: ubuntu-latest
runs-on: [self-hosted, linux, gcp]
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
steps:
- name: Free disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /usr/local/share/boost
df -h
- name: Checkout code
uses: actions/checkout@v4
with:
@@ -213,14 +335,6 @@ jobs:
distribution: "temurin"
java-version: "21"
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Install Firestore emulator
run: gcloud components install beta cloud-firestore-emulator --quiet
- name: Install dependencies
run: poetry install
@@ -243,6 +357,8 @@ jobs:
backend-lint:
name: "Backend - Code Quality"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true')
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -260,6 +376,14 @@ jobs:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-lint-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-lint-${{ runner.os }}-py3.13-
- name: Install dependencies
run: poetry install
@@ -269,9 +393,78 @@ jobs:
echo "✅ Basic syntax validation passed"
# ============================================
# DEPLOYMENTS (main branch only)
# CI GATE - Required status check for PRs
# Only runs on PRs, not on main
# ============================================
ci-gate:
name: "CI Gate"
runs-on: ubuntu-latest
needs: [frontend-unit-tests, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
if: github.ref != 'refs/heads/main' && always()
steps:
- name: Check results
run: |
results=(
"${{ needs.frontend-unit-tests.result }}"
"${{ needs.frontend-build.result }}"
"${{ needs.package-unit-tests.result }}"
"${{ needs.package-integration-tests.result }}"
"${{ needs.backend-unit-tests.result }}"
"${{ needs.backend-emulator-tests.result }}"
"${{ needs.backend-lint.result }}"
)
for result in "${results[@]}"; do
if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
echo "❌ CI failed"
exit 1
fi
done
echo "✅ CI passed"
# ============================================
# DEPLOYMENTS (main branch only)
# Tests already passed on PR, so we skip them here
# ============================================
package-build:
name: "Package - Build"
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install Poetry
uses: snok/install-poetry@v1
with:
virtualenvs-create: true
virtualenvs-in-project: true
- name: Cache Poetry dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-package-${{ runner.os }}-py3.13-${{ hashFiles('poetry.lock') }}
restore-keys: |
venv-package-${{ runner.os }}-py3.13-
- name: Build package
run: poetry build
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: package-dist
path: dist/
retention-days: 1
package-publish:
name: "Deploy - Publish to PyPI"
runs-on: ubuntu-latest
@@ -363,17 +556,16 @@ jobs:
name: "Deploy - Frontend (GitHub Pages)"
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Node.js
- name: Setup Node.js with caching
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Enable Yarn (for lyrics frontend)
run: corepack enable
@@ -390,6 +582,14 @@ jobs:
run: npm run build
# ===== Build Lyrics Review Frontend (Vite) =====
- name: Cache Yarn dependencies (lyrics frontend)
uses: actions/cache@v4
with:
path: lyrics_transcriber_temp/lyrics_transcriber/frontend/node_modules
key: yarn-lyrics-${{ runner.os }}-${{ hashFiles('lyrics_transcriber_temp/lyrics_transcriber/frontend/yarn.lock') }}
restore-keys: |
yarn-lyrics-${{ runner.os }}-
- name: Install lyrics frontend dependencies
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
run: yarn install --frozen-lockfile
@@ -426,20 +626,28 @@ jobs:
deploy-backend:
name: "Deploy - Backend (Cloud Run)"
runs-on: ubuntu-latest
needs: [backend-unit-tests, backend-emulator-tests, backend-lint]
runs-on: [self-hosted, linux, gcp]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
REPOSITORY: karaoke-repo
BASE_IMAGE_NAME: karaoke-backend-base
APP_IMAGE_NAME: karaoke-backend
steps:
# ============================================
# Check if deployment is needed FIRST
# ============================================
- name: Checkout code
uses: actions/checkout@v4
with:
submodules: false
fetch-depth: 0 # Need full history for tag check
- name: Set up Python (for version extraction)
- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: "3.13"
@@ -453,7 +661,7 @@ jobs:
VERSION=$(poetry version --short)
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "📋 Current version: $VERSION"
if git ls-remote --tags origin | grep -q "refs/tags/v$VERSION$"; then
echo "needs_deploy=false" >> $GITHUB_OUTPUT
echo "⏭️ Tag v$VERSION already exists - skipping backend deployment"
@@ -463,6 +671,8 @@ jobs:
echo "✨ Tag v$VERSION does not exist - proceeding with deployment"
fi
# No disk cleanup needed - self-hosted runner has 200GB SSD
- name: Authenticate to Google Cloud (Workload Identity)
if: steps.version_check.outputs.needs_deploy == 'true'
id: auth
@@ -477,21 +687,37 @@ jobs:
with:
version: "latest"
- name: Check if base image needs rebuild
- name: Configure Docker for Artifact Registry
if: steps.version_check.outputs.needs_deploy == 'true'
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
# ============================================
# Docker Build with BuildKit + Registry Cache
# This replaces expensive Cloud Build with free GitHub Actions
# Registry cache persists between builds for fast rebuilds
# ============================================
- name: Set up Docker Buildx
if: steps.version_check.outputs.needs_deploy == 'true'
uses: docker/setup-buildx-action@v3
- name: Calculate base image hash
if: steps.version_check.outputs.needs_deploy == 'true'
id: base-hash
run: |
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
echo "hash=$CURRENT_HASH" >> $GITHUB_OUTPUT
echo "📋 Base content hash: $CURRENT_HASH"
- name: Check if base image exists and is current
if: steps.version_check.outputs.needs_deploy == 'true'
id: check-base
run: |
echo "🔍 Checking if base image needs rebuild..."
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
echo "Current combined hash (poetry.lock + Dockerfile.base): $CURRENT_HASH"
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
BASE_IMAGE="us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest"
# Try to get the existing base image's hash label
EXISTING_HASH=""
if docker pull $BASE_IMAGE 2>/dev/null; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
@@ -500,10 +726,10 @@ jobs:
fi
echo "Existing base image hash: $EXISTING_HASH"
else
echo "⚠️ No existing base image found"
echo "⚠️ No existing base image found - will build"
fi
if [ "$CURRENT_HASH" != "$EXISTING_HASH" ]; then
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
else
@@ -511,88 +737,73 @@ jobs:
echo "needs_rebuild=false" >> $GITHUB_OUTPUT
fi
- name: Rebuild base image (if needed)
- name: Build and push base image (if needed)
if: steps.version_check.outputs.needs_deploy == 'true' && steps.check-base.outputs.needs_rebuild == 'true'
run: |
echo "🏗️ Rebuilding base image with updated dependencies..."
uses: docker/build-push-action@v6
with:
context: .
file: backend/Dockerfile.base
push: true
tags: |
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:${{ github.sha }}
# Registry cache for ALL layers (mode=max) - persists between builds
# ignore-error=true allows build to continue if cache doesn't exist yet
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,ignore-error=true
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:cache,mode=max,ignore-error=true
build-args: |
BUILD_DATE=${{ github.run_id }}
BUILD_ID=${{ github.sha }}
BASE_CONTENT_HASH=${{ steps.base-hash.outputs.hash }}
# Provenance disabled for smaller image and faster push
provenance: false
BASE_BUILD_ID=$(gcloud builds submit \
--config=cloudbuild-base.yaml \
--project=nomadkaraoke \
--format='value(id)' \
--async \
.)
- name: Build and push app image
if: steps.version_check.outputs.needs_deploy == 'true'
uses: docker/build-push-action@v6
with:
context: .
file: backend/Dockerfile
push: true
tags: |
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:latest
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}
${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:v${{ steps.version_check.outputs.version }}
# Registry cache for app layers
# ignore-error=true allows build to continue if cache doesn't exist yet
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,ignore-error=true
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:cache,mode=max,ignore-error=true
provenance: false
echo "Base Build ID: $BASE_BUILD_ID"
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BASE_BUILD_ID?project=nomadkaraoke"
# ============================================
# Deploy to Cloud Run
# ============================================
echo "Waiting for base image build to complete..."
while true; do
BUILD_STATUS=$(gcloud builds describe $BASE_BUILD_ID --project=nomadkaraoke --format='value(status)')
echo "Base build status: $BUILD_STATUS"
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
echo "✅ Base image build completed successfully"
break
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
echo "❌ Base image build failed with status: $BUILD_STATUS"
echo ""
echo "==================== BUILD LOGS ===================="
gcloud logging read "resource.type=build AND resource.labels.build_id=$BASE_BUILD_ID" \
--project=nomadkaraoke \
--format="value(textPayload)" \
--limit=200 \
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
echo "===================================================="
gcloud builds log $BASE_BUILD_ID --project=nomadkaraoke 2>&1 || true
exit 1
fi
sleep 30
done
- name: Trigger Cloud Build
- name: Deploy to Cloud Run
if: steps.version_check.outputs.needs_deploy == 'true'
run: |
echo "🚀 Triggering Cloud Build for deployment..."
echo "Commit: ${{ github.sha }}"
echo "Version: ${{ steps.version_check.outputs.version }}"
VERSION="${{ steps.version_check.outputs.version }}"
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
BUILD_ID=$(gcloud builds submit \
--config=cloudbuild.yaml \
--project=nomadkaraoke \
--format='value(id)' \
--async \
.)
echo "🚀 Deploying to Cloud Run..."
echo "Image: $IMAGE"
echo "Version: $VERSION"
echo "Build ID: $BUILD_ID"
echo "Build URL: https://console.cloud.google.com/cloud-build/builds/$BUILD_ID?project=nomadkaraoke"
echo "Waiting for build to complete..."
while true; do
BUILD_STATUS=$(gcloud builds describe $BUILD_ID --project=nomadkaraoke --format='value(status)')
echo "Build status: $BUILD_STATUS"
if [ "$BUILD_STATUS" = "SUCCESS" ]; then
echo "✅ Build completed successfully"
break
elif [ "$BUILD_STATUS" = "FAILURE" ] || [ "$BUILD_STATUS" = "CANCELLED" ] || [ "$BUILD_STATUS" = "TIMEOUT" ]; then
echo "❌ Build failed with status: $BUILD_STATUS"
echo ""
echo "==================== BUILD LOGS ===================="
gcloud logging read "resource.type=build AND resource.labels.build_id=$BUILD_ID" \
--project=nomadkaraoke \
--format="value(textPayload)" \
--limit=200 \
--order=asc || echo "Failed to retrieve logs from Cloud Logging"
echo "===================================================="
echo ""
gcloud builds log $BUILD_ID --project=nomadkaraoke 2>&1 || true
exit 1
fi
sleep 30
done
gcloud run deploy karaoke-backend \
--image $IMAGE \
--region us-central1 \
--platform managed \
--allow-unauthenticated \
--memory 16Gi \
--cpu 8 \
--cpu-boost \
--execution-environment gen2 \
--timeout 1800 \
--concurrency 1 \
--max-instances 50 \
--min-instances 0 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
- name: Verify deployment and version
if: steps.version_check.outputs.needs_deploy == 'true'
@@ -604,22 +815,22 @@ jobs:
echo "Service URL: $SERVICE_URL"
echo "Expected version: $EXPECTED_VERSION"
# Wait for new revision to be ready
echo "Waiting for new revision to be ready..."
sleep 15
# Basic health check
curl -f "$SERVICE_URL/api/health" || exit 1
echo "✅ Service is healthy"
# Version verification with retries (Cloud Run may take time to route to new revision)
MAX_RETRIES=6
RETRY_DELAY=10
for i in $(seq 1 $MAX_RETRIES); do
echo "Checking deployed version (attempt $i/$MAX_RETRIES)..."
DEPLOYED_VERSION=$(curl -sf "$SERVICE_URL/api/health/detailed" | jq -r '.version // empty')
if [ "$DEPLOYED_VERSION" = "$EXPECTED_VERSION" ]; then
echo "✅ Version verified: $DEPLOYED_VERSION"
break
@@ -643,7 +854,10 @@ jobs:
echo "Commit: ${{ github.sha }}"
echo "Branch: ${{ github.ref_name }}"
echo "Run: ${{ github.run_number }}"
echo ""
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
- name: Skipped deployment summary
if: steps.version_check.outputs.needs_deploy == 'false'
run: |
+6
View File
@@ -12,6 +12,9 @@ tests/outputs
/input
/output/
# Cloud Infra
.emulator-logs
# Next.js / React Frontend (./frontend)
node_modules/
frontend/node_modules/
@@ -224,3 +227,6 @@ cython_debug/
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
flacfetch
# Test output directories
Rick Astley*/
+73 -107
View File
@@ -1,131 +1,97 @@
# Project Guidelines for AI Assistants
# Karaoke-Gen AI Assistant Guidelines
This document contains guidelines and rules for AI assistants working on this codebase.
## Project Overview
## Core Principles
**Karaoke video generation platform** - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
**Always be testing.** Proactively evaluate, read, run, fix, and improve tests whenever you touch any part of the system—frontend, backend, infrastructure. Every code change should be thoroughly tested. Leave everything better tested than you found it.
- **Production**: <https://gen.nomadkaraoke.com> (frontend), <https://api.nomadkaraoke.com> (backend)
- **CLI**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
- **Repo**: <https://github.com/nomadkaraoke/karaoke-gen>
**Always be documenting.** Continuously read, write, improve, update, and organize markdown documentation. Reference existing docs and keep them accurate as the codebase evolves.
## Quick Reference
**Build for maintainability.** Keep code SOLID, modular, and clean. Favor simplicity and clarity. Future developers (and AI agents) will thank you.
| What | Where |
|------|-------|
| Current status | `docs/README.md` |
| Architecture | `docs/ARCHITECTURE.md` |
| Dev setup & testing | `docs/DEVELOPMENT.md` |
| API reference | `docs/API.md` |
| Past learnings | `docs/LESSONS-LEARNED.md` |
## Branching Workflow
## Tech Stack
**Never commit directly to `main`.** Always work on a feature branch.
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
- **Frontend**: Next.js on Cloudflare Pages
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
Create descriptive branch names like `feature/add-discord-notifications`, `fix/audio-sync-issue`, or `refactor/cleanup-api-routes`.
## Essential Rules
### Using Git Worktrees
**For non-trivial changes** (new features, multi-file edits, refactors):
- Create a git worktree in the parent folder before making any file changes:
### Git Workflow
- **Never commit directly to `main`** - always use feature branches
- **Use git worktrees** for all changes:
```bash
git worktree add ../karaoke-gen-feat-xyz feature/xyz
git worktree add -b feature/xyz ../karaoke-gen-feat-xyz main
```
- This keeps the main directory clean on `main` and enables parallel work on multiple features
- Name the worktree folder descriptively, e.g., `../karaoke-gen-fix-audio-sync`
**For trivial changes** (typos, single-line fixes):
- Creating a branch in the current directory is acceptable
- Use `git checkout -b fix/typo-xyz` before making changes
**Cleanup:** When done with a worktree, remove it with:
```bash
git worktree remove ../karaoke-gen-feat-xyz
```
### Creating a Pull Request
Once all changes are complete, tests pass, and commits are pushed, create a PR with:
1. **Summary** - Brief overview of the changes
2. **Changes Made** - List of specific modifications
3. **Testing** - How changes were tested (unit tests, manual testing)
4. **Related Issues** - Reference any GitHub issues (e.g., "Fixes #123")
Only merge after review and CI checks pass.
## Infrastructure Changes
All Google Cloud infrastructure (IAM, Cloud Storage, Cloud Run, Secret Manager, Artifact Registry, etc.) must be managed via **Pulumi IaC** in the `infrastructure/` folder.
### Rules
- **Reading/querying** with `gcloud` CLI is fine for investigation and debugging
- **Temporary changes** via `gcloud` CLI or Console are acceptable for urgent fixes or experimentation, but **must be cleaned up** and properly codified in the IaC afterward
- **Permanent changes** must go through Pulumi — never leave `gcloud` or Console changes as the final state
### Workflow
1. Make changes in `infrastructure/`
2. Run `pulumi preview` and verify the output
3. Run `pulumi up` to apply
### Authentication Issues
If you encounter any authentication issues with `gcloud` or `pulumi` (e.g., expired tokens, permission errors), **stop and notify the user** so they can run the appropriate auth commands in their external terminal:
```bash
gcloud auth login
gcloud auth application-default login
```
Do not attempt to work around auth issues or make assumptions about credentials.
## Version Management
When committing and pushing changes, always bump the patch version in `pyproject.toml`.
The version is located at `tool.poetry.version` in `pyproject.toml`. For example, if the current version is `0.70.3`, bump it to `0.70.4`.
This ensures every commit pushed to the repository has a unique version number for tracking and deployment purposes.
## Testing Requirements
**Before committing and pushing ANY changes, you MUST run the full test suite.**
### Backend Test Command
- Create PR with summary, changes, testing info
- Merge only after CI passes
### Testing (Required Before Commit)
```bash
# Backend (10 min timeout, limit output)
make test 2>&1 | tail -n 500
```
**Important for AI agents:**
- Set a **10 minute timeout** when running tests (they can take a while with emulators)
- Always pipe output through `tail -n 500` to limit output and preserve context window
- The tail output will show test results and any failures at the end
This runs all backend tests in order and fails fast if any test fails:
1. Unit tests (karaoke_gen package) with coverage check
2. Backend unit tests
3. E2E integration tests with emulators (auto-starts/stops emulators)
### Frontend Test Command
When working in the `frontend/` directory, run:
```bash
# Frontend
cd frontend && npm run test:all 2>&1 | tail -n 200
```
This runs:
1. **Unit tests** (Jest) - Component and utility tests
2. **E2E tests** (Playwright) - Browser-based integration tests
### Version Bumping
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
- Skip for docs-only changes
**Note:** E2E tests require `KARAOKE_ACCESS_TOKEN` environment variable to be set in `frontend/.env.local`.
### Infrastructure
- All GCP changes via Pulumi in `infrastructure/`
- `gcloud` CLI for reading/debugging only
- Stop and notify user on auth issues
Individual test commands:
- `npm run test:unit` - Run unit tests only
- `npm run test:e2e` - Run E2E tests only
- `npm run test:ci` - Run unit tests with coverage
## Documentation Maintenance
### Pre-commit Checklist
### Structure
```text
docs/
├── README.md # Current status + navigation (UPDATE THIS)
├── ARCHITECTURE.md # System design
├── DEVELOPMENT.md # Dev setup, testing, deployment
├── API.md # Backend API reference
├── LESSONS-LEARNED.md # Key insights for future agents
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
```
- [ ] `make test` passes (for backend changes)
- [ ] `cd frontend && npm run test:all` passes (for frontend changes)
- [ ] Version in `pyproject.toml` has been bumped
- [ ] `coderabbit review --plain` has been run to get comprehensive analysis and suggestions for cleaner, more maintainable code. Apply the feedback to improve accessibility, structure, and best practices.
### Rules
1. **Before merging PRs**: Run `/docs-review` to check if docs need updating
2. **Periodically**: Run `/docs-maintain` to verify docs are organized
3. **For significant work**: Create `docs/archive/YYYY-MM-DD-topic.md`
4. **Add learnings**: Update `docs/LESSONS-LEARNED.md` with insights
5. **Keep status current**: Update `docs/README.md` when project state changes
**Do not commit or push code that fails tests. Fix any test failures before proceeding.**
### What Goes Where
- **README.md**: Current status, known issues, quick links
- **ARCHITECTURE.md**: System design, data flow, tech decisions
- **DEVELOPMENT.md**: Setup, testing, deployment, CI/CD
- **API.md**: Endpoints, request/response formats
- **LESSONS-LEARNED.md**: What worked, what didn't, gotchas
- **archive/**: Completed work, session summaries, old plans
## Related Projects
### flacfetch
Location: `/Users/andrew/Projects/flacfetch`
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
## Pre-Commit Checklist
- [ ] Tests pass (`make test` / `npm run test:all`)
- [ ] Version bumped (if code changed)
- [ ] `/docs-review` run (check for doc updates)
- [ ] PR created with clear description
+72 -2
View File
@@ -71,8 +71,40 @@ export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
#### Option 2: Whisper via RunPod
Open-source alternative using OpenAI's Whisper model on RunPod infrastructure.
#### Option 2: Local Whisper (No Cloud Required)
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
```bash
# Install with local Whisper support
pip install "karaoke-gen[local-whisper]"
# Optional: Configure model size (tiny, base, small, medium, large)
export WHISPER_MODEL_SIZE="medium"
# Optional: Force specific device (cpu, cuda, mps)
export WHISPER_DEVICE="cpu"
```
**Model Size Guide:**
| Model | VRAM | Speed | Quality |
|-------|------|-------|---------|
| tiny | ~1GB | Fast | Lower |
| base | ~1GB | Fast | Basic |
| small | ~2GB | Medium | Good |
| medium | ~5GB | Slower | Better |
| large | ~10GB | Slowest | Best |
**CPU-Only Installation** (no GPU required):
```bash
# Pre-install CPU-only PyTorch first
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install "karaoke-gen[local-whisper]"
```
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
#### Option 3: Whisper via RunPod
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
```bash
export RUNPOD_API_KEY="your_runpod_key"
@@ -574,6 +606,44 @@ If the output video has quality problems:
- Check available codecs: `ffmpeg -codecs`
- For 4K output, ensure sufficient disk space (10GB+ per track)
### Local Whisper Issues
#### GPU Out of Memory
If you get CUDA out of memory errors:
```bash
# Use a smaller model
export WHISPER_MODEL_SIZE="small" # or "tiny"
# Or force CPU mode
export WHISPER_DEVICE="cpu"
```
#### Slow Transcription on CPU
CPU transcription is significantly slower than GPU. For faster processing:
- Use a smaller model (`tiny` or `base`)
- Consider using cloud transcription (AudioShake or RunPod)
- On Apple Silicon, the `small` model offers good speed/quality balance
#### Model Download Issues
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
- Check your internet connection
- Set a custom cache directory: `export WHISPER_CACHE_DIR="/path/with/space"`
- Models are cached in `~/.cache/whisper/` by default
#### whisper-timestamped Not Found
If you get "whisper-timestamped is not installed":
```bash
pip install "karaoke-gen[local-whisper]"
# Or install directly:
pip install whisper-timestamped
```
#### Disabling Local Whisper
If you want to disable local Whisper (e.g., to force cloud transcription):
```bash
export ENABLE_LOCAL_WHISPER="false"
```
---
## 🧪 Development
+68 -7
View File
@@ -85,9 +85,13 @@ class AudioSearchRequest(BaseModel):
# Auto-download mode
auto_download: bool = Field(False, description="Automatically select best result and download")
# Processing options (CDG/TXT require style config, disabled by default)
enable_cdg: bool = Field(False, description="Generate CDG+MP3 package (requires style config)")
enable_txt: bool = Field(False, description="Generate TXT+MP3 package (requires style config)")
# Theme configuration
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
@@ -180,6 +184,35 @@ class AudioSelectResponse(BaseModel):
selected_provider: str
def _resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool],
enable_txt: Optional[bool]
) -> Tuple[bool, bool]:
"""
Resolve CDG/TXT settings based on theme and explicit settings.
When a theme is selected, CDG and TXT are enabled by default.
Explicit True/False values always override the default.
Args:
theme_id: Theme identifier (if any)
enable_cdg: Explicit CDG setting (None means use default)
enable_txt: Explicit TXT setting (None means use default)
Returns:
Tuple of (resolved_enable_cdg, resolved_enable_txt)
"""
# Default based on whether theme is set
default_enabled = theme_id is not None
# Explicit values override defaults, None uses default
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
resolved_txt = enable_txt if enable_txt is not None else default_enabled
return resolved_cdg, resolved_txt
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
"""Extract metadata from request for job tracking."""
headers = dict(request.headers)
@@ -461,13 +494,20 @@ async def search_audio(
# Extract request metadata
request_metadata = extract_request_metadata(request, created_from="audio_search")
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Create job
job_create = JobCreate(
artist=body.artist,
title=body.title,
enable_cdg=body.enable_cdg,
enable_txt=body.enable_txt,
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=effective_brand_prefix,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=effective_youtube_description,
@@ -497,7 +537,28 @@ async def search_audio(
'audio_search_title': body.title,
'auto_download': body.auto_download,
})
# If theme is set and no custom style files are being uploaded, prepare theme style now
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
# can access the style configuration for preview videos
if body.theme_id and not body.style_files:
from backend.api.routes.file_upload import _prepare_theme_for_job
try:
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
job_id, body.theme_id, body.color_overrides
)
theme_update = {
'style_params_gcs_path': style_params_path,
'style_assets': theme_style_assets,
}
if youtube_desc and not effective_youtube_description:
theme_update['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, theme_update)
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
except Exception as e:
logger.warning(f"Failed to prepare theme '{body.theme_id}' for job {job_id}: {e}")
# Continue without theme - job can still be processed with defaults
# Handle style file uploads if provided
style_upload_urls: List[StyleUploadUrl] = []
style_assets = {}
+191 -28
View File
@@ -22,10 +22,12 @@ from typing import Optional, List, Dict, Any, Tuple
from pydantic import BaseModel, Field
from backend.models.job import JobCreate, JobStatus
from backend.models.theme import ColorOverrides
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.worker_service import get_worker_service
from backend.services.credential_manager import get_credential_manager, CredentialStatus
from backend.services.theme_service import get_theme_service
from backend.config import get_settings
from backend.version import VERSION
from backend.services.metrics import metrics
@@ -51,14 +53,18 @@ class CreateJobFromUrlRequest(BaseModel):
"""Request to create a job from a YouTube/online URL."""
# Required fields
url: str = Field(..., description="YouTube or other video URL to download audio from")
# Optional fields - will be auto-detected from URL if not provided
artist: Optional[str] = Field(None, description="Artist name (auto-detected from URL if not provided)")
title: Optional[str] = Field(None, description="Song title (auto-detected from URL if not provided)")
# Processing options (CDG/TXT require style config, disabled by default)
enable_cdg: bool = Field(False, description="Generate CDG+MP3 package (requires style config)")
enable_txt: bool = Field(False, description="Generate TXT+MP3 package (requires style config)")
# Theme configuration (use pre-made theme from GCS)
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
@@ -103,9 +109,13 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
title: str = Field(..., description="Song title")
files: List[FileUploadRequest] = Field(..., description="List of files to upload")
# Processing options (CDG/TXT require style config, disabled by default)
enable_cdg: bool = Field(False, description="Generate CDG+MP3 package (requires style config)")
enable_txt: bool = Field(False, description="Generate TXT+MP3 package (requires style config)")
# Theme configuration (use pre-made theme from GCS)
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
@@ -226,7 +236,7 @@ ALLOWED_FONT_EXTENSIONS = {'.ttf', '.otf', '.woff', '.woff2'}
async def _trigger_workers_parallel(job_id: str) -> None:
"""
Trigger both audio and lyrics workers in parallel.
FastAPI's BackgroundTasks runs async tasks sequentially, so we use
asyncio.gather to ensure both workers start at the same time.
"""
@@ -236,6 +246,83 @@ async def _trigger_workers_parallel(job_id: str) -> None:
)
def _resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool],
enable_txt: Optional[bool]
) -> Tuple[bool, bool]:
"""
Resolve CDG/TXT settings based on theme and explicit settings.
When a theme is selected, CDG and TXT are enabled by default.
Explicit True/False values always override the default.
Args:
theme_id: Theme identifier (if any)
enable_cdg: Explicit CDG setting (None means use default)
enable_txt: Explicit TXT setting (None means use default)
Returns:
Tuple of (resolved_enable_cdg, resolved_enable_txt)
"""
# Default based on whether theme is set
default_enabled = theme_id is not None
# Resolve with explicit override taking precedence
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
resolved_txt = enable_txt if enable_txt is not None else default_enabled
return resolved_cdg, resolved_txt
def _prepare_theme_for_job(
job_id: str,
theme_id: str,
color_overrides: Optional[Dict[str, str]] = None
) -> Tuple[str, Dict[str, str], Optional[str]]:
"""
Prepare theme style files for a job.
Args:
job_id: The job ID
theme_id: Theme identifier
color_overrides: Optional color override dict
Returns:
Tuple of (style_params_gcs_path, style_assets, youtube_description)
Raises:
HTTPException: If theme not found
"""
theme_service = get_theme_service()
# Verify theme exists
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=400,
detail=f"Theme not found: {theme_id}. Use GET /api/themes to list available themes."
)
# Convert dict to ColorOverrides model if provided
color_overrides_model = None
if color_overrides:
color_overrides_model = ColorOverrides(**color_overrides)
# Prepare job style from theme
style_params_path, style_assets = theme_service.prepare_job_style(
job_id=job_id,
theme_id=theme_id,
color_overrides=color_overrides_model
)
# Get YouTube description template if available
youtube_desc = theme_service.get_youtube_description(theme_id)
logger.info(f"Prepared theme '{theme_id}' for job {job_id}")
return style_params_path, style_assets, youtube_desc
@router.post("/jobs/upload")
async def upload_and_create_job(
request: Request,
@@ -682,9 +769,13 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
title: str = Field(..., description="Song title")
files: List[FinaliseOnlyFileRequest] = Field(..., description="List of prep output files to upload")
# Processing options (CDG/TXT require style config, disabled by default)
enable_cdg: bool = Field(False, description="Generate CDG+MP3 package (requires style config)")
enable_txt: bool = Field(False, description="Generate TXT+MP3 package (requires style config)")
# Theme configuration (use pre-made theme from GCS)
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
@@ -859,17 +950,27 @@ async def create_job_with_upload_urls(
# Extract request metadata for tracking
request_metadata = extract_request_metadata(request, created_from="signed_url_upload")
# Get original audio filename
audio_file = audio_files[0]
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Check if style_params is being uploaded (overrides theme)
has_style_params_upload = any(f.file_type == 'style_params' for f in body.files)
# Create job
job_create = JobCreate(
artist=body.artist,
title=body.title,
filename=audio_file.filename,
enable_cdg=body.enable_cdg,
enable_txt=body.enable_txt,
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=body.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
youtube_description=body.youtube_description,
@@ -889,11 +990,26 @@ async def create_job_with_upload_urls(
)
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
metrics.record_job_created(job_id, source="upload")
logger.info(f"Created job {job_id} for {body.artist} - {body.title} (signed URL upload flow)")
# If theme is set and no style_params uploaded, prepare theme style now
if body.theme_id and not has_style_params_upload:
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
job_id, body.theme_id, body.color_overrides
)
# Update job with theme style data
update_data = {
'style_params_gcs_path': style_params_path,
'style_assets': style_assets,
}
if youtube_desc and not body.youtube_description:
update_data['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, update_data)
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
# Generate signed upload URLs for each file
upload_urls = []
@@ -1245,19 +1361,26 @@ async def create_job_from_url(
# Extract request metadata for tracking
request_metadata = extract_request_metadata(request, created_from="url")
# Use provided artist/title or leave as None (will be auto-detected by audio worker)
artist = body.artist
title = body.title
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Create job with URL
job_create = JobCreate(
url=body.url,
artist=artist,
title=title,
filename=None, # No file uploaded
enable_cdg=body.enable_cdg,
enable_txt=body.enable_txt,
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=body.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
youtube_description=body.youtube_description,
@@ -1277,9 +1400,24 @@ async def create_job_from_url(
)
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
metrics.record_job_created(job_id, source="url")
# If theme is set, prepare theme style now
if body.theme_id:
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
job_id, body.theme_id, body.color_overrides
)
# Update job with theme style data
update_data = {
'style_params_gcs_path': style_params_path,
'style_assets': style_assets,
}
if youtube_desc and not body.youtube_description:
update_data['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, update_data)
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
logger.info(f"Created URL-based job {job_id} for URL: {body.url}")
if artist:
@@ -1462,14 +1600,24 @@ async def create_finalise_only_job(
# Extract request metadata
request_metadata = extract_request_metadata(request, created_from="finalise_only_upload")
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Check if style_params is being uploaded (overrides theme)
has_style_params_upload = any(f.file_type == 'style_params' for f in body.files)
# Create job with finalise_only=True
job_create = JobCreate(
artist=body.artist,
title=body.title,
filename="finalise_only", # No single audio file - using prep outputs
enable_cdg=body.enable_cdg,
enable_txt=body.enable_txt,
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=body.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
youtube_description=body.youtube_description,
@@ -1482,11 +1630,26 @@ async def create_finalise_only_job(
)
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
metrics.record_job_created(job_id, source="finalise")
logger.info(f"Created finalise-only job {job_id} for {body.artist} - {body.title}")
# If theme is set and no style_params uploaded, prepare theme style now
if body.theme_id and not has_style_params_upload:
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
job_id, body.theme_id, body.color_overrides
)
# Update job with theme style data
update_data = {
'style_params_gcs_path': style_params_path,
'style_assets': style_assets,
}
if youtube_desc and not body.youtube_description:
update_data['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, update_data)
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
# Generate signed upload URLs for each file
upload_urls = []
+162
View File
@@ -0,0 +1,162 @@
"""
Theme API routes for listing and retrieving theme information.
Themes are pre-created style configurations that users can select when creating jobs.
These endpoints are public (no authentication required) to allow theme selection
before job creation.
"""
import logging
from typing import Optional
from fastapi import APIRouter, HTTPException
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
from backend.services.theme_service import get_theme_service
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/themes", tags=["themes"])
@router.get("", response_model=ThemesListResponse)
async def list_themes() -> ThemesListResponse:
"""
List all available themes.
Returns a list of themes with their metadata and preview image URLs.
Preview URLs are signed and valid for 60 minutes.
This endpoint is public and does not require authentication.
"""
try:
theme_service = get_theme_service()
themes = theme_service.list_themes()
return ThemesListResponse(themes=themes)
except Exception as e:
logger.error(f"Error listing themes: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load themes"
)
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
async def get_theme(theme_id: str) -> ThemeDetailResponse:
"""
Get detailed information about a specific theme.
Returns the full theme details including style parameters.
This can be used to preview theme settings or inspect configuration.
Args:
theme_id: The unique identifier of the theme
Returns:
ThemeDetailResponse with full theme details
Raises:
404: Theme not found
"""
try:
theme_service = get_theme_service()
theme = theme_service.get_theme(theme_id)
if theme is None:
raise HTTPException(
status_code=404,
detail=f"Theme not found: {theme_id}"
)
return ThemeDetailResponse(theme=theme)
except HTTPException:
raise
except Exception as e:
logger.error(f"Error getting theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load theme"
)
@router.get("/{theme_id}/preview")
async def get_theme_preview(theme_id: str) -> dict:
"""
Get a signed URL for the theme's preview image.
This endpoint is useful when you need just the preview URL
without loading the full theme details.
Args:
theme_id: The unique identifier of the theme
Returns:
Object with preview_url field
Raises:
404: Theme not found or no preview available
"""
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=f"Theme not found: {theme_id}"
)
theme = theme_service.get_theme(theme_id)
if theme is None or theme.preview_url is None:
raise HTTPException(
status_code=404,
detail=f"No preview available for theme: {theme_id}"
)
return {"preview_url": theme.preview_url}
except HTTPException:
raise
except Exception as e:
logger.error(f"Error getting preview for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load theme preview"
)
@router.get("/{theme_id}/youtube-description")
async def get_theme_youtube_description(theme_id: str) -> dict:
"""
Get the YouTube description template for a theme.
Some themes include a default YouTube video description template
that can be used when uploading videos.
Args:
theme_id: The unique identifier of the theme
Returns:
Object with description field containing the template text,
or null if no template is available
Raises:
404: Theme not found
"""
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=f"Theme not found: {theme_id}"
)
description = theme_service.get_youtube_description(theme_id)
return {"description": description}
except HTTPException:
raise
except Exception as e:
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load YouTube description"
)
+2 -1
View File
@@ -7,7 +7,7 @@ from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
@@ -108,6 +108,7 @@ app.include_router(internal.router, prefix="/api") # Internal worker endpoints
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
app.include_router(auth.router, prefix="/api") # OAuth credential management
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
app.include_router(themes.router, prefix="/api") # Theme selection for styles
@app.get("/")
+24 -2
View File
@@ -205,7 +205,18 @@ class Job(BaseModel):
webhook_url: Optional[str] = None # Webhook for notifications
user_email: Optional[str] = None # Email for notifications
# Style configuration (uploaded files)
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
color_overrides: Dict[str, str] = Field(default_factory=dict)
"""
User color overrides applied on top of theme. Keys:
- artist_color: Hex color for artist name (#RRGGBB)
- title_color: Hex color for song title
- sung_lyrics_color: Hex color for highlighted lyrics
- unsung_lyrics_color: Hex color for unhighlighted lyrics
"""
# Style configuration (uploaded files - used when theme_id is not set)
style_params_gcs_path: Optional[str] = None # GCS path to style_params.json
style_assets: Dict[str, str] = Field(default_factory=dict)
"""
@@ -409,7 +420,18 @@ class JobCreate(BaseModel):
webhook_url: Optional[str] = None
user_email: Optional[str] = None
# Style configuration (will be populated after file upload)
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
color_overrides: Dict[str, str] = Field(default_factory=dict)
"""
User color overrides applied on top of theme. Keys:
- artist_color: Hex color for artist name (#RRGGBB)
- title_color: Hex color for song title
- sung_lyrics_color: Hex color for highlighted lyrics
- unsung_lyrics_color: Hex color for unhighlighted lyrics
"""
# Style configuration (will be populated after file upload, or from theme)
style_params_gcs_path: Optional[str] = None
style_assets: Dict[str, str] = Field(default_factory=dict)
+153
View File
@@ -0,0 +1,153 @@
"""
Theme data models for karaoke-gen.
Themes are pre-created style configurations stored in GCS that users can select
when creating karaoke jobs. Each theme includes:
- Style parameters (fonts, colors, backgrounds, layouts)
- Asset files (images, fonts)
- Preview images
- Optional YouTube description template
"""
from datetime import datetime
from typing import Any, Dict, List, Optional
from pydantic import BaseModel, Field
class ThemeSummary(BaseModel):
"""Summary of a theme for listing purposes."""
id: str = Field(..., description="Unique theme identifier (e.g., 'nomad', 'default')")
name: str = Field(..., description="Human-readable theme name")
description: str = Field(..., description="Brief description of the theme style")
preview_url: Optional[str] = Field(None, description="Signed URL for preview image")
thumbnail_url: Optional[str] = Field(None, description="Signed URL for smaller thumbnail")
is_default: bool = Field(False, description="Whether this is the default theme")
class ThemeMetadata(BaseModel):
"""Metadata for a single theme in the registry."""
id: str
name: str
description: str
is_default: bool = False
created_at: Optional[datetime] = None
class ThemeRegistry(BaseModel):
"""Registry of all available themes, stored as _metadata.json in GCS."""
version: int = 1
themes: List[ThemeMetadata] = Field(default_factory=list)
class ThemeDetail(BaseModel):
"""Full theme details including style parameters."""
id: str
name: str
description: str
preview_url: Optional[str] = None
is_default: bool = False
style_params: Dict[str, Any] = Field(default_factory=dict)
has_youtube_description: bool = False
class ColorOverrides(BaseModel):
"""
User-customizable color overrides that can be applied on top of a theme.
All colors are hex format (#RRGGBB). When applied:
- artist_color: Applied to intro.artist_color, end.artist_color, cdg.artist_color
- title_color: Applied to intro.title_color, end.title_color, cdg.title_color
- sung_lyrics_color: Applied to karaoke.primary_color (converted to RGBA), cdg.active_fill
- unsung_lyrics_color: Applied to karaoke.secondary_color (converted to RGBA), cdg.inactive_fill
"""
artist_color: Optional[str] = Field(
None,
pattern=r"^#[0-9A-Fa-f]{6}$",
description="Hex color for artist name on intro/end screens",
)
title_color: Optional[str] = Field(
None,
pattern=r"^#[0-9A-Fa-f]{6}$",
description="Hex color for song title on intro/end screens",
)
sung_lyrics_color: Optional[str] = Field(
None,
pattern=r"^#[0-9A-Fa-f]{6}$",
description="Hex color for lyrics being sung (highlighted)",
)
unsung_lyrics_color: Optional[str] = Field(
None,
pattern=r"^#[0-9A-Fa-f]{6}$",
description="Hex color for lyrics not yet sung",
)
def has_overrides(self) -> bool:
"""Check if any color overrides are set."""
return any(
[
self.artist_color,
self.title_color,
self.sung_lyrics_color,
self.unsung_lyrics_color,
]
)
def to_dict(self) -> Dict[str, str]:
"""Convert to dict, excluding None values."""
return {k: v for k, v in self.model_dump().items() if v is not None}
class ThemesListResponse(BaseModel):
"""Response from GET /api/themes endpoint."""
themes: List[ThemeSummary]
class ThemeDetailResponse(BaseModel):
"""Response from GET /api/themes/{theme_id} endpoint."""
theme: ThemeDetail
# Utility functions for color conversion
def hex_to_rgba(hex_color: str, alpha: int = 255) -> str:
"""
Convert hex color (#RRGGBB) to RGBA format for ASS subtitles.
ASS subtitle format uses "R, G, B, A" string format.
Args:
hex_color: Hex color string like "#7070F7"
alpha: Alpha value 0-255 (default 255 = opaque)
Returns:
RGBA string like "112, 112, 247, 255"
"""
hex_color = hex_color.lstrip("#")
r = int(hex_color[0:2], 16)
g = int(hex_color[2:4], 16)
b = int(hex_color[4:6], 16)
return f"{r}, {g}, {b}, {alpha}"
def rgba_to_hex(rgba_str: str) -> str:
"""
Convert RGBA format to hex color.
Args:
rgba_str: RGBA string like "112, 112, 247, 255"
Returns:
Hex color string like "#7070F7"
"""
parts = [int(x.strip()) for x in rgba_str.split(",")]
r, g, b = parts[0], parts[1], parts[2]
return f"#{r:02x}{g:02x}{b:02x}"
+5
View File
@@ -61,6 +61,11 @@ class JobManager:
discord_webhook_url=job_create.discord_webhook_url,
dropbox_path=job_create.dropbox_path,
gdrive_folder_id=job_create.gdrive_folder_id,
# Theme configuration
theme_id=job_create.theme_id,
color_overrides=job_create.color_overrides,
style_params_gcs_path=job_create.style_params_gcs_path,
style_assets=job_create.style_assets,
# Two-phase workflow (Batch 6)
prep_only=job_create.prep_only,
finalise_only=job_create.finalise_only,
+469
View File
@@ -0,0 +1,469 @@
"""
Theme service for managing pre-created style themes.
Themes are stored in GCS at themes/{theme_id}/ with:
- style_params.json: Complete style configuration
- preview.png: Preview image for theme selection UI
- youtube_description.txt: Optional YouTube description template
- assets/: Fonts, backgrounds, and other assets
The service provides:
- Theme listing with signed preview URLs
- Theme detail retrieval
- Color override application
- Job style preparation (copying theme to job folder)
"""
import copy
import json
import logging
import os
import tempfile
import threading
from datetime import datetime, timedelta
from typing import Any, Dict, List, Optional, Tuple
from backend.models.theme import (
ColorOverrides,
ThemeDetail,
ThemeMetadata,
ThemeRegistry,
ThemeSummary,
hex_to_rgba,
)
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
# GCS paths for themes
THEMES_PREFIX = "themes"
METADATA_FILE = f"{THEMES_PREFIX}/_metadata.json"
class ThemeService:
"""Service for managing themes from GCS."""
def __init__(self, storage: Optional[StorageService] = None):
"""
Initialize the theme service.
Args:
storage: StorageService instance (creates new one if not provided)
"""
self.storage = storage or StorageService()
self._metadata_cache: Optional[ThemeRegistry] = None
self._cache_time: Optional[datetime] = None
self.CACHE_TTL_SECONDS = 300 # 5 minute cache for theme metadata
def _get_theme_path(self, theme_id: str, filename: str = "") -> str:
"""Get the GCS path for a theme file."""
if filename:
return f"{THEMES_PREFIX}/{theme_id}/{filename}"
return f"{THEMES_PREFIX}/{theme_id}"
def _is_cache_valid(self) -> bool:
"""Check if the metadata cache is still valid."""
if self._metadata_cache is None or self._cache_time is None:
return False
age = datetime.now() - self._cache_time
return age.total_seconds() < self.CACHE_TTL_SECONDS
def _load_metadata(self, force_refresh: bool = False) -> ThemeRegistry:
"""
Load theme metadata from GCS with caching.
Args:
force_refresh: Force reload from GCS even if cache is valid
Returns:
ThemeRegistry containing all theme metadata
"""
if not force_refresh and self._is_cache_valid():
return self._metadata_cache # type: ignore
try:
data = self.storage.download_json(METADATA_FILE)
self._metadata_cache = ThemeRegistry(**data)
self._cache_time = datetime.now()
logger.info(f"Loaded theme metadata: {len(self._metadata_cache.themes)} themes")
return self._metadata_cache
except Exception as e:
logger.error(f"Failed to load theme metadata: {e}")
# Return empty registry on error
return ThemeRegistry(version=1, themes=[])
def list_themes(self) -> List[ThemeSummary]:
"""
List all available themes with signed preview URLs.
Returns:
List of ThemeSummary objects with preview URLs
"""
metadata = self._load_metadata()
summaries = []
for theme in metadata.themes:
try:
# Generate signed URLs for preview images
preview_path = self._get_theme_path(theme.id, "preview.png")
thumbnail_path = self._get_theme_path(theme.id, "preview_thumbnail.png")
preview_url = None
thumbnail_url = None
if self.storage.file_exists(preview_path):
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
# Try thumbnail, fall back to main preview
if self.storage.file_exists(thumbnail_path):
thumbnail_url = self.storage.generate_signed_url(thumbnail_path, expiration_minutes=60)
elif preview_url:
thumbnail_url = preview_url
summaries.append(
ThemeSummary(
id=theme.id,
name=theme.name,
description=theme.description,
preview_url=preview_url,
thumbnail_url=thumbnail_url,
is_default=theme.is_default,
)
)
except Exception as e:
logger.warning(f"Error processing theme {theme.id}: {e}")
# Still include theme but without preview URLs
summaries.append(
ThemeSummary(
id=theme.id,
name=theme.name,
description=theme.description,
is_default=theme.is_default,
)
)
return summaries
def get_theme(self, theme_id: str) -> Optional[ThemeDetail]:
"""
Get full theme details including style parameters.
Args:
theme_id: The theme identifier
Returns:
ThemeDetail if found, None otherwise
"""
metadata = self._load_metadata()
# Find theme in metadata
theme_meta = next((t for t in metadata.themes if t.id == theme_id), None)
if not theme_meta:
logger.warning(f"Theme not found: {theme_id}")
return None
try:
# Load style_params.json
style_params_path = self._get_theme_path(theme_id, "style_params.json")
style_params = self.storage.download_json(style_params_path)
# Check for YouTube description
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
has_youtube_desc = self.storage.file_exists(youtube_desc_path)
# Generate preview URL
preview_path = self._get_theme_path(theme_id, "preview.png")
preview_url = None
if self.storage.file_exists(preview_path):
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
return ThemeDetail(
id=theme_id,
name=theme_meta.name,
description=theme_meta.description,
preview_url=preview_url,
is_default=theme_meta.is_default,
style_params=style_params,
has_youtube_description=has_youtube_desc,
)
except Exception as e:
logger.error(f"Error loading theme {theme_id}: {e}")
return None
def get_theme_style_params(self, theme_id: str) -> Optional[Dict[str, Any]]:
"""
Get just the style_params.json for a theme.
Args:
theme_id: The theme identifier
Returns:
Style parameters dict if found, None otherwise
"""
try:
style_params_path = self._get_theme_path(theme_id, "style_params.json")
return self.storage.download_json(style_params_path)
except Exception as e:
logger.error(f"Error loading style params for theme {theme_id}: {e}")
return None
def get_youtube_description(self, theme_id: str) -> Optional[str]:
"""
Get the YouTube description template for a theme.
Args:
theme_id: The theme identifier
Returns:
YouTube description text if found, None otherwise
"""
try:
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
if not self.storage.file_exists(youtube_desc_path):
return None
# Download to temp file and read
with tempfile.NamedTemporaryFile(mode="w", delete=False, suffix=".txt") as tmp:
tmp_path = tmp.name
try:
self.storage.download_file(youtube_desc_path, tmp_path)
with open(tmp_path, "r") as f:
return f.read()
finally:
if os.path.exists(tmp_path):
os.remove(tmp_path)
except Exception as e:
logger.error(f"Error loading YouTube description for theme {theme_id}: {e}")
return None
def apply_color_overrides(
self, style_params: Dict[str, Any], overrides: ColorOverrides
) -> Dict[str, Any]:
"""
Apply user color overrides to theme style parameters.
Args:
style_params: Base style parameters from theme
overrides: User color overrides
Returns:
Modified style parameters with overrides applied
"""
if not overrides.has_overrides():
return style_params
# Deep copy to avoid modifying original
result = copy.deepcopy(style_params)
# Apply artist_color to intro, end, and CDG sections
if overrides.artist_color:
if "intro" in result:
result["intro"]["artist_color"] = overrides.artist_color
if "end" in result:
result["end"]["artist_color"] = overrides.artist_color
if "cdg" in result:
result["cdg"]["artist_color"] = overrides.artist_color
# Apply title_color to intro, end, and CDG sections
if overrides.title_color:
if "intro" in result:
result["intro"]["title_color"] = overrides.title_color
if "end" in result:
result["end"]["title_color"] = overrides.title_color
if "cdg" in result:
result["cdg"]["title_color"] = overrides.title_color
# Apply sung_lyrics_color (karaoke uses RGBA, CDG uses hex)
if overrides.sung_lyrics_color:
if "karaoke" in result:
result["karaoke"]["primary_color"] = hex_to_rgba(overrides.sung_lyrics_color)
if "cdg" in result:
result["cdg"]["active_fill"] = overrides.sung_lyrics_color
# Apply unsung_lyrics_color (karaoke uses RGBA, CDG uses hex)
if overrides.unsung_lyrics_color:
if "karaoke" in result:
result["karaoke"]["secondary_color"] = hex_to_rgba(overrides.unsung_lyrics_color)
if "cdg" in result:
result["cdg"]["inactive_fill"] = overrides.unsung_lyrics_color
return result
def prepare_job_style(
self,
job_id: str,
theme_id: str,
color_overrides: Optional[ColorOverrides] = None,
) -> Tuple[str, Dict[str, str]]:
"""
Prepare style files for a job by copying theme to job folder.
This method:
1. Loads the theme's style_params.json
2. Applies any color overrides
3. Updates asset paths to point to theme assets (shared, not copied)
4. Uploads modified style_params.json to job folder
5. Returns the GCS path and style_assets mapping
Args:
job_id: The job ID
theme_id: The theme to use
color_overrides: Optional color overrides
Returns:
Tuple of (style_params_gcs_path, style_assets dict)
Raises:
ValueError: If theme not found
"""
# Load theme style params
style_params = self.get_theme_style_params(theme_id)
if style_params is None:
raise ValueError(f"Theme not found: {theme_id}")
# Apply color overrides if provided
if color_overrides:
style_params = self.apply_color_overrides(style_params, color_overrides)
# Update asset paths to point to theme's shared assets
# Theme assets stay in themes/{theme_id}/assets/ - they're shared
style_assets = self._build_style_assets_mapping(theme_id, style_params)
# Update style_params to use theme asset paths (not job-specific)
style_params = self._update_asset_paths_in_style(theme_id, style_params)
# Upload modified style_params.json to job's style folder
job_style_path = f"uploads/{job_id}/style/style_params.json"
self.storage.upload_json(job_style_path, style_params)
logger.info(f"Prepared job {job_id} style from theme {theme_id}")
return job_style_path, style_assets
def _build_style_assets_mapping(
self, theme_id: str, style_params: Dict[str, Any]
) -> Dict[str, str]:
"""
Build the style_assets mapping for a theme.
Maps asset keys to their GCS paths in the theme folder.
"""
style_assets = {}
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
# Check each potential asset type
asset_checks = [
("intro_background", "intro", "background_image"),
("karaoke_background", "karaoke", "background_image"),
("end_background", "end", "background_image"),
("font", "intro", "font"), # Font is typically shared across sections
("cdg_instrumental_background", "cdg", "instrumental_background"),
("cdg_title_background", "cdg", "title_screen_background"),
("cdg_outro_background", "cdg", "outro_background"),
]
for asset_key, section, field in asset_checks:
if section in style_params and field in style_params[section]:
value = style_params[section][field]
if value and isinstance(value, str):
# Check if it's a theme asset path or already has the full path
if value.startswith(f"{THEMES_PREFIX}/"):
# Already a full theme path
style_assets[asset_key] = value
elif not value.startswith("/") and not value.startswith("gs://"):
# Relative path - prepend theme assets prefix
asset_path = f"{theme_assets_prefix}/{os.path.basename(value)}"
if self.storage.file_exists(asset_path):
style_assets[asset_key] = asset_path
return style_assets
def _update_asset_paths_in_style(
self, theme_id: str, style_params: Dict[str, Any]
) -> Dict[str, Any]:
"""
Update asset paths in style_params to use full GCS theme paths.
This ensures the style_loader can find assets when processing.
"""
result = copy.deepcopy(style_params)
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
# Fields that contain asset paths
path_fields = {
"intro": ["background_image", "font"],
"end": ["background_image", "font"],
"karaoke": ["background_image", "font_path"],
"cdg": [
"font_path",
"instrumental_background",
"title_screen_background",
"outro_background",
],
}
for section, fields in path_fields.items():
if section not in result:
continue
for field in fields:
if field not in result[section]:
continue
value = result[section][field]
if value and isinstance(value, str):
# Skip if already a full path or URL
if value.startswith(f"{THEMES_PREFIX}/") or value.startswith("gs://"):
continue
# Skip absolute local paths (shouldn't happen but be safe)
if value.startswith("/"):
continue
# Update to full theme asset path
result[section][field] = f"{theme_assets_prefix}/{os.path.basename(value)}"
return result
def theme_exists(self, theme_id: str) -> bool:
"""
Check if a theme exists.
Args:
theme_id: The theme identifier
Returns:
True if theme exists, False otherwise
"""
metadata = self._load_metadata()
return any(t.id == theme_id for t in metadata.themes)
def get_default_theme_id(self) -> Optional[str]:
"""
Get the ID of the default theme.
Returns:
Default theme ID if one exists, None otherwise
"""
metadata = self._load_metadata()
default_theme = next((t for t in metadata.themes if t.is_default), None)
return default_theme.id if default_theme else None
def invalidate_cache(self) -> None:
"""Force invalidation of the metadata cache."""
self._metadata_cache = None
self._cache_time = None
logger.info("Theme metadata cache invalidated")
# Singleton instance with thread-safe initialization
_theme_service: Optional[ThemeService] = None
_theme_service_lock = threading.Lock()
def get_theme_service() -> ThemeService:
"""Get or create the singleton ThemeService instance (thread-safe)."""
global _theme_service
if _theme_service is None:
with _theme_service_lock:
# Double-check after acquiring lock
if _theme_service is None:
_theme_service = ThemeService()
return _theme_service
@@ -330,7 +330,7 @@ class TestReviewEndpoints:
class TestRenderVideoWorker:
"""Test render video worker endpoint."""
def test_render_video_worker_endpoint_exists(self, client, auth_headers):
"""Test that render-video worker endpoint exists."""
# Create a job
@@ -340,9 +340,9 @@ class TestRenderVideoWorker:
json={"url": "https://youtube.com/watch?v=render-test"}
)
job_id = create_resp.json()["job_id"]
time.sleep(0.2)
# Test render-video worker endpoint exists
response = client.post(
"/api/internal/workers/render-video",
+248
View File
@@ -798,6 +798,254 @@ class TestRemoteDownloadPath:
assert result.filepath == "/tmp/test.flac"
class TestAudioSearchThemeSupport:
"""Test theme support in audio search requests.
These tests verify that theme_id and color_overrides are properly:
1. Accepted in AudioSearchRequest model
2. Passed through to JobCreate
3. Result in correct CDG/TXT defaults when theme is set
4. Theme style is PREPARED (copied to job folder) when theme_id is set
"""
def test_audio_search_request_accepts_theme_id(self):
"""Test AudioSearchRequest model has theme_id field."""
from backend.api.routes.audio_search import AudioSearchRequest
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
theme_id="nomad"
)
assert request.theme_id == "nomad"
def test_audio_search_request_accepts_color_overrides(self):
"""Test AudioSearchRequest model has color_overrides field."""
from backend.api.routes.audio_search import AudioSearchRequest
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
theme_id="nomad",
color_overrides={
"artist_color": "#ff0000",
"title_color": "#00ff00",
}
)
assert request.color_overrides["artist_color"] == "#ff0000"
assert request.color_overrides["title_color"] == "#00ff00"
def test_audio_search_request_theme_defaults_cdg_txt(self):
"""Test that when theme_id is set, CDG/TXT defaults to enabled.
This is the key behavior: selecting a theme should automatically
enable CDG and TXT output formats.
"""
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
# When theme_id is set, enable_cdg/enable_txt should default to True
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
theme_id="nomad"
# enable_cdg and enable_txt are None (not specified)
)
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
assert resolved_cdg is True, "CDG should be enabled by default when theme is set"
assert resolved_txt is True, "TXT should be enabled by default when theme is set"
def test_audio_search_request_no_theme_no_cdg_txt(self):
"""Test that without theme_id, CDG/TXT defaults to disabled."""
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song"
# No theme_id, no enable_cdg, no enable_txt
)
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
assert resolved_cdg is False, "CDG should be disabled by default without theme"
assert resolved_txt is False, "TXT should be disabled by default without theme"
def test_explicit_cdg_txt_overrides_theme_default(self):
"""Test that explicit enable_cdg/enable_txt values override theme defaults."""
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
# Theme set (would default to True), but explicitly disabled
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
theme_id="nomad",
enable_cdg=False,
enable_txt=False,
)
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
assert resolved_cdg is False, "Explicit False should override theme default"
assert resolved_txt is False, "Explicit False should override theme default"
def test_explicit_cdg_txt_enables_without_theme(self):
"""Test that explicit True enables CDG/TXT even without theme."""
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
# No theme (would default to False), but explicitly enabled
request = AudioSearchRequest(
artist="Test Artist",
title="Test Song",
enable_cdg=True,
enable_txt=True,
)
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
assert resolved_cdg is True, "Explicit True should enable CDG without theme"
assert resolved_txt is True, "Explicit True should enable TXT without theme"
def test_job_create_receives_theme_from_audio_search(self):
"""Test that JobCreate model can receive theme_id and color_overrides."""
job_create = JobCreate(
artist="Test Artist",
title="Test Song",
theme_id="nomad",
color_overrides={
"artist_color": "#ff0000",
"sung_lyrics_color": "#00ff00"
},
enable_cdg=True,
enable_txt=True,
)
assert job_create.theme_id == "nomad"
assert job_create.color_overrides["artist_color"] == "#ff0000"
assert job_create.color_overrides["sung_lyrics_color"] == "#00ff00"
assert job_create.enable_cdg is True
assert job_create.enable_txt is True
def test_job_model_stores_theme_configuration(self):
"""Test Job model stores theme configuration correctly."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.utcnow(),
updated_at=datetime.utcnow(),
artist="Test Artist",
title="Test Song",
theme_id="nomad",
color_overrides={
"artist_color": "#ff0000"
},
enable_cdg=True,
enable_txt=True,
)
assert job.theme_id == "nomad"
assert job.color_overrides["artist_color"] == "#ff0000"
assert job.enable_cdg is True
assert job.enable_txt is True
class TestAudioSearchThemePreparation:
"""Test that audio search endpoint properly prepares theme styles.
CRITICAL: This test class was added to catch the bug where audio_search.py
accepted theme_id but never called prepare_job_style(), resulting in jobs
with theme_id set but no style_params_gcs_path or style_assets.
The symptom: Preview videos show black background instead of themed background.
Job ID ffb0b8fa in production demonstrated this issue.
"""
def test_audio_search_with_theme_calls_prepare_job_style(self):
"""Test that creating a job via audio search with theme_id prepares the style.
CRITICAL: This test verifies that when a job is created via the audio-search
endpoint with a theme_id (and no custom style files), the code calls
prepare_job_style() to set:
1. style_params_gcs_path (pointing to the copied style_params.json)
2. style_assets (populated with asset mappings)
Without this, LyricsTranscriber won't have access to the theme's styles
and preview videos will have black backgrounds instead of themed ones.
"""
# Import the endpoint module to access internal functions
from backend.api.routes import audio_search as audio_search_module
# Check if there's an import of prepare_job_style from theme_service
source_code_path = audio_search_module.__file__
with open(source_code_path, 'r') as f:
source_code = f.read()
# Check for either:
# 1. Import of _prepare_theme_for_job from file_upload
# 2. Import of prepare_job_style from theme_service
# 3. Inline call to theme_service.prepare_job_style
has_theme_prep_import = (
'_prepare_theme_for_job' in source_code or
'prepare_job_style' in source_code
)
assert has_theme_prep_import, (
"audio_search.py does not import or call prepare_job_style() or _prepare_theme_for_job(). "
"When theme_id is provided without custom style files, the endpoint MUST call "
"prepare_job_style() to copy the theme's style_params.json to the job folder. "
"Without this, LyricsTranscriber won't have style configuration and preview "
"videos will have black backgrounds instead of themed ones."
)
def test_audio_search_code_handles_theme_preparation(self):
"""Verify audio_search.py has theme preparation logic similar to file_upload.py.
This test compares audio_search.py with file_upload.py to ensure they have
similar theme preparation patterns. file_upload.py correctly prepares themes,
audio_search.py should do the same.
"""
from backend.api.routes import audio_search as audio_search_module
from backend.api.routes import file_upload as file_upload_module
# Read source code of both modules
with open(audio_search_module.__file__, 'r') as f:
audio_search_code = f.read()
with open(file_upload_module.__file__, 'r') as f:
file_upload_code = f.read()
# file_upload.py has this pattern for theme preparation:
# if body.theme_id and not has_style_params_upload:
# style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(...)
assert '_prepare_theme_for_job' in file_upload_code, \
"file_upload.py should have _prepare_theme_for_job function"
# audio_search.py should have similar pattern
# Look for either the function or a similar conditional check
# STRICT CHECK: Look for actual theme preparation logic being CALLED, not just mentioned
has_theme_preparation_call = (
# Check for import or call of theme preparation function
'_prepare_theme_for_job(' in audio_search_code or
'prepare_job_style(' in audio_search_code
)
assert has_theme_preparation_call, (
"audio_search.py does not CALL theme preparation logic. "
"It accepts theme_id but never prepares the theme style like file_upload.py does. "
"Add a call to _prepare_theme_for_job() when theme_id is set and no style files are uploaded."
)
class TestFlacfetchIntegration:
"""
Integration tests that verify flacfetch imports work correctly.
+6 -6
View File
@@ -1123,9 +1123,9 @@ class TestFinaliseOnlyModels:
]
)
# CDG/TXT disabled by default (requires style config)
assert request.enable_cdg is False
assert request.enable_txt is False
# CDG/TXT default to None (server resolves based on theme_id)
assert request.enable_cdg is None
assert request.enable_txt is None
assert request.brand_prefix is None
assert request.keep_brand_code is None
assert request.enable_youtube_upload is False
@@ -1275,9 +1275,9 @@ class TestCreateJobFromUrlRequest:
assert request.url == "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
assert request.artist is None
assert request.title is None
# CDG/TXT disabled by default (requires style config)
assert request.enable_cdg is False
assert request.enable_txt is False
# CDG/TXT default to None (server resolves based on theme_id)
assert request.enable_cdg is None
assert request.enable_txt is None
def test_create_with_artist_and_title(self):
"""Test creating request with URL, artist, and title."""
+516
View File
@@ -0,0 +1,516 @@
"""
Tests for the theme service and API endpoints.
Tests theme listing, retrieval, color override application, and job style preparation.
"""
import json
import pytest
from unittest.mock import MagicMock, patch, Mock
from datetime import datetime
from backend.models.theme import (
ThemeSummary,
ThemeDetail,
ThemeMetadata,
ThemeRegistry,
ColorOverrides,
hex_to_rgba,
rgba_to_hex,
)
from backend.services.theme_service import ThemeService, get_theme_service
# =============================================================================
# Color Conversion Tests
# =============================================================================
class TestColorConversion:
"""Tests for hex/rgba color conversion utilities."""
def test_hex_to_rgba_basic(self):
"""Test basic hex to RGBA conversion."""
result = hex_to_rgba("#7070F7")
assert result == "112, 112, 247, 255"
def test_hex_to_rgba_with_custom_alpha(self):
"""Test hex to RGBA with custom alpha."""
result = hex_to_rgba("#FF0000", alpha=128)
assert result == "255, 0, 0, 128"
def test_hex_to_rgba_lowercase(self):
"""Test hex with lowercase letters."""
result = hex_to_rgba("#abcdef")
assert result == "171, 205, 239, 255"
def test_hex_to_rgba_without_hash(self):
"""Test hex without leading hash."""
result = hex_to_rgba("FFFFFF")
assert result == "255, 255, 255, 255"
def test_rgba_to_hex_basic(self):
"""Test basic RGBA to hex conversion."""
result = rgba_to_hex("112, 112, 247, 255")
assert result == "#7070f7"
def test_rgba_to_hex_ignores_alpha(self):
"""Test that alpha is ignored in conversion."""
result = rgba_to_hex("255, 0, 0, 128")
assert result == "#ff0000"
# =============================================================================
# ColorOverrides Model Tests
# =============================================================================
class TestColorOverrides:
"""Tests for ColorOverrides model."""
def test_has_overrides_empty(self):
"""Test has_overrides with no overrides set."""
overrides = ColorOverrides()
assert overrides.has_overrides() is False
def test_has_overrides_with_artist_color(self):
"""Test has_overrides with artist_color set."""
overrides = ColorOverrides(artist_color="#FF0000")
assert overrides.has_overrides() is True
def test_has_overrides_with_all_colors(self):
"""Test has_overrides with all colors set."""
overrides = ColorOverrides(
artist_color="#FF0000",
title_color="#00FF00",
sung_lyrics_color="#0000FF",
unsung_lyrics_color="#FFFF00",
)
assert overrides.has_overrides() is True
def test_to_dict_excludes_none(self):
"""Test to_dict excludes None values."""
overrides = ColorOverrides(artist_color="#FF0000")
result = overrides.to_dict()
assert result == {"artist_color": "#FF0000"}
assert "title_color" not in result
def test_color_validation_valid(self):
"""Test color validation accepts valid hex colors."""
overrides = ColorOverrides(artist_color="#AbCdEf")
assert overrides.artist_color == "#AbCdEf"
def test_color_validation_invalid(self):
"""Test color validation rejects invalid colors."""
with pytest.raises(ValueError):
ColorOverrides(artist_color="red")
with pytest.raises(ValueError):
ColorOverrides(artist_color="#GGG")
# =============================================================================
# ThemeService Tests
# =============================================================================
class TestThemeService:
"""Tests for ThemeService."""
@pytest.fixture
def mock_storage(self):
"""Create a mock storage service."""
storage = MagicMock()
return storage
@pytest.fixture
def sample_metadata(self):
"""Sample theme metadata for testing."""
return {
"version": 1,
"themes": [
{
"id": "nomad",
"name": "Nomad Karaoke",
"description": "Golden artist text, professional look",
"is_default": True,
},
{
"id": "default",
"name": "Default",
"description": "Clean white text on black background",
"is_default": False,
},
],
}
@pytest.fixture
def sample_style_params(self):
"""Sample style params for testing."""
return {
"intro": {
"artist_color": "#ffdf6b",
"title_color": "#ffffff",
"background_image": "intro_bg.png",
"font": "Montserrat-Bold.ttf",
},
"end": {
"artist_color": "#ffdf6b",
"title_color": "#ffffff",
"background_image": "end_bg.png",
"font": "Montserrat-Bold.ttf",
},
"karaoke": {
"primary_color": "112, 112, 247, 255",
"secondary_color": "255, 255, 255, 255",
"background_image": "karaoke_bg.png",
"font_path": "Montserrat-Bold.ttf",
},
"cdg": {
"active_fill": "#7070F7",
"inactive_fill": "#FFFFFF",
"font_path": "Montserrat-Bold.ttf",
},
}
def test_list_themes_returns_summaries(self, mock_storage, sample_metadata):
"""Test list_themes returns ThemeSummary objects."""
mock_storage.download_json.return_value = sample_metadata
mock_storage.file_exists.return_value = True
mock_storage.generate_signed_url.return_value = "https://signed-url.com/preview.png"
service = ThemeService(storage=mock_storage)
themes = service.list_themes()
assert len(themes) == 2
assert themes[0].id == "nomad"
assert themes[0].name == "Nomad Karaoke"
assert themes[0].is_default is True
assert themes[0].preview_url is not None
def test_list_themes_handles_missing_preview(self, mock_storage, sample_metadata):
"""Test list_themes handles missing preview images gracefully."""
mock_storage.download_json.return_value = sample_metadata
mock_storage.file_exists.return_value = False
service = ThemeService(storage=mock_storage)
themes = service.list_themes()
assert len(themes) == 2
assert themes[0].preview_url is None
def test_get_theme_returns_detail(self, mock_storage, sample_metadata, sample_style_params):
"""Test get_theme returns ThemeDetail with style params."""
mock_storage.download_json.side_effect = [sample_metadata, sample_style_params]
mock_storage.file_exists.return_value = True
mock_storage.generate_signed_url.return_value = "https://signed-url.com/preview.png"
service = ThemeService(storage=mock_storage)
theme = service.get_theme("nomad")
assert theme is not None
assert theme.id == "nomad"
assert theme.name == "Nomad Karaoke"
assert "intro" in theme.style_params
assert theme.style_params["intro"]["artist_color"] == "#ffdf6b"
def test_get_theme_returns_none_for_unknown(self, mock_storage, sample_metadata):
"""Test get_theme returns None for unknown theme ID."""
mock_storage.download_json.return_value = sample_metadata
service = ThemeService(storage=mock_storage)
theme = service.get_theme("unknown-theme")
assert theme is None
def test_theme_exists_true(self, mock_storage, sample_metadata):
"""Test theme_exists returns True for existing theme."""
mock_storage.download_json.return_value = sample_metadata
service = ThemeService(storage=mock_storage)
assert service.theme_exists("nomad") is True
def test_theme_exists_false(self, mock_storage, sample_metadata):
"""Test theme_exists returns False for unknown theme."""
mock_storage.download_json.return_value = sample_metadata
service = ThemeService(storage=mock_storage)
assert service.theme_exists("unknown") is False
def test_get_default_theme_id(self, mock_storage, sample_metadata):
"""Test get_default_theme_id returns the default theme."""
mock_storage.download_json.return_value = sample_metadata
service = ThemeService(storage=mock_storage)
assert service.get_default_theme_id() == "nomad"
def test_get_default_theme_id_no_default(self, mock_storage):
"""Test get_default_theme_id returns None when no default."""
mock_storage.download_json.return_value = {
"version": 1,
"themes": [{"id": "test", "name": "Test", "description": "Test", "is_default": False}],
}
service = ThemeService(storage=mock_storage)
assert service.get_default_theme_id() is None
def test_apply_color_overrides_no_changes(self, sample_style_params):
"""Test apply_color_overrides with empty overrides."""
service = ThemeService()
overrides = ColorOverrides()
result = service.apply_color_overrides(sample_style_params, overrides)
# Should return original (no deep copy needed when no changes)
assert result == sample_style_params
def test_apply_color_overrides_artist_color(self, sample_style_params):
"""Test apply_color_overrides applies artist_color."""
service = ThemeService()
overrides = ColorOverrides(artist_color="#FF0000")
result = service.apply_color_overrides(sample_style_params, overrides)
assert result["intro"]["artist_color"] == "#FF0000"
assert result["end"]["artist_color"] == "#FF0000"
assert result["cdg"]["artist_color"] == "#FF0000"
# Original should be unchanged
assert sample_style_params["intro"]["artist_color"] == "#ffdf6b"
def test_apply_color_overrides_sung_lyrics_color(self, sample_style_params):
"""Test apply_color_overrides applies sung_lyrics_color with conversion."""
service = ThemeService()
overrides = ColorOverrides(sung_lyrics_color="#FF0000")
result = service.apply_color_overrides(sample_style_params, overrides)
# Karaoke uses RGBA format
assert result["karaoke"]["primary_color"] == "255, 0, 0, 255"
# CDG uses hex
assert result["cdg"]["active_fill"] == "#FF0000"
def test_apply_color_overrides_unsung_lyrics_color(self, sample_style_params):
"""Test apply_color_overrides applies unsung_lyrics_color."""
service = ThemeService()
overrides = ColorOverrides(unsung_lyrics_color="#00FF00")
result = service.apply_color_overrides(sample_style_params, overrides)
assert result["karaoke"]["secondary_color"] == "0, 255, 0, 255"
assert result["cdg"]["inactive_fill"] == "#00FF00"
def test_metadata_cache(self, mock_storage, sample_metadata):
"""Test metadata caching works."""
mock_storage.download_json.return_value = sample_metadata
service = ThemeService(storage=mock_storage)
# First call loads from GCS
service.list_themes()
assert mock_storage.download_json.call_count == 1
# Second call uses cache
service.list_themes()
assert mock_storage.download_json.call_count == 1
def test_invalidate_cache(self, mock_storage, sample_metadata):
"""Test cache invalidation."""
mock_storage.download_json.return_value = sample_metadata
service = ThemeService(storage=mock_storage)
# Load metadata
service.list_themes()
assert mock_storage.download_json.call_count == 1
# Invalidate cache
service.invalidate_cache()
# Next call should reload
service.list_themes()
assert mock_storage.download_json.call_count == 2
def test_prepare_job_style(self, mock_storage, sample_metadata, sample_style_params):
"""Test prepare_job_style creates job style from theme."""
mock_storage.download_json.side_effect = [sample_metadata, sample_style_params]
mock_storage.file_exists.return_value = True
service = ThemeService(storage=mock_storage)
style_path, style_assets = service.prepare_job_style("job123", "nomad")
# Should upload modified style_params.json
mock_storage.upload_json.assert_called_once()
assert style_path == "uploads/job123/style/style_params.json"
def test_prepare_job_style_with_overrides(self, mock_storage, sample_metadata, sample_style_params):
"""Test prepare_job_style applies color overrides."""
# prepare_job_style calls get_theme_style_params which directly downloads style_params.json
# No metadata lookup happens unless we've cached it previously
mock_storage.download_json.return_value = sample_style_params
mock_storage.file_exists.return_value = True
service = ThemeService(storage=mock_storage)
overrides = ColorOverrides(artist_color="#FF0000")
style_path, style_assets = service.prepare_job_style("job123", "nomad", overrides)
# Check that upload_json was called with modified style params
call_args = mock_storage.upload_json.call_args
uploaded_path = call_args[0][0] # First positional arg is the path
uploaded_style = call_args[0][1] # Second positional arg is the data
assert uploaded_path == "uploads/job123/style/style_params.json"
# Check that artist_color override was applied
assert uploaded_style["intro"]["artist_color"] == "#FF0000"
assert uploaded_style["end"]["artist_color"] == "#FF0000"
def test_prepare_job_style_unknown_theme_raises(self, mock_storage, sample_metadata):
"""Test prepare_job_style raises for unknown theme."""
# First download_json returns metadata (no unknown theme)
# Second download_json for style_params.json should fail
def download_json_side_effect(path):
if "_metadata.json" in path:
return sample_metadata
# Unknown theme path - raise exception
raise Exception(f"File not found: {path}")
mock_storage.download_json.side_effect = download_json_side_effect
service = ThemeService(storage=mock_storage)
with pytest.raises(ValueError, match="Theme not found"):
service.prepare_job_style("job123", "unknown-theme")
def test_get_youtube_description(self, mock_storage, sample_metadata):
"""Test get_youtube_description returns template text."""
mock_storage.download_json.return_value = sample_metadata
mock_storage.file_exists.return_value = True
# Mock the download_file to write content to temp file
def mock_download(gcs_path, local_path):
with open(local_path, "w") as f:
f.write("Thank you for watching!")
mock_storage.download_file.side_effect = mock_download
service = ThemeService(storage=mock_storage)
desc = service.get_youtube_description("nomad")
assert desc == "Thank you for watching!"
def test_get_youtube_description_not_found(self, mock_storage, sample_metadata):
"""Test get_youtube_description returns None when no template."""
mock_storage.download_json.return_value = sample_metadata
mock_storage.file_exists.return_value = False
service = ThemeService(storage=mock_storage)
desc = service.get_youtube_description("nomad")
assert desc is None
# =============================================================================
# API Endpoint Tests
# =============================================================================
class TestThemeAPI:
"""Tests for theme API endpoints."""
@pytest.fixture
def sample_metadata(self):
"""Sample theme metadata."""
return {
"version": 1,
"themes": [
{
"id": "nomad",
"name": "Nomad Karaoke",
"description": "Golden artist text",
"is_default": True,
}
],
}
def test_list_themes_endpoint(self, test_client, sample_metadata):
"""Test GET /api/themes returns theme list."""
with patch("backend.api.routes.themes.get_theme_service") as mock_get_service:
mock_service = MagicMock()
mock_service.list_themes.return_value = [
ThemeSummary(
id="nomad",
name="Nomad Karaoke",
description="Golden artist text",
preview_url="https://example.com/preview.png",
is_default=True,
)
]
mock_get_service.return_value = mock_service
response = test_client.get("/api/themes")
assert response.status_code == 200
data = response.json()
assert len(data["themes"]) == 1
assert data["themes"][0]["id"] == "nomad"
assert data["themes"][0]["is_default"] is True
def test_get_theme_endpoint(self, test_client):
"""Test GET /api/themes/{theme_id} returns theme detail."""
with patch("backend.api.routes.themes.get_theme_service") as mock_get_service:
mock_service = MagicMock()
mock_service.get_theme.return_value = ThemeDetail(
id="nomad",
name="Nomad Karaoke",
description="Golden artist text",
is_default=True,
style_params={"intro": {"artist_color": "#ffdf6b"}},
)
mock_get_service.return_value = mock_service
response = test_client.get("/api/themes/nomad")
assert response.status_code == 200
data = response.json()
assert data["theme"]["id"] == "nomad"
assert "style_params" in data["theme"]
def test_get_theme_not_found(self, test_client):
"""Test GET /api/themes/{theme_id} returns 404 for unknown theme."""
with patch("backend.api.routes.themes.get_theme_service") as mock_get_service:
mock_service = MagicMock()
mock_service.get_theme.return_value = None
mock_get_service.return_value = mock_service
response = test_client.get("/api/themes/unknown")
assert response.status_code == 404
def test_get_theme_preview_endpoint(self, test_client):
"""Test GET /api/themes/{theme_id}/preview returns preview URL."""
with patch("backend.api.routes.themes.get_theme_service") as mock_get_service:
mock_service = MagicMock()
mock_service.theme_exists.return_value = True
mock_service.get_theme.return_value = ThemeDetail(
id="nomad",
name="Nomad Karaoke",
description="Golden artist text",
preview_url="https://example.com/preview.png",
is_default=True,
)
mock_get_service.return_value = mock_service
response = test_client.get("/api/themes/nomad/preview")
assert response.status_code == 200
data = response.json()
assert "preview_url" in data
def test_get_youtube_description_endpoint(self, test_client):
"""Test GET /api/themes/{theme_id}/youtube-description returns description."""
with patch("backend.api.routes.themes.get_theme_service") as mock_get_service:
mock_service = MagicMock()
mock_service.theme_exists.return_value = True
mock_service.get_youtube_description.return_value = "Thank you for watching!"
mock_get_service.return_value = mock_service
response = test_client.get("/api/themes/nomad/youtube-description")
assert response.status_code == 200
data = response.json()
assert data["description"] == "Thank you for watching!"
+183
View File
@@ -0,0 +1,183 @@
# API Reference
Base URL: `https://api.nomadkaraoke.com` (production) or `http://localhost:8000` (local)
## Authentication
All endpoints except `/health` and `/api/themes` require authentication.
```bash
curl -H "Authorization: Bearer YOUR_TOKEN" ...
```
## Endpoints
### Health
```http
GET /health
```
No auth required. Returns service status.
### Jobs
#### Create Job with Upload
```http
POST /api/jobs/upload
Content-Type: multipart/form-data
file: <audio file>
artist: "Artist Name"
title: "Song Title"
```
Returns job ID. Triggers async processing.
#### Get Job Status
```http
GET /api/jobs/{job_id}
```
Response includes:
- `status` - Current job state
- `file_urls` - Signed URLs for outputs (when ready)
- `state_data` - Worker progress details
- `timeline` - Processing history
#### List Jobs
```http
GET /api/jobs
GET /api/jobs?status=complete
GET /api/jobs?limit=10&offset=0
```
#### Delete Job
```http
DELETE /api/jobs/{job_id}
```
### Review
#### Get Correction Data
```http
GET /api/review/{job_id}/correction-data
```
Returns lyrics correction data for the review UI.
#### Complete Review
```http
POST /api/review/{job_id}/complete
Content-Type: application/json
{
"corrections": [...],
"corrected_segments": [...]
}
```
Saves corrections and triggers video rendering.
#### Generate Preview
```http
POST /api/review/{job_id}/preview-video
```
Generates preview video during review.
#### Stream Audio
```http
GET /api/review/{job_id}/audio/{stem_type}
```
Streams audio for review playback.
### Instrumental Selection
```http
POST /api/jobs/{job_id}/select-instrumental
Content-Type: application/json
{
"selection": "clean" // or "with_backing"
}
```
### Audio Search
```http
GET /api/audio-search?q=song+name
```
Search for songs (flacfetch integration).
### Themes
```http
GET /api/themes
```
No auth required. Returns available video themes.
### Internal (Admin Only)
These endpoints are used by workers and require admin tokens.
```http
POST /api/internal/jobs/{job_id}/trigger-audio
POST /api/internal/jobs/{job_id}/trigger-lyrics
POST /api/internal/jobs/{job_id}/trigger-screens
POST /api/internal/jobs/{job_id}/trigger-render-video
POST /api/internal/jobs/{job_id}/trigger-video
```
## Job States
| State | Description |
|-------|-------------|
| `pending` | Created, not started |
| `downloading` | Processing input |
| `separating_stage1` | Audio separation (1/2) |
| `separating_stage2` | Audio separation (2/2) |
| `transcribing` | Lyrics transcription |
| `generating_screens` | Title/end screens |
| `awaiting_review` | Waiting for human review |
| `in_review` | Human reviewing |
| `review_complete` | Review submitted |
| `rendering_video` | Generating karaoke video |
| `awaiting_instrumental_selection` | Waiting for selection |
| `instrumental_selected` | Selection made |
| `generating_video` | Final encoding |
| `complete` | Done |
| `failed` | Error |
## Error Responses
```json
{
"detail": "Error message"
}
```
Common status codes:
- `400` - Bad request
- `401` - Unauthorized
- `404` - Job not found
- `500` - Server error
## Rate Limits
No rate limits currently implemented.
## Webhooks
Not yet implemented. Jobs must be polled for status.
+162
View File
@@ -0,0 +1,162 @@
# Architecture
## System Overview
```text
┌─────────────────────────────────────────────────────────────────┐
│ Frontend (Cloudflare Pages) │
│ Next.js + TypeScript │
│ https://gen.nomadkaraoke.com │
└───────────────────────────┬─────────────────────────────────────┘
│ HTTPS
▼
┌─────────────────────────────────────────────────────────────────┐
│ Backend API (Cloud Run) │
│ FastAPI + Python 3.12 │
│ https://api.nomadkaraoke.com │
│ │
│ Workers: │
│ • Audio Worker - Stem separation via Modal API │
│ • Lyrics Worker - Transcription via AudioShake │
│ • Screens Worker - Title/end screen generation │
│ • Render Worker - Karaoke video with lyrics │
│ • Video Worker - Final assembly and encoding │
└───────────────────────────┬─────────────────────────────────────┘
│
┌─────────────┼─────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────────┐
│ Firestore│ │ GCS │ │Secret Manager│
│ (jobs) │ │ (files) │ │ (API keys) │
└──────────┘ └──────────┘ └──────────────┘
```
## Processing Pipeline
```text
1. UPLOAD
POST /api/jobs/upload
• Save audio to GCS
• Create job in Firestore
• Trigger parallel workers
2. PARALLEL PROCESSING
┌─────────────────┐ ┌─────────────────┐
│ Audio Worker │ │ Lyrics Worker │
│ Modal API │ │ AudioShake │
│ 2-stage sep │ │ Auto-correct │
└────────┬────────┘ └────────┬────────┘
└──────────┬───────────┘
▼
3. SCREENS GENERATION
• Title screen (5s video)
• End screen (5s video)
▼
4. HUMAN REVIEW (AWAITING_REVIEW)
• React UI for lyrics correction
• Preview video generation
• Submit corrections
▼
5. VIDEO RENDERING
• Merge corrections with original data
• Generate karaoke video with lyrics
▼
6. INSTRUMENTAL SELECTION
• User chooses: clean or with_backing
▼
7. FINAL VIDEO
• Remux with selected instrumental
• Concatenate: title + karaoke + end
• Encode to 4 formats
▼
8. COMPLETE
• User downloads via signed URLs
```
## Job States
| State | Description | Human Action |
|-------|-------------|--------------|
| `pending` | Job created | - |
| `downloading` | Processing audio | - |
| `separating_stage1` | Audio separation stage 1 | - |
| `separating_stage2` | Audio separation stage 2 | - |
| `transcribing` | Lyrics transcription | - |
| `generating_screens` | Creating title/end screens | - |
| `awaiting_review` | **Waiting for human** | Review lyrics |
| `in_review` | Human actively reviewing | Edit lyrics |
| `review_complete` | Review submitted | - |
| `rendering_video` | Generating karaoke video | - |
| `awaiting_instrumental_selection` | **Waiting for human** | Choose audio |
| `instrumental_selected` | Selection made | - |
| `generating_video` | Final encoding | - |
| `complete` | All done | Download files |
| `failed` | Error occurred | - |
## GCS File Structure
```text
jobs/{job_id}/
├── input/
│ └── original.flac # Uploaded audio
├── stems/
│ ├── vocals.flac
│ ├── instrumental_clean.flac
│ ├── instrumental_with_backing.flac
│ └── ...
├── lyrics/
│ ├── corrections.json # Auto-corrected
│ ├── corrections_updated.json # After human review
│ └── karaoke.ass
├── screens/
│ ├── title.mov
│ └── end.mov
├── videos/
│ └── with_vocals.mkv # Karaoke video
└── finals/
├── lossless_4k.mp4
├── lossless_4k.mkv
├── lossy_4k.mp4
└── lossy_720p.mp4
```
## LyricsTranscriber Integration
**Key Design Decision**: We use LyricsTranscriber as a **library**, not a server.
```text
CLI Mode (Original): Cloud Mode (Our Approach):
──────────────────── ──────────────────────────
LyricsTranscriber LyricsTranscriber
├── transcribe() ├── transcribe() ✅
├── correct() ├── correct() ✅
├── ReviewServer.start() │
│ └── BLOCKS (waits) │ (Skip server, save JSON)
│ │ ... ASYNC GAP ...
│ │ (Human reviews via React)
└── OutputGenerator │
└── OutputGenerator ✅
(called after review)
```
**Use**: `LyricsTranscriber`, `CorrectionResult`, `OutputGenerator`, `OutputConfig`
**Don't use**: `ReviewServer`, blocking waits
## External Services
| Service | Purpose | Required |
|---------|---------|----------|
| Modal | GPU audio separation | Yes |
| AudioShake | Lyrics transcription | Yes |
| Genius | Reference lyrics | Yes |
| YouTube API | Video upload | Optional |
## Tech Stack
- **Backend**: FastAPI, Python 3.12, Cloud Run
- **Frontend**: Next.js, TypeScript, Cloudflare Pages
- **Database**: Firestore
- **Storage**: Google Cloud Storage
- **Secrets**: Google Secret Manager
- **IaC**: Pulumi
- **CI/CD**: GitHub Actions
+216
View File
@@ -0,0 +1,216 @@
# Development Guide
## Quick Start
### Prerequisites
- Python 3.12
- Node.js 20+
- Docker (for GCS emulator)
- Google Cloud SDK with Firestore emulator
```bash
# Install Firestore emulator
gcloud components install cloud-firestore-emulator
```
### Backend Setup
```bash
# Create and activate venv
python3.12 -m venv venv
source venv/bin/activate
# Install dependencies
poetry install
# Copy environment template
cp .env.example .env
# Edit .env with required API keys
```
### Frontend Setup
```bash
cd frontend
npm install
cp .env.example .env.local
# Edit .env.local with KARAOKE_ACCESS_TOKEN
```
## Running Locally
### Backend with Emulators
```bash
# Start emulators and backend
./scripts/run-backend-local.sh --with-emulators
# Backend runs at http://localhost:8000
# API docs at http://localhost:8000/docs
```
### Frontend
```bash
cd frontend
npm run dev
# Runs at http://localhost:3000
```
## Testing
### Backend Tests
```bash
# Run all tests (required before commit)
make test 2>&1 | tail -n 500
# This runs in order:
# 1. Unit tests with coverage
# 2. Backend unit tests
# 3. E2E integration tests with emulators
```
**Important**: Set 10 minute timeout. Tests can take a while with emulators.
### Frontend Tests
```bash
cd frontend
npm run test:all 2>&1 | tail -n 200
# Individual commands:
npm run test:unit # Jest unit tests
npm run test:e2e # Playwright E2E tests
```
### Test Architecture
```
┌─────────────────────────────────┐
│ Unit Tests (Mocked) │ Fast, isolated
│ ~62 tests, <1s │
├─────────────────────────────────┤
│ Emulator Integration Tests │ Local Firestore/GCS
│ ~11 tests, ~2s │
├─────────────────────────────────┤
│ E2E Tests │ Full API + Frontend
│ Requires deployed backend │
└─────────────────────────────────┘
```
## Deployment
### Automatic (CI/CD)
Push to `main` triggers GitHub Actions:
1. Run tests
2. Build Docker image
3. Deploy to Cloud Run
4. Deploy frontend to Cloudflare Pages
### Manual Backend Deploy
```bash
# Preview infrastructure changes
cd infrastructure
pulumi preview
# Apply changes
pulumi up
```
### Manual Frontend Deploy
```bash
cd frontend
npm run build
npx wrangler pages deploy out
```
## Environment Variables
### Backend (.env)
```bash
# Required
AUDIOSHAKE_API_TOKEN=...
GENIUS_ACCESS_TOKEN=...
AUDIO_SEPARATOR_API_URL=... # Modal API endpoint
# GCP (auto-configured in Cloud Run)
GOOGLE_CLOUD_PROJECT=...
GCS_BUCKET_NAME=...
# Auth
ADMIN_TOKENS=token1,token2
```
### Frontend (.env.local)
```bash
NEXT_PUBLIC_API_URL=http://localhost:8000
KARAOKE_ACCESS_TOKEN=... # For E2E tests
```
## Key Scripts
| Script | Purpose |
|--------|---------|
| `make test` | Run all backend tests |
| `./scripts/run-backend-local.sh` | Start backend with emulators |
| `./scripts/start-emulators.sh` | Start emulators only |
| `./scripts/stop-emulators.sh` | Stop emulators |
## Project Structure
```
karaoke-gen/
├── backend/
│ ├── api/routes/ # FastAPI endpoints
│ ├── models/ # Pydantic models
│ ├── services/ # Business logic
│ ├── workers/ # Background workers
│ └── tests/ # Backend tests
├── frontend/
│ ├── src/
│ │ ├── app/ # Next.js pages
│ │ ├── components/ # React components
│ │ └── lib/ # Utilities
│ └── tests/ # Frontend tests
├── infrastructure/ # Pulumi IaC
├── karaoke_gen/ # CLI package
└── scripts/ # Dev scripts
```
## Troubleshooting
### Emulators Won't Start
```bash
# Check if already running
lsof -i :8080 # Firestore
lsof -i :4443 # GCS
# Stop them
./scripts/stop-emulators.sh
```
### Tests Fail with Auth Errors
Ensure `ADMIN_TOKENS` is set in environment or `.env` file.
### Docker Issues
```bash
# Ensure Docker is running
docker info
# For GCS emulator
docker pull fsouza/fake-gcs-server
```
### Coverage Below Target
Run `./scripts/run-tests.sh --coverage` and check `htmlcov/index.html` to see uncovered lines.
+151
View File
@@ -0,0 +1,151 @@
# Lessons Learned
Key insights for future AI agents working on this codebase.
## Architecture Decisions
### LyricsTranscriber: Library, Not Server
**Problem**: LyricsTranscriber's `ReviewServer` blocks waiting for human input - incompatible with async cloud architecture.
**Solution**: Use LyricsTranscriber as a library:
- Call `transcribe()` and `correct()` for processing
- Save `corrections.json` to GCS
- Skip `ReviewServer` entirely
- Call `OutputGenerator` only after human review completes
**What to use**: `LyricsTranscriber`, `CorrectionResult`, `OutputGenerator`, `OutputConfig`, `CorrectionOperations`
**What NOT to use**: `ReviewServer`, `server.start()`, blocking waits
### Video Generation Timing
**Wrong**: Generate video during lyrics worker, then review
**Right**: Lyrics worker saves JSON → Human reviews → Render worker generates video
Video must include human-corrected lyrics, so generation happens AFTER review.
### Parallel Processing
Audio separation and lyrics transcription run in parallel:
- Both can fail independently
- Progress tracked separately in `state_data`
- Screens generation waits for both to complete
## Common Gotchas
### Pydantic Model Fields
**Problem**: Setting a field that doesn't exist in the Pydantic model silently does nothing.
```python
# BAD - fails silently if field doesn't exist
job.input_media_gcs_path = "path/to/file"
# GOOD - explicit field in model
class Job(BaseModel):
input_media_gcs_path: Optional[str] = None
```
**Lesson**: Always verify Pydantic model has the field before setting it.
### Styles Configuration
ASS subtitle generator requires ALL style fields present and properly typed:
```python
required_fields = [
"font", "font_path", "ass_name", # ass_name often missed
"primary_color", "secondary_color", "outline_color", "back_color",
"bold", "italic", "underline", "strike_out",
"scale_x", "scale_y", "spacing", "angle",
"border_style", "outline", "shadow",
"margin_l", "margin_r", "margin_v", "encoding"
]
# Critical: font_path must be string, not None
"font_path": "" # Correct
"font_path": None # Causes ASS writer to fail
```
### Frontend Data Format
The React review UI sends partial correction data:
```python
# Frontend sends:
{"corrections": [...], "corrected_segments": [...]}
# Backend must merge with original corrections.json
```
### Firestore Consistency
**Problem**: Update job, trigger worker immediately, worker reads stale data.
**Solution**: Verify updates propagated before triggering workers:
```python
# Update job
await job_manager.update_job(job_id, updates)
# Verify update visible
job = await job_manager.get_job(job_id)
if job.input_media_gcs_path:
await trigger_worker()
```
## Testing Insights
### Emulator Tests Catch Real Bugs
Unit tests with mocks didn't catch the `input_media_gcs_path` bug. Emulator integration tests did because they use real Firestore behavior.
**Lesson**: Run `make test` which includes emulator tests before every commit.
### Test Isolation
Emulator tests need their own `conftest.py` that does NOT mock `google.cloud.*`. Mocks from unit tests can leak into integration tests.
## Deployment Notes
### Cloud Run Timeouts
Default 5-minute timeout may not be enough for video encoding. Consider:
- Cloud Run Jobs for long-running tasks
- Breaking encoding into smaller chunks
- Increasing timeout for specific operations
### Secret Manager Access
Cloud Run service account needs `Secret Manager Secret Accessor` role. This is managed via Pulumi in `infrastructure/`.
### CI/CD: GitHub Actions
Migrated from Cloud Build to GitHub Actions for:
- Simpler configuration
- Better visibility
- Faster feedback
## Performance Observations
| Operation | Duration |
|-----------|----------|
| Audio separation | 5-8 min |
| Lyrics transcription | 2-3 min |
| Auto-correction | 30 sec |
| Screens generation | 30 sec |
| Video rendering | 10-15 min |
| Final encoding | 5-10 min |
| **Total** | 30-50 min |
Human review time varies (5-15 min typical).
## What We'd Do Differently
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
2. **Use emulator tests from day one** - Faster feedback than deploying to test
3. **Design for async human review upfront** - Avoided rearchitecting after discovering ReviewServer blocks
4. **Keep docs minimal and current** - Less documentation, but always accurate
+318
View File
@@ -0,0 +1,318 @@
# Mobile UX Best Practices
This document outlines the mobile-first UX principles and implementation patterns used in this project to ensure a great experience across all screen sizes.
## Core Principles
### 1. Mobile-First Responsive Design
Design for the smallest screen first, then progressively enhance for larger screens.
```css
/* Base styles for mobile */
.element {
width: 100%;
padding: 12px;
}
/* Enhance for larger screens */
@media (min-width: 640px) {
.element {
width: 280px;
padding: 16px;
}
}
```
### 2. No Horizontal Overflow
The most critical mobile UX issue is horizontal scrolling. Always ensure content fits within the viewport width.
**Testing for overflow:**
```typescript
async function hasNoHorizontalOverflow(page: Page): Promise<boolean> {
return await page.evaluate(() => {
const docWidth = document.documentElement.scrollWidth;
const viewportWidth = window.innerWidth;
return docWidth <= viewportWidth + 2; // Small tolerance for rounding
});
}
```
**Common causes of horizontal overflow:**
- Fixed widths (e.g., `width: 340px`)
- Horizontal flex layouts without wrapping
- Images without `max-width: 100%`
- Tables without horizontal scroll containers
### 3. Touch Target Sizing
Interactive elements must be easily tappable. Follow these minimum sizes:
| Platform | Minimum Size | Recommended |
|----------|--------------|-------------|
| Apple iOS | 44x44px | 48x48px |
| Material Design | 48x48px | 48x48px |
| Web (general) | 32x32px | 44x44px |
**Implementation:**
```tsx
<Button className="min-h-[44px] px-3">
<Icon className="w-4 h-4" />
<span className="hidden sm:inline">Label</span>
</Button>
```
### 4. Progressive Disclosure
Show less content on mobile, with clear paths to access more:
- **Icon-only buttons on mobile, labels on desktop:**
```tsx
<Icon className="w-4 h-4 sm:mr-2" />
<span className="hidden sm:inline">Action Label</span>
```
- **Collapsed navigation on mobile**
- **Truncated text with expansion options**
## Layout Patterns
### Responsive Grid
Use responsive grid columns that stack on mobile:
```tsx
// Bad - always 2 columns
<div className="grid grid-cols-2 gap-4">
// Good - stack on mobile, 2 columns on larger screens
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 sm:gap-4">
```
### Flexible Containers
Avoid fixed widths. Use percentage-based or responsive widths:
```css
/* Bad */
.sidebar {
width: 340px;
}
/* Good */
.sidebar {
width: 100%;
}
@media (min-width: 768px) {
.sidebar {
width: 340px;
}
}
```
### Flex Wrap for Toolbars
Allow toolbar items to wrap on smaller screens:
```tsx
<div className="flex flex-wrap gap-2 items-center">
{/* Items will wrap naturally */}
</div>
```
## Typography
### Readable Font Sizes
Minimum font sizes for readability without zooming:
| Element | Minimum Size | Recommended |
|---------|--------------|-------------|
| Body text | 14px | 16px |
| Labels | 12px | 14px |
| Small text/badges | 11px | 12px |
### Responsive Headings
```tsx
<h1 className="text-base sm:text-xl font-bold truncate">
Title Text
</h1>
```
## Form Design
### Input Fields on Mobile
- Stack form fields vertically on mobile
- Use full-width inputs for easier tapping
- Ensure proper input type for mobile keyboard optimization
```tsx
<input
type="email" // Shows email keyboard
inputMode="numeric" // Shows number pad
className="w-full min-h-[44px]"
/>
```
### Labels and Placeholders
- Labels should be visible (not just placeholders)
- Adequate spacing between label and input
- Clear validation feedback
## Testing Strategy
### Automated Testing with Playwright
Test across multiple viewport sizes representing real devices:
```typescript
const MOBILE_VIEWPORTS = {
'iPhone SE': { width: 375, height: 667 },
'iPhone 14': { width: 390, height: 844 },
'Pixel 7': { width: 412, height: 915 },
'iPad Mini': { width: 768, height: 1024 },
};
for (const [name, viewport] of Object.entries(MOBILE_VIEWPORTS)) {
test(`page works on ${name}`, async ({ page }) => {
await page.setViewportSize(viewport);
await page.goto('/');
// Test for horizontal overflow
const hasOverflow = await page.evaluate(() =>
document.documentElement.scrollWidth > window.innerWidth + 2
);
expect(hasOverflow).toBe(false);
});
}
```
### Visual Regression Testing
Capture screenshots at different viewports to catch visual regressions:
```typescript
await expect(page).toHaveScreenshot(`homepage-${deviceName}.png`, {
fullPage: true,
maxDiffPixelRatio: 0.1,
});
```
### Touch Target Validation
```typescript
const buttons = page.locator('button, a, [role="button"]');
for (const button of await buttons.all()) {
const box = await button.boundingBox();
if (box) {
expect(box.width).toBeGreaterThanOrEqual(32);
expect(box.height).toBeGreaterThanOrEqual(32);
}
}
```
## Framework-Specific Guidelines
### Tailwind CSS (Next.js Frontend)
Use responsive prefixes consistently:
```plaintext
sm: 640px - Small tablets and up
md: 768px - Tablets and up
lg: 1024px - Laptops and up
xl: 1280px - Desktops and up
```
**Common patterns:**
```tsx
// Stack on mobile, side-by-side on larger
className="flex flex-col sm:flex-row"
// Hide on mobile, show on larger
className="hidden sm:block"
// Reduce padding on mobile
className="p-3 sm:p-4"
// Smaller font on mobile
className="text-xs sm:text-sm"
```
### Material UI (Lyrics Review UI)
Use the `useMediaQuery` hook:
```tsx
const theme = useTheme();
const isMobile = useMediaQuery(theme.breakpoints.down('md'));
return (
<Box sx={{
flexDirection: isMobile ? 'column' : 'row',
width: isMobile ? '100%' : '280px',
}}>
```
### Vanilla CSS (Instrumental Review)
Use CSS media queries:
```css
.element {
width: 100%;
}
@media (min-width: 768px) {
.element {
width: 340px;
}
}
```
## Checklist for New Features
Before merging any UI changes, verify:
- [ ] No horizontal overflow on any mobile viewport (320px - 480px)
- [ ] All interactive elements are at least 44x44px
- [ ] Text is readable without zooming (minimum 11px)
- [ ] Form inputs stack vertically on mobile
- [ ] Navigation is accessible on mobile
- [ ] Images scale properly with `max-width: 100%`
- [ ] Modals/dialogs fit within mobile viewport
- [ ] Buttons show icons-only on very small screens
- [ ] Playwright mobile tests pass
## Common Fixes
### Fix: Horizontal Overflow
1. **Check for fixed widths** - Replace with responsive values
2. **Add flex-wrap** - Allow content to wrap
3. **Add overflow-x: hidden** - On containers (use sparingly)
4. **Check images** - Add `max-width: 100%`
### Fix: Touch Target Too Small
1. **Add minimum height** - `min-h-[44px]`
2. **Add padding** - `p-3` or `px-4 py-3`
3. **Use icon-only on mobile** - Hide text labels
### Fix: Text Too Small
1. **Use responsive font sizes** - `text-xs sm:text-sm`
2. **Set minimum body font size** - At least 14px
3. **Increase line height** - For better readability
## Resources
- [Apple Human Interface Guidelines - Touch Targets](https://developer.apple.com/design/human-interface-guidelines/accessibility)
- [Material Design - Touch Targets](https://m3.material.io/foundations/accessible-design/accessibility-basics)
- [Tailwind CSS Responsive Design](https://tailwindcss.com/docs/responsive-design)
- [Playwright Testing](https://playwright.dev/docs/test-mobile)
+33 -177
View File
@@ -1,193 +1,49 @@
# Karaoke-Gen Documentation
This directory contains all documentation for the karaoke-gen project, organized by purpose.
> **Status**: Production-ready. First full E2E flow completed 2025-12-28.
## 📋 Current Status (Updated: 2025-12-08)
## What Works
**Project:** Cloud-hosted karaoke generation with async human review
**Backend:** 🔄 Phase 1.3 (85%) - Implementing review architecture
**Frontend:** ⏸️ Not started (waiting for backend review flow)
**Deployment:** ✅ Cloud Run with custom domain (api.nomadkaraoke.com)
**Testing:** 🔄 Local emulator testing in progress
- **Web App**: https://gen.nomadkaraoke.com - Upload audio, review lyrics, download videos
- **Backend API**: https://api.nomadkaraoke.com - FastAPI on Cloud Run
- **CLI Tools**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
### Key Discovery
## Current State
We identified that the `LyricsTranscriber` library's `ReviewServer` **blocks** waiting for human input, which doesn't work in our async cloud architecture. The solution is to:
| Component | Status |
|-----------|--------|
| Audio upload & separation | Working |
| Lyrics transcription | Working |
| Human lyrics review | Working |
| Preview video generation | Working |
| Instrumental selection | Working |
| Multi-format encoding | Working |
| Token-based auth | Working |
| Payment flow | Not started |
1. Use LyricsTranscriber for transcription + auto-correction
2. Save corrections to GCS for async human review
3. Use `OutputGenerator` to render video **after** review completes
## Known Issues
See `00-current-plan/CURRENT-STATUS.md` for full details.
- CDG format generation requires additional style configuration
- Long audio files (>10 min) may timeout on some workers
---
## Quick Links
## 📂 Documentation Structure
| Doc | Purpose |
|-----|---------|
| [ARCHITECTURE.md](ARCHITECTURE.md) | System design, data flow, integrations |
| [DEVELOPMENT.md](DEVELOPMENT.md) | Local setup, testing, deployment |
| [API.md](API.md) | Backend endpoint reference |
| [LESSONS-LEARNED.md](LESSONS-LEARNED.md) | Key insights for future agents |
| [archive/](archive/) | Historical documentation |
### `00-current-plan/` - **START HERE**
Active migration plan and current status.
| File | Purpose |
|------|---------|
| **`CURRENT-STATUS.md`** | Where we are now, what's working, what's next |
| **`WHATS-NEXT.md`** | Detailed plan for review architecture implementation |
| **`WORKER-IMPLEMENTATION-PLAN.md`** | Worker responsibilities and implementation details |
| `MASTER-PLAN.md` | Overall migration strategy |
| `CLOUD-MIGRATION-REQUIREMENTS.md` | Architectural analysis from CLI workflow |
### `01-reference/` - Reference Documentation
Core system documentation.
| File | Purpose |
|------|---------|
| **`ARCHITECTURE.md`** | Cloud + CLI architecture with diagrams |
| `KARAOKE-GEN-CLI-WORKFLOW.md` | CLI workflow breakdown (8 stages) |
| `API-MANUAL-TESTING.md` | How to test backend API with curl |
| `AUTHENTICATION.md` | Token-based auth system |
| `CLOUD-RUN-ARCHITECTURE.md` | Cloud Run concepts |
### `02-implementation-history/` - Session Logs
Historical implementation records.
### `03-deployment/` - Deployment Guides
How to deploy and operate the system.
| File | Purpose |
|------|---------|
| **`EMULATOR-TESTING.md`** | Local testing with Firestore/GCS emulators |
| **`SETUP-VENV.md`** | Python venv setup |
| **`LOCAL-VALIDATION.md`** | Validate before deploying |
| **`OBSERVABILITY-GUIDE.md`** | Debug production issues |
### `04-testing/` - Testing Guides
| File | Purpose |
|------|---------|
| **`TESTING-GUIDE.md`** | Complete testing guide |
---
## 🚀 Quick Start
### I want to understand the current state:
1. Read `00-current-plan/CURRENT-STATUS.md` - where we are now
2. Read `00-current-plan/WHATS-NEXT.md` - what needs to be done
3. Read `01-reference/ARCHITECTURE.md` - system architecture
### I want to work on the backend:
1. Read `03-deployment/EMULATOR-TESTING.md` - local testing setup
2. Read `00-current-plan/WORKER-IMPLEMENTATION-PLAN.md` - worker details
3. Run `./scripts/run-backend-local.sh --with-emulators`
### I want to test:
```bash
# Start backend with emulators
./scripts/run-backend-local.sh --with-emulators
# Upload test file
curl -X POST http://localhost:8000/api/jobs/upload \
-F "file=@tests/data/waterloo10sec.flac" \
-F "artist=ABBA" \
-F "title=Waterloo"
# Check status
curl http://localhost:8000/api/jobs/{job_id}
```
---
## 🎯 Current Phase: Review Architecture
**Goal:** Implement async human review workflow
**What Works:**
- ✅ File upload → GCS
- ✅ Audio separation via Modal API
- ✅ Lyrics transcription via AudioShake
- ✅ Auto-correction via LyricsTranscriber
- ✅ Title/end screen generation
- ✅ Instrumental selection endpoint
**What's Being Implemented:**
- 🔄 Review API endpoints (`GET/POST /api/jobs/{id}/review`)
- 🔄 Render Video Worker (uses `OutputGenerator` after review)
- 🔄 Updated state machine with `RENDERING_VIDEO` state
**The Key Insight:**
```
WRONG: Lyrics Worker → Generate Video → Review → ???
RIGHT: Lyrics Worker → Save JSON → Human Review → Render Video → Final Assembly
```
Video generation happens AFTER human review, not during the lyrics worker.
---
## 📝 Key Technical Decisions
### Why Not Use LyricsTranscriber's ReviewServer?
The `ReviewServer` class (in `lyrics_transcriber/review/server.py`):
- Starts a local web server on port 8000
- Opens a browser window
- **BLOCKS** the thread until human submits corrections
- Designed for local CLI operation
This doesn't work in Cloud Run where:
- We can't open browsers
- Workers must be non-blocking
- Human review happens asynchronously via React frontend
**Solution:** Use the data structures and `OutputGenerator` directly, skip the server.
### Why Separate Video Rendering?
## Workflow Summary
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Lyrics Worker │ ──▶ │ Human Reviews │ ──▶ │ Render Worker │
│ │ │ (React UI) │ │ │
│ • Transcribe │ │ • Corrects │ │ • OutputGenerator│
│ • Auto-correct │ │ • Saves JSON │ │ • with_vocals.mkv│
│ • Save JSON │ │ │ │ │
│ • NO VIDEO │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
Upload Audio → Parallel Processing → Human Review → Video Generation → Download
│ │
(Modal + AudioShake) (LyricsTranscriber UI)
```
The video must include the human-corrected lyrics, so it can only be generated AFTER review.
## For AI Agents
---
## 🔄 Job State Flow
```
PENDING → DOWNLOADING → [parallel audio+lyrics] → GENERATING_SCREENS
↓
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE
↓
RENDERING_VIDEO (NEW)
↓
AWAITING_INSTRUMENTAL_SELECTION → INSTRUMENTAL_SELECTED
↓
GENERATING_VIDEO → COMPLETE
```
Two human interaction points:
1. **AWAITING_REVIEW** - User corrects lyrics
2. **AWAITING_INSTRUMENTAL_SELECTION** - User chooses audio track
---
## 📞 Getting Help
1. Check `00-current-plan/CURRENT-STATUS.md` first
2. Look for specific guides in `03-deployment/`
3. Review session history in `02-implementation-history/`
### Common Issues
| Issue | Solution |
|-------|----------|
| Emulator not starting | Check ports 8080, 4443 are free |
| "Missing lyrics video" | Review hasn't happened yet - expected behavior |
| Workers not progressing | Check `state_data` in Firestore |
| FFmpeg errors | Ensure FFmpeg is in PATH |
Start with `CLAUDE.md` in the repo root for essential rules, then reference these docs as needed. Keep this README current when project state changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,197 @@
# GitHub Actions Docker Builds - Cost Optimization
**Date:** 2025-12-28
**Problem:** Cloud Build costing $100+/month with slow builds
**Solution:** Move Docker builds to GitHub Actions with registry caching
---
## Overview
This document describes the migration from Google Cloud Build to GitHub Actions for Docker image builds. This change eliminates Cloud Build costs while improving build speeds through better caching.
## Cost Comparison
| Approach | Monthly Cost | Build Speed | Caching |
|----------|-------------|-------------|---------|
| **Cloud Build (N1_HIGHCPU_32)** | ~$100+ | 5-15 min | Limited |
| **GitHub Actions + Registry Cache** | **$0** | 2-5 min | Excellent |
### Why Cloud Build Was Expensive
1. **N1_HIGHCPU_32 pricing**: ~$0.064/build-minute (21x more than default)
2. **No persistent layer cache**: Each build re-downloaded dependencies
3. **Cold start overhead**: Spinning up build VMs added latency
4. **Large dependency tree**: torch, transformers, etc. = ~500 packages
### Why GitHub Actions Is Free
- **Public repositories**: Unlimited build minutes on GitHub-hosted runners
- **Registry cache**: Layers cached in Artifact Registry, persist between builds
- **BuildKit mode=max**: Caches ALL intermediate layers, not just final
---
## Architecture
```
┌─────────────────────────────────────────────────────────────────────────┐
│ NEW BUILD PIPELINE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ GitHub Actions (Free) │
│ ├── docker/setup-buildx-action (BuildKit) │
│ ├── google-github-actions/auth (Workload Identity) │
│ └── docker/build-push-action │
│ ├── cache-from: type=registry,ref=...karaoke-backend:cache │
│ └── cache-to: type=registry,ref=...karaoke-backend:cache,mode=max│
│ │
│ Artifact Registry │
│ ├── karaoke-backend:latest (production image) │
│ ├── karaoke-backend:cache (build cache - ALL layers) │
│ ├── karaoke-backend-base:latest (base image with deps) │
│ └── karaoke-backend-base:cache (base cache) │
│ │
│ Build Times (Expected): │
│ ├── First build (cold): ~8-10 min │
│ ├── Code-only changes: ~2-3 min (cache hit) │
│ └── Dependency changes: ~6-8 min (partial cache hit) │
│ │
└─────────────────────────────────────────────────────────────────────────┘
```
---
## How It Works
### Two-Stage Build Pattern
We maintain the existing two-stage build pattern for efficiency:
1. **Base Image** (`karaoke-backend-base`)
- Contains Python 3.11, ffmpeg, system dependencies
- All pip dependencies pre-installed
- Only rebuilt when `poetry.lock` or `Dockerfile.base` changes
- Hash-based change detection via image labels
2. **App Image** (`karaoke-backend`)
- Built FROM the base image
- Contains only application code
- Fast rebuilds (~2 min) for code-only changes
### Registry Cache
The key optimization is using Artifact Registry as a persistent layer cache:
```yaml
- uses: docker/build-push-action@v6
with:
cache-from: type=registry,ref=.../karaoke-backend:cache
cache-to: type=registry,ref=.../karaoke-backend:cache,mode=max
```
**Benefits:**
- `mode=max` caches ALL layers, not just the final image
- Cache persists between CI runs (unlike GitHub Actions cache which has 10GB limit)
- Layers only re-built when inputs change
---
## Files Changed
### New Files
- **`.dockerignore`** - Reduces build context from ~1GB to ~10MB
### Modified Files
- **`.github/workflows/ci.yml`** - `deploy-backend` job rewritten to use:
- `docker/setup-buildx-action@v3`
- `docker/build-push-action@v6`
- Registry caching to Artifact Registry
### Kept (as fallback)
- **`cloudbuild.yaml`** - Still works for manual triggers
- **`cloudbuild-base.yaml`** - Still works for manual base image builds
---
## Manual Builds (Fallback)
If GitHub Actions fails, you can still use Cloud Build:
```bash
# Build base image (only if dependencies changed)
gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
# Build and deploy app image
gcloud builds submit --config=cloudbuild.yaml --project=nomadkaraoke
```
---
## Monitoring
### Check Build Cache Usage
View cache images in Artifact Registry:
```bash
gcloud artifacts docker images list \
us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo \
--include-tags
```
You should see `*:cache` tags for both base and app images.
### Compare Build Times
- **GitHub Actions**: Check workflow run times in Actions tab
- **Previous Cloud Build**: ~5-15 min per build
- **Expected with cache**: ~2-3 min for code changes
---
## Troubleshooting
### Cache Miss (Slow Build)
If builds are slow despite no dependency changes:
1. Check if cache image exists:
```bash
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:cache
```
2. Force cache rebuild by pushing to `main` with a small change
### Disk Space Issues
GitHub Actions runners have limited disk. If base image builds fail:
1. The workflow includes `jlumbroso/free-disk-space` action
2. It removes Android SDK, Haskell, .NET, etc. to free ~20GB
### Authentication Issues
If `docker push` fails with permission errors:
1. Check Workload Identity is configured correctly
2. Verify service account has `roles/artifactregistry.writer`
---
## Future Improvements
1. **Parallel base/app builds**: If base is unchanged, start app build immediately
2. **GitHub Actions cache fallback**: Use GHA cache when registry is slow
3. **Build metrics**: Track build times and cache hit rates
---
## References
- [Docker Build with GitHub Actions](https://docs.docker.com/build/ci/github-actions/)
- [Registry Cache Backend](https://docs.docker.com/build/cache/backends/registry/)
- [Cache Management](https://docs.docker.com/build/ci/github-actions/cache/)
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,233 @@
# Job Status Display Improvement Plan
**Status:** Planned
**Created:** 2025-12-27
**Priority:** HIGH - User experience improvement
## Problem Statement
Users see generic status messages like "Downloading..." that persist for too long, even after the download is complete and other processing stages are happening. This makes it hard to understand:
1. What stage the job is actually at
2. How much progress has been made
3. How much longer the job might take
## Current Implementation
### Backend Status System
The backend has 28 distinct job statuses organized into 8 processing stages:
```
Stage 1: Setup (PENDING)
Stage 2: Audio Search (SEARCHING_AUDIO, AWAITING_AUDIO_SELECTION, DOWNLOADING_AUDIO)
Stage 3: Download (DOWNLOADING)
Stage 4: Parallel Processing
- Audio: SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE
- Lyrics: TRANSCRIBING → CORRECTING → LYRICS_COMPLETE
Stage 5: Screen Generation (GENERATING_SCREENS, APPLYING_PADDING)
Stage 6: Lyrics Review (AWAITING_REVIEW, IN_REVIEW, REVIEW_COMPLETE) ⚠️ BLOCKING
Stage 7: Video Rendering (RENDERING_VIDEO)
Stage 8: Instrumental Selection (AWAITING_INSTRUMENTAL_SELECTION) ⚠️ BLOCKING
Stage 9: Final Processing (INSTRUMENTAL_SELECTED, GENERATING_VIDEO, ENCODING, PACKAGING)
Stage 10: Distribution (UPLOADING, NOTIFYING, COMPLETE)
```
### Key Findings
1. **Progress field exists but unused**: The `Job` model has a `progress: number` field (0-100) that's returned by the API but NOT displayed in the frontend
2. **Parallel processing complicates status**: Audio and lyrics workers run in parallel, updating `state_data.audio_progress` and `state_data.lyrics_progress` independently
3. **Status labels are static**: Frontend shows "Downloading..." but doesn't reflect that parallel workers are running
4. **10-second polling**: Frontend polls every 10 seconds, so status updates are not real-time
## Proposed Solution
### 1. Step-Based Progress Display
Show a `[X/10] Status...` format that gives users a clear sense of overall progress:
```
[1/10] Setting up...
[2/10] Searching for audio...
[3/10] Downloading audio...
[4/10] Processing audio & lyrics... ← Parallel stage
[5/10] Generating screens...
[6/10] Needs review ← Blocking
[7/10] Rendering video...
[8/10] Select instrumental ← Blocking
[9/10] Encoding final video...
[10/10] Uploading...
```
### 2. Status-to-Step Mapping
```typescript
const STATUS_STEP_MAP: Record<string, { step: number; total: number; label: string }> = {
// Step 1: Setup
pending: { step: 1, total: 10, label: "Setting up" },
// Step 2: Audio Search (optional)
searching_audio: { step: 2, total: 10, label: "Searching for audio" },
awaiting_audio_selection: { step: 2, total: 10, label: "Select audio source" },
// Step 3: Download
downloading_audio: { step: 3, total: 10, label: "Downloading audio" },
downloading: { step: 3, total: 10, label: "Downloading" },
// Step 4: Parallel Processing (Audio + Lyrics)
separating_stage1: { step: 4, total: 10, label: "Separating audio (1/2)" },
separating_stage2: { step: 4, total: 10, label: "Separating audio (2/2)" },
audio_complete: { step: 4, total: 10, label: "Audio ready" },
transcribing: { step: 4, total: 10, label: "Transcribing lyrics" },
correcting: { step: 4, total: 10, label: "Correcting lyrics" },
lyrics_complete: { step: 4, total: 10, label: "Lyrics ready" },
// Step 5: Screen Generation
generating_screens: { step: 5, total: 10, label: "Generating screens" },
applying_padding: { step: 5, total: 10, label: "Syncing countdown" },
// Step 6: Review (BLOCKING)
awaiting_review: { step: 6, total: 10, label: "Needs review" },
in_review: { step: 6, total: 10, label: "In review" },
// Step 7: Video Rendering
review_complete: { step: 7, total: 10, label: "Review complete" },
rendering_video: { step: 7, total: 10, label: "Rendering video" },
// Step 8: Instrumental Selection (BLOCKING)
awaiting_instrumental_selection: { step: 8, total: 10, label: "Select instrumental" },
// Step 9: Final Encoding
instrumental_selected: { step: 9, total: 10, label: "Instrumental selected" },
generating_video: { step: 9, total: 10, label: "Generating final video" },
encoding: { step: 9, total: 10, label: "Encoding video" },
packaging: { step: 9, total: 10, label: "Packaging files" },
// Step 10: Distribution
uploading: { step: 10, total: 10, label: "Uploading" },
notifying: { step: 10, total: 10, label: "Sending notifications" },
complete: { step: 10, total: 10, label: "Complete" },
// Terminal states
failed: { step: 0, total: 10, label: "Failed" },
cancelled: { step: 0, total: 10, label: "Cancelled" },
prep_complete: { step: 10, total: 10, label: "Prep complete" },
};
```
### 3. Enhanced Parallel Processing Display
When audio and lyrics are processing in parallel, show both activities:
```typescript
// Check state_data for parallel progress
if (job.state_data?.audio_progress && job.state_data?.lyrics_progress) {
const audioStatus = job.state_data.audio_progress.stage;
const lyricsStatus = job.state_data.lyrics_progress.stage;
// Show combined status like "Audio: separating, Lyrics: transcribing"
if (audioStatus !== 'audio_complete' || lyricsStatus !== 'lyrics_complete') {
return {
step: 4,
total: 10,
label: buildParallelLabel(audioStatus, lyricsStatus)
};
}
}
```
### 4. Visual Progress Bar (Optional Enhancement)
Add a thin progress bar below the status text:
```tsx
<div className="w-full h-1 bg-slate-700 rounded mt-1">
<div
className="h-full bg-blue-500 rounded transition-all"
style={{ width: `${(step / total) * 100}%` }}
/>
</div>
```
## Implementation Steps
### Phase 1: Core Changes (MVP)
1. **Create status utility** (`frontend/lib/job-status.ts`):
- Define `STATUS_STEP_MAP` constant
- Create `getJobStep(job: Job)` function
- Create `formatStepStatus(step, total, label)` function
2. **Update JobCard component**:
- Import status utility
- Replace static `StatusText` with step-based format
- Keep existing color coding from `statusConfig`
3. **Add unit tests** (`frontend/__tests__/job-status.test.ts`):
- Test all status mappings
- Test parallel processing detection
- Test edge cases (unknown status, null values)
### Phase 2: Enhanced Parallel Processing (Optional)
1. **Add parallel progress detection**:
- Check `state_data.audio_progress` and `state_data.lyrics_progress`
- Build combined label showing both activities
2. **Add visual progress bar**:
- Thin progress indicator based on step number
- Smooth transitions between steps
### Phase 3: Backend Improvements (Future)
1. **Add estimated time remaining**:
- Track average duration per stage from historical data
- Return ETA in API response
2. **WebSocket for real-time updates**:
- Replace polling with push notifications
- Smoother progress updates during long operations
## Files to Modify
| File | Changes |
|------|---------|
| `frontend/lib/job-status.ts` | NEW: Status utility functions |
| `frontend/components/job/JobCard.tsx` | Update status display |
| `frontend/__tests__/job-status.test.ts` | NEW: Unit tests |
| `frontend/lib/api.ts` | Add `state_data` types (if needed) |
## Testing Plan
1. **Unit tests**:
- All status mappings return correct step/label
- Unknown statuses handled gracefully
- Parallel progress detection works
2. **Integration tests** (E2E):
- Verify status displays correctly at each stage
- Verify step numbers increment as job progresses
3. **Manual testing**:
- Submit a job and observe status progression
- Verify parallel processing shows meaningful status
## Success Criteria
- [ ] All statuses show `[X/10] Label` format
- [ ] Step numbers progress logically (never decrease except on retry)
- [ ] Parallel processing shows combined activity
- [ ] Blocking statuses clearly indicated for user action
- [ ] All existing tests pass
- [ ] New unit tests have >80% coverage
## Rollback Plan
The changes are frontend-only and purely cosmetic. If issues arise:
1. Revert the JobCard changes to use the old `statusConfig` approach
2. The backend API remains unchanged
## Notes
- The 10-step system is a simplification of the 28 backend statuses for user comprehension
- Parallel processing (step 4) groups multiple backend statuses into one user-facing step
- Blocking steps (6, 8) require user action and should be visually distinct
- The `progress` field from backend could be used for sub-step progress bars in the future
Loaded 100 of 160 files, more files were not shown because too many files have changed in this diff. Show more