Compare commits

...
Author SHA1 Message Date
Andrew Beveridge e333fb2ae0 Removed flacfetch submodule 2025-12-23 00:31:51 -05:00
Andrew Beveridge 3dd2ee4a86 Switch back to pypi flacfetch 2025-12-23 00:23:24 -05:00
Andrew Beveridge 70b1410c0e infra: add missing ops-api-url secret
Version: 0.75.23
2025-12-23 00:10:34 -05:00
Andrew Beveridge 067052448f infra: grant flacfetch repo access to Workload Identity and Compute SSH permissions
- Add nomadkaraoke/flacfetch to workload identity user binding
- Grant compute.instanceAdmin.v1 for VM management
- Grant compute.osLogin for SSH access
- Grant iap.tunnelResourceAccessor for secure SSH tunneling

This fixes the 'iam.serviceAccounts.getAccessToken' permission denied error
in flacfetch's deploy-to-gcp CI job.

Version: 0.75.22
2025-12-22 23:54:21 -05:00
Andrew Beveridge a5bca8781a Updated poetry lockfile 2025-12-22 23:48:49 -05:00
Andrew Beveridge 7d1b9ce960 fix: use local flacfetch submodule for builds and CI
- Update pyproject.toml to use path dependency for flacfetch
- Update CI workflow to checkout submodules for package tests
- Bump version to 0.75.21
2025-12-22 23:42:54 -05:00
Andrew Beveridge 311e0a4ad2 refactor: update to use RED naming and env var URLs for private trackers
Updates to align with flacfetch 0.8.0 breaking changes:
- Renamed "Redacted" to "RED" throughout codebase
- Updated REDACTED_API_KEY references to RED_API_KEY
- Updated redacted_api_key parameter to red_api_key
- Added RED_API_URL and OPS_API_URL env var requirements
- Updated infrastructure config for new environment variables
- Updated all tests and documentation

This improves security by requiring private tracker URLs to be
configured via environment variables rather than hardcoded.

Version: 0.75.20
2025-12-22 23:30:19 -05:00
Andrew Beveridge 70977f4807 Bump for deploy 2025-12-22 17:41:31 -05:00
Andrew Beveridge 11c8d8d58e Disable codecov on main 2025-12-22 17:40:42 -05:00
Andrew Beveridge ☄️ c15673b27e Merge pull request #47 from nomadkaraoke/feature/review-auth-tokens
feat: Job-scoped review tokens for lyrics review authentication
2025-12-22 17:29:38 -05:00
Andrew Beveridge 913efcf72a fix: Fix test fixtures and auth headers for backend tests
- Remove mock_auth_service fixture dependency from test_internal_api.py
  and test_upload_api.py (now using autouse mock_auth_dependency from conftest)
- Add upload_auth_headers fixture for file upload tests (no Content-Type header
  to avoid breaking multipart form data)
- Add service._remote_client = None to audio search tests to prevent hitting
  remote flacfetch service
- Add proper skipif decorator for TestRemoteCLIStyleParsing when flacfetch
  module is not properly installed
- Add headers=auth_headers to emulator integration test for annotations endpoint
- Fix headers=upload_auth_headers usage in all file upload tests
2025-12-22 17:19:17 -05:00
Andrew Beveridge ad0a6c3475 feat: Add instrumental review token authentication
- Add instrumental_token and instrumental_token_expires_at to Job model
- Generate instrumental token when entering AWAITING_INSTRUMENTAL_SELECTION
- Create require_instrumental_auth dependency for instrumental endpoints
- Update instrumental endpoints to use require_instrumental_auth:
  - GET /instrumental-options
  - GET /instrumental-analysis
  - GET /audio-stream/{stem_type}
  - POST /create-custom-instrumental
  - GET /waveform-data
  - POST /select-instrumental
- Update remote CLI to build correct instrumental review URL with token
- Modify instrumental review HTML to:
  - Accept baseApiUrl and instrumentalToken from URL params
  - Use cloud API endpoints when in cloud mode
  - Include token in all API requests
- Add CI step to deploy instrumental UI to gen.nomadkaraoke.com/instrumental/
- Add mock for require_instrumental_auth in test fixtures

The instrumental review UI is now accessible at:
gen.nomadkaraoke.com/instrumental/?baseApiUrl=...&instrumentalToken=...
2025-12-22 16:31:48 -05:00
Andrew Beveridge cd15024c42 feat: Add job-scoped review tokens for lyrics review auth
- Add review_token and review_token_expires_at fields to Job model
- Create require_review_auth dependency that accepts either:
  - Full user authentication (admin/user token)
  - Job-specific review token (48-hour expiry)
- Generate review token automatically when job enters AWAITING_REVIEW
- Update all review endpoints to use require_review_auth
- Update remote CLI to include reviewToken in review UI URL
- Update lyrics review frontend to extract and use token from URL
- Add lyrics frontend deployment to gen.nomadkaraoke.com/lyrics/
- Update default review UI URL to gen.nomadkaraoke.com/lyrics
- Fix test auth fixtures and add auth_headers where missing

This enables the lyrics review UI to authenticate without requiring
users to manually enter tokens, while maintaining security through
job-scoped tokens with automatic expiration.
2025-12-22 16:20:10 -05:00
Andrew Beveridge ☄️ 9906d4bb63 Merge pull request #46 from nomadkaraoke/feature/instrumental-review-original-audio-tab
Add Original Audio tab to Instrumental Review UI
2025-12-22 14:56:37 -05:00
Andrew Beveridge 9e28f7bd99 Merge branch 'main' into feature/instrumental-review-original-audio-tab
Resolved conflict in pyproject.toml:
- Updated version to 0.75.16 (after main's 0.75.15)
2025-12-22 14:38:09 -05:00
Andrew Beveridge ☄️ 3ca62aa555 Merge pull request #45 from nomadkaraoke/security/require-auth-all-endpoints
security: Require authentication for all major backend API endpoints
2025-12-22 14:36:12 -05:00
Andrew Beveridge 8b808b0bf0 tests: fix missing auth headers in emulator review endpoint tests
Fixed 2 failing emulator tests that were missing auth_headers:

1. test_review_correction_data_wrong_status
   - GET /api/review/{job_id}/correction-data now includes headers=auth_headers

2. test_review_preview_video_stub
   - POST /api/review/{job_id}/preview-video now includes headers=auth_headers

These tests were getting 401 Unauthorized instead of expected 400 Bad Request
because review endpoints require authentication.

Verified locally: All 27 emulator tests now pass
  poetry run pytest backend/tests/emulator/ -v
  ✅ 27 passed, 341 warnings in 8.79s
2025-12-22 14:23:16 -05:00
Andrew Beveridge 6985ab31b3 tests: fix auth mock to properly skip emulator tests
The mock_auth_dependency fixture was interfering with emulator tests.

Changes:
1. Check for FIRESTORE_EMULATOR_HOST env var to detect emulator environment
2. Skip both 'emulator' and 'integration' in test path
3. This allows emulator tests to use REAL auth with ADMIN_TOKENS

Emulator tests need real auth because they set:
  ADMIN_TOKENS="test-admin-token"

And call endpoints with:
  Authorization: Bearer test-admin-token

The real AuthService should validate this token.
2025-12-22 14:08:14 -05:00
Andrew Beveridge 706e414e36 chore: remove temporary test debugging files 2025-12-22 13:56:52 -05:00
Andrew Beveridge 5ace8b0575 tests: fix auth mocking for all unit tests
Fixed authentication in unit tests after adding require_auth to 33+ endpoints:

1. Added missing imports (Depends, Tuple) to route files:
   - file_upload.py, audio_search.py, review.py

2. Updated conftest.py with proper auth mocking:
   - Use FastAPI's dependency_overrides system
   - Mock require_auth and require_admin to return valid admin credentials
   - Apply automatically to all non-emulator tests

3. Auto-fixed 6 test files to include auth_headers:
   - test_jobs_api.py
   - test_upload_api.py
   - test_api_routes.py
   - test_internal_api.py
   - test_emulator_integration.py (both copies)

4. Fixed test client fixtures:
   - Removed mock_auth_service parameter (replaced by global fixture)
   - Tests now use dependency_overrides instead of patching

Tests verified passing locally for test_jobs_api.py.
Let CI run the full test suite to verify all tests pass.

Related: PR #45
2025-12-22 13:56:23 -05:00
Andrew Beveridge bc7d1a2308 tests: add mock auth service and documentation for test fixes
Added mock_auth_service and auth_headers fixtures to conftest.py to support
testing authenticated endpoints.

The tests are currently failing because they don't include authentication
headers in their requests. This commit:

1. Adds global test fixtures for mock authentication
2. Documents the remaining test fixes needed in TEST-FIXES-NEEDED.md

Remaining work: Update 60+ test methods across multiple test files to:
- Add `auth_headers` parameter to test methods
- Pass `headers=auth_headers` to all client.get/post/etc() calls

See docs/TEST-FIXES-NEEDED.md for detailed instructions and an auto-fix script.

Related: PR #45
2025-12-22 13:31:50 -05:00
Andrew Beveridge 1741470ddb feat: add Original Audio tab to Instrumental Review UI
- Added original audio path detection in gen_cli.py
- Pass original_audio_path to InstrumentalReviewServer
- Updated test-instrumental-review.py script with --original argument
- Added tests for original audio functionality
- Bumped version to 0.75.12

The Instrumental Review UI now shows an 'Original Audio' tab when the
original audio file is available, allowing users to compare the full
song against backing vocals and instrumentals when deciding which
regions to mute.

Closes #[issue_number] (if applicable)
2025-12-22 13:24:34 -05:00
Andrew Beveridge ee9ea7c0c5 security: require authentication for all major backend API endpoints
**Critical Security Fix**

Previously, most backend API endpoints were publicly accessible without any
authentication, allowing anyone to:
- List all jobs
- View job details
- Create new jobs
- Cancel/delete jobs
- Access job outputs

This commit adds `Depends(require_auth)` to 33+ endpoints across the backend.

## Endpoints Secured (33+)

### jobs.py (20 endpoints) ✅
- Job CRUD: create, get, list, delete, bulk_delete
- Review flow: get_review_data, start_review, submit_corrections, complete_review
- Instrumental: get_instrumental_options, get_instrumental_analysis, stream_audio,
  create_custom_instrumental, get_waveform_data, select_instrumental
- Downloads: get_download_urls, download_file
- Management: cancel_job, retry_job, get_worker_logs

### file_upload.py (6 endpoints) ✅
- upload_and_create_job - Direct file upload
- create_job_with_upload_urls - Signed URL flow (step 1)
- mark_uploads_complete - Signed URL flow (step 2)
- create_job_from_url - YouTube/URL submission
- create_finalise_only_job - Prep-only continuation
- mark_finalise_uploads_complete - Finalise uploads validation

### audio_search.py (3 endpoints) ✅
- search_audio - Search by artist/title
- get_audio_search_results - Get search results
- select_audio_source - Select and download audio

### review.py (4/12 endpoints) ⚠️
- get_correction_data - Get lyrics for review
- complete_review - Submit corrected lyrics
- add_lyrics - Add custom lyrics source
- generate_preview_video - Generate preview

## Remaining Work

See `docs/SECURITY-AUTH-TODO.md` for:
- Remaining review.py endpoints (audio streaming, handlers, annotations)
- auth.py endpoints (OAuth credential management - needs admin auth)
- internal.py endpoints (worker triggers - CRITICAL, needs admin auth)
- health.py detailed endpoint

## Frontend Impact

The frontend already has `AuthBanner` component for token input and stores
tokens in localStorage. All API calls in `frontend/lib/api.ts` include the
Authorization header when a token is present.

**Users must now provide a valid authentication token to use the API.**

## Migration Notes

- Health checks (`/health`, `/readiness`) remain unauthenticated for k8s/monitoring
- Internal worker endpoints currently check `X-Admin-Token` but should use formal
  `Depends(require_admin)` - tracked in SECURITY-AUTH-TODO.md
- Review UI audio streaming may need query param token support

Closes: Part of authentication enforcement initiative
2025-12-22 12:01:45 -05:00
Andrew Beveridge 91b2421c37 docs: merge frontend guide documentation 2025-12-22 11:43:11 -05:00
Andrew Beveridge 06748776e3 docs: add comprehensive frontend guide
- Complete overview of frontend architecture and deployment
- Authentication system documentation
- All features explained (job submission, monitoring, interactive elements)
- Component structure and key files
- Local development guide
- Deployment process and CI/CD
- Common issues and troubleshooting
- API integration patterns
- Future enhancement ideas

This provides a complete reference for understanding and working with the frontend.
2025-12-22 11:42:48 -05:00
Andrew Beveridge ☄️ 91f51d6a8c feat: Match frontend styling to instrumental review UI and add authentication
- Updated color scheme to match instrumental review UI (neutral dark theme with blue accents)
- Replaced Geist fonts with system font stack for faster loading
- Fixed API URL from gen.nomadkaraoke.com to api.nomadkaraoke.com
- Added authentication support with token management (localStorage)
- Created AuthBanner component for token input
- All API requests now include Authorization header when authenticated

This resolves frontend API 404 errors and provides consistent styling across all karaoke-gen UIs.
2025-12-22 11:41:25 -05:00
Andrew Beveridge f2e2617e43 fix: update frontend to use correct API URL (api.nomadkaraoke.com) and add authentication support
- Changed API_BASE_URL from gen.nomadkaraoke.com to api.nomadkaraoke.com
- Added token management (localStorage)
- Created AuthBanner component for token input
- All API calls now include Authorization header when token is present
- Fixed 404 errors - gen.nomadkaraoke.com serves frontend, api.nomadkaraoke.com is backend

This resolves the issue where the frontend was trying to call API endpoints on the frontend domain instead of the dedicated backend API domain.
2025-12-22 11:26:43 -05:00
Andrew Beveridge 198ffc2469 feat: match frontend styling to instrumental review UI
- Update color scheme to match instrumental review UI dark theme
- Replace Geist fonts with system font stack (-apple-system, BlinkMacSystemFont, etc.)
- Use #0f0f0f background, #1a1a1a cards, #3b82f6 primary blue
- Maintain Tailwind compatibility with CSS variable mappings
- Remove sidebar color variables (not used in this UI)

This creates a more consistent visual experience across all karaoke-gen UIs.
2025-12-22 10:59:27 -05:00
Andrew Beveridge 0a03a93948 fix: Add frontend/lib directory to git (was ignored)
- Update .gitignore to exclude Python lib/ but include frontend/lib/
- Add all frontend lib files: api.ts, types.ts, utils.ts, auth.ts, jobs.ts
- Add API client tests
- Verified build succeeds locally before commit
2025-12-22 10:44:13 -05:00
Andrew Beveridge 501c301f4f fix: Add legacy-peer-deps flag for npm ci in frontend deployment
- Update package-lock.json with testing library dependencies
- React 19 compatibility requires legacy-peer-deps flag
- Fixes frontend deployment CI failure
2025-12-22 10:27:46 -05:00
Andrew Beveridge ☄️ 867a519f6c feat: Add comprehensive modular frontend with full interactive features
- Modular component architecture with JobCard, JobActions, JobLogs, OutputLinks, InstrumentalSelector, AudioSearchDialog, and JobSubmission
- API enhancements for audio search, job logs, and retry functionality
- Interactive features: lyrics review, audio search, instrumental selection, job logs, output links
- Testing infrastructure with Jest and React Testing Library
- All CodeRabbit review comments addressed
- Clean architecture with components <200 lines each
2025-12-22 10:18:29 -05:00
Andrew Beveridge d5db0c955d docs(monitoring): add comprehensive debugging guide to Job Inspector
Adds collapsible guide sections with:
- Job pipeline flow explanation with all worker stages
- Step-by-step debugging walkthrough
- Common issues and their causes
- Useful gcloud CLI commands with copy buttons
- Log patterns to look for (success, auth failure, timeout, etc.)

Guide state (open/closed) is persisted to localStorage.
2025-12-22 02:50:53 -05:00
Andrew Beveridge 6a371a01a9 feat(monitoring): add Google account selector to Job Inspector
Adds authuser parameter to all GCP console URLs. Users with multiple
Google accounts can select which account to use (0-5), defaulting to 1.
Selection is saved to localStorage for persistence.
2025-12-22 02:48:44 -05:00
Andrew Beveridge 4d771c9db1 fix(auth): use X-Admin-Token header for Cloud Tasks auth
Cloud Tasks OIDC token overwrites the Authorization header when oidc_token
is specified in the task. This fix uses a custom X-Admin-Token header for
application-level authentication, while keeping OIDC for Cloud Run auth.

Also improves Job Inspector query to search both JSON and text log formats.
2025-12-22 02:47:28 -05:00
Andrew Beveridge 5e46cf57fb fix(infra): fix Pulumi alert policy API typos
- AlertPolicyConditionConditionAbsenceArgs -> AlertPolicyConditionConditionAbsentArgs
- autoClose minimum is 30 minutes (1800s), not 10 minutes
2025-12-22 02:15:53 -05:00
Andrew Beveridge ed3e6ae039 fix(infra): add Cloud Build IAM permission to deploy as backend SA
Cloud Build compute service account needs iam.serviceAccountUser role on
the karaoke-backend service account to deploy Cloud Run services.

Without this, Cloud Build fails with:
  PERMISSION_DENIED: Permission 'iam.serviceaccounts.actAs' denied on
  service account karaoke-backend@nomadkaraoke.iam.gserviceaccount.com
2025-12-22 02:13:12 -05:00
Andrew Beveridge 4fe7d84789 Bump version for re-deployment (retry after transient CI failure) 2025-12-22 02:11:38 -05:00
Andrew Beveridge fdd84bad36 Bump version for release 2025-12-22 01:52:14 -05:00
Andrew Beveridge f52964441a fix: correct indentation in video_worker.py breaking CI
Two code blocks had incorrect indentation (16 spaces instead of 12),
causing an IndentationError that broke emulator integration tests.
2025-12-22 01:49:08 -05:00
Andrew Beveridge 92affebded feat(monitoring): add Job Inspector HTML page with simple job ID input 2025-12-22 01:37:52 -05:00
Andrew Beveridge 3221c2626b BUmp version for release with observability 2025-12-22 01:35:18 -05:00
Andrew Beveridge 2ea21b0e51 docs(monitoring): add Quick Start guide for job debugging 2025-12-22 01:30:21 -05:00
Andrew Beveridge 76fa4befd8 fix(monitoring): use ALIGN_PERCENTILE for CPU/Memory metrics, improve log filter 2025-12-22 01:26:44 -05:00
Andrew Beveridge 583dd323dc fix(monitoring): remove $schema field from dashboard JSON 2025-12-22 01:20:41 -05:00
Andrew Beveridge ef2ecb4c55 feat(instrumental-review): auto-find available port and auto-close browser
- Add automatic port finding to support concurrent CLI instances
  - If port 8765 is in use, tries 8766, 8767, etc.
  - Prevents "address already in use" errors when running multiple jobs
- Auto-close browser window 2 seconds after selection submission
  - Shows countdown "Closing in 2s..." before closing
  - Falls back to manual close message if browser blocks window.close()
- Add tests for port finding functionality
2025-12-22 01:15:51 -05:00
Andrew Beveridge 6d28e09fce feat(observability): implement full observability plan
Phase 1: Core Tracing
- Add job_span helper with automatic job_id attribute
- Propagate trace context through Cloud Tasks headers
- Extract trace context in worker endpoints

Phase 2: Worker Instrumentation
- Add detailed nested spans to all workers (audio, lyrics, screens, render, video)
- Log WORKER_START/WORKER_END with timing and status
- Track external API calls (Modal, AudioShake)

Phase 3: Structured Logging
- Add StructuredFormatter for JSON logs with trace correlation
- Auto-configure based on environment (Cloud Run vs local)
- Add JobLogAdapter for automatic job context

Phase 4: Custom Metrics
- Add MetricsService for job, worker, GCS, and external API metrics
- Emit metrics as structured log entries for log-based metrics
- Add time_external_api context manager for API timing

Phase 5: Dashboard & Alerting
- Add Cloud Monitoring dashboard with request, latency, queue metrics
- Add alert policies for error rate, queue backlog, memory, availability
- Add monitoring README with setup instructions

Files created:
- backend/services/structured_logging.py
- backend/services/metrics.py
- infrastructure/monitoring/dashboard.json
- infrastructure/monitoring/README.md
2025-12-22 01:05:03 -05:00
Andrew Beveridge 565586aca4 Added Cloud Tasks service account, plan for observability 2025-12-22 00:45:24 -05:00
Andrew Beveridge 9147e2f21b fix: pass gcs_path to remote download to fix FileNotFoundError
The _download_and_start_processing function wasn't passing gcs_path
to audio_search_service.download(), causing the remote flacfetch VM
to download locally without uploading to GCS. The backend then tried
to open a VM-local path that doesn't exist on Cloud Run.

- Add gcs_path parameter to download call for torrent providers
- Add 4 API route tests for download orchestration (TestAudioSearchApiRouteDownload)
- Update TESTING-GAPS.md with analysis of why this bug wasn't caught
- Document new testing pattern: test API route orchestration, not just services
2025-12-22 00:01:10 -05:00
Andrew Beveridge 9ad20e3a36 Updated dependencies, bumped version 2025-12-21 23:41:58 -05:00
Andrew Beveridge 85a7f62b01 Add comprehensive tests for backend services
New test files:
- test_storage_service.py (17 tests) - GCS operations
- test_audio_analysis_service.py (9 tests) - waveform analysis
- test_audio_editing_service.py (13 tests) - mute regions, custom instrumentals
- test_youtube_service.py (12 tests) - credential management
- test_dropbox_service.py (14 tests) - OAuth, file upload, shared links
- test_gdrive_service.py (13 tests) - OAuth, folder creation, file upload

Documentation:
- docs/00-current-plan/TESTING-GAPS.md - testing status and implementation plan

Total: 78 new tests covering previously untested backend services
2025-12-21 23:35:17 -05:00
Andrew Beveridge 09f9e15ff0 test: add comprehensive tests for FlacfetchClient (26 tests)
Tests cover:
- FlacfetchClient initialization and header formatting
- health_check() success, degraded, unhealthy, and network error cases
- search() with results and no results
- download() with GCS path and custom filename
- get_download_status() success and not found
- wait_for_download() complete, seeding, failed, cancelled, timeout
- Progress callback invocation
- get_flacfetch_client() singleton factory behavior

This fills a critical testing gap identified during the extra_info bug fix.
2025-12-21 23:23:36 -05:00
Andrew Beveridge dd62d30fa9 fix: remove extra_info attribute access bug in audio download path
The download() method was trying to access result.extra_info which doesn't
exist on AudioSearchResult. The fix simplifies the remote routing logic
to just check provider type (Redacted/OPS).

Added 7 new tests for remote download path coverage:
- test_download_uses_remote_for_redacted_provider
- test_download_uses_remote_for_ops_provider
- test_download_uses_local_for_youtube_even_with_remote_client
- test_download_uses_local_when_no_remote_client
- test_download_uses_local_when_no_remote_search_id
- test_torrent_provider_routing_checks_both_conditions
- test_download_does_not_access_nonexistent_attributes

Bumps version to 0.74.2
2025-12-21 23:21:04 -05:00
Andrew Beveridge adfd3de651 docs: update TORRENT-SERVICE-PLAN.md with implementation status
- Mark all phases and success criteria as complete
- Document deployment automation (VM auto-deploy, version-gated backend)
- List all key files modified in both repos
- Add testing commands and known issues
- Update timestamps and status to Complete
2025-12-21 23:12:31 -05:00
Andrew Beveridge 871e97cae7 feat: version-gated backend deployments with verification
- Only deploy backend when version changes (check if tag exists)
- Skip deployment if version already deployed, with helpful message
- Verify deployed version matches expected version after deployment
- Retry version check with backoff to handle Cloud Run routing delays
- Bump version to 0.74.1 to trigger deployment
2025-12-21 23:08:55 -05:00
Andrew Beveridge 03e1aec641 fix: resolve nest_asyncio compatibility with uvloop in Cloud Run
- Use --loop asyncio in uvicorn CMD to avoid uvloop incompatibility
- Apply nest_asyncio only when running inside an async context
- Fix AudioSearchResult field mapping for remote flacfetch responses
- Make url field optional in AudioSearchResultResponse for remote results

Tested locally with Docker - search returns 39 results via remote flacfetch
2025-12-21 22:52:55 -05:00
Andrew Beveridge 40948c2ada chore: update poetry.lock for nest-asyncio 2025-12-21 21:59:08 -05:00
Andrew Beveridge 2182bf7e58 fix: add nest_asyncio to fix event loop conflict in audio search
The remote flacfetch client uses async methods but is called from sync
code within FastAPI's event loop. This caused 'Cannot run the event
loop while another loop is running' errors.

nest_asyncio allows nested event loops, fixing the conflict.
2025-12-21 21:54:06 -05:00
Andrew Beveridge ed723dd6d0 Merge branch 'feature/flacfetch-http-api' 2025-12-21 21:33:41 -05:00
Andrew Beveridge ddc4460065 chore: update flacfetch submodule to v0.7.3 2025-12-21 21:31:40 -05:00
Andrew Beveridge b0f3d15112 feat: configure flacfetch remote service via secrets
- Add flacfetch-api-url secret to infrastructure (stores VM IP as secret)
- Add FLACFETCH_API_URL and FLACFETCH_API_KEY to Cloud Run deployment
- Both values pulled from Secret Manager (no IPs in repo)
2025-12-21 21:31:35 -05:00
Andrew Beveridge ☄️ 213d3b4311 Merge pull request #41 from nomadkaraoke/feature/flacfetch-http-api
feat: Add flacfetch HTTP API integration for torrent downloads
2025-12-21 19:18:15 -05:00
Andrew Beveridge 753499db01 chore: Update flacfetch submodule to feature/http-api branch
Updates submodule to include HTTP API implementation.
2025-12-20 17:38:38 -05:00
Andrew Beveridge 1b6e682860 feat: Add flacfetch HTTP API for torrent downloads
This commit implements a dedicated flacfetch HTTP API service to handle
torrent downloads from private trackers (Redacted, OPS), replacing the
non-functional Transmission daemon in Cloud Run.

## Flacfetch API (v0.5.0)
- Add FastAPI-based HTTP API module (flacfetch/api/)
- Implement endpoints: /search, /download, /torrents, /health
- Add API key authentication (X-API-Key header)
- Add download manager with progress tracking and GCS uploads
- Add disk manager with automatic cleanup by oldest/largest/ratio
- Add keep_seeding mode to TorrentDownloader
- Add 'serve' CLI subcommand for running API server
- Add [api] optional dependencies (FastAPI, Uvicorn, httpx, GCS)
- Add comprehensive unit tests (158 tests, 57% coverage)

## Infrastructure (Pulumi)
- Add flacfetch VM (e2-small) with static IP for tracker whitelist
- Add firewall rules for BitTorrent (51413) and API (8080)
- Add service account with GCS and Secret Manager permissions
- Add startup script for automated deployment

## Backend Integration
- Add FlacfetchClient for remote API communication
- Update AudioSearchService to use remote flacfetch when configured
- Add FLACFETCH_API_URL and FLACFETCH_API_KEY config
- Add flacfetch health check to /health/detailed endpoint

## Backend Cleanup
- Remove Transmission daemon from Dockerfile.base
- Remove startup script with transmission-daemon
- Simplify CMD to direct uvicorn invocation

Closes #torrent-service-implementation
2025-12-20 17:37:38 -05:00
Andrew Beveridge ☄️ 9baba12280 Merge pull request #40 from nomadkaraoke/feat/interruptible-download-with-cleanup
feat: Add interruptible download with Ctrl+C support and torrent cleanup
2025-12-20 15:06:41 -05:00
Andrew Beveridge 24b7b75ab2 feat: Add torrent service plan and improve test coverage to 70%+
- Add docs/00-current-plan/TORRENT-SERVICE-PLAN.md with detailed implementation plan
- Add 8 tests for JobMonitor._show_download_progress()
- Add test for _check_interrupt() function
- Add test for AudioSearchResult.to_dict() with full Release fields
- Add test for interactive_select fallback on CLIHandler error
- Add test coverage for AudioSearchResult.from_dict() with seeders/target_file
- Coverage increased from 69.51% to 70.15%
2025-12-20 14:57:03 -05:00
Andrew Beveridge 0b59b7abfc feat: Add verbose download progress to remote CLI
- Remote CLI now shows detailed torrent progress during download_audio status
- Health endpoint returns detailed torrent info (progress, peers, speed, stalled)
- CLI polls health endpoint every 5s during download and shows:
  - Progress percentage
  - Download speed
  - Peer count
  - Stalled warning when no peers connected
- Helps diagnose issues like Cloud Run torrent connectivity problems
2025-12-20 14:33:47 -05:00
Andrew Beveridge 942cbb4f66 feat: Add interruptible download with Ctrl+C support and torrent cleanup
- Wrap torrent downloads in _interruptible_download() to handle Ctrl+C
- Check for interrupt every 200ms during download
- Clean up incomplete torrents from Transmission when cancelled
- Add _cleanup_transmission_torrents() to remove matching incomplete torrents
- Fix empty filepath check (use 'not filepath' instead of 'is None')
- Add BASE_IMAGE arg to Dockerfile for local testing
- Add comprehensive tests for interruptible download and cleanup (10 new tests)
2025-12-20 14:25:48 -05:00
Andrew Beveridge 96383a60d4 Deleted flacfetch code from this repo 2025-12-20 13:45:05 -05:00
Andrew Beveridge 0cf14be531 fix: Rebuild base image when Dockerfile.base changes (not just poetry.lock)
CI was only checking poetry.lock hash to decide if base image needs rebuild.
Now it checks a combined hash of poetry.lock + Dockerfile.base, so changes
to the Dockerfile (like adding Transmission) will trigger a rebuild.

- Update CI to hash both files
- Update cloudbuild-base.yaml to use new combined hash
- Update Dockerfile.base label to base.content.hash
- Bump version to 0.73.3
2025-12-20 13:44:36 -05:00
Andrew Beveridge 95edd82555 Merge branch 'backend-parity-fixes-one' 2025-12-20 13:39:11 -05:00
Andrew Beveridge 6e02ca3056 feat: Add Transmission daemon for tracker downloads on Cloud Run
- Install transmission-daemon in base Docker image
- Add startup script to launch Transmission before uvicorn
- Add _check_transmission_available() to only enable tracker providers
  when Transmission is actually available (graceful fallback to YouTube)
- Add /api/health/detailed endpoint to check Transmission status
- Add comprehensive tests for Transmission availability checking
- Bump version to 0.73.2
2025-12-20 13:38:24 -05:00
Andrew Beveridge ☄️ 966e8cc963 Merge pull request #38 from nomadkaraoke/backend-parity-fixes-one
feat: Add flacfetch API keys to cloud backend and improve audio selection display
2025-12-20 03:28:04 -05:00
Andrew Beveridge 04b1c8fd3e fix: Fix failing tests after flacfetch refactor
- Add create_test_release helper to create real Release objects for tests
- Update interactive_select tests to use real Release objects instead of MagicMocks
  (flacfetch's print_releases does type comparisons that fail with MagicMock)
- Update test assertions to match flacfetch's CLIHandler output messages
- Update test_cli_handler_works_with_real_releases to use format_release_line
2025-12-20 03:18:28 -05:00
Andrew Beveridge f47b126c08 fix: Address CodeRabbit review comments
- Fix undefined print_search_results in _basic_interactive_select (use flacfetch.print_releases)
- Remove unused title variable in handle_audio_selection
2025-12-20 03:07:36 -05:00
Andrew Beveridge 4438293b6d chore: Remove unnecessary backwards compatibility wrappers 2025-12-20 02:59:46 -05:00
Andrew Beveridge de4e92d9bf Updated flacfetch 2025-12-20 02:53:11 -05:00
Andrew Beveridge e7f88b60d0 refactor: Use flacfetch's shared display functions for audio selection
- Updated karaoke-gen to use flacfetch's print_releases and format_release_line
- Remote CLI now converts API results to Release-compatible dicts for display
- Backend API now returns full Release data for rich display on remote CLI
- Simplified audio_fetcher.py to wrap flacfetch functions instead of duplicating
- This ensures consistent, colorized output between local and remote CLIs
2025-12-20 02:38:09 -05:00
Andrew Beveridge fbb1cd7299 feat: Add flacfetch API keys to cloud backend and improve audio selection display
- Add REDACTED_API_KEY and OPS_API_KEY secrets to infrastructure (Pulumi)
- Update cloudbuild.yaml to pass secrets as env vars to Cloud Run
- Add seeders and target_file fields to AudioSearchResult
- Create shared format_search_result_line() and print_search_results() functions
- Update remote CLI to use shared formatting (consistent with local CLI)
- Add artist/title to /results endpoint response for header display
- Color-coded providers, seeder counts, and quality in selection display

This enables flacfetch audio search to work in cloud backend (search Redacted/OPS
trackers, not just YouTube) and provides rich, informative output matching local CLI.
2025-12-20 02:14:25 -05:00
Andrew Beveridge ☄️ 77820a44c9 Merge pull request #37 from nomadkaraoke/update-deps
Update deps
2025-12-20 02:02:50 -05:00
Andrew Beveridge e92ddd8609 Updated dependencies, bumped version 2025-12-20 01:50:27 -05:00
Andrew Beveridge a2bf2230ab Merge branch 'main' of github.com:nomadkaraoke/karaoke-gen 2025-12-20 01:49:33 -05:00
Andrew Beveridge 75c6879b7e Added codecov, bumped for release 2025-12-20 01:49:01 -05:00
Andrew Beveridge ☄️ f51ef68179 Merge pull request #36 from nomadkaraoke/refactor/unify-audio-fetcher
refactor: Unify audio fetcher code between local CLI and backend
2025-12-20 01:33:44 -05:00
Andrew Beveridge 8a5420f5f8 test: Add comprehensive tests for audio_fetcher new features
Add tests for:
- AudioSearchResult.to_dict() and from_dict() methods
- AudioSearchResult.index field
- AudioFetchResult.to_dict() and from_dict() methods
- FlacFetchAudioFetcher.select_best() method
- FlacFetcher alias
- search() setting index on results
- Roundtrip serialization tests
- Edge cases (empty lists, missing fields, exception handling)

These tests improve coverage for the code sharing between
local CLI and cloud backend.
2025-12-20 01:23:27 -05:00
Andrew Beveridge c9b7f1bb1a refactor: Unify audio fetcher code between local CLI and backend
This eliminates code duplication between karaoke_gen/audio_fetcher.py
and backend/services/audio_search_service.py.

Changes:
- Add to_dict()/from_dict() methods to AudioSearchResult and AudioFetchResult
- Add select_best() standalone method to FlacFetcher
- Add index field to AudioSearchResult for API selection
- Add FlacFetcher alias for FlacFetchAudioFetcher
- Refactor backend AudioSearchService to delegate to karaoke_gen.audio_fetcher
- Update karaoke_gen/__init__.py with lazy imports to avoid heavy dependencies
- Update backend tests to work with shared classes
- Add integration tests verifying shared classes are identical

Benefits:
- Single source of truth for flacfetch integration
- ~150 lines of duplicate code removed from backend
- Bug fixes apply everywhere automatically
- Prevents future divergence like YouTubeProvider -> YoutubeProvider

Resolves: docs/00-current-plan/REFACTOR-AUDIO-FETCHER.md
2025-12-20 01:12:50 -05:00
Andrew Beveridge d9da1dd360 docs: Add audio fetcher refactoring plan to eliminate code duplication
Proposes unifying backend/services/audio_search_service.py with
karaoke_gen/audio_fetcher.py to prevent future bugs like the
YouTubeProvider -> YoutubeProvider rename issue.
2025-12-20 01:05:18 -05:00
Andrew Beveridge a8fa6b5f46 test: Add flacfetch integration tests to catch import errors
These tests verify the actual flacfetch imports work, not just mocks.
Would have caught the YouTubeProvider -> YoutubeProvider rename.
2025-12-20 01:01:51 -05:00
Andrew Beveridge ebe0121938 fix: Update YoutubeProvider import in backend audio_search_service.py
The flacfetch 0.2.0 renamed YouTubeProvider to YoutubeProvider (lowercase 'tube').
PR #28 fixed this in karaoke_gen/audio_fetcher.py but missed the backend version.

This caused 'flacfetch library is not installed' errors when using the
audio search feature remotely, even though flacfetch was actually installed.

The ImportError from the renamed class was being caught and converted to
the generic 'not installed' error message.
2025-12-20 01:00:22 -05:00
Andrew Beveridge ☄️ 01db9a764a fix: countdown padding synchronization for instrumental audio (#35)
When LyricsTranscriber adds 3-second countdown padding to vocals,
the instrumental audio must be padded to match. This was broken when
transcription was skipped due to existing files.

Changes:
- Add _detect_countdown_padding_from_lrc() to detect padding from existing LRC
- Update early returns in transcribe_lyrics() to include padding info
- Add _scan_directory_for_instrumentals() to find existing instrumentals
- Fix counting bug in apply_countdown_padding_to_instrumentals
- Add countdown_padding_seconds safety net to KaraokeFinalise
- Store padding metadata in lyrics_worker job state_data
- Pass padding info to video_worker for instrumental sync
- Add comprehensive unit tests for new functionality
2025-12-19 18:49:03 -05:00
114 changed files with 22416 additions and 3811 deletions

No files matched your search

+135 -18
View File
@@ -25,6 +25,8 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.13
uses: actions/setup-python@v5
@@ -65,6 +67,8 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.13
uses: actions/setup-python@v5
@@ -97,6 +101,8 @@ jobs:
needs: [package-unit-tests, package-integration-tests]
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.13
uses: actions/setup-python@v5
@@ -275,6 +281,8 @@ jobs:
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python 3.13
uses: actions/setup-python@v5
@@ -364,21 +372,55 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Enable Yarn (for lyrics frontend)
run: corepack enable
# ===== Build Main Frontend (Next.js) =====
- name: Install main frontend dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Build main frontend (Next.js static export)
working-directory: frontend
env:
NEXT_PUBLIC_API_URL: https://api.nomadkaraoke.com
run: npm run build
# ===== Build Lyrics Review Frontend (Vite) =====
- name: Install lyrics frontend dependencies
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
run: yarn install --frozen-lockfile
- name: Build lyrics frontend (Vite)
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
env:
# Set base path for subpath deployment
VITE_BASE_PATH: /lyrics/
run: yarn build --base=/lyrics/
- name: Copy lyrics frontend to main output
run: |
mkdir -p frontend/out/lyrics
cp -r lyrics_transcriber_temp/lyrics_transcriber/frontend/dist/* frontend/out/lyrics/
# ===== Copy Instrumental Review UI (Static HTML) =====
- name: Copy instrumental review UI to main output
run: |
mkdir -p frontend/out/instrumental
cp karaoke_gen/instrumental_review/static/index.html frontend/out/instrumental/
- name: Setup Pages
uses: actions/configure-pages@v4
- name: Prepare frontend files
run: |
echo "📁 Preparing frontend files for deployment..."
mkdir -p _site
cp -r frontend/* _site/
echo "✅ Frontend files copied to _site/"
ls -la _site/
- name: Upload artifact
uses: actions/upload-pages-artifact@v3
with:
path: "_site"
path: "frontend/out"
- name: Deploy to GitHub Pages
id: deployment
@@ -397,8 +439,34 @@ jobs:
uses: actions/checkout@v4
with:
submodules: false
fetch-depth: 0 # Need full history for tag check
- name: Set up Python (for version extraction)
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install Poetry
uses: snok/install-poetry@v1
- name: Extract version and check if deployment needed
id: version_check
run: |
VERSION=$(poetry version --short)
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "📋 Current version: $VERSION"
if git ls-remote --tags origin | grep -q "refs/tags/v$VERSION$"; then
echo "needs_deploy=false" >> $GITHUB_OUTPUT
echo "⏭️ Tag v$VERSION already exists - skipping backend deployment"
echo " (Bump version in pyproject.toml to trigger a new deployment)"
else
echo "needs_deploy=true" >> $GITHUB_OUTPUT
echo "✨ Tag v$VERSION does not exist - proceeding with deployment"
fi
- name: Authenticate to Google Cloud (Workload Identity)
if: steps.version_check.outputs.needs_deploy == 'true'
id: auth
uses: google-github-actions/auth@v2
with:
@@ -406,17 +474,20 @@ jobs:
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
if: steps.version_check.outputs.needs_deploy == 'true'
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Check if base image needs rebuild
if: steps.version_check.outputs.needs_deploy == 'true'
id: check-base
run: |
echo "🔍 Checking if base image needs rebuild..."
CURRENT_HASH=$(sha256sum poetry.lock | cut -d' ' -f1)
echo "Current poetry.lock hash: $CURRENT_HASH"
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
CURRENT_HASH=$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
echo "Current combined hash (poetry.lock + Dockerfile.base): $CURRENT_HASH"
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
@@ -424,14 +495,18 @@ jobs:
EXISTING_HASH=""
if docker pull $BASE_IMAGE 2>/dev/null; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
if [ -z "$EXISTING_HASH" ]; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
fi
echo "Existing base image hash: $EXISTING_HASH"
else
echo "⚠️ No existing base image found"
fi
if [ "$CURRENT_HASH" != "$EXISTING_HASH" ]; then
echo "🔄 Base image needs rebuild (poetry.lock changed)"
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
else
echo "✅ Base image is up to date"
@@ -439,7 +514,7 @@ jobs:
fi
- name: Rebuild base image (if needed)
if: steps.check-base.outputs.needs_rebuild == 'true'
if: steps.version_check.outputs.needs_deploy == 'true' && steps.check-base.outputs.needs_rebuild == 'true'
run: |
echo "🏗️ Rebuilding base image with updated dependencies..."
@@ -479,9 +554,11 @@ jobs:
done
- name: Trigger Cloud Build
if: steps.version_check.outputs.needs_deploy == 'true'
run: |
echo "🚀 Triggering Cloud Build for deployment..."
echo "Commit: ${{ github.sha }}"
echo "Version: ${{ steps.version_check.outputs.version }}"
BUILD_ID=$(gcloud builds submit \
--config=cloudbuild.yaml \
@@ -519,21 +596,61 @@ jobs:
sleep 30
done
- name: Verify deployment
- name: Verify deployment and version
if: steps.version_check.outputs.needs_deploy == 'true'
run: |
EXPECTED_VERSION="${{ steps.version_check.outputs.version }}"
SERVICE_URL=$(gcloud run services describe karaoke-backend \
--region us-central1 \
--format 'value(status.url)')
echo "Service URL: $SERVICE_URL"
sleep 10
echo "Expected version: $EXPECTED_VERSION"
# Wait for new revision to be ready
echo "Waiting for new revision to be ready..."
sleep 15
# Basic health check
curl -f "$SERVICE_URL/api/health" || exit 1
echo "✅ Deployment verified - service is healthy"
echo "✅ Service is healthy"
# Version verification with retries (Cloud Run may take time to route to new revision)
MAX_RETRIES=6
RETRY_DELAY=10
for i in $(seq 1 $MAX_RETRIES); do
echo "Checking deployed version (attempt $i/$MAX_RETRIES)..."
DEPLOYED_VERSION=$(curl -sf "$SERVICE_URL/api/health/detailed" | jq -r '.version // empty')
if [ "$DEPLOYED_VERSION" = "$EXPECTED_VERSION" ]; then
echo "✅ Version verified: $DEPLOYED_VERSION"
break
else
echo "⚠️ Version mismatch: deployed=$DEPLOYED_VERSION, expected=$EXPECTED_VERSION"
if [ $i -lt $MAX_RETRIES ]; then
echo " Retrying in ${RETRY_DELAY}s..."
sleep $RETRY_DELAY
else
echo "❌ Version verification failed after $MAX_RETRIES attempts"
exit 1
fi
fi
done
- name: Summary
if: success()
if: steps.version_check.outputs.needs_deploy == 'true' && success()
run: |
echo "🎉 Deployment successful!"
echo "Version: ${{ steps.version_check.outputs.version }}"
echo "Commit: ${{ github.sha }}"
echo "Branch: ${{ github.ref_name }}"
echo "Run: ${{ github.run_number }}"
- name: Skipped deployment summary
if: steps.version_check.outputs.needs_deploy == 'false'
run: |
echo "⏭️ Backend deployment skipped - version ${{ steps.version_check.outputs.version }} already deployed"
echo ""
echo "To deploy a new version:"
echo " 1. Update version in pyproject.toml"
echo " 2. Commit and push to main"
+3
View File
@@ -73,8 +73,11 @@ dist/
downloads/
eggs/
.eggs/
# Python lib directories (not frontend)
lib/
lib64/
# But allow frontend lib
!frontend/lib/
parts/
sdist/
var/
+2
View File
@@ -183,6 +183,8 @@ karaoke-gen-remote \
| `REVIEW_UI_URL` | Lyrics review UI URL | `https://lyrics.nomadkaraoke.com` |
| `POLL_INTERVAL` | Seconds between status polls | `5` |
**Note:** The `REVIEW_UI_URL` defaults to the hosted lyrics review UI. For local development, set it to `http://localhost:5173` if you're running the frontend dev server.
### Authentication
The backend uses token-based authentication for admin operations (bulk delete, internal worker triggers). For basic job submission and monitoring, authentication is optional.
+8 -3
View File
@@ -11,7 +11,9 @@
# gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
# Use pre-built base image with all dependencies
FROM us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest
# Override with --build-arg BASE_IMAGE=karaoke-backend-base:local for local testing
ARG BASE_IMAGE=us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest
FROM ${BASE_IMAGE}
# Set working directory (should already exist from base)
WORKDIR /app
@@ -34,5 +36,8 @@ ENV PORT=8080
# Expose port
EXPOSE 8080
# Run the application
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8080"]
# Run uvicorn directly
# Note: Transmission daemon removed - torrent downloads now handled by
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
# Using --loop asyncio (instead of uvloop) to support nest_asyncio for remote flacfetch calls
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8080", "--loop", "asyncio"]
+6 -3
View File
@@ -7,12 +7,15 @@
FROM python:3.11-slim
# Install system dependencies required by karaoke-gen
# Note: Transmission daemon removed - torrent downloads now handled by
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
RUN apt-get update && apt-get install -y \
ffmpeg \
libsndfile1 \
libsox-dev \
sox \
build-essential \
curl \
&& rm -rf /var/lib/apt/lists/*
# Set working directory
@@ -34,9 +37,9 @@ RUN python -m spacy download en_core_web_sm
# Label with build info
ARG BUILD_DATE
ARG BUILD_ID
ARG POETRY_LOCK_HASH
ARG BASE_CONTENT_HASH
LABEL org.opencontainers.image.created="${BUILD_DATE}"
LABEL org.opencontainers.image.revision="${BUILD_ID}"
LABEL org.opencontainers.image.title="karaoke-backend-base"
LABEL org.opencontainers.image.description="Base image with Python dependencies for karaoke-backend"
LABEL poetry.lock.hash="${POETRY_LOCK_HASH}"
LABEL org.opencontainers.image.description="Base image with Python dependencies and system tools for karaoke-backend"
LABEL base.content.hash="${BASE_CONTENT_HASH}"
+214 -7
View File
@@ -2,8 +2,10 @@
FastAPI dependencies for authentication and authorization.
"""
import logging
from typing import Optional, Tuple
from fastapi import Depends, HTTPException, Header, Query, Request
import secrets
from datetime import datetime, timedelta, UTC
from typing import Optional, Tuple, Callable
from fastapi import Depends, HTTPException, Header, Query, Request, Path
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from backend.services.auth_service import get_auth_service, UserType, AuthService
@@ -12,6 +14,16 @@ from backend.services.auth_service import get_auth_service, UserType, AuthServic
logger = logging.getLogger(__name__)
def generate_review_token() -> str:
"""Generate a cryptographically secure review token."""
return secrets.token_urlsafe(32)
def get_review_token_expiry(hours: int = 24) -> datetime:
"""Get expiry time for a review token."""
return datetime.now(UTC) + timedelta(hours=hours)
# HTTP Bearer security scheme (Authorization: Bearer <token>)
security = HTTPBearer(auto_error=False)
@@ -24,14 +36,21 @@ async def get_token_from_request(
"""
Extract access token from request.
Supports two methods:
1. Authorization header: Authorization: Bearer <token>
2. Query parameter: ?token=<token> (for download links)
Supports three methods (in priority order):
1. X-Admin-Token header: Used by Cloud Tasks (since OIDC overwrites Authorization)
2. Authorization header: Authorization: Bearer <token>
3. Query parameter: ?token=<token> (for download links)
Returns:
Token string or None
"""
# Try Authorization header first
# Check X-Admin-Token header first (used by Cloud Tasks)
# Cloud Tasks OIDC token overwrites Authorization header, so we use a custom header
admin_token = request.headers.get("X-Admin-Token")
if admin_token:
return admin_token
# Try Authorization header
if credentials:
return credentials.credentials
@@ -70,7 +89,15 @@ async def require_auth(
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
if not is_valid:
logger.warning(f"Authentication failed: {message}")
# Log more details for debugging token issues
auth_header = request.headers.get("Authorization", "")
logger.warning(
f"Authentication failed: {message}. "
f"Token provided: {bool(token_str)}, "
f"Token length: {len(token_str) if token_str else 0}, "
f"Auth header present: {bool(auth_header)}, "
f"Header prefix: {auth_header[:20] if auth_header else 'none'}..."
)
raise HTTPException(
status_code=401,
detail=f"Authentication failed: {message}"
@@ -136,3 +163,183 @@ def optional_auth(
return None
return token_str, user_type, remaining_uses
def require_review_auth_factory(job_id_param: str = "job_id"):
"""
Factory to create a review authentication dependency.
Accepts either:
1. Full user authentication (admin/user token)
2. Job-specific review token (only valid for the specific job)
Args:
job_id_param: Name of the path parameter containing the job ID
Returns:
Dependency function that validates review access
"""
async def require_review_auth(
request: Request,
job_id: str = Path(...),
review_token: Optional[str] = Query(None, description="Job-specific review token"),
auth_service: AuthService = Depends(get_auth_service),
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
token: Optional[str] = Query(None, alias="token", description="Full access token")
) -> Tuple[str, str]:
"""
Validate review access for a job.
Returns:
(job_id, auth_type) where auth_type is "full" or "review_token"
Raises:
HTTPException: 401 if authentication fails
"""
# Import here to avoid circular dependency
from backend.services.job_manager import JobManager
# Try full authentication first
full_token = None
if credentials:
full_token = credentials.credentials
elif token:
full_token = token
if full_token:
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
if is_valid:
logger.info(f"Review access granted via full auth ({user_type}) for job {job_id}")
return job_id, "full"
# Try review token
if review_token:
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Validate review token matches
if job.review_token and secrets.compare_digest(job.review_token, review_token):
# Check expiry if set
if job.review_token_expires_at:
now = datetime.now(UTC)
# Handle timezone-naive datetimes from Firestore
expiry = job.review_token_expires_at
if expiry.tzinfo is None:
expiry = expiry.replace(tzinfo=UTC)
if now > expiry:
logger.warning(f"Review token expired for job {job_id}")
raise HTTPException(
status_code=401,
detail="Review token has expired. Please request a new review link."
)
logger.info(f"Review access granted via review_token for job {job_id}")
return job_id, "review_token"
else:
logger.warning(f"Invalid review token for job {job_id}")
# No valid authentication
raise HTTPException(
status_code=401,
detail="Authentication required. Provide either a full access token or a valid review_token for this job."
)
return require_review_auth
# Default instance for most review endpoints
require_review_auth = require_review_auth_factory()
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
"""
Factory to create an instrumental review authentication dependency.
Accepts either:
1. Full user authentication (admin/user token)
2. Job-specific instrumental token (only valid for the specific job)
Args:
job_id_param: Name of the path parameter containing the job ID
Returns:
Dependency function that validates instrumental review access
"""
async def require_instrumental_auth(
request: Request,
job_id: str = Path(...),
instrumental_token: Optional[str] = Query(None, description="Job-specific instrumental review token"),
auth_service: AuthService = Depends(get_auth_service),
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
token: Optional[str] = Query(None, alias="token", description="Full access token")
) -> Tuple[str, str]:
"""
Validate instrumental review access for a job.
Returns:
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
Raises:
HTTPException: 401 if authentication fails
"""
# Import here to avoid circular dependency
from backend.services.job_manager import JobManager
# Try full authentication first
full_token = None
if credentials:
full_token = credentials.credentials
elif token:
full_token = token
if full_token:
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
if is_valid:
logger.info(f"Instrumental access granted via full auth ({user_type}) for job {job_id}")
return job_id, "full"
# Try instrumental token
if instrumental_token:
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Validate instrumental token matches
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
# Check expiry if set
if job.instrumental_token_expires_at:
now = datetime.now(UTC)
# Handle timezone-naive datetimes from Firestore
expiry = job.instrumental_token_expires_at
if expiry.tzinfo is None:
expiry = expiry.replace(tzinfo=UTC)
if now > expiry:
logger.warning(f"Instrumental token expired for job {job_id}")
raise HTTPException(
status_code=401,
detail="Instrumental review token has expired. Please request a new review link."
)
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
return job_id, "instrumental_token"
else:
logger.warning(f"Invalid instrumental token for job {job_id}")
# No valid authentication
raise HTTPException(
status_code=401,
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
)
return require_instrumental_auth
# Default instance for most instrumental review endpoints
require_instrumental_auth = require_instrumental_auth_factory()
+153 -47
View File
@@ -15,9 +15,9 @@ import asyncio
import logging
import os
import tempfile
from typing import Optional, List, Dict, Any
from typing import Optional, List, Dict, Any, Tuple
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
from pydantic import BaseModel, Field
from backend.models.job import JobCreate, JobStatus
@@ -34,6 +34,8 @@ from backend.services.audio_search_service import (
)
from backend.config import get_settings
from backend.version import VERSION
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType
logger = logging.getLogger(__name__)
router = APIRouter(tags=["audio-search"])
@@ -82,15 +84,40 @@ class AudioSearchRequest(BaseModel):
class AudioSearchResultResponse(BaseModel):
"""A single audio search result."""
"""A single audio search result.
Contains all fields needed for rich display using flacfetch's
shared formatting functions.
"""
index: int
title: str
artist: str
provider: str
url: str
duration: Optional[int] = None
quality: Optional[str] = None
source_id: Optional[str] = None
provider: str # Maps to source_name in Release
url: Optional[str] = None # Maps to download_url in Release (may be None for remote)
duration: Optional[int] = None # Maps to duration_seconds in Release
quality: Optional[str] = None # Stringified quality
source_id: Optional[str] = None # Maps to info_hash in Release
seeders: Optional[int] = None
target_file: Optional[str] = None
# Additional fields for rich display (from Release.to_dict())
year: Optional[int] = None
label: Optional[str] = None
edition_info: Optional[str] = None
release_type: Optional[str] = None
channel: Optional[str] = None # For YouTube
view_count: Optional[int] = None # For YouTube
size_bytes: Optional[int] = None
target_file_size: Optional[int] = None
track_pattern: Optional[str] = None
match_score: Optional[float] = None
# Pre-computed display fields
formatted_size: Optional[str] = None
formatted_duration: Optional[str] = None
formatted_views: Optional[str] = None
is_lossless: Optional[bool] = None
quality_str: Optional[str] = None
# Full quality object for Release.from_dict()
quality_data: Optional[Dict[str, Any]] = None
class AudioSearchResponse(BaseModel):
@@ -168,6 +195,10 @@ async def _download_and_start_processing(
This is called either:
- Immediately after search if auto_download=True
- When user calls the select endpoint
For remote flacfetch downloads (torrent sources), the file is downloaded
on the flacfetch VM and uploaded directly to GCS. For local downloads
(YouTube), the file is downloaded locally and then uploaded to GCS.
"""
job = job_manager.get_job(job_id)
if not job:
@@ -199,33 +230,71 @@ async def _download_and_start_processing(
)
try:
# Download to temp directory
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
# Determine if this is a remote torrent download
is_torrent_source = selected.get('provider') in ['RED', 'OPS']
is_remote_enabled = audio_search_service.is_remote_enabled()
result = audio_search_service.download(
result_index=selection_index,
output_dir=temp_dir,
)
# Upload to GCS
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(
f,
audio_gcs_path,
content_type='audio/flac' # flacfetch typically returns FLAC
# For remote torrent downloads, have flacfetch VM upload directly to GCS
if is_torrent_source and is_remote_enabled:
# Generate GCS path for remote upload
gcs_destination = f"uploads/{job_id}/audio/"
logger.info(f"Using remote download with GCS upload to: {gcs_destination}")
result = audio_search_service.download(
result_index=selection_index,
output_dir="", # Not used for remote
gcs_path=gcs_destination,
)
logger.info(f"Uploaded audio to GCS: {audio_gcs_path}")
# Clean up temp file
try:
os.remove(result.filepath)
os.rmdir(temp_dir)
except Exception as e:
logger.warning(f"Failed to clean up temp files: {e}")
# For remote downloads, filepath is already the GCS path
if result.filepath.startswith("gs://"):
# Extract the path portion after the bucket name
# Format: gs://bucket/uploads/job_id/audio/filename.flac
parts = result.filepath.replace("gs://", "").split("/", 1)
if len(parts) == 2:
audio_gcs_path = parts[1]
else:
audio_gcs_path = result.filepath
filename = os.path.basename(result.filepath)
else:
# Fallback: treat as local path (shouldn't happen for remote)
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
else:
# Local download (YouTube or fallback)
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
result = audio_search_service.download(
result_index=selection_index,
output_dir=temp_dir,
)
# Upload to GCS
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(
f,
audio_gcs_path,
content_type='audio/flac' # flacfetch typically returns FLAC
)
logger.info(f"Uploaded audio to GCS: {audio_gcs_path}")
# Clean up temp file
try:
os.remove(result.filepath)
os.rmdir(temp_dir)
except Exception as e:
logger.warning(f"Failed to clean up temp files: {e}")
# Update job with GCS path and transition to DOWNLOADING
job_manager.update_job(job_id, {
@@ -263,6 +332,7 @@ async def search_audio(
request: Request,
background_tasks: BackgroundTasks,
body: AudioSearchRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Search for audio by artist and title, creating a new job.
@@ -418,20 +488,50 @@ async def search_audio(
message=f"Found {len(search_results)} audio sources. Waiting for selection."
)
# Convert to response format
result_responses = [
AudioSearchResultResponse(
index=r.index,
title=r.title,
artist=r.artist,
provider=r.provider,
url=r.url,
duration=r.duration,
quality=r.quality,
source_id=r.source_id,
# Convert to response format with full Release data for rich display
result_responses = []
for r in search_results:
# Get full serialized data from raw_result if available
raw_dict = {}
if r.raw_result:
try:
raw_dict = r.raw_result.to_dict()
except AttributeError:
pass # Not a Release object
result_responses.append(
AudioSearchResultResponse(
index=r.index,
title=r.title,
artist=r.artist,
provider=r.provider,
url=r.url,
duration=r.duration,
quality=r.quality,
source_id=r.source_id,
seeders=raw_dict.get('seeders') or r.seeders,
target_file=raw_dict.get('target_file') or r.target_file,
# Additional fields for rich display
year=raw_dict.get('year'),
label=raw_dict.get('label'),
edition_info=raw_dict.get('edition_info'),
release_type=raw_dict.get('release_type'),
channel=raw_dict.get('channel'),
view_count=raw_dict.get('view_count'),
size_bytes=raw_dict.get('size_bytes'),
target_file_size=raw_dict.get('target_file_size'),
track_pattern=raw_dict.get('track_pattern'),
match_score=raw_dict.get('match_score'),
# Pre-computed display fields
formatted_size=raw_dict.get('formatted_size'),
formatted_duration=raw_dict.get('formatted_duration'),
formatted_views=raw_dict.get('formatted_views'),
is_lossless=raw_dict.get('is_lossless'),
quality_str=raw_dict.get('quality_str'),
# Full quality object for Release.from_dict()
quality_data=raw_dict.get('quality'),
)
)
for r in search_results
]
return AudioSearchResponse(
status="awaiting_selection",
@@ -451,7 +551,10 @@ async def search_audio(
@router.get("/audio-search/{job_id}/results")
async def get_audio_search_results(job_id: str):
async def get_audio_search_results(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Get audio search results for a job.
@@ -473,6 +576,8 @@ async def get_audio_search_results(job_id: str):
"status": "success",
"job_id": job_id,
"job_status": job.status,
"artist": job.audio_search_artist or job.artist,
"title": job.audio_search_title or job.title,
"results": search_results,
"results_count": len(search_results),
}
@@ -483,6 +588,7 @@ async def select_audio_source(
job_id: str,
background_tasks: BackgroundTasks,
body: AudioSelectRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Select an audio source and start job processing.
+22 -1
View File
@@ -15,7 +15,7 @@ import json
import logging
import tempfile
import os
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Request, Body
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Request, Body, Depends
from pathlib import Path
from typing import Optional, List, Dict, Any, Tuple
@@ -28,6 +28,9 @@ from backend.services.worker_service import get_worker_service
from backend.services.credential_manager import get_credential_manager, CredentialStatus
from backend.config import get_settings
from backend.version import VERSION
from backend.services.metrics import metrics
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType
logger = logging.getLogger(__name__)
router = APIRouter(tags=["jobs"])
@@ -237,6 +240,7 @@ async def _trigger_workers_parallel(job_id: str) -> None:
async def upload_and_create_job(
request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_auth),
# Required fields
file: UploadFile = File(..., description="Audio file to process"),
artist: str = Form(..., description="Artist name"),
@@ -428,6 +432,9 @@ async def upload_and_create_job(
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
metrics.record_job_created(job_id, source="upload")
logger.info(f"Created job {job_id} for {artist} - {title}")
# Upload main audio file to GCS
@@ -770,6 +777,7 @@ def _get_gcs_path_for_file(job_id: str, file_type: str, filename: str) -> str:
async def create_job_with_upload_urls(
request: Request,
body: CreateJobWithUploadUrlsRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Create a karaoke generation job and return signed URLs for direct file upload to GCS.
@@ -882,6 +890,9 @@ async def create_job_with_upload_urls(
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
metrics.record_job_created(job_id, source="upload")
logger.info(f"Created job {job_id} for {body.artist} - {body.title} (signed URL upload flow)")
# Generate signed upload URLs for each file
@@ -925,6 +936,7 @@ async def mark_uploads_complete(
job_id: str,
background_tasks: BackgroundTasks,
body: UploadsCompleteRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Mark file uploads as complete and start job processing.
@@ -1170,6 +1182,7 @@ async def create_job_from_url(
request: Request,
background_tasks: BackgroundTasks,
body: CreateJobFromUrlRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Create a karaoke generation job from a YouTube or other online video URL.
@@ -1265,6 +1278,9 @@ async def create_job_from_url(
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
metrics.record_job_created(job_id, source="url")
logger.info(f"Created URL-based job {job_id} for URL: {body.url}")
if artist:
logger.info(f" Artist: {artist}")
@@ -1344,6 +1360,7 @@ def _get_gcs_path_for_finalise_file(job_id: str, file_type: str, filename: str)
async def create_finalise_only_job(
request: Request,
body: CreateFinaliseOnlyJobRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Create a finalise-only job for continuing from a local prep phase.
@@ -1466,6 +1483,9 @@ async def create_finalise_only_job(
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
metrics.record_job_created(job_id, source="finalise")
logger.info(f"Created finalise-only job {job_id} for {body.artist} - {body.title}")
# Generate signed upload URLs for each file
@@ -1509,6 +1529,7 @@ async def mark_finalise_uploads_complete(
job_id: str,
background_tasks: BackgroundTasks,
body: UploadsCompleteRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Mark finalise-only file uploads as complete and start video generation.
+116 -1
View File
@@ -1,10 +1,64 @@
"""
Health check routes.
"""
import os
import logging
from fastapi import APIRouter
from typing import Dict
from typing import Dict, Any, Optional
from backend.version import VERSION
from backend.services.flacfetch_client import get_flacfetch_client
router = APIRouter()
logger = logging.getLogger(__name__)
def check_transmission_status() -> Dict[str, Any]:
"""Check if Transmission daemon is available and responsive."""
try:
import transmission_rpc
host = os.environ.get("TRANSMISSION_HOST", "localhost")
port = int(os.environ.get("TRANSMISSION_PORT", "9091"))
client = transmission_rpc.Client(host=host, port=port, timeout=5)
stats = client.session_stats()
# Get detailed torrent info
torrents = client.get_torrents()
torrent_details = []
for t in torrents:
torrent_info = {
"name": t.name,
"progress": round(t.progress, 1),
"status": str(t.status),
"peers": t.peers_connected if hasattr(t, 'peers_connected') else 0,
"download_speed": round(t.rate_download / 1024, 1) if hasattr(t, 'rate_download') else 0, # KB/s
}
# Check if stalled (downloading but no progress and no peers)
if t.status.downloading and t.progress < 100 and torrent_info['peers'] == 0:
torrent_info['stalled'] = True
torrent_details.append(torrent_info)
return {
"available": True,
"host": host,
"port": port,
"download_dir": stats.download_dir if hasattr(stats, 'download_dir') else None,
"active_torrent_count": stats.active_torrent_count if hasattr(stats, 'active_torrent_count') else 0,
"torrents": torrent_details,
}
except ImportError as e:
return {
"available": False,
"error": f"transmission_rpc not installed: {e}",
}
except Exception as e:
return {
"available": False,
"host": os.environ.get("TRANSMISSION_HOST", "localhost"),
"port": int(os.environ.get("TRANSMISSION_PORT", "9091")),
"error": str(e),
}
@router.get("/health")
@@ -16,6 +70,67 @@ async def health_check() -> Dict[str, str]:
}
async def check_flacfetch_service_status() -> Dict[str, Any]:
"""Check if remote flacfetch service is available and healthy."""
client = get_flacfetch_client()
if not client:
return {
"configured": False,
"message": "Remote flacfetch service not configured (FLACFETCH_API_URL not set)",
}
try:
health = await client.health_check()
return {
"configured": True,
"available": True,
"status": health.get("status"),
"version": health.get("version"),
"transmission": health.get("transmission", {}),
"disk": health.get("disk", {}),
"providers": health.get("providers", {}),
}
except Exception as e:
return {
"configured": True,
"available": False,
"error": str(e),
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
Detailed health check including dependencies.
Use this to debug issues with Transmission, flacfetch service, etc.
"""
transmission_status = check_transmission_status()
flacfetch_status = await check_flacfetch_service_status()
# Log for debugging
if not transmission_status.get("available"):
logger.warning(f"Local Transmission not available: {transmission_status.get('error')}")
else:
logger.info(f"Local Transmission available at {transmission_status.get('host')}:{transmission_status.get('port')}")
if flacfetch_status.get("configured") and not flacfetch_status.get("available"):
logger.warning(f"Remote flacfetch service not available: {flacfetch_status.get('error')}")
elif flacfetch_status.get("available"):
logger.info(f"Remote flacfetch service healthy: {flacfetch_status.get('status')}")
return {
"status": "healthy",
"service": "karaoke-gen-backend",
"version": VERSION,
"dependencies": {
"transmission_local": transmission_status,
"flacfetch_remote": flacfetch_status,
}
}
@router.get("/readiness")
async def readiness_check() -> Dict[str, str]:
"""Readiness check endpoint for Cloud Run."""
+66 -9
View File
@@ -6,11 +6,17 @@ They are protected by admin authentication.
With Cloud Tasks integration, these endpoints may be called multiple times
(retry on failure). Idempotency checks prevent duplicate processing.
Observability:
- Extracts trace context from incoming requests (propagated via Cloud Tasks)
- Creates worker spans linked to the original request trace
- All logs include job_id for easy filtering in Cloud Logging
"""
import logging
import asyncio
import time
from typing import Tuple, Optional
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
from pydantic import BaseModel
from backend.workers.audio_worker import process_audio_separation
@@ -21,6 +27,12 @@ from backend.workers.render_video_worker import process_render_video
from backend.api.dependencies import require_admin
from backend.services.auth_service import UserType
from backend.services.job_manager import JobManager
from backend.services.tracing import (
extract_trace_context,
start_span_with_context,
add_span_attribute,
add_span_event,
)
logger = logging.getLogger(__name__)
@@ -57,7 +69,7 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
job = job_manager.get_job(job_id)
if not job:
logger.warning(f"Job {job_id} not found for {worker_name} worker")
logger.warning(f"[job:{job_id}] Job not found for {worker_name} worker")
return WorkerResponse(
status="not_found",
job_id=job_id,
@@ -70,7 +82,7 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
stage = worker_progress.get('stage')
if stage == 'running':
logger.info(f"{worker_name.capitalize()} worker already running for job {job_id}, skipping")
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already running, skipping")
return WorkerResponse(
status="already_running",
job_id=job_id,
@@ -78,7 +90,7 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
)
if stage == 'complete':
logger.info(f"{worker_name.capitalize()} worker already complete for job {job_id}, skipping")
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already complete, skipping")
return WorkerResponse(
status="already_complete",
job_id=job_id,
@@ -93,6 +105,7 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
@router.post("/workers/audio", response_model=WorkerResponse)
async def trigger_audio_worker(
request: WorkerRequest,
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
@@ -107,11 +120,18 @@ async def trigger_audio_worker(
The worker runs in the background and updates job state as it progresses.
"""
job_id = request.job_id
logger.info(f"Triggering audio worker for job {job_id}")
# Extract trace context from incoming request (propagated via Cloud Tasks)
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=audio")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "audio")
# Idempotency check
skip_response = _check_worker_idempotency(job_id, "audio")
if skip_response:
add_span_event("worker_skipped", {"reason": skip_response.status})
return skip_response
# Add task to background tasks
@@ -119,6 +139,7 @@ async def trigger_audio_worker(
# while the worker continues processing
background_tasks.add_task(process_audio_separation, job_id)
add_span_event("worker_started")
return WorkerResponse(
status="started",
job_id=job_id,
@@ -129,6 +150,7 @@ async def trigger_audio_worker(
@router.post("/workers/lyrics", response_model=WorkerResponse)
async def trigger_lyrics_worker(
request: WorkerRequest,
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
@@ -143,16 +165,24 @@ async def trigger_lyrics_worker(
The worker runs in the background and updates job state as it progresses.
"""
job_id = request.job_id
logger.info(f"Triggering lyrics worker for job {job_id}")
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=lyrics")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "lyrics")
# Idempotency check
skip_response = _check_worker_idempotency(job_id, "lyrics")
if skip_response:
add_span_event("worker_skipped", {"reason": skip_response.status})
return skip_response
# Add task to background tasks
background_tasks.add_task(process_lyrics_transcription, job_id)
add_span_event("worker_started")
return WorkerResponse(
status="started",
job_id=job_id,
@@ -163,6 +193,7 @@ async def trigger_lyrics_worker(
@router.post("/workers/screens", response_model=WorkerResponse)
async def trigger_screens_worker(
request: WorkerRequest,
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
@@ -174,16 +205,24 @@ async def trigger_screens_worker(
Idempotency: If worker is already running or complete, returns early.
"""
job_id = request.job_id
logger.info(f"Triggering screens worker for job {job_id}")
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=screens")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "screens")
# Idempotency check
skip_response = _check_worker_idempotency(job_id, "screens")
if skip_response:
add_span_event("worker_skipped", {"reason": skip_response.status})
return skip_response
# Add task to background tasks
background_tasks.add_task(generate_screens, job_id)
add_span_event("worker_started")
return WorkerResponse(
status="started",
job_id=job_id,
@@ -194,6 +233,7 @@ async def trigger_screens_worker(
@router.post("/workers/video", response_model=WorkerResponse)
async def trigger_video_worker(
request: WorkerRequest,
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
@@ -206,16 +246,24 @@ async def trigger_video_worker(
Idempotency: If worker is already running or complete, returns early.
"""
job_id = request.job_id
logger.info(f"Triggering video worker for job {job_id}")
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=video")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "video")
# Idempotency check
skip_response = _check_worker_idempotency(job_id, "video")
if skip_response:
add_span_event("worker_skipped", {"reason": skip_response.status})
return skip_response
# Add task to background tasks
background_tasks.add_task(generate_video, job_id)
add_span_event("worker_started")
return WorkerResponse(
status="started",
job_id=job_id,
@@ -226,6 +274,7 @@ async def trigger_video_worker(
@router.post("/workers/render-video", response_model=WorkerResponse)
async def trigger_render_video_worker(
request: WorkerRequest,
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
@@ -242,16 +291,24 @@ async def trigger_render_video_worker(
Next state: AWAITING_INSTRUMENTAL_SELECTION
"""
job_id = request.job_id
logger.info(f"Triggering render-video worker for job {job_id}")
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=render-video")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "render-video")
# Idempotency check
skip_response = _check_worker_idempotency(job_id, "render")
if skip_response:
add_span_event("worker_skipped", {"reason": skip_response.status})
return skip_response
# Add task to background tasks
background_tasks.add_task(process_render_video, job_id)
add_span_event("worker_started")
return WorkerResponse(
status="started",
job_id=job_id,
+83 -19
View File
@@ -26,8 +26,9 @@ from backend.services.job_manager import JobManager
from backend.services.worker_service import get_worker_service
from backend.services.storage_service import StorageService
from backend.config import get_settings
from backend.api.dependencies import require_admin
from backend.api.dependencies import require_admin, require_auth, require_instrumental_auth
from backend.services.auth_service import UserType
from backend.services.metrics import metrics
logger = logging.getLogger(__name__)
@@ -55,7 +56,8 @@ async def _trigger_workers_parallel(job_id: str) -> None:
@router.post("", response_model=JobResponse)
async def create_job(
request: URLSubmissionRequest,
background_tasks: BackgroundTasks
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> JobResponse:
"""
Create a new karaoke generation job from a URL.
@@ -81,6 +83,9 @@ async def create_job(
)
job = job_manager.create_job(job_create)
# Record job creation metric
metrics.record_job_created(job.job_id, source="url")
# Trigger both workers in parallel using asyncio.gather
# (FastAPI's BackgroundTasks runs async tasks sequentially)
background_tasks.add_task(_trigger_workers_parallel, job.job_id)
@@ -102,7 +107,10 @@ async def create_job(
@router.get("/{job_id}", response_model=Job)
async def get_job(job_id: str) -> Job:
async def get_job(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> Job:
"""Get job status and details."""
job = job_manager.get_job(job_id)
if not job:
@@ -122,7 +130,8 @@ async def list_jobs(
client_id: Optional[str] = None,
created_after: Optional[str] = None,
created_before: Optional[str] = None,
limit: int = 100
limit: int = 100,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> List[Job]:
"""
List jobs with optional filters.
@@ -174,7 +183,11 @@ async def list_jobs(
@router.delete("/{job_id}")
async def delete_job(job_id: str, delete_files: bool = True) -> dict:
async def delete_job(
job_id: str,
delete_files: bool = True,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> dict:
"""Delete a job and optionally its output files."""
try:
job = job_manager.get_job(job_id)
@@ -301,7 +314,10 @@ async def bulk_delete_jobs(
# ============================================================================
@router.get("/{job_id}/review-data")
async def get_review_data(job_id: str) -> Dict[str, Any]:
async def get_review_data(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> Dict[str, Any]:
"""
Get data needed for lyrics review interface.
@@ -342,7 +358,11 @@ async def get_review_data(job_id: str) -> Dict[str, Any]:
@router.post("/{job_id}/start-review")
async def start_review(job_id: str, request: StartReviewRequest) -> dict:
async def start_review(
job_id: str,
request: StartReviewRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> dict:
"""
Mark job as IN_REVIEW (user opened review interface).
@@ -364,7 +384,8 @@ async def start_review(job_id: str, request: StartReviewRequest) -> dict:
async def submit_corrections(
job_id: str,
submission: CorrectionsSubmission,
background_tasks: BackgroundTasks
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> dict:
"""
Save corrected lyrics during human review.
@@ -422,7 +443,8 @@ async def submit_corrections(
@router.post("/{job_id}/complete-review")
async def complete_review(
job_id: str,
background_tasks: BackgroundTasks
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> dict:
"""
Complete the human review and trigger video rendering.
@@ -470,13 +492,18 @@ async def complete_review(
@router.get("/{job_id}/instrumental-options")
async def get_instrumental_options(job_id: str) -> Dict[str, Any]:
async def get_instrumental_options(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
) -> Dict[str, Any]:
"""
Get instrumental audio options for user selection.
Returns signed URLs for both options:
1. Clean instrumental (no backing vocals)
2. Instrumental with backing vocals
Accepts either full auth token or job-specific instrumental_token.
"""
job = job_manager.get_job(job_id)
if not job:
@@ -525,7 +552,10 @@ async def get_instrumental_options(job_id: str) -> Dict[str, Any]:
@router.get("/{job_id}/instrumental-analysis")
async def get_instrumental_analysis(job_id: str) -> Dict[str, Any]:
async def get_instrumental_analysis(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
) -> Dict[str, Any]:
"""
Get audio analysis data for instrumental selection.
@@ -536,6 +566,8 @@ async def get_instrumental_analysis(job_id: str) -> Dict[str, Any]:
This endpoint enables intelligent instrumental selection by providing
detailed analysis of the backing vocals track.
Accepts either full auth token or job-specific instrumental_token.
"""
job = job_manager.get_job(job_id)
if not job:
@@ -599,7 +631,11 @@ async def get_instrumental_analysis(job_id: str) -> Dict[str, Any]:
@router.get("/{job_id}/audio-stream/{stem_type}")
async def stream_audio(job_id: str, stem_type: str):
async def stream_audio(
job_id: str,
stem_type: str,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
):
"""
Stream an audio file for playback in the browser.
@@ -611,6 +647,8 @@ async def stream_audio(job_id: str, stem_type: str):
Returns audio as a streaming response with proper headers for
browser audio playback.
Accepts either full auth token or job-specific instrumental_token.
"""
from fastapi.responses import StreamingResponse
import tempfile
@@ -687,6 +725,7 @@ async def stream_audio(job_id: str, stem_type: str):
async def create_custom_instrumental(
job_id: str,
request: CreateCustomInstrumentalRequest,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
) -> Dict[str, Any]:
"""
Create a custom instrumental by muting regions of backing vocals.
@@ -698,6 +737,8 @@ async def create_custom_instrumental(
After calling this endpoint, the user can select "custom" in the
select-instrumental endpoint.
Accepts either full auth token or job-specific instrumental_token.
"""
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
@@ -776,7 +817,11 @@ async def create_custom_instrumental(
@router.get("/{job_id}/waveform-data")
async def get_waveform_data(job_id: str, num_points: int = 500) -> Dict[str, Any]:
async def get_waveform_data(
job_id: str,
num_points: int = 500,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
) -> Dict[str, Any]:
"""
Get waveform amplitude data for client-side rendering.
@@ -793,6 +838,8 @@ async def get_waveform_data(job_id: str, num_points: int = 500) -> Dict[str, Any
"duration_seconds": 180.5,
"num_points": 500
}
Accepts either full auth token or job-specific instrumental_token.
"""
from backend.services.audio_analysis_service import AudioAnalysisService
@@ -833,13 +880,16 @@ async def get_waveform_data(job_id: str, num_points: int = 500) -> Dict[str, Any
async def select_instrumental(
job_id: str,
selection: InstrumentalSelection,
background_tasks: BackgroundTasks
background_tasks: BackgroundTasks,
auth_info: Tuple[str, str] = Depends(require_instrumental_auth)
) -> dict:
"""
Submit instrumental selection.
This is the SECOND critical human-in-the-loop interaction point.
After selection, the job proceeds to video generation.
Accepts either full auth token or job-specific instrumental_token.
"""
job = job_manager.get_job(job_id)
if not job:
@@ -881,7 +931,10 @@ async def select_instrumental(
@router.get("/{job_id}/download-urls")
async def get_download_urls(job_id: str) -> dict:
async def get_download_urls(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> dict:
"""
Get download URLs for all job output files.
@@ -923,7 +976,12 @@ async def get_download_urls(job_id: str) -> dict:
@router.get("/{job_id}/download/{category}/{file_key}")
async def download_file(job_id: str, category: str, file_key: str):
async def download_file(
job_id: str,
category: str,
file_key: str,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
):
"""
Stream download a specific file from a completed job.
@@ -1010,7 +1068,11 @@ async def download_file(job_id: str, category: str, file_key: str):
@router.post("/{job_id}/cancel")
async def cancel_job(job_id: str, request: CancelJobRequest) -> dict:
async def cancel_job(
job_id: str,
request: CancelJobRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> dict:
"""
Cancel a job.
@@ -1031,7 +1093,8 @@ async def cancel_job(job_id: str, request: CancelJobRequest) -> dict:
@router.post("/{job_id}/retry")
async def retry_job(
job_id: str,
background_tasks: BackgroundTasks
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> dict:
"""
Retry a failed job from the last successful checkpoint.
@@ -1185,7 +1248,8 @@ async def retry_job(
async def get_worker_logs(
job_id: str,
since_index: int = 0,
worker: Optional[str] = None
worker: Optional[str] = None,
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
) -> Dict[str, Any]:
"""
Get worker logs for debugging.
+51 -12
View File
@@ -16,9 +16,9 @@ import json
import os
import tempfile
from pathlib import Path
from typing import Dict, Any, Set
from typing import Dict, Any, Set, Tuple
from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, HTTPException, Request, Depends
from fastapi.responses import FileResponse, StreamingResponse
from starlette.background import BackgroundTask
@@ -27,6 +27,8 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_logging import job_log_context, JobLogger
from backend.services.tracing import create_span, add_span_attribute, add_span_event
from backend.api.dependencies import require_auth, require_review_auth
from backend.services.auth_service import UserType
# LyricsTranscriber imports for preview generation
from lyrics_transcriber.types import CorrectionResult
@@ -63,7 +65,10 @@ async def ping(job_id: str):
@router.get("/{job_id}/correction-data")
async def get_correction_data(job_id: str):
async def get_correction_data(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Get correction data for the review interface.
@@ -129,14 +134,21 @@ async def get_correction_data(job_id: str):
@router.get("/{job_id}/audio/{audio_hash}")
async def get_audio_with_hash(job_id: str, audio_hash: str):
async def get_audio_with_hash(
job_id: str,
audio_hash: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""Stream the audio file for playback (with hash parameter)."""
return await _stream_audio(job_id)
@router.get("/{job_id}/audio/")
@router.get("/{job_id}/audio")
async def get_audio_no_hash(job_id: str):
async def get_audio_no_hash(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""Stream the audio file for playback (without hash parameter)."""
return await _stream_audio(job_id)
@@ -189,7 +201,11 @@ async def _stream_audio(job_id: str):
@router.post("/{job_id}/complete")
async def complete_review(job_id: str, updated_data: Dict[str, Any]):
async def complete_review(
job_id: str,
updated_data: Dict[str, Any],
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Complete the review and save corrected lyrics.
@@ -244,7 +260,11 @@ async def complete_review(job_id: str, updated_data: Dict[str, Any]):
@router.post("/{job_id}/handlers")
async def update_handlers(job_id: str, enabled_handlers: list):
async def update_handlers(
job_id: str,
enabled_handlers: list,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Update enabled correction handlers (optional feature).
@@ -256,7 +276,11 @@ async def update_handlers(job_id: str, enabled_handlers: list):
@router.post("/{job_id}/add-lyrics")
async def add_lyrics(job_id: str, data: Dict[str, str]):
async def add_lyrics(
job_id: str,
data: Dict[str, str],
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Add custom lyrics source and rerun correction.
@@ -354,7 +378,11 @@ async def add_lyrics(job_id: str, data: Dict[str, str]):
@router.post("/{job_id}/preview-video")
async def generate_preview_video(job_id: str, updated_data: Dict[str, Any]):
async def generate_preview_video(
job_id: str,
updated_data: Dict[str, Any],
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Generate a preview video with the current corrections.
@@ -469,7 +497,11 @@ async def generate_preview_video(job_id: str, updated_data: Dict[str, Any]):
@router.get("/{job_id}/preview-video/{preview_hash}")
async def get_preview_video(job_id: str, preview_hash: str):
async def get_preview_video(
job_id: str,
preview_hash: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""Stream the generated preview video."""
storage = StorageService()
@@ -510,7 +542,11 @@ async def get_preview_video(job_id: str, preview_hash: str):
@router.post("/{job_id}/v1/annotations")
async def submit_annotation(job_id: str, annotation: Dict[str, Any]):
async def submit_annotation(
job_id: str,
annotation: Dict[str, Any],
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Submit a correction annotation for ML training data.
@@ -522,7 +558,10 @@ async def submit_annotation(job_id: str, annotation: Dict[str, Any]):
@router.get("/{job_id}/v1/annotations/stats")
async def get_annotation_stats(job_id: str):
async def get_annotation_stats(
job_id: str,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""Get annotation statistics."""
return {
"total_annotations": 0,
+6
View File
@@ -56,6 +56,12 @@ class Settings(BaseSettings):
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
# Flacfetch remote service (for torrent downloads)
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
# Default distribution settings (can be overridden per-request)
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
+4 -5
View File
@@ -9,16 +9,15 @@ from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.version import VERSION
# Configure logging with trace ID support
logging.basicConfig(
level=getattr(logging, settings.log_level.upper()),
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
# Configure structured logging (JSON in Cloud Run, human-readable locally)
# This must happen before any logging calls
setup_structured_logging()
logger = logging.getLogger(__name__)
# Initialize OpenTelemetry tracing (must happen before app creation)
+6
View File
@@ -251,6 +251,12 @@ class Job(BaseModel):
finalise_only: bool = False # Skip prep, run only finalisation (requires uploaded prep outputs)
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
# Review authentication (Batch 7)
review_token: Optional[str] = None # Job-scoped token for lyrics review UI access (generated when entering AWAITING_REVIEW)
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
# Processing state
track_output_dir: Optional[str] = None # Local output directory (temp)
audio_hash: Optional[str] = None # Hash for deduplication
+331 -198
View File
@@ -1,79 +1,69 @@
"""
Audio search service for finding and downloading audio files.
This service integrates with flacfetch to search for audio from various sources
(YouTube, music trackers, etc.) and download the selected audio file.
This service integrates with karaoke_gen.audio_fetcher (which wraps flacfetch)
to search for audio from various sources (YouTube, music trackers, etc.)
and download the selected audio file.
This is a thin wrapper that adds backend-specific functionality:
- Caching raw results for API-based selection flow
- Firestore-compatible serialization
- Singleton pattern for service lifecycle
- Remote flacfetch service integration for torrent downloads
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
HTTP API for search and download operations. This is required for torrent
downloads since Cloud Run doesn't support BitTorrent peer connections.
"""
import asyncio
import logging
import os
import tempfile
from dataclasses import dataclass, asdict
from typing import List, Optional, Dict, Any
from typing import List, Optional
import nest_asyncio
# Import shared classes from karaoke_gen.audio_fetcher - single source of truth
# Note: Import directly from audio_fetcher module, not karaoke_gen.__init__
# to avoid pulling in heavier dependencies like lyrics_transcriber
from karaoke_gen.audio_fetcher import (
FlacFetcher,
AudioSearchResult,
AudioFetchResult,
AudioFetcherError,
NoResultsError,
DownloadError,
)
from .flacfetch_client import get_flacfetch_client, FlacfetchServiceError
logger = logging.getLogger(__name__)
@dataclass
class AudioSearchResult:
"""
Represents a single search result for audio.
This is a serializable representation of a search result that can be
stored in Firestore and sent to clients.
"""
title: str
artist: str
provider: str
url: str
duration: Optional[int] = None # Duration in seconds
quality: Optional[str] = None # e.g., "FLAC", "320kbps", etc.
source_id: Optional[str] = None # Unique ID from the source
index: int = 0 # Index in the results list (for selection)
def to_dict(self) -> Dict[str, Any]:
"""Convert to dict for JSON serialization."""
return asdict(self)
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioSearchResult":
"""Create from dict."""
return cls(**data)
# Re-export exception under backend's naming convention for compatibility
AudioSearchError = AudioFetcherError
@dataclass
class AudioDownloadResult:
"""Result of downloading an audio file."""
filepath: str
artist: str
title: str
provider: str
duration: Optional[int] = None
quality: Optional[str] = None
class AudioSearchError(Exception):
"""Base exception for audio search errors."""
pass
class NoResultsError(AudioSearchError):
"""Raised when no search results are found."""
pass
class DownloadError(AudioSearchError):
"""Raised when download fails."""
pass
# Also alias AudioFetchResult as AudioDownloadResult for backwards compatibility
AudioDownloadResult = AudioFetchResult
class AudioSearchService:
"""
Service for searching and downloading audio files using flacfetch.
Service for searching and downloading audio files.
This service is used by the backend to:
1. Search for audio by artist/title
2. Return results for user selection
3. Download the selected audio to a local file
This is a thin wrapper around karaoke_gen.audio_fetcher.FlacFetcher that adds
backend-specific functionality:
- Caches raw results for API-based selection (client picks by index)
- Provides Firestore-compatible serialization
- Manages service lifecycle as singleton
- Remote flacfetch service integration for torrent downloads
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
HTTP API for search and download operations. This enables torrent downloads
which are not possible in Cloud Run due to network restrictions.
The actual flacfetch integration is in karaoke_gen.audio_fetcher.FlacFetcher,
which is shared between local CLI and cloud backend.
"""
# Sentinel value to indicate "use environment variable"
@@ -81,72 +71,64 @@ class AudioSearchService:
def __init__(
self,
redacted_api_key: Optional[str] = _USE_ENV,
red_api_key: Optional[str] = _USE_ENV,
red_api_url: Optional[str] = _USE_ENV,
ops_api_key: Optional[str] = _USE_ENV,
ops_api_url: Optional[str] = _USE_ENV,
):
"""
Initialize the audio search service.
Args:
redacted_api_key: API key for Redacted tracker (optional, uses env if not provided)
red_api_key: API key for RED tracker (optional, uses env if not provided)
red_api_url: Base URL for RED tracker API (optional, uses env if not provided)
ops_api_key: API key for OPS tracker (optional, uses env if not provided)
ops_api_url: Base URL for OPS tracker API (optional, uses env if not provided)
"""
# Use environment variables if not explicitly provided
if redacted_api_key is self._USE_ENV:
self._redacted_api_key = os.environ.get("REDACTED_API_KEY")
else:
self._redacted_api_key = redacted_api_key
if red_api_key is self._USE_ENV:
red_api_key = os.environ.get("RED_API_KEY")
if red_api_url is self._USE_ENV:
red_api_url = os.environ.get("RED_API_URL")
if ops_api_key is self._USE_ENV:
self._ops_api_key = os.environ.get("OPS_API_KEY")
else:
self._ops_api_key = ops_api_key
self._manager = None
self._raw_results = {} # Cache raw results for download
def _get_manager(self):
"""Lazily initialize and return the FetchManager."""
if self._manager is None:
try:
from flacfetch.core.manager import FetchManager
from flacfetch.providers.youtube import YouTubeProvider
except ImportError as e:
logger.error(f"flacfetch not installed: {e}")
raise AudioSearchError(
"flacfetch library is not installed. "
"Install with: pip install flacfetch"
) from e
self._manager = FetchManager()
# Add providers based on available API keys
if self._redacted_api_key:
try:
from flacfetch.providers.redacted import RedactedProvider
self._manager.add_provider(RedactedProvider(api_key=self._redacted_api_key))
logger.debug("Added Redacted provider")
except ImportError:
logger.warning("Redacted provider not available")
if self._ops_api_key:
try:
from flacfetch.providers.ops import OPSProvider
self._manager.add_provider(OPSProvider(api_key=self._ops_api_key))
logger.debug("Added OPS provider")
except ImportError:
logger.warning("OPS provider not available")
# Always add YouTube as a fallback provider
self._manager.add_provider(YouTubeProvider())
logger.debug("Added YouTube provider")
ops_api_key = os.environ.get("OPS_API_KEY")
if ops_api_url is self._USE_ENV:
ops_api_url = os.environ.get("OPS_API_URL")
return self._manager
# Check for remote flacfetch client
self._remote_client = get_flacfetch_client()
# Log which mode we're using
if self._remote_client:
logger.info("AudioSearchService using REMOTE flacfetch service (torrent downloads enabled)")
else:
logger.info("AudioSearchService using LOCAL flacfetch (YouTube only in Cloud Run)")
# Delegate to shared FlacFetcher implementation (for local mode or fallback)
self._fetcher = FlacFetcher(
red_api_key=red_api_key,
red_api_url=red_api_url,
ops_api_key=ops_api_key,
ops_api_url=ops_api_url,
)
# Cache search results for API-based selection flow
# Key: index, Value: AudioSearchResult (with raw_result)
self._cached_results: List[AudioSearchResult] = []
# Cache for remote search results (search_id -> results)
self._remote_search_id: Optional[str] = None
def search(self, artist: str, title: str) -> List[AudioSearchResult]:
"""
Search for audio matching the given artist and title.
Results are cached internally for later download via download().
If a remote flacfetch service is configured (FLACFETCH_API_URL), uses that
for better torrent download support. Otherwise falls back to local flacfetch.
Args:
artist: The artist name to search for
title: The track title to search for
@@ -156,49 +138,94 @@ class AudioSearchService:
Raises:
NoResultsError: If no results are found
AudioSearchError: For other errors
AudioSearchError: For other errors (e.g., flacfetch not installed)
"""
try:
from flacfetch.core.models import TrackQuery
except ImportError as e:
raise AudioSearchError("flacfetch not installed") from e
manager = self._get_manager()
query = TrackQuery(artist=artist, title=title)
logger.info(f"Searching for audio: {artist} - {title}")
try:
results = manager.search(query)
# Try remote flacfetch service first if configured
if self._remote_client:
return self._search_remote(artist, title)
# Fallback to local flacfetch
results = self._fetcher.search(artist, title)
# Cache results for later download
self._cached_results = results
self._remote_search_id = None
logger.info(f"Found {len(results)} results for: {artist} - {title}")
return results
except (NoResultsError, AudioFetcherError):
# Re-raise these as-is
raise
except Exception as e:
logger.error(f"Search failed: {e}")
raise AudioSearchError(f"Search failed: {e}") from e
def _search_remote(self, artist: str, title: str) -> List[AudioSearchResult]:
"""
Search using the remote flacfetch service.
if not results:
raise NoResultsError(f"No results found for: {artist} - {title}")
# Convert to our AudioSearchResult format and cache raw results
search_results = []
self._raw_results.clear()
for i, result in enumerate(results):
search_result = AudioSearchResult(
title=getattr(result, "title", title),
artist=getattr(result, "artist", artist),
provider=getattr(result, "provider", "Unknown"),
url=getattr(result, "url", ""),
duration=getattr(result, "duration", None),
quality=getattr(result, "quality", None),
source_id=getattr(result, "id", str(i)),
index=i,
)
search_results.append(search_result)
Runs async code in a sync context for compatibility with existing API.
"""
try:
# Enable nested event loops (needed when called from FastAPI async context)
# Apply to the current loop if one exists, otherwise it's a no-op
try:
loop = asyncio.get_running_loop()
nest_asyncio.apply(loop)
except RuntimeError:
pass # No running loop, nest_asyncio not needed
# Cache raw result for later download
self._raw_results[i] = result
logger.info(f"Found {len(search_results)} results for: {artist} - {title}")
return search_results
# Run async search in sync context
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
response = loop.run_until_complete(
self._remote_client.search(artist, title)
)
finally:
loop.close()
# Check for empty results
if not response.get("results"):
raise NoResultsError(f"No results found for: {artist} - {title}")
# Store search_id for download
self._remote_search_id = response.get("search_id")
# Convert remote results to AudioSearchResult objects
results = []
for item in response.get("results", []):
result = AudioSearchResult(
title=item.get("title", "Unknown"),
artist=item.get("artist", "Unknown"),
url=item.get("download_url", ""), # May be empty for remote
provider=item.get("provider", "Unknown"),
duration=item.get("duration_seconds"),
quality=item.get("quality", ""),
source_id=item.get("info_hash"),
index=item.get("index", 0),
seeders=item.get("seeders"),
target_file=item.get("target_file"),
# Store full remote data in raw_result for rich display
raw_result=item,
)
results.append(result)
# Cache results
self._cached_results = results
logger.info(f"Found {len(results)} results from remote flacfetch for: {artist} - {title}")
return results
except FlacfetchServiceError as e:
logger.warning(f"Remote flacfetch search failed, falling back to local: {e}")
# Fallback to local search
self._remote_search_id = None
results = self._fetcher.search(artist, title)
self._cached_results = results
return results
def select_best(self, results: List[AudioSearchResult]) -> int:
"""
@@ -212,90 +239,142 @@ class AudioSearchService:
Returns:
Index of the best result
"""
if not results:
return 0
manager = self._get_manager()
# Get raw results for the indices we have
raw_results = [self._raw_results.get(r.index) for r in results if r.index in self._raw_results]
if raw_results:
try:
best = manager.select_best(raw_results)
# Find index of best result
for i, raw in enumerate(raw_results):
if raw == best:
return results[i].index
except Exception as e:
logger.warning(f"select_best failed, using first result: {e}")
# Fallback: return first result
return results[0].index if results else 0
return self._fetcher.select_best(results)
def download(
self,
result_index: int,
output_dir: str,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> AudioDownloadResult:
"""
Download audio from a search result.
Download audio from a cached search result.
This method uses the cached results from the last search() call.
The API flow is:
1. Client calls search() -> gets list of results
2. Client picks an index
3. Client calls download(index) -> gets downloaded file
If a remote flacfetch service is configured and the search was performed
remotely (for torrent sources), uses the remote service for download.
Args:
result_index: Index of the result to download (from search results)
output_dir: Directory to save the downloaded file
output_filename: Optional filename (without extension)
gcs_path: Optional GCS path for remote uploads (e.g., "uploads/job123/audio/")
Returns:
AudioDownloadResult with the downloaded file path
Raises:
DownloadError: If download fails
DownloadError: If download fails or no cached result for index
"""
if result_index not in self._raw_results:
raise DownloadError(f"No cached result for index {result_index}. Run search() first.")
if result_index < 0 or result_index >= len(self._cached_results):
raise DownloadError(
f"No cached result for index {result_index}. "
f"Available indices: 0-{len(self._cached_results) - 1}. "
"Run search() first."
)
raw_result = self._raw_results[result_index]
manager = self._get_manager()
result = self._cached_results[result_index]
# Ensure output directory exists
os.makedirs(output_dir, exist_ok=True)
logger.info(f"Downloading: {result.artist} - {result.title} from {result.provider}")
# Get metadata from raw result
artist = getattr(raw_result, "artist", "Unknown")
title = getattr(raw_result, "title", "Unknown")
provider = getattr(raw_result, "provider", "Unknown")
# Check if this was a remote search (torrent sources need remote download)
if self._remote_search_id and self._remote_client:
# Check if this is a torrent source that needs remote handling
is_torrent = result.provider in ["RED", "OPS"]
if is_torrent:
return self._download_remote(result_index, output_dir, output_filename, gcs_path)
# Generate filename if not provided
if output_filename is None:
output_filename = f"{artist} - {title}"
# Delegate to shared FlacFetcher (local download)
fetch_result = self._fetcher.download(result, output_dir, output_filename)
logger.info(f"Downloading: {artist} - {title} from {provider}")
logger.info(f"Downloaded to: {fetch_result.filepath}")
return fetch_result
def _download_remote(
self,
result_index: int,
output_dir: str,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> AudioDownloadResult:
"""
Download using the remote flacfetch service.
The remote service downloads via torrent and optionally uploads to GCS.
"""
if not self._remote_search_id:
raise DownloadError("No remote search ID - run search() first")
result = self._cached_results[result_index]
try:
filepath = manager.download(
raw_result,
output_path=output_dir,
output_filename=output_filename,
)
# Enable nested event loops (needed when called from FastAPI async context)
try:
running_loop = asyncio.get_running_loop()
nest_asyncio.apply(running_loop)
except RuntimeError:
pass # No running loop, nest_asyncio not needed
if filepath is None:
raise DownloadError(f"Download returned no file path for: {artist} - {title}")
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
# Start download
download_id = loop.run_until_complete(
self._remote_client.download(
search_id=self._remote_search_id,
result_index=result_index,
output_filename=output_filename,
gcs_path=gcs_path,
)
)
logger.info(f"Remote download started: {download_id}")
# Wait for completion with progress logging
def log_progress(status):
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
final_status = loop.run_until_complete(
self._remote_client.wait_for_download(
download_id,
timeout=600, # 10 minute timeout
progress_callback=log_progress,
)
)
finally:
loop.close()
logger.info(f"Downloaded to: {filepath}")
# Determine file path
filepath = final_status.get("gcs_path") or final_status.get("output_path")
if not filepath:
raise DownloadError("Remote download completed but no file path returned")
logger.info(f"Remote download complete: {filepath}")
return AudioDownloadResult(
filepath=filepath,
artist=artist,
title=title,
provider=provider,
duration=getattr(raw_result, "duration", None),
quality=getattr(raw_result, "quality", None),
artist=result.artist,
title=result.title,
provider=result.provider,
quality=result.quality,
)
except FlacfetchServiceError as e:
raise DownloadError(f"Remote download failed: {e}") from e
except Exception as e:
raise DownloadError(f"Failed to download {artist} - {title}: {e}") from e
logger.error(f"Remote download error: {e}")
raise DownloadError(f"Remote download failed: {e}") from e
def search_and_download_auto(
self,
@@ -303,6 +382,7 @@ class AudioSearchService:
title: str,
output_dir: str,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> AudioDownloadResult:
"""
Search for audio and automatically download the best result.
@@ -315,6 +395,7 @@ class AudioSearchService:
title: The track title to search for
output_dir: Directory to save the downloaded file
output_filename: Optional filename (without extension)
gcs_path: Optional GCS path for remote uploads
Returns:
AudioDownloadResult with the downloaded file path
@@ -328,10 +409,64 @@ class AudioSearchService:
# Select best
best_index = self.select_best(results)
logger.info(f"Auto-selected result {best_index}: {results[best_index].artist} - {results[best_index].title} ({results[best_index].provider})")
best_result = results[best_index]
logger.info(
f"Auto-selected result {best_index}: "
f"{best_result.artist} - {best_result.title} ({best_result.provider})"
)
# Download
return self.download(best_index, output_dir, output_filename)
return self.download(best_index, output_dir, output_filename, gcs_path)
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
"""
Async version of search for use in async routes.
Note: Currently wraps sync search in executor. Future optimization
could make this fully async.
"""
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, self.search, artist, title)
async def download_async(
self,
result_index: int,
output_dir: str,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> AudioDownloadResult:
"""
Async version of download for use in async routes.
Note: Currently wraps sync download in executor. Future optimization
could make this fully async.
"""
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
None,
lambda: self.download(result_index, output_dir, output_filename, gcs_path)
)
def is_remote_enabled(self) -> bool:
"""Check if remote flacfetch service is configured."""
return self._remote_client is not None
async def check_remote_health(self) -> Optional[dict]:
"""
Check health of remote flacfetch service.
Returns:
Health status dict if remote service is configured and healthy,
None if not configured or unhealthy.
"""
if not self._remote_client:
return None
try:
return await self._remote_client.health_check()
except Exception as e:
logger.warning(f"Remote flacfetch health check failed: {e}")
return None
# Singleton instance
@@ -344,5 +479,3 @@ def get_audio_search_service() -> AudioSearchService:
if _audio_search_service is None:
_audio_search_service = AudioSearchService()
return _audio_search_service
+10
View File
@@ -68,6 +68,16 @@ class AuthService:
logger.info("Admin token validated")
return True, UserType.ADMIN, -1, "Admin access granted"
# Debug: Log token comparison info (only first/last few chars for security)
if self.admin_tokens:
expected_prefix = self.admin_tokens[0][:8] if len(self.admin_tokens[0]) >= 8 else self.admin_tokens[0]
provided_prefix = token[:8] if len(token) >= 8 else token
logger.debug(
f"Token mismatch: expected prefix '{expected_prefix}...', "
f"got '{provided_prefix}...', "
f"expected len={len(self.admin_tokens[0])}, got len={len(token)}"
)
# Check stored tokens in Firestore
token_data = self.firestore.get_token(token)
+412
View File
@@ -0,0 +1,412 @@
"""
Client for the remote flacfetch HTTP API service.
This client communicates with a dedicated flacfetch VM that handles:
- BitTorrent downloads from private trackers (RED, OPS)
- YouTube downloads
- GCS uploads of downloaded files
The flacfetch service provides:
- Full peer connectivity for torrents (not possible in Cloud Run)
- Indefinite seeding of completed torrents
- Automatic disk cleanup
Usage:
client = get_flacfetch_client()
if client:
# Search for audio
search_result = await client.search("ABBA", "Waterloo")
# Download the best result
download_id = await client.download(
search_id=search_result["search_id"],
result_index=0,
gcs_path="uploads/job123/audio/",
)
# Wait for completion
result = await client.wait_for_download(download_id)
print(f"Downloaded to: {result['gcs_path']}")
"""
import asyncio
import logging
from typing import Any, Dict, List, Optional
import httpx
logger = logging.getLogger(__name__)
class FlacfetchServiceError(Exception):
"""Error communicating with or returned from flacfetch service."""
pass
class FlacfetchClient:
"""
Client for remote flacfetch HTTP API.
All endpoints require authentication via X-API-Key header.
"""
def __init__(
self,
base_url: str,
api_key: str,
timeout: int = 60,
):
"""
Initialize flacfetch client.
Args:
base_url: Base URL of flacfetch service (e.g., http://10.0.0.5:8080)
api_key: API key for authentication
timeout: Default request timeout in seconds
"""
self.base_url = base_url.rstrip('/')
self.api_key = api_key
self.timeout = timeout
logger.info(f"FlacfetchClient initialized with base_url={self.base_url}")
def _headers(self) -> Dict[str, str]:
"""Get request headers with authentication."""
return {
"X-API-Key": self.api_key,
"Content-Type": "application/json",
}
async def health_check(self) -> Dict[str, Any]:
"""
Check if flacfetch service is healthy.
Returns:
Health status dict with transmission, disk, and provider info
Raises:
FlacfetchServiceError: If service is unhealthy or unreachable
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/health",
headers=self._headers(),
timeout=10,
)
resp.raise_for_status()
data = resp.json()
if data.get("status") not in ["healthy", "degraded"]:
raise FlacfetchServiceError(f"Service unhealthy: {data}")
return data
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
async def search(
self,
artist: str,
title: str,
) -> Dict[str, Any]:
"""
Search for audio matching artist and title.
Args:
artist: Artist name
title: Track title
Returns:
Search response dict with search_id and results list
Raises:
FlacfetchServiceError: On search failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.post(
f"{self.base_url}/search",
headers=self._headers(),
json={"artist": artist, "title": title},
timeout=self.timeout,
)
if resp.status_code == 404:
# No results found - return empty results
return {
"search_id": None,
"artist": artist,
"title": title,
"results": [],
"results_count": 0,
}
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Search request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
async def download(
self,
search_id: str,
result_index: int,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""
Start downloading an audio file.
Args:
search_id: Search ID from previous search
result_index: Index of result to download
output_filename: Optional custom filename (without extension)
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
Returns:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure
"""
try:
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
"""
Get the current status of a download.
Args:
download_id: Download ID from start_download
Returns:
Status dict with progress, speed, path info
Raises:
FlacfetchServiceError: On status check failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/download/{download_id}/status",
headers=self._headers(),
timeout=10,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Status request failed: {e}")
except httpx.HTTPStatusError as e:
if e.response.status_code == 404:
raise FlacfetchServiceError(f"Download not found: {download_id}")
raise FlacfetchServiceError(f"Status check failed: {e.response.status_code}")
async def wait_for_download(
self,
download_id: str,
timeout: int = 600,
poll_interval: float = 2.0,
progress_callback: Optional[callable] = None,
) -> Dict[str, Any]:
"""
Wait for a download to complete.
Args:
download_id: Download ID to wait for
timeout: Maximum wait time in seconds
poll_interval: Time between status checks
progress_callback: Optional callback(status_dict) for progress updates
Returns:
Final status dict with output_path or gcs_path
Raises:
FlacfetchServiceError: On download failure or timeout
"""
elapsed = 0.0
while elapsed < timeout:
status = await self.get_download_status(download_id)
if progress_callback:
try:
progress_callback(status)
except Exception as e:
logger.warning(f"Progress callback error: {e}")
download_status = status.get("status")
if download_status in ["complete", "seeding"]:
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
return status
elif download_status == "failed":
error = status.get("error", "Unknown error")
raise FlacfetchServiceError(f"Download failed: {error}")
elif download_status == "cancelled":
raise FlacfetchServiceError("Download was cancelled")
# Log progress
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
peers = status.get("peers", 0)
logger.debug(
f"Download {download_id}: {progress:.1f}% "
f"({speed:.1f} KB/s, {peers} peers)"
)
await asyncio.sleep(poll_interval)
elapsed += poll_interval
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
async def list_torrents(self) -> Dict[str, Any]:
"""
List all torrents in Transmission.
Returns:
Torrent list response with torrent info and totals
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/torrents",
headers=self._headers(),
timeout=30,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"List torrents failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
async def delete_torrent(
self,
torrent_id: int,
delete_data: bool = True,
) -> Dict[str, Any]:
"""
Delete a torrent from Transmission.
Args:
torrent_id: Transmission torrent ID
delete_data: Also delete downloaded files
Returns:
Delete response
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.delete(
f"{self.base_url}/torrents/{torrent_id}",
headers=self._headers(),
params={"delete_data": str(delete_data).lower()},
timeout=30,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
async def cleanup_torrents(
self,
strategy: str = "oldest",
target_free_gb: float = 10.0,
) -> Dict[str, Any]:
"""
Trigger disk cleanup.
Args:
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
target_free_gb: Target free space to achieve
Returns:
Cleanup response with removed count and freed space
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.post(
f"{self.base_url}/torrents/cleanup",
headers=self._headers(),
json={
"strategy": strategy,
"target_free_gb": target_free_gb,
},
timeout=60,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
# Singleton client instance
_client: Optional[FlacfetchClient] = None
def get_flacfetch_client() -> Optional[FlacfetchClient]:
"""
Get the flacfetch client if configured.
Returns:
FlacfetchClient instance if FLACFETCH_API_URL and FLACFETCH_API_KEY are set,
None otherwise (indicating local-only mode).
"""
global _client
if _client is None:
from backend.config import get_settings
settings = get_settings()
if settings.flacfetch_api_url and settings.flacfetch_api_key:
_client = FlacfetchClient(
base_url=settings.flacfetch_api_url,
api_key=settings.flacfetch_api_key,
)
logger.info("FlacfetchClient initialized for remote service")
else:
logger.debug("FlacfetchClient not configured (missing URL or API key)")
return _client
def reset_flacfetch_client():
"""Reset the singleton client (for testing)."""
global _client
_client = None
+32
View File
@@ -280,6 +280,24 @@ class JobManager:
if progress is not None:
updates['progress'] = progress
# Generate review token when entering AWAITING_REVIEW state
if new_status == JobStatus.AWAITING_REVIEW:
from backend.api.dependencies import generate_review_token, get_review_token_expiry
review_token = generate_review_token()
review_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
updates['review_token'] = review_token
updates['review_token_expires_at'] = review_token_expires
logger.info(f"Generated review token for job {job_id}, expires in 48 hours")
# Generate instrumental token when entering AWAITING_INSTRUMENTAL_SELECTION state
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
from backend.api.dependencies import generate_review_token, get_review_token_expiry
instrumental_token = generate_review_token() # Reuse same token generator
instrumental_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
updates['instrumental_token'] = instrumental_token
updates['instrumental_token_expires_at'] = instrumental_token_expires
logger.info(f"Generated instrumental token for job {job_id}, expires in 48 hours")
# If we have state_data_updates, merge them with existing state_data
merged_state_data = None
if state_data_updates:
@@ -304,6 +322,20 @@ class JobManager:
message=message
)
# Apply review token update separately if generated
if new_status == JobStatus.AWAITING_REVIEW and 'review_token' in updates:
self.firestore.update_job(job_id, {
'review_token': updates['review_token'],
'review_token_expires_at': updates['review_token_expires_at']
})
# Apply instrumental token update separately if generated
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION and 'instrumental_token' in updates:
self.firestore.update_job(job_id, {
'instrumental_token': updates['instrumental_token'],
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
})
logger.info(f"Job {job_id} transitioned to {new_status}")
return True
+413
View File
@@ -0,0 +1,413 @@
"""
Custom metrics for Cloud Monitoring.
This module provides application-level metrics for tracking job processing,
worker performance, and external API usage. Metrics can be viewed in:
1. Cloud Logging (via structured log entries)
2. Cloud Monitoring (via log-based metrics or OpenTelemetry)
The metrics service uses a pragmatic approach:
- Always emits metrics as structured log entries (works immediately)
- Uses the same JSON format as Cloud Logging
- Can be enhanced with OpenTelemetry metrics exporters when available
Usage:
from backend.services.metrics import metrics
# Record a job completion
metrics.record_job_completed("abc123", source="upload")
# Record worker duration
metrics.record_worker_duration("audio", 45.2, success=True)
# Record external API call
with metrics.time_external_api("modal"):
response = await modal_client.separate_audio(...)
"""
import logging
import time
from contextlib import contextmanager
from dataclasses import dataclass, field
from typing import Any, Dict, Optional
from backend.services.tracing import get_current_trace_id, get_current_span_id
logger = logging.getLogger("metrics")
@dataclass
class MetricLabels:
"""Common metric labels."""
job_id: Optional[str] = None
worker: Optional[str] = None
status: Optional[str] = None
source: Optional[str] = None
api: Optional[str] = None
operation: Optional[str] = None
bucket: Optional[str] = None
def to_dict(self) -> Dict[str, Any]:
"""Convert to dict, excluding None values."""
return {k: v for k, v in self.__dict__.items() if v is not None}
class MetricsService:
"""
Application metrics service.
Emits metrics as structured log entries that can be:
1. Queried directly in Cloud Logging
2. Converted to Cloud Monitoring metrics via log-based metrics
3. Exported via OpenTelemetry (when configured)
"""
def __init__(self):
"""Initialize the metrics service."""
self._logger = logging.getLogger("metrics")
# Ensure metrics logger outputs at INFO level
self._logger.setLevel(logging.INFO)
def _emit_metric(self, metric_name: str, metric_type: str, value: float, labels: Dict[str, Any]) -> None:
"""
Emit a metric as a structured log entry.
The log format is designed to be easily parsed by Cloud Logging
and converted to log-based metrics.
Args:
metric_name: Name of the metric (e.g., "jobs_total")
metric_type: Type of metric (counter, histogram, gauge)
value: Metric value
labels: Metric labels/dimensions
"""
# Add trace context if available
trace_id = get_current_trace_id()
span_id = get_current_span_id()
# Build metric entry
metric_entry = {
"metric_name": metric_name,
"metric_type": metric_type,
"metric_value": value,
**labels,
}
if trace_id:
metric_entry["trace_id"] = trace_id
if span_id:
metric_entry["span_id"] = span_id
# Emit as structured log entry
# Use INFO level so metrics always show up
self._logger.info(
f"METRIC {metric_name}={value}",
extra=metric_entry
)
# =========================================
# Job Metrics
# =========================================
def record_job_created(self, job_id: str, source: str = "unknown") -> None:
"""
Record a new job creation.
Args:
job_id: Job ID
source: Job source (upload, url, search)
"""
self._emit_metric(
metric_name="jobs_total",
metric_type="counter",
value=1,
labels={"job_id": job_id, "status": "created", "source": source}
)
def record_job_completed(self, job_id: str, source: str = "unknown") -> None:
"""
Record a job completion.
Args:
job_id: Job ID
source: Job source (upload, url, search)
"""
self._emit_metric(
metric_name="jobs_total",
metric_type="counter",
value=1,
labels={"job_id": job_id, "status": "completed", "source": source}
)
def record_job_failed(self, job_id: str, source: str = "unknown", error: Optional[str] = None) -> None:
"""
Record a job failure.
Args:
job_id: Job ID
source: Job source (upload, url, search)
error: Optional error message
"""
labels = {"job_id": job_id, "status": "failed", "source": source}
if error:
labels["error"] = error[:200] # Truncate long errors
self._emit_metric(
metric_name="jobs_total",
metric_type="counter",
value=1,
labels=labels
)
def record_job_duration(self, job_id: str, duration_seconds: float, source: str = "unknown") -> None:
"""
Record total job processing duration.
Args:
job_id: Job ID
duration_seconds: Total processing time in seconds
source: Job source (upload, url, search)
"""
self._emit_metric(
metric_name="job_duration_seconds",
metric_type="histogram",
value=duration_seconds,
labels={"job_id": job_id, "source": source}
)
# =========================================
# Worker Metrics
# =========================================
def record_worker_started(self, worker: str, job_id: str) -> None:
"""
Record a worker invocation start.
Args:
worker: Worker name (audio, lyrics, screens, video, render_video)
job_id: Job ID
"""
self._emit_metric(
metric_name="worker_invocations_total",
metric_type="counter",
value=1,
labels={"worker": worker, "job_id": job_id, "status": "started"}
)
def record_worker_duration(self, worker: str, duration_seconds: float, success: bool, job_id: Optional[str] = None) -> None:
"""
Record worker processing duration.
Args:
worker: Worker name
duration_seconds: Processing time in seconds
success: Whether worker completed successfully
job_id: Optional job ID
"""
labels = {
"worker": worker,
"success": str(success).lower(),
}
if job_id:
labels["job_id"] = job_id
self._emit_metric(
metric_name="job_stage_duration_seconds",
metric_type="histogram",
value=duration_seconds,
labels=labels
)
# Also emit a counter for success/failure tracking
self._emit_metric(
metric_name="worker_invocations_total",
metric_type="counter",
value=1,
labels={"worker": worker, "success": str(success).lower(), "job_id": job_id or "unknown"}
)
# =========================================
# GCS Metrics
# =========================================
def record_gcs_operation(
self,
operation: str,
bucket: str,
success: bool,
size_bytes: Optional[int] = None,
duration_seconds: Optional[float] = None,
job_id: Optional[str] = None,
) -> None:
"""
Record a GCS operation.
Args:
operation: Operation type (upload, download, delete)
bucket: GCS bucket name
success: Whether operation succeeded
size_bytes: Optional file size in bytes
duration_seconds: Optional operation duration
job_id: Optional job ID
"""
labels = {
"operation": operation,
"bucket": bucket,
"success": str(success).lower(),
}
if job_id:
labels["job_id"] = job_id
self._emit_metric(
metric_name="gcs_operations_total",
metric_type="counter",
value=1,
labels=labels
)
if size_bytes is not None:
self._emit_metric(
metric_name="gcs_operation_bytes",
metric_type="histogram",
value=size_bytes,
labels={**labels, "operation": operation}
)
if duration_seconds is not None:
self._emit_metric(
metric_name="gcs_operation_duration_seconds",
metric_type="histogram",
value=duration_seconds,
labels={**labels, "operation": operation}
)
# =========================================
# External API Metrics
# =========================================
def record_external_api_call(
self,
api: str,
success: bool,
duration_seconds: float,
job_id: Optional[str] = None,
error: Optional[str] = None,
) -> None:
"""
Record an external API call.
Args:
api: API name (modal, audioshake, genius, spotify)
success: Whether call succeeded
duration_seconds: API call duration
job_id: Optional job ID
error: Optional error message
"""
labels = {
"api": api,
"success": str(success).lower(),
}
if job_id:
labels["job_id"] = job_id
if error:
labels["error"] = error[:100] # Truncate
self._emit_metric(
metric_name="external_api_calls_total",
metric_type="counter",
value=1,
labels=labels
)
self._emit_metric(
metric_name="external_api_duration_seconds",
metric_type="histogram",
value=duration_seconds,
labels={"api": api, "success": str(success).lower()}
)
@contextmanager
def time_external_api(self, api: str, job_id: Optional[str] = None):
"""
Context manager to time an external API call.
Usage:
with metrics.time_external_api("modal", job_id) as timer:
response = await client.call_api()
timer.set_success(True)
Args:
api: API name
job_id: Optional job ID
Yields:
Timer object with set_success() method
"""
timer = _ApiTimer()
start_time = time.time()
try:
yield timer
except Exception as e:
timer.set_success(False)
timer.error = str(e)
raise
finally:
duration = time.time() - start_time
self.record_external_api_call(
api=api,
success=timer.success,
duration_seconds=duration,
job_id=job_id,
error=timer.error,
)
@contextmanager
def time_worker(self, worker: str, job_id: str):
"""
Context manager to time a worker execution.
Usage:
with metrics.time_worker("audio", job_id) as timer:
await process_audio()
timer.set_success(True)
Args:
worker: Worker name
job_id: Job ID
Yields:
Timer object with set_success() method
"""
timer = _ApiTimer()
self.record_worker_started(worker, job_id)
start_time = time.time()
try:
yield timer
except Exception as e:
timer.set_success(False)
raise
finally:
duration = time.time() - start_time
self.record_worker_duration(
worker=worker,
duration_seconds=duration,
success=timer.success,
job_id=job_id,
)
class _ApiTimer:
"""Helper class for tracking API call success state."""
def __init__(self):
self.success = True # Assume success unless set otherwise
self.error: Optional[str] = None
def set_success(self, success: bool) -> None:
self.success = success
# Global metrics instance
metrics = MetricsService()
+246
View File
@@ -0,0 +1,246 @@
"""
Structured logging with trace correlation for Cloud Logging.
This module provides JSON-formatted logging that integrates with Google Cloud Logging
and correlates logs with OpenTelemetry traces in Cloud Trace.
When running in Cloud Run:
- Logs are output as JSON for Cloud Logging to parse
- Each log entry includes trace ID and span ID for correlation
- Custom fields (job_id, worker) are preserved in the log structure
Usage:
# In main.py, before any logging:
from backend.services.structured_logging import setup_structured_logging
setup_structured_logging()
# Then use standard logging:
logger = logging.getLogger(__name__)
logger.info("Processing started", extra={"job_id": "abc123"})
"""
import json
import logging
import os
import sys
from datetime import datetime
from typing import Any, Dict, Optional
# Cloud Logging severity mapping (compatible with GCP)
# https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity
SEVERITY_MAP = {
"DEBUG": "DEBUG",
"INFO": "INFO",
"WARNING": "WARNING",
"ERROR": "ERROR",
"CRITICAL": "CRITICAL",
}
class StructuredFormatter(logging.Formatter):
"""
JSON formatter with trace correlation for Google Cloud Logging.
Produces log entries in the format expected by Cloud Logging, including:
- logging.googleapis.com/trace: Links to Cloud Trace
- logging.googleapis.com/spanId: Current span ID
- severity: Cloud Logging severity level
- Custom fields passed via 'extra' dict
"""
def __init__(self, project_id: Optional[str] = None):
"""
Initialize the structured formatter.
Args:
project_id: GCP project ID for trace URLs (auto-detected if not provided)
"""
super().__init__()
self.project_id = project_id or os.environ.get("GOOGLE_CLOUD_PROJECT") or os.environ.get("GCP_PROJECT")
def format(self, record: logging.LogRecord) -> str:
"""
Format a log record as JSON.
Args:
record: The log record to format
Returns:
JSON string for Cloud Logging
"""
from backend.services.tracing import get_current_trace_id, get_current_span_id
# Build base log entry
log_entry: Dict[str, Any] = {
"timestamp": datetime.utcfromtimestamp(record.created).isoformat() + "Z",
"severity": SEVERITY_MAP.get(record.levelname, record.levelname),
"message": record.getMessage(),
"logger": record.name,
}
# Add trace correlation if available
trace_id = get_current_trace_id()
span_id = get_current_span_id()
if trace_id and self.project_id:
# Format for Cloud Logging trace correlation
log_entry["logging.googleapis.com/trace"] = f"projects/{self.project_id}/traces/{trace_id}"
if span_id:
log_entry["logging.googleapis.com/spanId"] = span_id
# Add custom fields from 'extra' dict
# Common fields we want to extract from log records
custom_fields = ["job_id", "worker", "operation", "duration", "status", "error"]
for field in custom_fields:
value = getattr(record, field, None)
if value is not None:
log_entry[field] = value
# Add source location for debugging
if record.pathname and record.lineno:
log_entry["logging.googleapis.com/sourceLocation"] = {
"file": record.pathname,
"line": record.lineno,
"function": record.funcName,
}
# Add exception info if present
if record.exc_info:
log_entry["exception"] = self.formatException(record.exc_info)
# Remove None values for cleaner output
return json.dumps({k: v for k, v in log_entry.items() if v is not None})
class HumanReadableFormatter(logging.Formatter):
"""
Human-readable formatter for local development.
Includes trace context when available but outputs in traditional format.
"""
def format(self, record: logging.LogRecord) -> str:
"""Format a log record for human readability."""
from backend.services.tracing import get_current_trace_id
# Build base message
timestamp = datetime.utcfromtimestamp(record.created).strftime("%Y-%m-%d %H:%M:%S")
# Add job_id prefix if available
job_id = getattr(record, "job_id", None)
job_prefix = f"[job:{job_id[:8]}] " if job_id else ""
# Add trace ID suffix if available
trace_id = get_current_trace_id()
trace_suffix = f" [trace:{trace_id[:8]}]" if trace_id else ""
message = f"{timestamp} - {record.name} - {record.levelname} - {job_prefix}{record.getMessage()}{trace_suffix}"
# Add exception info if present
if record.exc_info:
message += "\n" + self.formatException(record.exc_info)
return message
def is_cloud_run() -> bool:
"""Check if we're running in Cloud Run."""
return os.environ.get("K_SERVICE") is not None
def setup_structured_logging(force_json: bool = False, log_level: Optional[str] = None) -> None:
"""
Configure structured logging for the application.
In Cloud Run: Uses JSON format with trace correlation for Cloud Logging
In development: Uses human-readable format with optional trace context
Args:
force_json: Force JSON output even in development
log_level: Override log level (default from settings or INFO)
"""
from backend.config import settings
# Determine log level
level = log_level or getattr(settings, "log_level", "INFO")
level = getattr(logging, level.upper(), logging.INFO)
# Choose formatter based on environment
if is_cloud_run() or force_json:
formatter = StructuredFormatter()
else:
formatter = HumanReadableFormatter()
# Create handler
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(formatter)
handler.setLevel(level)
# Configure root logger
root_logger = logging.getLogger()
root_logger.setLevel(level)
# Remove existing handlers and add our handler
root_logger.handlers = [handler]
# Also configure uvicorn loggers to use our format
for logger_name in ["uvicorn", "uvicorn.access", "uvicorn.error"]:
logger = logging.getLogger(logger_name)
logger.handlers = [handler]
logger.propagate = False
class JobLogAdapter(logging.LoggerAdapter):
"""
Logger adapter that automatically adds job context to log records.
Usage:
logger = logging.getLogger(__name__)
job_logger = JobLogAdapter(logger, job_id="abc123", worker="audio")
job_logger.info("Processing started") # Automatically includes job_id and worker
"""
def __init__(self, logger: logging.Logger, job_id: str, worker: Optional[str] = None, **extra):
"""
Initialize the adapter.
Args:
logger: Base logger instance
job_id: Job ID to include in all log records
worker: Optional worker name
**extra: Additional fields to include in all log records
"""
super().__init__(logger, {"job_id": job_id, "worker": worker, **extra})
def process(self, msg: str, kwargs: Dict[str, Any]) -> tuple:
"""Process a logging call to add extra context."""
# Merge extra dict from adapter with any extra passed to the log call
extra = kwargs.get("extra", {})
extra.update(self.extra)
kwargs["extra"] = extra
return msg, kwargs
def get_job_logger(job_id: str, worker: Optional[str] = None, name: Optional[str] = None) -> JobLogAdapter:
"""
Get a logger adapter configured for a specific job.
This is a convenience function for creating job-specific loggers.
Args:
job_id: Job ID to include in all log records
worker: Optional worker name
name: Logger name (defaults to "job")
Returns:
JobLogAdapter configured for the job
Usage:
logger = get_job_logger("abc123", "audio")
logger.info("Starting audio separation")
logger.error("Failed to process", extra={"error": str(e)})
"""
base_logger = logging.getLogger(name or "job")
return JobLogAdapter(base_logger, job_id=job_id, worker=worker)
+161
View File
@@ -19,6 +19,25 @@ Usage:
with tracer.start_as_current_span("my-operation") as span:
span.set_attribute("job_id", job_id)
# ... do work ...
# For job-specific spans with automatic job_id attribute
from backend.services.tracing import job_span
with job_span("audio-worker", job_id) as span:
# job_id is automatically set as an attribute
span.set_attribute("stage", "separation")
# ... do work ...
# For propagating trace context through Cloud Tasks
from backend.services.tracing import inject_trace_context, extract_trace_context
# In worker_service.py (when creating task):
headers = inject_trace_context({})
# In internal.py (when receiving task):
context = extract_trace_context(request.headers)
with tracer.start_as_current_span("worker", context=context):
...
"""
import os
import logging
@@ -380,3 +399,145 @@ def get_current_span_id() -> Optional[str]:
if ctx.is_valid:
return format(ctx.span_id, '016x')
return None
# =====================================================
# Job-Aware Tracing Utilities
# =====================================================
@contextmanager
def job_span(
name: str,
job_id: str,
attributes: Optional[Dict[str, Any]] = None,
kind: trace.SpanKind = trace.SpanKind.INTERNAL,
):
"""
Create a traced span with job_id automatically set as an attribute.
This is the preferred way to create spans in job processing workers.
The job_id attribute enables filtering spans by job in Cloud Trace.
Usage:
with job_span("audio-separation", job_id) as span:
span.set_attribute("stage", "clean_instrumental")
# ... do work ...
Args:
name: Span name (appears in Cloud Trace)
job_id: Job ID to attach to the span
attributes: Additional span attributes
kind: Span kind (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER)
Yields:
The active span
"""
all_attributes = {
"job_id": job_id,
"service.operation": name,
}
if attributes:
all_attributes.update(attributes)
with create_span(name, all_attributes, kind) as span:
yield span
def inject_trace_context(headers: Dict[str, str]) -> Dict[str, str]:
"""
Inject current trace context into HTTP headers.
Use this when creating Cloud Tasks or making HTTP calls to propagate
the trace context so child spans link to the parent trace.
Usage:
headers = inject_trace_context({
"Content-Type": "application/json",
"Authorization": "Bearer token",
})
# Use headers in HTTP request or Cloud Task
Args:
headers: Existing headers dict to add trace context to
Returns:
Headers dict with traceparent/tracestate headers added
"""
propagator = TraceContextTextMapPropagator()
carrier = dict(headers) # Make a copy
propagator.inject(carrier)
return carrier
def extract_trace_context(headers: Dict[str, str]) -> Optional[trace.Context]:
"""
Extract trace context from HTTP headers.
Use this when receiving Cloud Tasks or HTTP requests to continue
the trace from the parent span.
Usage:
context = extract_trace_context(request.headers)
with tracer.start_as_current_span("worker", context=context):
# ... processing ...
Args:
headers: HTTP headers containing traceparent/tracestate
Returns:
Context object for use with start_as_current_span, or None
"""
propagator = TraceContextTextMapPropagator()
return propagator.extract(headers)
def start_span_with_context(
name: str,
context: Optional[trace.Context],
job_id: Optional[str] = None,
attributes: Optional[Dict[str, Any]] = None,
):
"""
Start a span as child of the given context (or current context if None).
This is useful for starting worker spans that should link to the
trace from the original API request.
Usage:
context = extract_trace_context(request.headers)
with start_span_with_context("audio-worker", context, job_id) as span:
# ... processing ...
Args:
name: Span name
context: Parent context (from extract_trace_context) or None
job_id: Optional job ID to set as attribute
attributes: Additional span attributes
Returns:
Context manager yielding the span
"""
tracer = get_tracer()
all_attributes = attributes or {}
if job_id:
all_attributes["job_id"] = job_id
all_attributes["service.operation"] = name
@contextmanager
def _span():
kwargs = {"kind": trace.SpanKind.INTERNAL}
if context is not None:
kwargs["context"] = context
with tracer.start_as_current_span(name, **kwargs) as span:
for key, value in all_attributes.items():
span.set_attribute(key, value)
try:
yield span
except Exception as e:
span.set_status(Status(StatusCode.ERROR, str(e)))
span.record_exception(e)
raise
return _span()
+42 -14
View File
@@ -10,6 +10,10 @@ SOLID Principles:
- Single Responsibility: Only handles worker coordination
- Dependency Inversion: Depends on HTTP abstraction, not implementation
- Open/Closed: Can add new workers without modifying existing code
Observability:
- Propagates trace context through Cloud Tasks for distributed tracing
- All worker invocations create spans linked to original request trace
"""
import logging
import os
@@ -18,6 +22,7 @@ from typing import Optional
import httpx
from backend.config import get_settings
from backend.services.tracing import inject_trace_context
logger = logging.getLogger(__name__)
@@ -180,6 +185,9 @@ class WorkerService:
- Deduplication (via task name if needed)
- OIDC authentication for Cloud Run
Observability:
- Injects trace context headers so worker spans link to parent trace
Args:
worker_type: Worker type
job_id: Job ID to process
@@ -205,14 +213,33 @@ class WorkerService:
# Build queue path
parent = self.tasks_client.queue_path(project, location, queue_name)
# Build base headers
headers = {
"Content-Type": "application/json",
}
# Add admin auth via custom header
# NOTE: We use X-Admin-Token instead of Authorization because Cloud Tasks
# OIDC token overwrites the Authorization header when oidc_token is specified.
# The OIDC token handles Cloud Run authentication (allows Cloud Tasks to invoke
# the service), while X-Admin-Token handles application-level authentication.
if self._admin_token:
headers["X-Admin-Token"] = self._admin_token
logger.debug(
f"[job:{job_id}] Using admin token for Cloud Task auth via X-Admin-Token, "
f"token prefix: {self._admin_token[:8]}..., len={len(self._admin_token)}"
)
# Inject trace context for distributed tracing
# This allows worker spans to link back to the original request trace
headers = inject_trace_context(headers)
# Build task payload
task = {
"http_request": {
"http_method": tasks_v2.HttpMethod.POST,
"url": f"{self._base_url}/api/internal/workers/{worker_type}",
"headers": {
"Content-Type": "application/json",
},
"headers": headers,
"body": json.dumps({"job_id": job_id}).encode(),
# Use OIDC token for Cloud Run authentication
# This allows Cloud Tasks to invoke the Cloud Run service
@@ -222,23 +249,18 @@ class WorkerService:
},
}
# Add admin auth header if available
# This is needed for the internal endpoint's require_admin dependency
if self._admin_token:
task["http_request"]["headers"]["Authorization"] = f"Bearer {self._admin_token}"
# Create task
# Note: We don't set a task name, allowing Cloud Tasks to generate one
# This prevents duplicate task errors on retries
response = self.tasks_client.create_task(parent=parent, task=task)
logger.info(
f"Created Cloud Task for {worker_type} worker, job {job_id}: {response.name}"
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name}"
)
return True
except Exception as e:
logger.error(
f"Failed to enqueue Cloud Task for {worker_type}/{job_id}: {e}",
f"[job:{job_id}] Failed to enqueue Cloud Task for {worker_type}: {e}",
exc_info=True
)
return False
@@ -255,6 +277,9 @@ class WorkerService:
This is the original implementation - direct HTTP POST to internal endpoint.
The endpoint adds the worker function to FastAPI BackgroundTasks.
Observability:
- Injects trace context headers so worker spans link to parent trace
Args:
worker_type: Worker type
job_id: Job ID to process
@@ -269,6 +294,9 @@ class WorkerService:
if self._admin_token:
headers["Authorization"] = f"Bearer {self._admin_token}"
# Inject trace context for distributed tracing
headers = inject_trace_context(headers)
async with httpx.AsyncClient(timeout=timeout_seconds) as client:
url = f"{self._base_url}/api/internal/workers/{worker_type}"
@@ -279,22 +307,22 @@ class WorkerService:
)
if response.status_code == 200:
logger.info(f"Successfully triggered {worker_type} worker for job {job_id}")
logger.info(f"[job:{job_id}] Successfully triggered {worker_type} worker")
return True
else:
logger.error(
f"Failed to trigger {worker_type} worker for job {job_id}: "
f"[job:{job_id}] Failed to trigger {worker_type} worker: "
f"HTTP {response.status_code} - {response.text}"
)
return False
except httpx.TimeoutException:
logger.error(f"Timeout triggering {worker_type} worker for job {job_id}")
logger.error(f"[job:{job_id}] Timeout triggering {worker_type} worker")
return False
except Exception as e:
logger.error(
f"Error triggering {worker_type} worker for job {job_id}: {e}",
f"[job:{job_id}] Error triggering {worker_type} worker: {e}",
exc_info=True
)
return False
+57
View File
@@ -34,6 +34,63 @@ if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
from backend.models.job import Job, JobStatus, JobCreate
@pytest.fixture(autouse=True)
def mock_auth_dependency(request):
"""Mock the require_auth dependency for all tests using FastAPI's dependency override system."""
# Skip for emulator tests and integration tests which use real auth
test_path = str(request.fspath)
if 'emulator' in test_path or 'integration' in test_path:
yield
return
# Skip if FIRESTORE_EMULATOR_HOST is set (running in emulator environment)
import os
if os.environ.get('FIRESTORE_EMULATOR_HOST'):
yield
return
from backend.services.auth_service import UserType
from backend.api.dependencies import require_auth, require_admin, require_review_auth, require_instrumental_auth
from backend.main import app
# Create mock auth functions
async def mock_require_auth():
"""Mock require_auth to always return valid admin credentials."""
return ("test-admin-token", UserType.ADMIN, 999)
async def mock_require_admin():
"""Mock require_admin to always return valid admin credentials."""
return ("test-admin-token", UserType.ADMIN, 999)
async def mock_require_review_auth(job_id: str = "test123"):
"""Mock require_review_auth to always return valid review access."""
return (job_id, "full")
async def mock_require_instrumental_auth(job_id: str = "test123"):
"""Mock require_instrumental_auth to always return valid instrumental review access."""
return (job_id, "full")
# Use FastAPI's dependency override system
app.dependency_overrides[require_auth] = mock_require_auth
app.dependency_overrides[require_admin] = mock_require_admin
app.dependency_overrides[require_review_auth] = mock_require_review_auth
app.dependency_overrides[require_instrumental_auth] = mock_require_instrumental_auth
yield
# Clean up after test
app.dependency_overrides.clear()
@pytest.fixture
def auth_headers():
"""Get authentication headers for test requests."""
return {
"Authorization": "Bearer test-admin-token",
"Content-Type": "application/json"
}
@pytest.fixture
def mock_firestore():
"""Mock Firestore client for unit tests."""
@@ -27,15 +27,15 @@ pytestmark = pytest.mark.skipif(
class TestEmulatorBasics:
"""Basic emulator connectivity tests."""
def test_health_endpoint(self, client):
def test_health_endpoint(self, client, auth_headers):
"""Test health endpoint works."""
response = client.get("/api/health")
response = client.get("/api/health", )
assert response.status_code == 200
assert response.json()["status"] == "healthy"
def test_root_endpoint(self, client):
def test_root_endpoint(self, client, auth_headers):
"""Test root endpoint works."""
response = client.get("/")
response = client.get("/", )
assert response.status_code == 200
assert response.json()["service"] == "karaoke-gen-backend"
@@ -281,13 +281,13 @@ class TestReviewEndpoints:
time.sleep(0.2)
# Should fail - job not in AWAITING_REVIEW status
response = client.get(f"/api/review/{job_id}/correction-data")
response = client.get(f"/api/review/{job_id}/correction-data", headers=auth_headers)
assert response.status_code == 400
assert "not ready for review" in response.json()["detail"].lower()
def test_review_audio_no_job(self, client):
def test_review_audio_no_job(self, client, auth_headers):
"""Test audio endpoint returns 404 for nonexistent job."""
response = client.get("/api/review/nonexistent-job/audio/")
response = client.get("/api/review/nonexistent-job/audio/", headers=auth_headers)
assert response.status_code == 404
def test_review_preview_video_stub(self, client, auth_headers):
@@ -303,6 +303,7 @@ class TestReviewEndpoints:
# Preview video should return error since job not ready
response = client.post(
f"/api/review/{job_id}/preview-video",
headers=auth_headers,
json={"corrections": [], "corrected_segments": []}
)
# Should return 400 (job not in AWAITING_REVIEW state)
@@ -320,6 +321,7 @@ class TestReviewEndpoints:
# Annotations endpoint should accept data (even if it just logs it)
response = client.post(
f"/api/review/{job_id}/v1/annotations",
headers=auth_headers,
json={"type": "test", "data": "test annotation"}
)
assert response.status_code == 200
+6 -6
View File
@@ -28,20 +28,20 @@ class TestHealthRoutes:
from backend.main import app
return TestClient(app)
def test_health_endpoint_returns_200(self, client):
def test_health_endpoint_returns_200(self, client, auth_headers):
"""Test /api/health returns 200 OK."""
response = client.get("/api/health")
response = client.get("/api/health", )
assert response.status_code == 200
def test_health_endpoint_returns_healthy_status(self, client):
def test_health_endpoint_returns_healthy_status(self, client, auth_headers):
"""Test health endpoint returns healthy status."""
response = client.get("/api/health")
response = client.get("/api/health", )
data = response.json()
assert data["status"] == "healthy"
def test_root_endpoint_returns_200(self, client):
def test_root_endpoint_returns_200(self, client, auth_headers):
"""Test root endpoint returns 200."""
response = client.get("/")
response = client.get("/", )
assert response.status_code == 200
@@ -0,0 +1,294 @@
"""
Tests for audio_analysis_service.py - Audio analysis for backing vocals.
These tests mock the storage service and shared karaoke_gen modules to verify:
- GCS file download/upload operations
- Audio analysis delegation to shared AudioAnalyzer
- Waveform generation delegation to shared WaveformGenerator
"""
import pytest
from unittest.mock import Mock, MagicMock, patch, ANY
import tempfile
import os
class TestAudioAnalysisServiceInit:
"""Test AudioAnalysisService initialization."""
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_init_creates_dependencies(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test initialization creates storage service and analyzers."""
from backend.services.audio_analysis_service import AudioAnalysisService
service = AudioAnalysisService()
mock_storage_class.assert_called_once()
mock_analyzer_class.assert_called_once_with(
silence_threshold_db=-40.0,
min_segment_duration_ms=100,
)
mock_waveform_class.assert_called_once()
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_init_with_custom_params(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test initialization with custom threshold and duration."""
from backend.services.audio_analysis_service import AudioAnalysisService
service = AudioAnalysisService(
silence_threshold_db=-50.0,
min_segment_duration_ms=200,
)
mock_analyzer_class.assert_called_once_with(
silence_threshold_db=-50.0,
min_segment_duration_ms=200,
)
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_init_with_provided_storage_service(
self, mock_waveform_class, mock_analyzer_class
):
"""Test initialization with externally provided storage service."""
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
service = AudioAnalysisService(storage_service=mock_storage)
assert service.storage_service is mock_storage
class TestAnalyzeBackingVocals:
"""Test analyze_backing_vocals method."""
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_analyze_downloads_and_analyzes(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test analyze_backing_vocals downloads file and runs analysis."""
from backend.services.audio_analysis_service import AudioAnalysisService
# Setup mocks
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.has_audible_content = True
mock_result.segment_count = 5
mock_result.recommended_selection = Mock(value="with_backing")
mock_analyzer = Mock()
mock_analyzer.analyze.return_value = mock_result
mock_analyzer_class.return_value = mock_analyzer
service = AudioAnalysisService()
# Call method
result = service.analyze_backing_vocals(
gcs_audio_path="uploads/job123/backing_vocals.flac",
job_id="job123",
)
# Verify storage download was called
mock_storage.download_file.assert_called_once()
call_args = mock_storage.download_file.call_args
assert call_args[0][0] == "uploads/job123/backing_vocals.flac"
# Verify analyzer was called
mock_analyzer.analyze.assert_called_once()
# Verify result
assert result is mock_result
assert result.has_audible_content is True
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_analyze_cleans_up_temp_files(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test that temp files are cleaned up after analysis."""
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.has_audible_content = False
mock_result.segment_count = 0
mock_result.recommended_selection = Mock(value="clean")
mock_analyzer = Mock()
mock_analyzer.analyze.return_value = mock_result
mock_analyzer_class.return_value = mock_analyzer
service = AudioAnalysisService()
# Call method
service.analyze_backing_vocals(
gcs_audio_path="uploads/job123/backing.flac",
job_id="job123",
)
# Temp directory should be cleaned up (no assertion needed,
# TemporaryDirectory context manager handles this)
# Just verify the method completes without error
class TestAnalyzeAndGenerateWaveform:
"""Test analyze_and_generate_waveform method."""
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_analyze_and_generate_waveform(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test analyze_and_generate_waveform does both operations."""
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.has_audible_content = True
mock_result.audible_segments = [Mock(), Mock()]
mock_analyzer = Mock()
mock_analyzer.analyze.return_value = mock_result
mock_analyzer.silence_threshold_db = -40.0
mock_analyzer_class.return_value = mock_analyzer
mock_waveform = Mock()
mock_waveform_class.return_value = mock_waveform
service = AudioAnalysisService()
# Call method
result, waveform_path = service.analyze_and_generate_waveform(
gcs_audio_path="uploads/job123/backing.flac",
job_id="job123",
gcs_waveform_destination="uploads/job123/waveform.png",
)
# Verify analysis was performed
mock_analyzer.analyze.assert_called_once()
# Verify waveform was generated
mock_waveform.generate.assert_called_once()
call_kwargs = mock_waveform.generate.call_args.kwargs
assert call_kwargs["segments"] == mock_result.audible_segments
assert call_kwargs["show_time_axis"] is True
# Verify waveform was uploaded
assert mock_storage.upload_file.call_count == 1
# Verify return values
assert result is mock_result
assert waveform_path == "uploads/job123/waveform.png"
class TestGetWaveformData:
"""Test get_waveform_data method."""
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_get_waveform_data(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test get_waveform_data returns amplitude data."""
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_waveform = Mock()
mock_waveform.generate_data_only.return_value = (
[0.1, 0.3, 0.5, 0.3, 0.1], # amplitudes
180.5, # duration
)
mock_waveform_class.return_value = mock_waveform
service = AudioAnalysisService()
amplitudes, duration = service.get_waveform_data(
gcs_audio_path="uploads/job123/backing.flac",
job_id="job123",
num_points=500,
)
# Verify storage download
mock_storage.download_file.assert_called_once()
# Verify waveform generator called
mock_waveform.generate_data_only.assert_called_once()
call_kwargs = mock_waveform.generate_data_only.call_args.kwargs
assert call_kwargs["num_points"] == 500
# Verify return values
assert amplitudes == [0.1, 0.3, 0.5, 0.3, 0.1]
assert duration == 180.5
class TestGenerateWaveformWithMutes:
"""Test generate_waveform_with_mutes method."""
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_generate_waveform_with_mutes(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
"""Test generate_waveform_with_mutes highlights mute regions."""
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.audible_segments = [Mock(), Mock()]
mock_analyzer = Mock()
mock_analyzer.analyze.return_value = mock_result
mock_analyzer_class.return_value = mock_analyzer
mock_waveform = Mock()
mock_waveform_class.return_value = mock_waveform
# Create mock mute regions
mock_mute1 = Mock()
mock_mute2 = Mock()
mute_regions = [mock_mute1, mock_mute2]
service = AudioAnalysisService()
result_path = service.generate_waveform_with_mutes(
gcs_audio_path="uploads/job123/backing.flac",
job_id="job123",
gcs_waveform_destination="uploads/job123/waveform_muted.png",
mute_regions=mute_regions,
)
# Verify waveform generator was called with mute_regions
mock_waveform.generate.assert_called_once()
call_kwargs = mock_waveform.generate.call_args.kwargs
assert call_kwargs["mute_regions"] == mute_regions
assert call_kwargs["segments"] == mock_result.audible_segments
# Verify upload
mock_storage.upload_file.assert_called_once()
# Verify return
assert result_path == "uploads/job123/waveform_muted.png"
+386
View File
@@ -0,0 +1,386 @@
"""
Tests for audio_editing_service.py - Custom instrumental creation.
These tests mock the storage service and shared karaoke_gen modules to verify:
- GCS file download/upload operations
- Audio editing delegation to shared AudioEditor
- Mute region validation
"""
import pytest
from unittest.mock import Mock, MagicMock, patch, ANY
class TestAudioEditingServiceInit:
"""Test AudioEditingService initialization."""
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_init_creates_dependencies(self, mock_editor_class, mock_storage_class):
"""Test initialization creates storage service and editor."""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService()
mock_storage_class.assert_called_once()
mock_editor_class.assert_called_once_with(output_format="flac")
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_init_with_custom_format(self, mock_editor_class, mock_storage_class):
"""Test initialization with custom output format."""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService(output_format="wav")
mock_editor_class.assert_called_once_with(output_format="wav")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_init_with_provided_storage_service(self, mock_editor_class):
"""Test initialization with externally provided storage service."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
service = AudioEditingService(storage_service=mock_storage)
assert service.storage_service is mock_storage
class TestCreateCustomInstrumental:
"""Test create_custom_instrumental method."""
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_create_custom_instrumental_success(
self, mock_editor_class, mock_storage_class
):
"""Test creating a custom instrumental successfully."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.total_muted_duration_seconds = 15.5
mock_result.mute_regions_applied = [Mock(), Mock(), Mock()]
mock_result.output_path = None # Will be set by service
mock_editor = Mock()
mock_editor.create_custom_instrumental.return_value = mock_result
mock_editor_class.return_value = mock_editor
service = AudioEditingService()
# Create mock mute regions
mock_mute1 = Mock()
mock_mute2 = Mock()
mute_regions = [mock_mute1, mock_mute2]
result = service.create_custom_instrumental(
gcs_clean_instrumental_path="uploads/job123/clean.flac",
gcs_backing_vocals_path="uploads/job123/backing.flac",
mute_regions=mute_regions,
gcs_output_path="uploads/job123/custom.flac",
job_id="job123",
)
# Verify downloads
assert mock_storage.download_file.call_count == 2
# Verify editor was called
mock_editor.create_custom_instrumental.assert_called_once()
call_kwargs = mock_editor.create_custom_instrumental.call_args.kwargs
assert call_kwargs["mute_regions"] == mute_regions
# Verify upload
mock_storage.upload_file.assert_called_once()
# Verify result path was updated
assert result.output_path == "uploads/job123/custom.flac"
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_create_custom_instrumental_downloads_both_files(
self, mock_editor_class, mock_storage_class
):
"""Test that both input files are downloaded."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_result = Mock()
mock_result.total_muted_duration_seconds = 0
mock_result.mute_regions_applied = []
mock_editor = Mock()
mock_editor.create_custom_instrumental.return_value = mock_result
mock_editor_class.return_value = mock_editor
service = AudioEditingService()
service.create_custom_instrumental(
gcs_clean_instrumental_path="uploads/job123/clean.flac",
gcs_backing_vocals_path="uploads/job123/backing.flac",
mute_regions=[],
gcs_output_path="uploads/job123/output.flac",
job_id="job123",
)
# Verify both files were downloaded
download_calls = mock_storage.download_file.call_args_list
assert len(download_calls) == 2
# First call should be clean instrumental
assert download_calls[0][0][0] == "uploads/job123/clean.flac"
# Second call should be backing vocals
assert download_calls[1][0][0] == "uploads/job123/backing.flac"
class TestCreatePreview:
"""Test create_preview method."""
@patch("pydub.AudioSegment")
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_create_preview_success(
self, mock_editor_class, mock_storage_class, mock_audio_segment
):
"""Test creating a preview successfully."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_preview = Mock()
mock_preview.export = Mock()
mock_preview.__getitem__ = Mock(return_value=mock_preview)
mock_editor = Mock()
mock_editor.preview_with_mutes.return_value = mock_preview
mock_editor_class.return_value = mock_editor
service = AudioEditingService()
result = service.create_preview(
gcs_clean_instrumental_path="uploads/job123/clean.flac",
gcs_backing_vocals_path="uploads/job123/backing.flac",
mute_regions=[Mock(), Mock()],
gcs_preview_path="uploads/job123/preview.flac",
job_id="job123",
)
# Verify editor was called
mock_editor.preview_with_mutes.assert_called_once()
# Verify result
assert result == "uploads/job123/preview.flac"
@patch("pydub.AudioSegment")
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_create_preview_with_duration_limit(
self, mock_editor_class, mock_storage_class, mock_audio_segment
):
"""Test creating a preview with duration limit."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_preview = Mock()
mock_truncated = Mock()
mock_preview.__getitem__ = Mock(return_value=mock_truncated)
mock_truncated.export = Mock()
mock_editor = Mock()
mock_editor.preview_with_mutes.return_value = mock_preview
mock_editor_class.return_value = mock_editor
service = AudioEditingService()
service.create_preview(
gcs_clean_instrumental_path="uploads/job123/clean.flac",
gcs_backing_vocals_path="uploads/job123/backing.flac",
mute_regions=[],
gcs_preview_path="uploads/job123/preview.flac",
job_id="job123",
preview_duration_seconds=30.0,
)
# Verify preview was truncated (30 seconds = 30000ms)
mock_preview.__getitem__.assert_called_once_with(slice(None, 30000))
class TestMuteBackingVocalsOnly:
"""Test mute_backing_vocals_only method."""
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_mute_backing_vocals_only(self, mock_editor_class, mock_storage_class):
"""Test muting backing vocals without combining."""
from backend.services.audio_editing_service import AudioEditingService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_editor = Mock()
mock_editor_class.return_value = mock_editor
service = AudioEditingService()
mute_regions = [Mock(), Mock()]
result = service.mute_backing_vocals_only(
gcs_backing_vocals_path="uploads/job123/backing.flac",
mute_regions=mute_regions,
gcs_output_path="uploads/job123/muted_backing.flac",
job_id="job123",
)
# Verify only one file was downloaded
mock_storage.download_file.assert_called_once()
assert mock_storage.download_file.call_args[0][0] == "uploads/job123/backing.flac"
# Verify editor method was called
mock_editor.apply_mute_to_single_track.assert_called_once()
call_kwargs = mock_editor.apply_mute_to_single_track.call_args.kwargs
assert call_kwargs["mute_regions"] == mute_regions
# Verify upload
mock_storage.upload_file.assert_called_once()
# Verify result
assert result == "uploads/job123/muted_backing.flac"
class TestValidateMuteRegions:
"""Test validate_mute_regions method.
Note: MuteRegion uses pydantic validation that rejects invalid values at
creation time. These tests use Mock objects to test the validation logic
in the service, simulating edge cases that would normally be rejected.
"""
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_valid_mute_regions(self, mock_editor_class, mock_storage_class):
"""Test validation passes for valid mute regions."""
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
service = AudioEditingService()
mute_regions = [
MuteRegion(start_seconds=10.0, end_seconds=15.0),
MuteRegion(start_seconds=30.0, end_seconds=45.0),
]
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
assert errors == []
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_negative_start_seconds(self, mock_editor_class, mock_storage_class):
"""Test validation catches negative start time.
Uses Mock since MuteRegion pydantic model rejects negative values.
"""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService()
# Use mock to simulate invalid region (pydantic would reject this)
mock_region = Mock()
mock_region.start_seconds = -5.0
mock_region.end_seconds = 10.0
errors = service.validate_mute_regions([mock_region], total_duration_seconds=180.0)
assert len(errors) == 1
assert "cannot be negative" in errors[0]
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_end_before_start(self, mock_editor_class, mock_storage_class):
"""Test validation catches end time before start time."""
from backend.services.audio_editing_service import AudioEditingService
service = AudioEditingService()
# Use mock to simulate invalid region
mock_region = Mock()
mock_region.start_seconds = 30.0
mock_region.end_seconds = 20.0
errors = service.validate_mute_regions([mock_region], total_duration_seconds=180.0)
assert len(errors) == 1
assert "must be after" in errors[0]
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_start_exceeds_duration(self, mock_editor_class, mock_storage_class):
"""Test validation catches start time exceeding audio duration."""
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
service = AudioEditingService()
mute_regions = [
MuteRegion(start_seconds=200.0, end_seconds=210.0),
]
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
assert len(errors) == 1
assert "exceeds audio duration" in errors[0]
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_end_exceeds_duration_is_warning_not_error(
self, mock_editor_class, mock_storage_class
):
"""Test that end time exceeding duration is logged as warning, not error."""
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
service = AudioEditingService()
mute_regions = [
MuteRegion(start_seconds=170.0, end_seconds=200.0), # End exceeds 180s
]
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
# Should not be an error (will be clamped)
assert len(errors) == 0
@patch("backend.services.audio_editing_service.StorageService")
@patch("backend.services.audio_editing_service.AudioEditor")
def test_multiple_errors(self, mock_editor_class, mock_storage_class):
"""Test validation reports multiple errors."""
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
service = AudioEditingService()
# Use mocks for invalid regions
mock_region1 = Mock()
mock_region1.start_seconds = -5.0
mock_region1.end_seconds = 10.0
mock_region2 = Mock()
mock_region2.start_seconds = 30.0
mock_region2.end_seconds = 20.0 # End before start
mock_region3 = Mock()
mock_region3.start_seconds = 200.0 # Exceeds duration
mock_region3.end_seconds = 210.0
mute_regions = [mock_region1, mock_region2, mock_region3]
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
assert len(errors) == 3
+785 -97
View File
@@ -99,7 +99,11 @@ class TestJobStatusAudioSearchStates:
class TestAudioSearchResult:
"""Test AudioSearchResult dataclass."""
"""Test AudioSearchResult dataclass.
AudioSearchResult is now imported from karaoke_gen.audio_fetcher.
These tests verify the backend can use it correctly.
"""
def test_create_audio_search_result(self):
"""Test creating an AudioSearchResult."""
@@ -140,6 +144,8 @@ class TestAudioSearchResult:
assert data['provider'] == "YouTube"
assert data['url'] == "https://youtube.com/watch?v=abc123"
assert data['index'] == 0
# raw_result should NOT be in serialized dict
assert 'raw_result' not in data
def test_from_dict(self):
"""Test creating from dict."""
@@ -160,64 +166,67 @@ class TestAudioSearchResult:
assert result.artist == "ABBA"
assert result.provider == "YouTube"
assert result.index == 0
assert result.raw_result is None # Not set from dict
class TestAudioSearchServiceInit:
"""Test AudioSearchService initialization."""
@patch.dict('os.environ', {'REDACTED_API_KEY': '', 'OPS_API_KEY': ''}, clear=False)
@patch.dict('os.environ', {'RED_API_KEY': '', 'RED_API_URL': '', 'OPS_API_KEY': '', 'OPS_API_URL': ''}, clear=False)
def test_init_with_no_keys(self):
"""Test initialization without API keys."""
# Pass explicit None to override environment
service = AudioSearchService(redacted_api_key=None, ops_api_key=None)
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
# When explicitly passed None, it should be None (not from env)
assert service._redacted_api_key is None
assert service._ops_api_key is None
assert service._manager is None
# Service should have a FlacFetcher internally
assert service._fetcher is not None
assert service._cached_results == []
def test_init_with_keys(self):
"""Test initialization with API keys."""
"""Test initialization with API keys and URLs."""
service = AudioSearchService(
redacted_api_key="test_redacted_key",
red_api_key="test_red_key",
red_api_url="https://red.url",
ops_api_key="test_ops_key",
ops_api_url="https://ops.url",
)
assert service._redacted_api_key == "test_redacted_key"
assert service._ops_api_key == "test_ops_key"
# Keys and URLs are passed to FlacFetcher
assert service._fetcher._red_api_key == "test_red_key"
assert service._fetcher._red_api_url == "https://red.url"
assert service._fetcher._ops_api_key == "test_ops_key"
assert service._fetcher._ops_api_url == "https://ops.url"
def test_init_reads_from_environment(self):
"""Test initialization reads keys from environment variables."""
import os
# Just test that the service can be initialized
service = AudioSearchService()
# The service should have API keys from env or None
assert service._manager is None # Not yet initialized
# The service should have a FlacFetcher
assert service._fetcher is not None
class TestAudioSearchServiceSearch:
"""Test AudioSearchService.search() method."""
@patch('backend.services.audio_search_service.AudioSearchService._get_manager')
def test_search_returns_results(self, mock_get_manager):
def test_search_returns_results(self):
"""Test search returns AudioSearchResult list."""
# Mock the flacfetch result
mock_result = Mock()
mock_result.title = "Waterloo"
mock_result.artist = "ABBA"
mock_result.provider = "YouTube"
mock_result.url = "https://youtube.com/watch?v=abc123"
mock_result.duration = 180
mock_result.quality = "FLAC"
mock_result.id = "abc123"
# Create mock result
mock_result = AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="YouTube",
url="https://youtube.com/watch?v=abc123",
duration=180,
quality="FLAC",
source_id="abc123",
index=0,
)
mock_manager = Mock()
mock_manager.search.return_value = [mock_result]
mock_get_manager.return_value = mock_manager
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
service._fetcher.search.return_value = [mock_result]
service = AudioSearchService()
results = service.search("ABBA", "Waterloo")
assert len(results) == 1
@@ -226,40 +235,38 @@ class TestAudioSearchServiceSearch:
assert results[0].provider == "YouTube"
assert results[0].index == 0
@patch('backend.services.audio_search_service.AudioSearchService._get_manager')
def test_search_no_results_raises_error(self, mock_get_manager):
def test_search_no_results_raises_error(self):
"""Test search raises NoResultsError when no results."""
mock_manager = Mock()
mock_manager.search.return_value = []
mock_get_manager.return_value = mock_manager
service = AudioSearchService()
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._remote_client = None # Force local mode to use mocked fetcher
service._fetcher = Mock()
service._fetcher.search.side_effect = NoResultsError("No results found")
with pytest.raises(NoResultsError) as exc_info:
service.search("Unknown Artist", "Unknown Song")
assert "No results found" in str(exc_info.value)
@patch('backend.services.audio_search_service.AudioSearchService._get_manager')
def test_search_multiple_results(self, mock_get_manager):
def test_search_multiple_results(self):
"""Test search returns multiple results with correct indices."""
mock_results = []
for i in range(3):
mock_result = Mock()
mock_result.title = f"Song {i}"
mock_result.artist = "Artist"
mock_result.provider = ["YouTube", "Redacted", "OPS"][i]
mock_result.url = f"https://example.com/{i}"
mock_result.duration = 180 + i * 10
mock_result.quality = ["320kbps", "FLAC", "FLAC"][i]
mock_result.id = str(i)
mock_results.append(mock_result)
mock_results.append(AudioSearchResult(
title=f"Song {i}",
artist="Artist",
provider=["YouTube", "RED", "OPS"][i],
url=f"https://example.com/{i}",
duration=180 + i * 10,
quality=["320kbps", "FLAC", "FLAC"][i],
source_id=str(i),
index=i,
))
mock_manager = Mock()
mock_manager.search.return_value = mock_results
mock_get_manager.return_value = mock_manager
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._remote_client = None # Force local mode to use mocked fetcher
service._fetcher = Mock()
service._fetcher.search.return_value = mock_results
service = AudioSearchService()
results = service.search("Artist", "Song")
assert len(results) == 3
@@ -271,34 +278,39 @@ class TestAudioSearchServiceSearch:
class TestAudioSearchServiceSelectBest:
"""Test AudioSearchService.select_best() method."""
@patch('backend.services.audio_search_service.AudioSearchService._get_manager')
def test_select_best_uses_manager(self, mock_get_manager):
"""Test select_best uses flacfetch's select_best."""
# First need to do a search to populate cache
mock_result = Mock()
mock_result.title = "Waterloo"
mock_result.artist = "ABBA"
mock_result.provider = "YouTube"
mock_result.url = "https://youtube.com/watch?v=abc123"
mock_result.duration = 180
mock_result.quality = "FLAC"
mock_result.id = "abc123"
def test_select_best_delegates_to_fetcher(self):
"""Test select_best uses FlacFetcher's select_best."""
mock_results = [
AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="YouTube",
url="https://youtube.com/watch?v=abc123",
index=0,
),
AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="RED",
url="https://example.com/456",
index=1,
),
]
mock_manager = Mock()
mock_manager.search.return_value = [mock_result]
mock_manager.select_best.return_value = mock_result
mock_get_manager.return_value = mock_manager
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
service._fetcher.select_best.return_value = 1
service = AudioSearchService()
results = service.search("ABBA", "Waterloo")
best_index = service.select_best(mock_results)
best_index = service.select_best(results)
assert best_index == 0
assert best_index == 1
service._fetcher.select_best.assert_called_once_with(mock_results)
def test_select_best_empty_list_returns_zero(self):
"""Test select_best returns 0 for empty list."""
service = AudioSearchService()
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
service._fetcher.select_best.return_value = 0
best_index = service.select_best([])
@@ -308,24 +320,18 @@ class TestAudioSearchServiceSelectBest:
class TestAudioSearchServiceDownload:
"""Test AudioSearchService.download() method."""
@patch('backend.services.audio_search_service.AudioSearchService._get_manager')
def test_download_without_search_raises_error(self, mock_get_manager):
def test_download_without_search_raises_error(self):
"""Test download without prior search raises error."""
mock_manager = Mock()
mock_get_manager.return_value = mock_manager
service = AudioSearchService()
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
with pytest.raises(DownloadError) as exc_info:
service.download(0, "/tmp")
assert "No cached result" in str(exc_info.value)
@patch('backend.services.audio_search_service.AudioSearchService._get_manager')
def test_download_after_search(self, mock_get_manager):
def test_download_after_search(self):
"""Test download after search works."""
import tempfile
import os
# Create a temp file to simulate downloaded file
temp_dir = tempfile.mkdtemp()
@@ -333,23 +339,33 @@ class TestAudioSearchServiceDownload:
with open(temp_file, 'w') as f:
f.write("test")
mock_result = Mock()
mock_result.title = "Waterloo"
mock_result.artist = "ABBA"
mock_result.provider = "YouTube"
mock_result.url = "https://youtube.com/watch?v=abc123"
mock_result.duration = 180
mock_result.quality = "FLAC"
mock_result.id = "abc123"
mock_result = AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="YouTube",
url="https://youtube.com/watch?v=abc123",
duration=180,
quality="FLAC",
source_id="abc123",
index=0,
)
mock_manager = Mock()
mock_manager.search.return_value = [mock_result]
mock_manager.download.return_value = temp_file
mock_get_manager.return_value = mock_manager
mock_fetch_result = AudioDownloadResult(
filepath=temp_file,
artist="ABBA",
title="Waterloo",
provider="YouTube",
duration=180,
quality="FLAC",
)
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._remote_client = None # Force local mode to use mocked fetcher
service._fetcher = Mock()
service._fetcher.search.return_value = [mock_result]
service._fetcher.download.return_value = mock_fetch_result
service = AudioSearchService()
service.search("ABBA", "Waterloo")
result = service.download(0, temp_dir)
assert result.filepath == temp_file
@@ -360,6 +376,27 @@ class TestAudioSearchServiceDownload:
# Cleanup
os.remove(temp_file)
os.rmdir(temp_dir)
def test_download_invalid_index_raises_error(self):
"""Test download with invalid index raises error."""
mock_result = AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="YouTube",
url="https://youtube.com/watch?v=abc123",
index=0,
)
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
service._fetcher.search.return_value = [mock_result]
service.search("ABBA", "Waterloo")
with pytest.raises(DownloadError) as exc_info:
service.download(99, "/tmp")
assert "No cached result for index 99" in str(exc_info.value)
class TestGetAudioSearchService:
@@ -384,3 +421,654 @@ class TestGetAudioSearchService:
assert service1 is service2
class TestRemoteDownloadPath:
"""Test remote download functionality when flacfetch service is configured.
These tests verify the code path that uses the remote flacfetch service
for torrent downloads (RED/OPS providers). This would have caught
the 'extra_info' attribute error.
The tests directly set up the service state without calling search() to avoid
the complexity of mocking async remote search operations.
"""
def test_download_uses_remote_for_red_provider(self):
"""Test download routes to remote service for RED provider.
This test would have caught: 'AudioSearchResult' object has no attribute 'extra_info'
"""
# Create a RED result (torrent source)
mock_result = AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="RED", # Torrent provider - should use remote
url="", # No URL for torrent sources
quality="FLAC 16bit CD",
seeders=50,
target_file="Waterloo.flac",
index=0,
)
# Create service with mocked remote client
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
# Directly set cached results (simulating after search)
service._cached_results = [mock_result]
service._remote_search_id = "remote_search_123" # Remote search was performed
# Mock the remote client
mock_remote_client = Mock()
service._remote_client = mock_remote_client
# Mock the remote download method
mock_download_result = AudioDownloadResult(
filepath="gs://bucket/uploads/job123/audio/Waterloo.flac",
artist="ABBA",
title="Waterloo",
provider="RED",
quality="FLAC 16bit CD",
)
service._download_remote = Mock(return_value=mock_download_result)
# Call download - should route to remote
result = service.download(0, "/tmp", gcs_path="uploads/job123/audio/")
# Verify remote download was called
service._download_remote.assert_called_once_with(0, "/tmp", None, "uploads/job123/audio/")
assert result.filepath == "gs://bucket/uploads/job123/audio/Waterloo.flac"
def test_download_uses_remote_for_ops_provider(self):
"""Test download routes to remote service for OPS provider."""
mock_result = AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="OPS", # Torrent provider
url="",
quality="FLAC 16bit CD",
seeders=30,
index=0,
)
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
# Directly set cached results
service._cached_results = [mock_result]
service._remote_search_id = "remote_search_456"
# Mock remote client
mock_remote_client = Mock()
service._remote_client = mock_remote_client
# Mock remote download
mock_download_result = AudioDownloadResult(
filepath="gs://bucket/test.flac",
artist="ABBA",
title="Waterloo",
provider="OPS",
quality="FLAC",
)
service._download_remote = Mock(return_value=mock_download_result)
# Call download
result = service.download(0, "/tmp")
# Should use remote for OPS
service._download_remote.assert_called_once()
def test_download_uses_local_for_youtube_even_with_remote_client(self):
"""Test YouTube downloads use local even when remote client is configured."""
mock_result = AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="YouTube", # NOT a torrent provider
url="https://youtube.com/watch?v=abc123",
quality="Opus 128kbps",
index=0,
)
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
# Directly set cached results
service._cached_results = [mock_result]
service._remote_search_id = "remote_search_789" # Remote search was performed
# Mock remote client (configured but shouldn't be used for YouTube)
mock_remote_client = Mock()
service._remote_client = mock_remote_client
# Mock local download
mock_fetch_result = AudioDownloadResult(
filepath="/tmp/test.opus",
artist="ABBA",
title="Waterloo",
provider="YouTube",
quality="Opus 128kbps",
)
service._fetcher.download.return_value = mock_fetch_result
# Mock _download_remote to ensure it's NOT called
service._download_remote = Mock()
# Call download
result = service.download(0, "/tmp")
# Should use local for YouTube
service._download_remote.assert_not_called()
service._fetcher.download.assert_called_once()
def test_download_uses_local_when_no_remote_client(self):
"""Test download uses local when remote client is not configured."""
mock_result = AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="RED", # Would normally use remote
url="",
quality="FLAC",
index=0,
)
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
# Directly set cached results
service._cached_results = [mock_result]
# No remote client
service._remote_client = None
service._remote_search_id = None # No remote search
# Mock local download
mock_fetch_result = AudioDownloadResult(
filepath="/tmp/test.flac",
artist="ABBA",
title="Waterloo",
provider="RED",
quality="FLAC",
)
service._fetcher.download.return_value = mock_fetch_result
# Call download
result = service.download(0, "/tmp")
# Should use local even though it's RED
service._fetcher.download.assert_called_once()
def test_download_uses_local_when_no_remote_search_id(self):
"""Test download uses local when search wasn't done remotely."""
mock_result = AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="RED",
url="",
quality="FLAC",
index=0,
)
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
# Directly set cached results
service._cached_results = [mock_result]
# Remote client configured but search was local (fallback scenario)
service._remote_client = Mock()
service._remote_search_id = None # Search was local, not remote
# Mock local download
mock_fetch_result = AudioDownloadResult(
filepath="/tmp/test.flac",
artist="ABBA",
title="Waterloo",
provider="RED",
quality="FLAC",
)
service._fetcher.download.return_value = mock_fetch_result
# Call download
result = service.download(0, "/tmp")
# Should use local since no remote search ID
service._fetcher.download.assert_called_once()
def test_torrent_provider_routing_checks_both_conditions(self):
"""Test that torrent provider routing requires BOTH remote_search_id AND remote_client.
This test verifies the logical AND condition in the download routing.
"""
mock_result = AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="RED",
url="",
quality="FLAC",
index=0,
)
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
service._cached_results = [mock_result]
# Test: remote_client set but no remote_search_id -> should use local
service._remote_client = Mock()
service._remote_search_id = None
mock_fetch_result = AudioDownloadResult(
filepath="/tmp/test.flac", artist="ABBA", title="Waterloo",
provider="RED", quality="FLAC",
)
service._fetcher.download.return_value = mock_fetch_result
service._download_remote = Mock()
service.download(0, "/tmp")
service._download_remote.assert_not_called()
service._fetcher.download.assert_called_once()
# Reset mocks
service._fetcher.download.reset_mock()
service._download_remote.reset_mock()
# Test: remote_search_id set but no remote_client -> should use local
service._remote_client = None
service._remote_search_id = "search_123"
service.download(0, "/tmp")
service._download_remote.assert_not_called()
service._fetcher.download.assert_called_once()
def test_download_does_not_access_nonexistent_attributes(self):
"""Test that download() doesn't try to access nonexistent attributes.
This test would have caught the 'extra_info' attribute error:
AttributeError: 'AudioSearchResult' object has no attribute 'extra_info'
AudioSearchResult only has: title, artist, url, provider, duration,
quality, source_id, index, seeders, target_file, raw_result
"""
mock_result = AudioSearchResult(
title="Waterloo",
artist="ABBA",
provider="RED",
url="",
quality="FLAC 16bit CD",
seeders=50,
index=0,
)
# Verify AudioSearchResult doesn't have extra_info
assert not hasattr(mock_result, 'extra_info')
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
service._fetcher = Mock()
service._cached_results = [mock_result]
service._remote_search_id = "search_123"
service._remote_client = Mock()
# Mock _download_remote to avoid actual network call
mock_download_result = AudioDownloadResult(
filepath="/tmp/test.flac",
artist="ABBA",
title="Waterloo",
provider="RED",
quality="FLAC",
)
service._download_remote = Mock(return_value=mock_download_result)
# This should NOT raise AttributeError: 'AudioSearchResult' object has no attribute 'extra_info'
result = service.download(0, "/tmp")
# Should succeed and use remote download for RED provider
service._download_remote.assert_called_once()
assert result.filepath == "/tmp/test.flac"
class TestFlacfetchIntegration:
"""
Integration tests that verify flacfetch imports work correctly.
These tests ensure the backend code is compatible with the installed
flacfetch version and would catch issues like renamed classes.
Note: Some tests are marked with pytest.importorskip for flacfetch
since it may not be installed in all test environments (e.g., CI).
"""
def test_flacfetch_imports_work(self):
"""Test that all flacfetch imports in audio_search_service work.
This test would have caught the YouTubeProvider -> YoutubeProvider rename.
"""
flacfetch = pytest.importorskip("flacfetch")
# These imports match what karaoke_gen.audio_fetcher does
from flacfetch.core.manager import FetchManager
from flacfetch.providers.youtube import YoutubeProvider
from flacfetch.core.models import TrackQuery
# Verify the classes exist and are callable
assert FetchManager is not None
assert YoutubeProvider is not None
assert TrackQuery is not None
def test_karaoke_gen_audio_fetcher_imports_work(self):
"""Test that karaoke_gen.audio_fetcher imports work.
Since backend now imports from karaoke_gen, this is the critical test.
This should work even without flacfetch installed (lazy import).
"""
from karaoke_gen.audio_fetcher import (
FlacFetcher,
AudioSearchResult,
AudioFetchResult,
AudioFetcherError,
NoResultsError,
DownloadError,
)
assert FlacFetcher is not None
assert AudioSearchResult is not None
assert AudioFetchResult is not None
assert AudioFetcherError is not None
assert NoResultsError is not None
assert DownloadError is not None
def test_audio_search_service_can_initialize_fetcher(self):
"""Test AudioSearchService can initialize its FlacFetcher.
This verifies the actual initialization code path works.
"""
pytest.importorskip("flacfetch")
service = AudioSearchService(
red_api_key=None,
red_api_url=None,
ops_api_key=None,
)
# Verify FlacFetcher was initialized
assert service._fetcher is not None
# FlacFetcher can initialize its manager (tests actual flacfetch)
manager = service._fetcher._get_manager()
assert manager is not None
def test_shared_classes_are_same_as_karaoke_gen(self):
"""Verify backend uses the same classes as karaoke_gen."""
from karaoke_gen.audio_fetcher import (
AudioSearchResult as KGAudioSearchResult,
AudioFetchResult as KGAudioFetchResult,
NoResultsError as KGNoResultsError,
DownloadError as KGDownloadError,
)
# These should be the exact same classes, not copies
assert AudioSearchResult is KGAudioSearchResult
assert AudioDownloadResult is KGAudioFetchResult
assert NoResultsError is KGNoResultsError
assert DownloadError is KGDownloadError
class TestAudioSearchApiRouteDownload:
"""
Tests for _download_and_start_processing function in audio_search routes.
This tests the API layer's handling of the download flow, including:
- Correct routing between local and remote downloads
- GCS path handling for remote downloads
- Error handling
"""
@pytest.fixture
def mock_job_manager(self):
"""Mock the job manager singleton."""
with patch('backend.api.routes.audio_search.job_manager') as mock:
yield mock
@pytest.fixture
def mock_storage_service(self):
"""Mock the storage service singleton."""
with patch('backend.api.routes.audio_search.storage_service') as mock:
yield mock
def test_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
"""
Test that remote torrent downloads include GCS path for direct upload.
This was the bug: we were calling download() without gcs_path for
torrent sources, causing flacfetch VM to return a local path that
Cloud Run couldn't access.
"""
from backend.api.routes.audio_search import _download_and_start_processing
import asyncio
# Setup mock job with RED search result
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Unwanted',
'artist': 'Avril Lavigne',
'provider': 'RED', # Torrent source
'quality': 'FLAC 16bit CD',
}]
}
mock_job.audio_search_artist = 'Avril Lavigne'
mock_job.audio_search_title = 'Unwanted'
mock_job_manager.get_job.return_value = mock_job
# Create mock audio search service with remote client
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = True
# Mock download to return GCS path
mock_download_result = Mock()
mock_download_result.filepath = "gs://bucket/uploads/job123/audio/Avril Lavigne - Unwanted.flac"
mock_audio_service.download.return_value = mock_download_result
# Create mock background tasks
mock_bg_tasks = Mock()
# Run the async function
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
job_id="job123",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=mock_bg_tasks,
)
)
finally:
loop.close()
# CRITICAL: Verify download was called with gcs_path for remote torrent source
mock_audio_service.download.assert_called_once()
call_kwargs = mock_audio_service.download.call_args.kwargs
# The gcs_path should be set for remote torrent downloads
assert 'gcs_path' in call_kwargs
assert call_kwargs['gcs_path'] == "uploads/job123/audio/"
def test_local_youtube_download_does_not_pass_gcs_path(self, mock_job_manager, mock_storage_service):
"""Test that YouTube downloads don't use gcs_path (download locally, upload manually)."""
from backend.api.routes.audio_search import _download_and_start_processing
import asyncio
import tempfile
# Setup mock job with YouTube search result
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Unwanted',
'artist': 'Avril Lavigne',
'provider': 'YouTube', # NOT a torrent source
'quality': 'Opus 128kbps',
}]
}
mock_job.audio_search_artist = 'Avril Lavigne'
mock_job.audio_search_title = 'Unwanted'
mock_job_manager.get_job.return_value = mock_job
# Create mock audio search service with remote client (but YouTube doesn't use it)
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = True
# Create a temp file to simulate downloaded file
temp_file = tempfile.NamedTemporaryFile(suffix='.opus', delete=False)
temp_file.write(b'fake audio data')
temp_file.close()
try:
# Mock download to return local path
mock_download_result = Mock()
mock_download_result.filepath = temp_file.name
mock_audio_service.download.return_value = mock_download_result
# Create mock background tasks
mock_bg_tasks = Mock()
# Run the async function
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
job_id="job456",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=mock_bg_tasks,
)
)
finally:
loop.close()
# YouTube should NOT have gcs_path in kwargs
mock_audio_service.download.assert_called_once()
call_kwargs = mock_audio_service.download.call_args.kwargs
# For YouTube, gcs_path should NOT be set (or should be None)
assert call_kwargs.get('gcs_path') is None
# Instead, storage service should be called to upload
mock_storage_service.upload_fileobj.assert_called_once()
finally:
import os
try:
os.unlink(temp_file.name)
except:
pass
def test_handles_gcs_path_response_correctly(self, mock_job_manager, mock_storage_service):
"""Test that GCS path responses are parsed correctly."""
from backend.api.routes.audio_search import _download_and_start_processing
import asyncio
# Setup mock job with RED search result
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Test',
'artist': 'Test Artist',
'provider': 'RED',
'quality': 'FLAC',
}]
}
mock_job_manager.get_job.return_value = mock_job
# Create mock audio search service
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = True
# Mock download to return full GCS path
mock_download_result = Mock()
mock_download_result.filepath = "gs://karaoke-gen-bucket/uploads/job789/audio/test.flac"
mock_audio_service.download.return_value = mock_download_result
# Run the async function
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
job_id="job789",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=Mock(),
)
)
finally:
loop.close()
# Verify job was updated with correct GCS path (without gs://bucket/ prefix)
update_calls = mock_job_manager.update_job.call_args_list
# Find the call that sets input_media_gcs_path
gcs_path_set = False
for call in update_calls:
if 'input_media_gcs_path' in call.args[1]:
gcs_path = call.args[1]['input_media_gcs_path']
# The path stored should be the relative path, not the full gs:// URL
assert gcs_path == "uploads/job789/audio/test.flac"
gcs_path_set = True
break
assert gcs_path_set, "input_media_gcs_path was not set in job update"
def test_remote_disabled_always_uses_local(self, mock_job_manager, mock_storage_service):
"""Test that when remote is disabled, even torrent sources use local download."""
from backend.api.routes.audio_search import _download_and_start_processing
import asyncio
import tempfile
# Setup mock job with RED search result
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Test',
'artist': 'Test',
'provider': 'RED', # Torrent source, but remote is disabled
'quality': 'FLAC',
}]
}
mock_job_manager.get_job.return_value = mock_job
# Create temp file
temp_file = tempfile.NamedTemporaryFile(suffix='.flac', delete=False)
temp_file.write(b'fake')
temp_file.close()
try:
# Create mock audio search service WITHOUT remote client
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = False # REMOTE DISABLED
mock_download_result = Mock()
mock_download_result.filepath = temp_file.name
mock_audio_service.download.return_value = mock_download_result
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
job_id="job_no_remote",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=Mock(),
)
)
finally:
loop.close()
# When remote is disabled, gcs_path should NOT be passed
mock_audio_service.download.assert_called_once()
call_kwargs = mock_audio_service.download.call_args.kwargs
assert call_kwargs.get('gcs_path') is None
# And storage should upload manually
mock_storage_service.upload_fileobj.assert_called_once()
finally:
import os
try:
os.unlink(temp_file.name)
except:
pass
+408
View File
@@ -0,0 +1,408 @@
"""
Tests for dropbox_service.py - Dropbox file operations.
These tests mock the Dropbox SDK and Secret Manager to verify:
- Credential loading from Secret Manager
- Folder listing and brand code calculation
- File and folder uploads
- Shared link creation
"""
import json
import os
import pytest
from unittest.mock import Mock, MagicMock, patch
class TestDropboxServiceInit:
"""Test DropboxService initialization."""
def test_init_creates_service(self):
"""Test initialization creates service with no client."""
from backend.services.dropbox_service import DropboxService
service = DropboxService()
assert service._client is None
assert service._is_configured is False
class TestLoadCredentials:
"""Test _load_credentials method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_load_credentials_success(self, mock_sm_client_class):
"""Test successful credential loading from Secret Manager."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({
"access_token": "access-token-123",
"refresh_token": "refresh-token-456",
"app_key": "app-key",
"app_secret": "app-secret",
}).encode("UTF-8")
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
creds = service._load_credentials()
assert creds is not None
assert creds["access_token"] == "access-token-123"
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_load_credentials_failure(self, mock_sm_client_class):
"""Test handling when Secret Manager fails."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_sm_client.access_secret_version.side_effect = Exception("Access denied")
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
creds = service._load_credentials()
assert creds is None
class TestIsConfigured:
"""Test is_configured property."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_is_configured_true(self, mock_sm_client_class):
"""Test is_configured returns True when credentials available."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({
"access_token": "token"
}).encode("UTF-8")
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
assert service.is_configured is True
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_is_configured_false_no_token(self, mock_sm_client_class):
"""Test is_configured returns False when no access_token."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({
"refresh_token": "refresh" # Missing access_token
}).encode("UTF-8")
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
assert service.is_configured is False
class TestDropboxClient:
"""Test client property."""
@patch("dropbox.Dropbox")
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_client_creates_dropbox_instance(self, mock_sm_client_class, mock_dropbox_class):
"""Test client property creates Dropbox SDK client."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({
"access_token": "token",
"refresh_token": "refresh",
"app_key": "key",
"app_secret": "secret",
}).encode("UTF-8")
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
mock_dropbox = Mock()
mock_dropbox_class.return_value = mock_dropbox
service = DropboxService()
client = service.client
mock_dropbox_class.assert_called_once_with(
oauth2_access_token="token",
oauth2_refresh_token="refresh",
app_key="key",
app_secret="secret",
)
assert client == mock_dropbox
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_client_raises_on_missing_credentials(self, mock_sm_client_class):
"""Test client raises RuntimeError when credentials missing."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_sm_client.access_secret_version.side_effect = Exception("Not found")
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
with pytest.raises(RuntimeError) as exc_info:
_ = service.client
assert "not configured" in str(exc_info.value)
class TestListFolders:
"""Test list_folders method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_list_folders(self, mock_sm_client_class):
"""Test listing folders at a path."""
from backend.services.dropbox_service import DropboxService
from dropbox.files import FolderMetadata
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
# Create mock folder entries
mock_folder1 = Mock(spec=FolderMetadata)
mock_folder1.name = "NOMAD-0001"
mock_folder2 = Mock(spec=FolderMetadata)
mock_folder2.name = "NOMAD-0002"
mock_file = Mock() # Not a FolderMetadata
mock_result = Mock()
mock_result.entries = [mock_folder1, mock_folder2, mock_file]
mock_result.has_more = False
service = DropboxService()
# Directly set the client to avoid needing to mock the whole init chain
mock_dropbox = Mock()
mock_dropbox.files_list_folder.return_value = mock_result
service._client = mock_dropbox
folders = service.list_folders("/Karaoke/Tracks")
mock_dropbox.files_list_folder.assert_called_once_with("/Karaoke/Tracks")
assert folders == ["NOMAD-0001", "NOMAD-0002"]
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_list_folders_adds_leading_slash(self, mock_sm_client_class):
"""Test that path without leading slash gets one added."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
mock_result = Mock()
mock_result.entries = []
mock_result.has_more = False
service = DropboxService()
mock_dropbox = Mock()
mock_dropbox.files_list_folder.return_value = mock_result
service._client = mock_dropbox
service.list_folders("path/without/slash")
mock_dropbox.files_list_folder.assert_called_once_with("/path/without/slash")
class TestGetNextBrandCode:
"""Test get_next_brand_code method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_get_next_brand_code(self, mock_sm_client_class):
"""Test calculating next brand code from existing folders."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
# Mock list_folders to return existing codes
with patch.object(service, "list_folders") as mock_list:
mock_list.return_value = [
"NOMAD-1161",
"NOMAD-1162",
"NOMAD-1163",
"Other Folder",
"NOMAD-0001",
]
next_code = service.get_next_brand_code("/path", "NOMAD")
assert next_code == "NOMAD-1164"
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_get_next_brand_code_empty_folder(self, mock_sm_client_class):
"""Test brand code calculation with no existing codes."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
service = DropboxService()
with patch.object(service, "list_folders") as mock_list:
mock_list.return_value = []
next_code = service.get_next_brand_code("/path", "BRAND")
assert next_code == "BRAND-0001"
class TestUploadFile:
"""Test upload_file method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_upload_small_file(self, mock_sm_client_class, tmp_path):
"""Test uploading a small file directly."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
# Create test file
test_file = tmp_path / "test.txt"
test_file.write_text("Small file content")
service = DropboxService()
mock_dropbox = Mock()
service._client = mock_dropbox
service.upload_file(str(test_file), "/Uploads/test.txt")
mock_dropbox.files_upload.assert_called_once()
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_upload_file_adds_leading_slash(self, mock_sm_client_class, tmp_path):
"""Test upload adds leading slash to remote path."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
test_file = tmp_path / "test.txt"
test_file.write_text("content")
service = DropboxService()
mock_dropbox = Mock()
service._client = mock_dropbox
service.upload_file(str(test_file), "uploads/test.txt")
# Check that the path has leading slash
call_args = mock_dropbox.files_upload.call_args
assert call_args[0][1] == "/uploads/test.txt"
class TestUploadFolder:
"""Test upload_folder method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_upload_folder(self, mock_sm_client_class, tmp_path):
"""Test uploading a folder with multiple files."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
# Create test folder with files
(tmp_path / "file1.txt").write_text("content1")
(tmp_path / "file2.txt").write_text("content2")
(tmp_path / "subdir").mkdir() # Should be skipped
service = DropboxService()
with patch.object(service, "upload_file") as mock_upload:
service.upload_folder(str(tmp_path), "/Uploads/folder")
# Should upload 2 files (not subdir)
assert mock_upload.call_count == 2
class TestCreateSharedLink:
"""Test create_shared_link method."""
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_create_shared_link_new(self, mock_sm_client_class):
"""Test creating a new shared link."""
from backend.services.dropbox_service import DropboxService
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
mock_link = Mock()
mock_link.url = "https://dropbox.com/s/abc123/file.mp4"
service = DropboxService()
mock_dropbox = Mock()
mock_dropbox.sharing_create_shared_link_with_settings.return_value = mock_link
service._client = mock_dropbox
url = service.create_shared_link("/Uploads/file.mp4")
assert url == "https://dropbox.com/s/abc123/file.mp4"
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
def test_create_shared_link_existing(self, mock_sm_client_class):
"""Test getting existing shared link when one already exists."""
from backend.services.dropbox_service import DropboxService
from dropbox.exceptions import ApiError
mock_sm_client = Mock()
mock_response = Mock()
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
mock_sm_client.access_secret_version.return_value = mock_response
mock_sm_client_class.return_value = mock_sm_client
# Simulate "link already exists" error
mock_error = Mock()
mock_error.is_shared_link_already_exists.return_value = True
mock_existing_link = Mock()
mock_existing_link.url = "https://dropbox.com/s/existing/file.mp4"
mock_links_result = Mock()
mock_links_result.links = [mock_existing_link]
service = DropboxService()
mock_dropbox = Mock()
mock_dropbox.sharing_create_shared_link_with_settings.side_effect = \
ApiError("req_id", mock_error, "message", "headers")
mock_dropbox.sharing_list_shared_links.return_value = mock_links_result
service._client = mock_dropbox
url = service.create_shared_link("/Uploads/file.mp4")
assert url == "https://dropbox.com/s/existing/file.mp4"
+4 -4
View File
@@ -96,15 +96,15 @@ def auth_headers():
class TestEmulatorBasics:
"""Basic emulator connectivity tests."""
def test_health_endpoint(self, client):
def test_health_endpoint(self, client, auth_headers):
"""Test health endpoint works."""
response = client.get("/api/health")
response = client.get("/api/health", )
assert response.status_code == 200
assert response.json()["status"] == "healthy"
def test_root_endpoint(self, client):
def test_root_endpoint(self, client, auth_headers):
"""Test root endpoint works."""
response = client.get("/")
response = client.get("/", )
assert response.status_code == 200
assert response.json()["service"] == "karaoke-gen-backend"
+632
View File
@@ -0,0 +1,632 @@
"""
Tests for the flacfetch remote HTTP API client.
This module tests FlacfetchClient which handles communication with the
dedicated flacfetch VM for torrent/audio downloads.
These tests use mocked HTTP responses to verify:
- Request formatting (headers, payloads)
- Response parsing
- Error handling
- State management (singleton pattern)
"""
import pytest
from unittest.mock import AsyncMock, Mock, patch, MagicMock
import asyncio
import httpx
from backend.services.flacfetch_client import (
FlacfetchClient,
FlacfetchServiceError,
get_flacfetch_client,
reset_flacfetch_client,
)
class TestFlacfetchClientInit:
"""Test FlacfetchClient initialization."""
def test_init_with_required_params(self):
"""Test client initializes with base_url and api_key."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key-123",
)
assert client.base_url == "http://localhost:8080"
assert client.api_key == "test-key-123"
assert client.timeout == 60 # Default timeout
def test_init_strips_trailing_slash(self):
"""Test base_url trailing slash is stripped."""
client = FlacfetchClient(
base_url="http://localhost:8080/",
api_key="test-key",
)
assert client.base_url == "http://localhost:8080"
def test_init_with_custom_timeout(self):
"""Test custom timeout is set."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
timeout=120,
)
assert client.timeout == 120
def test_headers_include_api_key(self):
"""Test _headers() includes X-API-Key."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="secret-key-456",
)
headers = client._headers()
assert headers["X-API-Key"] == "secret-key-456"
assert headers["Content-Type"] == "application/json"
class TestFlacfetchClientHealthCheck:
"""Test health_check() method."""
@pytest.mark.asyncio
async def test_health_check_success(self):
"""Test successful health check."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {
"status": "healthy",
"transmission": {"available": True, "version": "4.0.5"},
"disk": {"free_gb": 15.5},
}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
result = await client.health_check()
assert result["status"] == "healthy"
assert result["transmission"]["available"] is True
@pytest.mark.asyncio
async def test_health_check_degraded_status(self):
"""Test health check with degraded status still succeeds."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {"status": "degraded"}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
result = await client.health_check()
assert result["status"] == "degraded"
@pytest.mark.asyncio
async def test_health_check_unhealthy_raises_error(self):
"""Test health check with unhealthy status raises error."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {"status": "unhealthy", "error": "Transmission down"}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
with pytest.raises(FlacfetchServiceError) as exc_info:
await client.health_check()
assert "unhealthy" in str(exc_info.value)
@pytest.mark.asyncio
async def test_health_check_network_error(self):
"""Test health check handles network errors."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(side_effect=httpx.RequestError("Connection refused"))
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
with pytest.raises(FlacfetchServiceError) as exc_info:
await client.health_check()
assert "Cannot reach" in str(exc_info.value)
class TestFlacfetchClientSearch:
"""Test search() method."""
@pytest.mark.asyncio
async def test_search_success(self):
"""Test successful search returns results."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"search_id": "search_abc123",
"results": [
{
"title": "Waterloo",
"artist": "ABBA",
"provider": "RED",
"quality": "FLAC 16bit CD",
"seeders": 50,
"index": 0,
}
],
"results_count": 1,
}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
result = await client.search("ABBA", "Waterloo")
assert result["search_id"] == "search_abc123"
assert len(result["results"]) == 1
assert result["results"][0]["title"] == "Waterloo"
# Verify request was made correctly
mock_client.post.assert_called_once()
call_args = mock_client.post.call_args
assert call_args.kwargs["json"] == {"artist": "ABBA", "title": "Waterloo"}
@pytest.mark.asyncio
async def test_search_no_results_returns_empty(self):
"""Test search with no results returns empty list."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.status_code = 404
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
result = await client.search("Unknown", "Artist")
assert result["results"] == []
assert result["search_id"] is None
class TestFlacfetchClientDownload:
"""Test download() method."""
@pytest.mark.asyncio
async def test_download_starts_successfully(self):
"""Test download() returns download_id."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {
"download_id": "dl_xyz789",
"status": "queued",
}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
download_id = await client.download(
search_id="search_abc123",
result_index=0,
)
assert download_id == "dl_xyz789"
@pytest.mark.asyncio
async def test_download_with_gcs_path(self):
"""Test download() includes GCS upload parameters."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {"download_id": "dl_xyz789"}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
await client.download(
search_id="search_abc123",
result_index=0,
gcs_path="uploads/job123/audio/",
)
# Verify GCS params were included
call_args = mock_client.post.call_args
payload = call_args.kwargs["json"]
assert payload["upload_to_gcs"] is True
assert payload["gcs_path"] == "uploads/job123/audio/"
@pytest.mark.asyncio
async def test_download_with_custom_filename(self):
"""Test download() with custom output filename."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {"download_id": "dl_xyz789"}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.post = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
await client.download(
search_id="search_abc123",
result_index=0,
output_filename="ABBA - Waterloo",
)
call_args = mock_client.post.call_args
payload = call_args.kwargs["json"]
assert payload["output_filename"] == "ABBA - Waterloo"
class TestFlacfetchClientGetDownloadStatus:
"""Test get_download_status() method."""
@pytest.mark.asyncio
async def test_get_download_status_success(self):
"""Test getting download status."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.json.return_value = {
"download_id": "dl_xyz789",
"status": "downloading",
"progress": 45.5,
"download_speed_kbps": 1250.0,
"peers": 5,
}
mock_response.raise_for_status = Mock()
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
status = await client.get_download_status("dl_xyz789")
assert status["status"] == "downloading"
assert status["progress"] == 45.5
@pytest.mark.asyncio
async def test_get_download_status_not_found(self):
"""Test getting status for non-existent download."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
mock_response = Mock()
mock_response.status_code = 404
mock_response.raise_for_status = Mock(
side_effect=httpx.HTTPStatusError("Not found", request=Mock(), response=mock_response)
)
with patch("httpx.AsyncClient") as mock_client_class:
mock_client = AsyncMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
with pytest.raises(FlacfetchServiceError) as exc_info:
await client.get_download_status("nonexistent")
assert "not found" in str(exc_info.value).lower()
class TestFlacfetchClientWaitForDownload:
"""Test wait_for_download() method."""
@pytest.mark.asyncio
async def test_wait_for_download_completes(self):
"""Test waiting for download to complete."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
# Mock get_download_status to return complete status
async def mock_get_status(download_id):
return {
"download_id": download_id,
"status": "complete",
"gcs_path": "gs://bucket/uploads/test.flac",
}
client.get_download_status = mock_get_status
result = await client.wait_for_download("dl_xyz789", timeout=10)
assert result["status"] == "complete"
assert result["gcs_path"] == "gs://bucket/uploads/test.flac"
@pytest.mark.asyncio
async def test_wait_for_download_seeding_status_completes(self):
"""Test that 'seeding' status is treated as complete."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
async def mock_get_status(download_id):
return {
"download_id": download_id,
"status": "seeding",
"output_path": "/downloads/test.flac",
}
client.get_download_status = mock_get_status
result = await client.wait_for_download("dl_xyz789", timeout=10)
assert result["status"] == "seeding"
@pytest.mark.asyncio
async def test_wait_for_download_failed(self):
"""Test wait raises error when download fails."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
async def mock_get_status(download_id):
return {
"download_id": download_id,
"status": "failed",
"error": "Torrent download failed: no peers",
}
client.get_download_status = mock_get_status
with pytest.raises(FlacfetchServiceError) as exc_info:
await client.wait_for_download("dl_xyz789", timeout=10)
assert "failed" in str(exc_info.value).lower()
assert "no peers" in str(exc_info.value)
@pytest.mark.asyncio
async def test_wait_for_download_cancelled(self):
"""Test wait raises error when download is cancelled."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
async def mock_get_status(download_id):
return {
"download_id": download_id,
"status": "cancelled",
}
client.get_download_status = mock_get_status
with pytest.raises(FlacfetchServiceError) as exc_info:
await client.wait_for_download("dl_xyz789", timeout=10)
assert "cancelled" in str(exc_info.value).lower()
@pytest.mark.asyncio
async def test_wait_for_download_timeout(self):
"""Test wait raises error on timeout."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
call_count = 0
async def mock_get_status(download_id):
nonlocal call_count
call_count += 1
return {
"download_id": download_id,
"status": "downloading",
"progress": 10.0 * call_count,
}
client.get_download_status = mock_get_status
with pytest.raises(FlacfetchServiceError) as exc_info:
# Very short timeout with short poll interval
await client.wait_for_download("dl_xyz789", timeout=0.2, poll_interval=0.1)
assert "timed out" in str(exc_info.value).lower()
@pytest.mark.asyncio
async def test_wait_for_download_calls_progress_callback(self):
"""Test progress callback is called during wait."""
client = FlacfetchClient(
base_url="http://localhost:8080",
api_key="test-key",
)
call_count = 0
async def mock_get_status(download_id):
nonlocal call_count
call_count += 1
if call_count >= 3:
return {"status": "complete", "gcs_path": "gs://test"}
return {"status": "downloading", "progress": 33.0 * call_count}
client.get_download_status = mock_get_status
progress_updates = []
def progress_callback(status):
progress_updates.append(status)
await client.wait_for_download(
"dl_xyz789",
timeout=10,
poll_interval=0.01,
progress_callback=progress_callback,
)
# Should have received progress updates
assert len(progress_updates) >= 2
class TestGetFlacfetchClient:
"""Test get_flacfetch_client() singleton factory."""
def test_returns_none_when_not_configured(self):
"""Test returns None when FLACFETCH_API_URL not set."""
reset_flacfetch_client()
with patch("backend.config.get_settings") as mock_settings:
mock_settings.return_value = Mock(
flacfetch_api_url=None,
flacfetch_api_key=None,
)
client = get_flacfetch_client()
assert client is None
def test_returns_none_when_key_missing(self):
"""Test returns None when API key not set."""
reset_flacfetch_client()
with patch("backend.config.get_settings") as mock_settings:
mock_settings.return_value = Mock(
flacfetch_api_url="http://localhost:8080",
flacfetch_api_key=None,
)
client = get_flacfetch_client()
assert client is None
def test_returns_client_when_configured(self):
"""Test returns FlacfetchClient when both URL and key are set."""
reset_flacfetch_client()
with patch("backend.config.get_settings") as mock_settings:
mock_settings.return_value = Mock(
flacfetch_api_url="http://10.0.0.5:8080",
flacfetch_api_key="secret-key",
)
client = get_flacfetch_client()
assert client is not None
assert isinstance(client, FlacfetchClient)
assert client.base_url == "http://10.0.0.5:8080"
assert client.api_key == "secret-key"
def test_returns_same_instance(self):
"""Test singleton returns same instance."""
reset_flacfetch_client()
with patch("backend.config.get_settings") as mock_settings:
mock_settings.return_value = Mock(
flacfetch_api_url="http://localhost:8080",
flacfetch_api_key="test-key",
)
client1 = get_flacfetch_client()
client2 = get_flacfetch_client()
assert client1 is client2
def test_reset_clears_singleton(self):
"""Test reset_flacfetch_client() clears the singleton."""
reset_flacfetch_client()
with patch("backend.config.get_settings") as mock_settings:
mock_settings.return_value = Mock(
flacfetch_api_url="http://localhost:8080",
flacfetch_api_key="test-key",
)
client1 = get_flacfetch_client()
reset_flacfetch_client()
# Change the settings
mock_settings.return_value = Mock(
flacfetch_api_url="http://other:8080",
flacfetch_api_key="other-key",
)
client2 = get_flacfetch_client()
assert client1 is not client2
assert client2.base_url == "http://other:8080"
+524
View File
@@ -0,0 +1,524 @@
"""
Tests for gdrive_service.py - Google Drive file operations.
These tests mock the Google API client and Secret Manager to verify:
- Credential loading from Secret Manager
- Folder creation and lookup
- File uploads with proper MIME types
- Public share folder structure uploads
"""
import json
import os
import pytest
from unittest.mock import Mock, MagicMock, patch
class TestGoogleDriveServiceInit:
"""Test GoogleDriveService initialization."""
@patch("backend.services.gdrive_service.get_settings")
def test_init_creates_service(self, mock_get_settings):
"""Test initialization creates service with settings."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
assert service.settings == mock_settings
assert service._service is None
assert service._credentials_data is None
assert service._loaded is False
class TestLoadCredentials:
"""Test _load_credentials method."""
@patch("backend.services.gdrive_service.get_settings")
def test_load_credentials_success(self, mock_get_settings):
"""Test successful credential loading from Secret Manager."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"token": "access-token",
"refresh_token": "refresh-token",
"client_id": "client-id",
"client_secret": "client-secret",
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
creds = service._load_credentials()
assert creds is not None
assert creds["refresh_token"] == "refresh-token"
mock_settings.get_secret.assert_called_once_with("gdrive-oauth-credentials")
@patch("backend.services.gdrive_service.get_settings")
def test_load_credentials_fallback_to_youtube(self, mock_get_settings):
"""Test fallback to YouTube credentials when Drive creds not found."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
# First call returns None (Drive creds not found)
# Second call returns YouTube creds
mock_settings.get_secret.side_effect = [
None,
json.dumps({
"refresh_token": "youtube-token",
"client_id": "youtube-id",
"client_secret": "youtube-secret",
})
]
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
creds = service._load_credentials()
assert creds is not None
assert creds["refresh_token"] == "youtube-token"
assert mock_settings.get_secret.call_count == 2
@patch("backend.services.gdrive_service.get_settings")
def test_load_credentials_not_found(self, mock_get_settings):
"""Test handling when no credentials found."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
creds = service._load_credentials()
assert creds is None
assert service._loaded is True
@patch("backend.services.gdrive_service.get_settings")
def test_load_credentials_missing_required_fields(self, mock_get_settings):
"""Test handling when credentials missing required fields."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"token": "access-token",
# Missing: refresh_token, client_id, client_secret
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
creds = service._load_credentials()
assert creds is None
@patch("backend.services.gdrive_service.get_settings")
def test_load_credentials_cached(self, mock_get_settings):
"""Test credentials are cached after first load."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
creds1 = service._load_credentials()
creds2 = service._load_credentials()
assert creds1 == creds2
assert mock_settings.get_secret.call_count == 1
class TestIsConfigured:
"""Test is_configured property."""
@patch("backend.services.gdrive_service.get_settings")
def test_is_configured_true(self, mock_get_settings):
"""Test is_configured returns True when credentials exist."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
assert service.is_configured is True
@patch("backend.services.gdrive_service.get_settings")
def test_is_configured_false(self, mock_get_settings):
"""Test is_configured returns False when no credentials."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
assert service.is_configured is False
class TestDriveService:
"""Test service property."""
@patch("googleapiclient.discovery.build")
@patch("google.oauth2.credentials.Credentials")
@patch("backend.services.gdrive_service.get_settings")
def test_service_creates_drive_client(
self, mock_get_settings, mock_creds_class, mock_build
):
"""Test service property creates Google Drive API client."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"token": "access-token",
"refresh_token": "refresh-token",
"client_id": "client-id",
"client_secret": "client-secret",
})
mock_get_settings.return_value = mock_settings
mock_creds = Mock()
mock_creds_class.return_value = mock_creds
mock_drive_service = Mock()
mock_build.return_value = mock_drive_service
service = GoogleDriveService()
result = service.service
mock_build.assert_called_once_with("drive", "v3", credentials=mock_creds)
assert result == mock_drive_service
@patch("backend.services.gdrive_service.get_settings")
def test_service_raises_on_missing_credentials(self, mock_get_settings):
"""Test service raises RuntimeError when credentials missing."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
with pytest.raises(RuntimeError) as exc_info:
_ = service.service
assert "not configured" in str(exc_info.value)
class TestGetOrCreateFolder:
"""Test get_or_create_folder method."""
@patch("backend.services.gdrive_service.get_settings")
def test_get_existing_folder(self, mock_get_settings):
"""Test finding an existing folder."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
# Mock the Drive API
mock_files_api = Mock()
mock_list_result = Mock()
mock_list_result.execute.return_value = {
"files": [{"id": "folder-id-123", "name": "MP4"}]
}
mock_files_api.list.return_value = mock_list_result
mock_drive = Mock()
mock_drive.files.return_value = mock_files_api
service._service = mock_drive
service._loaded = True
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
folder_id = service.get_or_create_folder("parent-123", "MP4")
assert folder_id == "folder-id-123"
@patch("backend.services.gdrive_service.get_settings")
def test_create_new_folder(self, mock_get_settings):
"""Test creating a new folder when it doesn't exist."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = GoogleDriveService()
# Mock the Drive API
mock_files_api = Mock()
# list returns empty (folder doesn't exist)
mock_list_result = Mock()
mock_list_result.execute.return_value = {"files": []}
mock_files_api.list.return_value = mock_list_result
# create returns new folder
mock_create_result = Mock()
mock_create_result.execute.return_value = {"id": "new-folder-id"}
mock_files_api.create.return_value = mock_create_result
mock_drive = Mock()
mock_drive.files.return_value = mock_files_api
service._service = mock_drive
service._loaded = True
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
folder_id = service.get_or_create_folder("parent-123", "NewFolder")
assert folder_id == "new-folder-id"
mock_files_api.create.assert_called_once()
class TestUploadFile:
"""Test upload_file method."""
@patch("googleapiclient.http.MediaFileUpload")
@patch("backend.services.gdrive_service.get_settings")
def test_upload_file_success(self, mock_get_settings, mock_media, tmp_path):
"""Test uploading a file to Drive."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
# Create test file
test_file = tmp_path / "test.mp4"
test_file.write_bytes(b"video content")
service = GoogleDriveService()
mock_files_api = Mock()
# list returns empty (no existing file)
mock_list_result = Mock()
mock_list_result.execute.return_value = {"files": []}
mock_files_api.list.return_value = mock_list_result
# create returns new file
mock_create_result = Mock()
mock_create_result.execute.return_value = {"id": "file-id-123"}
mock_files_api.create.return_value = mock_create_result
mock_drive = Mock()
mock_drive.files.return_value = mock_files_api
service._service = mock_drive
service._loaded = True
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
file_id = service.upload_file(str(test_file), "parent-123", "video.mp4")
assert file_id == "file-id-123"
mock_media.assert_called_once()
# Verify MIME type was set correctly
call_kwargs = mock_media.call_args.kwargs
assert call_kwargs["mimetype"] == "video/mp4"
@patch("googleapiclient.http.MediaFileUpload")
@patch("backend.services.gdrive_service.get_settings")
def test_upload_file_replaces_existing(self, mock_get_settings, mock_media, tmp_path):
"""Test upload deletes existing file before upload."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
test_file = tmp_path / "test.mp4"
test_file.write_bytes(b"content")
service = GoogleDriveService()
mock_files_api = Mock()
# list returns existing file
mock_list_result = Mock()
mock_list_result.execute.return_value = {
"files": [{"id": "existing-file-id"}]
}
mock_files_api.list.return_value = mock_list_result
mock_delete_result = Mock()
mock_delete_result.execute.return_value = {}
mock_files_api.delete.return_value = mock_delete_result
mock_create_result = Mock()
mock_create_result.execute.return_value = {"id": "new-file-id"}
mock_files_api.create.return_value = mock_create_result
mock_drive = Mock()
mock_drive.files.return_value = mock_files_api
service._service = mock_drive
service._loaded = True
service._credentials_data = {"refresh_token": "token", "client_id": "id", "client_secret": "secret"}
file_id = service.upload_file(str(test_file), "parent-123", "video.mp4")
# Should have deleted existing file
mock_files_api.delete.assert_called_once_with(fileId="existing-file-id")
assert file_id == "new-file-id"
class TestUploadToPublicShare:
"""Test upload_to_public_share method."""
@patch("backend.services.gdrive_service.get_settings")
def test_upload_to_public_share(self, mock_get_settings, tmp_path):
"""Test uploading files to public share folder structure."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
# Create test files
mp4_file = tmp_path / "output.mp4"
mp4_file.write_bytes(b"4k video")
mp4_720_file = tmp_path / "output_720p.mp4"
mp4_720_file.write_bytes(b"720p video")
cdg_file = tmp_path / "output.zip"
cdg_file.write_bytes(b"cdg package")
service = GoogleDriveService()
# Mock methods
with patch.object(service, "get_or_create_folder") as mock_get_folder:
with patch.object(service, "upload_file") as mock_upload:
mock_get_folder.side_effect = [
"mp4-folder-id",
"mp4-720-folder-id",
"cdg-folder-id",
]
mock_upload.side_effect = [
"mp4-file-id",
"720p-file-id",
"cdg-file-id",
]
output_files = {
"final_karaoke_lossy_mp4": str(mp4_file),
"final_karaoke_lossy_720p_mp4": str(mp4_720_file),
"final_karaoke_cdg_zip": str(cdg_file),
}
result = service.upload_to_public_share(
root_folder_id="root-123",
brand_code="NOMAD-1163",
base_name="Artist - Title",
output_files=output_files,
)
# Should have created/found 3 folders
assert mock_get_folder.call_count == 3
# Should have uploaded 3 files
assert mock_upload.call_count == 3
# Check result
assert result["mp4"] == "mp4-file-id"
assert result["mp4_720p"] == "720p-file-id"
assert result["cdg"] == "cdg-file-id"
@patch("backend.services.gdrive_service.get_settings")
def test_upload_to_public_share_skips_missing_files(
self, mock_get_settings, tmp_path
):
"""Test upload skips files that don't exist."""
from backend.services.gdrive_service import GoogleDriveService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
# Only create one file
mp4_file = tmp_path / "output.mp4"
mp4_file.write_bytes(b"video")
service = GoogleDriveService()
with patch.object(service, "get_or_create_folder") as mock_get_folder:
with patch.object(service, "upload_file") as mock_upload:
mock_get_folder.return_value = "folder-id"
mock_upload.return_value = "file-id"
output_files = {
"final_karaoke_lossy_mp4": str(mp4_file),
"final_karaoke_lossy_720p_mp4": "/nonexistent/file.mp4",
"final_karaoke_cdg_zip": None,
}
result = service.upload_to_public_share(
root_folder_id="root-123",
brand_code="CODE",
base_name="Name",
output_files=output_files,
)
# Should only upload the one file that exists
assert mock_upload.call_count == 1
assert len(result) == 1
class TestGetGdriveService:
"""Test get_gdrive_service singleton."""
@patch("backend.services.gdrive_service.get_settings")
def test_get_gdrive_service_singleton(self, mock_get_settings):
"""Test get_gdrive_service returns singleton instance."""
from backend.services.gdrive_service import get_gdrive_service
import backend.services.gdrive_service as gdrive_module
# Reset singleton
gdrive_module._gdrive_service = None
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service1 = get_gdrive_service()
service2 = get_gdrive_service()
assert service1 is service2
+5 -5
View File
@@ -98,7 +98,7 @@ class TestAudioWorkerEndpoint:
)
assert response.status_code == 422
def test_audio_worker_requires_auth(self, client):
def test_audio_worker_requires_auth(self, client, auth_headers):
"""Test audio worker requires authentication."""
response = client.post(
"/api/internal/workers/audio",
@@ -128,7 +128,7 @@ class TestLyricsWorkerEndpoint:
)
assert response.status_code == 422
def test_lyrics_worker_requires_auth(self, client):
def test_lyrics_worker_requires_auth(self, client, auth_headers):
"""Test lyrics worker requires authentication."""
response = client.post(
"/api/internal/workers/lyrics",
@@ -158,7 +158,7 @@ class TestScreensWorkerEndpoint:
)
assert response.status_code == 422
def test_screens_worker_requires_auth(self, client):
def test_screens_worker_requires_auth(self, client, auth_headers):
"""Test screens worker requires authentication."""
response = client.post(
"/api/internal/workers/screens",
@@ -188,7 +188,7 @@ class TestVideoWorkerEndpoint:
)
assert response.status_code == 422
def test_video_worker_requires_auth(self, client):
def test_video_worker_requires_auth(self, client, auth_headers):
"""Test video worker requires authentication."""
response = client.post(
"/api/internal/workers/video",
@@ -218,7 +218,7 @@ class TestRenderVideoWorkerEndpoint:
)
assert response.status_code == 422
def test_render_video_worker_requires_auth(self, client):
def test_render_video_worker_requires_auth(self, client, auth_headers):
"""Test render-video worker requires authentication."""
response = client.post(
"/api/internal/workers/render-video",
+38 -32
View File
@@ -68,20 +68,20 @@ def client(mock_job_manager, mock_worker_service):
class TestGetJob:
"""Tests for GET /api/jobs/{job_id}."""
def test_get_job_returns_200(self, client, mock_job_manager, mock_job):
def test_get_job_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
"""Test getting an existing job returns 200."""
response = client.get("/api/jobs/test123")
response = client.get("/api/jobs/test123", headers=auth_headers)
assert response.status_code == 200
def test_get_job_returns_job_data(self, client, mock_job_manager, mock_job):
def test_get_job_returns_job_data(self, client, mock_job_manager, mock_job, auth_headers):
"""Test response contains job data."""
response = client.get("/api/jobs/test123")
response = client.get("/api/jobs/test123", headers=auth_headers)
data = response.json()
assert data["job_id"] == "test123"
assert data["status"] == "pending"
assert data["artist"] == "Test Artist"
def test_get_nonexistent_job_returns_404(self, mock_worker_service):
def test_get_nonexistent_job_returns_404(self, mock_worker_service, auth_headers):
"""Test getting non-existent job returns 404."""
mock_job_manager = MagicMock()
mock_job_manager.get_job.return_value = None
@@ -95,58 +95,60 @@ class TestGetJob:
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.get("/api/jobs/nonexistent")
response = client.get("/api/jobs/nonexistent", headers=auth_headers)
assert response.status_code == 404
class TestListJobs:
"""Tests for GET /api/jobs."""
def test_list_jobs_returns_200(self, client):
def test_list_jobs_returns_200(self, client, auth_headers):
"""Test listing jobs returns 200."""
response = client.get("/api/jobs")
response = client.get("/api/jobs", headers=auth_headers)
assert response.status_code == 200
def test_list_jobs_returns_array(self, client, mock_job_manager):
def test_list_jobs_returns_array(self, client, mock_job_manager, auth_headers):
"""Test response is an array of jobs."""
response = client.get("/api/jobs")
response = client.get("/api/jobs", headers=auth_headers)
data = response.json()
assert isinstance(data, list)
assert len(data) == 1
assert data[0]["job_id"] == "test123"
def test_list_jobs_with_status_filter(self, client, mock_job_manager):
def test_list_jobs_with_status_filter(self, client, mock_job_manager, auth_headers):
"""Test listing jobs with status filter."""
response = client.get("/api/jobs?status=pending")
response = client.get("/api/jobs?status=pending", headers=auth_headers)
assert response.status_code == 200
def test_list_jobs_with_limit(self, client, mock_job_manager):
def test_list_jobs_with_limit(self, client, mock_job_manager, auth_headers):
"""Test listing jobs with limit."""
response = client.get("/api/jobs?limit=10")
response = client.get("/api/jobs?limit=10", headers=auth_headers)
assert response.status_code == 200
class TestCreateJob:
"""Tests for POST /api/jobs."""
def test_create_job_with_url_returns_200(self, client, mock_job_manager):
def test_create_job_with_url_returns_200(self, client, mock_job_manager, auth_headers):
"""Test creating job with URL returns 200."""
response = client.post(
"/api/jobs",
json={"url": "https://youtube.com/watch?v=test123"}
json={"url": "https://youtube.com/watch?v=test123"},
headers=auth_headers
)
assert response.status_code == 200
def test_create_job_returns_job_id(self, client, mock_job_manager):
def test_create_job_returns_job_id(self, client, mock_job_manager, auth_headers):
"""Test create response contains job_id."""
response = client.post(
"/api/jobs",
json={"url": "https://youtube.com/watch?v=test"}
json={"url": "https://youtube.com/watch?v=test"},
headers=auth_headers
)
data = response.json()
assert "job_id" in data
def test_create_job_with_artist_title(self, client, mock_job_manager):
def test_create_job_with_artist_title(self, client, mock_job_manager, auth_headers):
"""Test creating job with artist and title."""
response = client.post(
"/api/jobs",
@@ -154,7 +156,8 @@ class TestCreateJob:
"url": "https://youtube.com/watch?v=test",
"artist": "Test Artist",
"title": "Test Song"
}
},
headers=auth_headers
)
assert response.status_code == 200
@@ -162,12 +165,12 @@ class TestCreateJob:
class TestDeleteJob:
"""Tests for DELETE /api/jobs/{job_id}."""
def test_delete_job_returns_200(self, client, mock_job_manager):
def test_delete_job_returns_200(self, client, mock_job_manager, auth_headers):
"""Test deleting job returns 200."""
response = client.delete("/api/jobs/test123")
response = client.delete("/api/jobs/test123", headers=auth_headers)
assert response.status_code == 200
def test_delete_nonexistent_job(self, mock_worker_service):
def test_delete_nonexistent_job(self, mock_worker_service, auth_headers):
"""Test deleting non-existent job."""
mock_job_manager = MagicMock()
mock_job_manager.delete_job.return_value = False
@@ -181,7 +184,7 @@ class TestDeleteJob:
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
from backend.main import app
client = TestClient(app)
response = client.delete("/api/jobs/nonexistent")
response = client.delete("/api/jobs/nonexistent", headers=auth_headers)
# Either 200 or 404 depending on implementation
assert response.status_code in [200, 404]
@@ -189,12 +192,13 @@ class TestDeleteJob:
class TestCancelJob:
"""Tests for POST /api/jobs/{job_id}/cancel."""
def test_cancel_job_returns_200(self, client, mock_job_manager, mock_job):
def test_cancel_job_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
"""Test cancelling job returns 200."""
mock_job_manager.cancel_job.return_value = mock_job
response = client.post(
"/api/jobs/test123/cancel",
json={}
json={},
headers=auth_headers
)
assert response.status_code == 200
@@ -202,7 +206,7 @@ class TestCancelJob:
class TestSubmitCorrections:
"""Tests for POST /api/jobs/{job_id}/corrections."""
def test_submit_corrections_returns_200(self, client, mock_job_manager, mock_job):
def test_submit_corrections_returns_200(self, client, mock_job_manager, mock_job, auth_headers):
"""Test submitting corrections returns 200."""
# Job needs to be in AWAITING_REVIEW or IN_REVIEW status
review_job = Job(
@@ -222,7 +226,8 @@ class TestSubmitCorrections:
"lines": [],
"metadata": {"source": "test"}
}
}
},
headers=auth_headers
)
# Should be 200 or validation error
assert response.status_code in [200, 400, 422]
@@ -231,11 +236,12 @@ class TestSubmitCorrections:
class TestSelectInstrumental:
"""Tests for POST /api/jobs/{job_id}/select-instrumental."""
def test_select_instrumental_requires_selection_field(self, client, mock_job_manager, mock_job):
def test_select_instrumental_requires_selection_field(self, client, mock_job_manager, mock_job, auth_headers):
"""Test instrumental selection requires selection field."""
response = client.post(
"/api/jobs/test123/select-instrumental",
json={} # Missing selection field
json={}, # Missing selection field
headers=auth_headers
)
# Missing field should cause validation error
assert response.status_code == 422
@@ -244,9 +250,9 @@ class TestSelectInstrumental:
class TestStartReview:
"""Tests for POST /api/jobs/{job_id}/start-review."""
def test_start_review_endpoint_exists(self, client, mock_job_manager, mock_job):
def test_start_review_endpoint_exists(self, client, mock_job_manager, mock_job, auth_headers):
"""Test start review endpoint exists."""
response = client.post("/api/jobs/test123/start-review")
response = client.post("/api/jobs/test123/start-review", headers=auth_headers)
# Should not be 404 or 405
assert response.status_code not in [404, 405]
+448
View File
@@ -0,0 +1,448 @@
"""
Tests for storage_service.py - Google Cloud Storage operations.
These tests mock the GCS client to verify:
- Upload operations (file, fileobj, JSON)
- Download operations (file, JSON)
- Signed URL generation
- File operations (delete, list, exists)
"""
import json
import pytest
from unittest.mock import Mock, MagicMock, patch
from io import BytesIO
from backend.services.storage_service import StorageService
class TestStorageServiceInit:
"""Test StorageService initialization."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_init_creates_client_and_bucket(self, mock_settings, mock_client_class):
"""Test initialization creates GCS client and gets bucket."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_client = Mock()
mock_bucket = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
mock_client_class.assert_called_once_with(project="test-project")
mock_client.bucket.assert_called_once_with("test-bucket")
assert service.client == mock_client
assert service.bucket == mock_bucket
class TestStorageServiceUpload:
"""Test upload operations."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_upload_file(self, mock_settings, mock_client_class):
"""Test uploading a file to GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
result = service.upload_file("/local/path/file.flac", "uploads/job123/file.flac")
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
mock_blob.upload_from_filename.assert_called_once_with("/local/path/file.flac")
assert result == "uploads/job123/file.flac"
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_upload_file_raises_on_error(self, mock_settings, mock_client_class):
"""Test upload_file raises exception on GCS error."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.upload_from_filename.side_effect = Exception("GCS error")
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
with pytest.raises(Exception) as exc_info:
service.upload_file("/local/path/file.flac", "dest/file.flac")
assert "GCS error" in str(exc_info.value)
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_upload_fileobj(self, mock_settings, mock_client_class):
"""Test uploading a file object to GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
file_obj = BytesIO(b"test content")
result = service.upload_fileobj(file_obj, "uploads/test.txt", content_type="text/plain")
mock_bucket.blob.assert_called_once_with("uploads/test.txt")
assert mock_blob.content_type == "text/plain"
mock_blob.upload_from_file.assert_called_once_with(file_obj, rewind=True)
assert result == "uploads/test.txt"
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_upload_json(self, mock_settings, mock_client_class):
"""Test uploading JSON data to GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
data = {"key": "value", "nested": {"data": True}}
result = service.upload_json("data/config.json", data)
mock_bucket.blob.assert_called_once_with("data/config.json")
assert mock_blob.content_type == "application/json"
# Check the uploaded content
call_args = mock_blob.upload_from_string.call_args
uploaded_content = call_args[0][0]
assert json.loads(uploaded_content) == data
assert result == "data/config.json"
class TestStorageServiceDownload:
"""Test download operations."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_download_file(self, mock_settings, mock_client_class):
"""Test downloading a file from GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
result = service.download_file("uploads/job123/file.flac", "/local/path/file.flac")
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
mock_blob.download_to_filename.assert_called_once_with("/local/path/file.flac")
assert result == "/local/path/file.flac"
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_download_file_raises_on_error(self, mock_settings, mock_client_class):
"""Test download_file raises exception on GCS error."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.download_to_filename.side_effect = Exception("File not found")
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
with pytest.raises(Exception) as exc_info:
service.download_file("missing/file.flac", "/local/path")
assert "File not found" in str(exc_info.value)
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_download_json(self, mock_settings, mock_client_class):
"""Test downloading and parsing JSON from GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.download_as_text.return_value = '{"key": "value", "count": 42}'
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
result = service.download_json("data/config.json")
assert result == {"key": "value", "count": 42}
class TestStorageServiceDelete:
"""Test delete operations."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_delete_file(self, mock_settings, mock_client_class):
"""Test deleting a single file from GCS."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
service.delete_file("uploads/job123/file.flac")
mock_bucket.blob.assert_called_once_with("uploads/job123/file.flac")
mock_blob.delete.assert_called_once()
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_delete_folder(self, mock_settings, mock_client_class):
"""Test deleting all files with a prefix (folder)."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
# Create mock blobs
mock_blob1 = Mock()
mock_blob1.name = "uploads/job123/file1.flac"
mock_blob2 = Mock()
mock_blob2.name = "uploads/job123/file2.flac"
mock_blob3 = Mock()
mock_blob3.name = "uploads/job123/subdir/file3.flac"
mock_bucket = Mock()
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2, mock_blob3]
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
count = service.delete_folder("uploads/job123/")
mock_bucket.list_blobs.assert_called_once_with(prefix="uploads/job123/")
mock_blob1.delete.assert_called_once()
mock_blob2.delete.assert_called_once()
mock_blob3.delete.assert_called_once()
assert count == 3
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_delete_folder_handles_errors_gracefully(self, mock_settings, mock_client_class):
"""Test delete_folder continues even if some deletes fail."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob1 = Mock()
mock_blob1.name = "file1.flac"
mock_blob2 = Mock()
mock_blob2.name = "file2.flac"
mock_blob2.delete.side_effect = Exception("Permission denied")
mock_blob3 = Mock()
mock_blob3.name = "file3.flac"
mock_bucket = Mock()
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2, mock_blob3]
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
count = service.delete_folder("uploads/")
# Should have deleted 2 files (blob2 failed)
assert count == 2
class TestStorageServiceFileOperations:
"""Test file listing and existence checks."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_list_files(self, mock_settings, mock_client_class):
"""Test listing files with a prefix."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob1 = Mock()
mock_blob1.name = "uploads/job123/audio.flac"
mock_blob2 = Mock()
mock_blob2.name = "uploads/job123/lyrics.txt"
mock_bucket = Mock()
mock_bucket.list_blobs.return_value = [mock_blob1, mock_blob2]
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
files = service.list_files("uploads/job123/")
assert files == ["uploads/job123/audio.flac", "uploads/job123/lyrics.txt"]
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_file_exists_returns_true(self, mock_settings, mock_client_class):
"""Test file_exists returns True for existing file."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.exists.return_value = True
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
result = service.file_exists("uploads/job123/file.flac")
assert result is True
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_file_exists_returns_false(self, mock_settings, mock_client_class):
"""Test file_exists returns False for missing file."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.exists.return_value = False
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
service = StorageService()
result = service.file_exists("missing/file.flac")
assert result is False
class TestStorageServiceSignedUrls:
"""Test signed URL generation."""
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_generate_signed_url(self, mock_settings, mock_client_class):
"""Test generating a signed download URL."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
# Mock google.auth.default to return credentials without service_account_email
with patch("google.auth.default") as mock_auth_default:
mock_credentials = Mock(spec=[]) # No service_account_email attr
mock_auth_default.return_value = (mock_credentials, "test-project")
service = StorageService()
url = service.generate_signed_url("uploads/file.flac", expiration_minutes=30)
assert url == "https://storage.googleapis.com/signed-url"
mock_blob.generate_signed_url.assert_called_once()
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
assert call_kwargs["method"] == "GET"
assert call_kwargs["version"] == "v4"
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_generate_signed_upload_url(self, mock_settings, mock_client_class):
"""Test generating a signed upload URL."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-upload-url"
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
with patch("google.auth.default") as mock_auth_default:
mock_credentials = Mock(spec=[])
mock_auth_default.return_value = (mock_credentials, "test-project")
service = StorageService()
url = service.generate_signed_upload_url(
"uploads/file.flac",
content_type="audio/flac",
expiration_minutes=15
)
assert url == "https://storage.googleapis.com/signed-upload-url"
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
assert call_kwargs["method"] == "PUT"
assert call_kwargs["headers"] == {"Content-Type": "audio/flac"}
@patch("backend.services.storage_service.storage.Client")
@patch("backend.services.storage_service.settings")
def test_signed_url_with_service_account(self, mock_settings, mock_client_class):
"""Test signed URL generation with service account credentials."""
mock_settings.google_cloud_project = "test-project"
mock_settings.gcs_bucket_name = "test-bucket"
mock_blob = Mock()
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
mock_client = Mock()
mock_client.bucket.return_value = mock_bucket
mock_client_class.return_value = mock_client
with patch("google.auth.default") as mock_auth_default:
with patch("google.auth.transport.requests.Request") as mock_request:
# Mock credentials with service_account_email
mock_credentials = Mock()
mock_credentials.service_account_email = "sa@project.iam.gserviceaccount.com"
mock_credentials.token = "access-token-123"
mock_auth_default.return_value = (mock_credentials, "test-project")
service = StorageService()
url = service.generate_signed_url("uploads/file.flac")
# Should have refreshed credentials and used IAM signing
mock_credentials.refresh.assert_called_once()
call_kwargs = mock_blob.generate_signed_url.call_args.kwargs
assert call_kwargs["service_account_email"] == "sa@project.iam.gserviceaccount.com"
assert call_kwargs["access_token"] == "access-token-123"
+9 -2
View File
@@ -16,12 +16,20 @@ from unittest.mock import Mock, MagicMock, patch, AsyncMock
from io import BytesIO
# Test the remote CLI's style parsing
# Skip this class if flacfetch is not properly installed (remote_cli depends on it)
try:
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, Config
_remote_cli_available = True
except ImportError:
_remote_cli_available = False
@pytest.mark.skipif(not _remote_cli_available, reason="flacfetch not properly installed")
class TestRemoteCLIStyleParsing:
"""Test that remote CLI correctly parses style_params.json."""
def test_parse_style_params_extracts_file_paths(self):
"""Test that _parse_style_params extracts all file references."""
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, Config
# Create a mock style_params.json
with tempfile.TemporaryDirectory() as temp_dir:
@@ -79,7 +87,6 @@ class TestRemoteCLIStyleParsing:
def test_parse_style_params_handles_missing_files(self):
"""Test that _parse_style_params ignores non-existent files."""
from karaoke_gen.utils.remote_cli import RemoteKaraokeClient, Config
with tempfile.TemporaryDirectory() as temp_dir:
style_params = {
+42 -17
View File
@@ -10,6 +10,14 @@ from io import BytesIO
from backend.models.job import Job, JobStatus
@pytest.fixture
def upload_auth_headers():
"""Auth headers for file upload tests (no Content-Type, let multipart work)."""
return {
"Authorization": "Bearer test-admin-token"
}
@pytest.fixture
def mock_job():
"""Create a standard mock job."""
@@ -65,37 +73,41 @@ def client(mock_services):
class TestFileUploadEndpoint:
"""Tests for POST /api/jobs/upload."""
def test_upload_flac_returns_200(self, client, mock_services):
def test_upload_flac_returns_200(self, client, mock_services, upload_auth_headers):
"""Test uploading FLAC file returns 200."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.flac", BytesIO(b"fake audio"), "audio/flac")},
data={"artist": "Test Artist", "title": "Test Song"}
)
assert response.status_code == 200
def test_upload_mp3_returns_200(self, client, mock_services):
def test_upload_mp3_returns_200(self, client, mock_services, upload_auth_headers):
"""Test uploading MP3 file returns 200."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.mp3", BytesIO(b"fake audio"), "audio/mpeg")},
data={"artist": "Test Artist", "title": "Test Song"}
)
assert response.status_code == 200
def test_upload_wav_returns_200(self, client, mock_services):
def test_upload_wav_returns_200(self, client, mock_services, upload_auth_headers):
"""Test uploading WAV file returns 200."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.wav", BytesIO(b"fake audio"), "audio/wav")},
data={"artist": "Test Artist", "title": "Test Song"}
)
assert response.status_code == 200
def test_upload_returns_job_id(self, client, mock_services):
def test_upload_returns_job_id(self, client, mock_services, upload_auth_headers):
"""Test upload response contains job_id."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.flac", BytesIO(b"fake audio"), "audio/flac")},
data={"artist": "Test", "title": "Song"}
)
@@ -103,10 +115,11 @@ class TestFileUploadEndpoint:
assert "job_id" in data
assert data["status"] == "success"
def test_upload_returns_filename(self, client, mock_services):
def test_upload_returns_filename(self, client, mock_services, upload_auth_headers):
"""Test upload response contains original filename."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("my_song.flac", BytesIO(b"fake audio"), "audio/flac")},
data={"artist": "Test", "title": "Song"}
)
@@ -114,84 +127,93 @@ class TestFileUploadEndpoint:
assert "filename" in data
assert data["filename"] == "my_song.flac"
def test_upload_rejects_txt_file(self, client, mock_services):
def test_upload_rejects_txt_file(self, client, mock_services, upload_auth_headers):
"""Test upload rejects text files."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.txt", BytesIO(b"not audio"), "text/plain")},
data={"artist": "Test", "title": "Song"}
)
assert response.status_code == 400
def test_upload_rejects_pdf_file(self, client, mock_services):
def test_upload_rejects_pdf_file(self, client, mock_services, upload_auth_headers):
"""Test upload rejects PDF files."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("doc.pdf", BytesIO(b"pdf content"), "application/pdf")},
data={"artist": "Test", "title": "Song"}
)
assert response.status_code == 400
def test_upload_rejects_exe_file(self, client, mock_services):
def test_upload_rejects_exe_file(self, client, mock_services, upload_auth_headers):
"""Test upload rejects executable files."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("app.exe", BytesIO(b"exe content"), "application/octet-stream")},
data={"artist": "Test", "title": "Song"}
)
assert response.status_code == 400
def test_upload_requires_artist(self, client, mock_services):
def test_upload_requires_artist(self, client, mock_services, upload_auth_headers):
"""Test upload requires artist field."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.flac", BytesIO(b"audio"), "audio/flac")},
data={"title": "Song"} # Missing artist
)
assert response.status_code == 422
def test_upload_requires_title(self, client, mock_services):
def test_upload_requires_title(self, client, mock_services, upload_auth_headers):
"""Test upload requires title field."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.flac", BytesIO(b"audio"), "audio/flac")},
data={"artist": "Artist"} # Missing title
)
assert response.status_code == 422
def test_upload_requires_file(self, client, mock_services):
def test_upload_requires_file(self, client, mock_services, upload_auth_headers):
"""Test upload requires file."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
data={"artist": "Artist", "title": "Song"}
# Missing file
)
assert response.status_code == 422
def test_upload_triggers_workers(self, client, mock_services):
def test_upload_triggers_workers(self, client, mock_services, upload_auth_headers):
"""Test upload triggers audio and lyrics workers."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.flac", BytesIO(b"audio"), "audio/flac")},
data={"artist": "Test", "title": "Song"}
)
assert response.status_code == 200
# Workers should be triggered in background
def test_upload_creates_job(self, client, mock_services):
def test_upload_creates_job(self, client, mock_services, upload_auth_headers):
"""Test upload creates job in job manager."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.flac", BytesIO(b"audio"), "audio/flac")},
data={"artist": "Test", "title": "Song"}
)
assert response.status_code == 200
mock_services['job_manager'].create_job.assert_called()
def test_upload_stores_file_to_gcs(self, client, mock_services):
def test_upload_stores_file_to_gcs(self, client, mock_services, upload_auth_headers):
"""Test upload stores file to GCS."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.flac", BytesIO(b"audio"), "audio/flac")},
data={"artist": "Test", "title": "Song"}
)
@@ -202,28 +224,31 @@ class TestFileUploadEndpoint:
class TestUploadValidation:
"""Tests for upload validation logic."""
def test_upload_accepts_m4a(self, client, mock_services):
def test_upload_accepts_m4a(self, client, mock_services, upload_auth_headers):
"""Test upload accepts M4A files."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.m4a", BytesIO(b"audio"), "audio/mp4")},
data={"artist": "Test", "title": "Song"}
)
assert response.status_code == 200
def test_upload_accepts_ogg(self, client, mock_services):
def test_upload_accepts_ogg(self, client, mock_services, upload_auth_headers):
"""Test upload accepts OGG files."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.ogg", BytesIO(b"audio"), "audio/ogg")},
data={"artist": "Test", "title": "Song"}
)
assert response.status_code == 200
def test_upload_accepts_aac(self, client, mock_services):
def test_upload_accepts_aac(self, client, mock_services, upload_auth_headers):
"""Test upload accepts AAC files."""
response = client.post(
"/api/jobs/upload",
headers=upload_auth_headers,
files={"file": ("test.aac", BytesIO(b"audio"), "audio/aac")},
data={"artist": "Test", "title": "Song"}
)
+247
View File
@@ -0,0 +1,247 @@
"""
Tests for youtube_service.py - YouTube credential management.
These tests mock Secret Manager to verify:
- Credential loading and parsing
- Validation of required fields
- Error handling for missing/invalid credentials
"""
import json
import pytest
from unittest.mock import Mock, patch
class TestYouTubeServiceInit:
"""Test YouTubeService initialization."""
@patch("backend.services.youtube_service.get_settings")
def test_init_creates_service(self, mock_get_settings):
"""Test initialization creates service with settings."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_get_settings.return_value = mock_settings
service = YouTubeService()
assert service.settings == mock_settings
assert service._credentials is None
assert service._loaded is False
class TestLoadCredentials:
"""Test load_credentials method."""
@patch("backend.services.youtube_service.get_settings")
def test_load_credentials_success(self, mock_get_settings):
"""Test successful credential loading from Secret Manager."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"token": "access-token-123",
"refresh_token": "refresh-token-456",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "client-id.apps.googleusercontent.com",
"client_secret": "client-secret-789",
"scopes": ["https://www.googleapis.com/auth/youtube.upload"],
})
mock_get_settings.return_value = mock_settings
service = YouTubeService()
result = service.load_credentials()
assert result is True
assert service._credentials is not None
assert service._credentials["refresh_token"] == "refresh-token-456"
mock_settings.get_secret.assert_called_once_with("youtube-oauth-credentials")
@patch("backend.services.youtube_service.get_settings")
def test_load_credentials_not_found(self, mock_get_settings):
"""Test handling when credentials are not in Secret Manager."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service = YouTubeService()
result = service.load_credentials()
assert result is False
assert service._credentials is None
assert service._loaded is True
@patch("backend.services.youtube_service.get_settings")
def test_load_credentials_missing_required_fields(self, mock_get_settings):
"""Test handling when credentials are missing required fields."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"token": "access-token",
# Missing: refresh_token, token_uri, client_id, client_secret
})
mock_get_settings.return_value = mock_settings
service = YouTubeService()
result = service.load_credentials()
assert result is False
assert service._credentials is None
@patch("backend.services.youtube_service.get_settings")
def test_load_credentials_invalid_json(self, mock_get_settings):
"""Test handling of invalid JSON in credentials."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_settings.get_secret.return_value = "not valid json {"
mock_get_settings.return_value = mock_settings
service = YouTubeService()
result = service.load_credentials()
assert result is False
assert service._credentials is None
@patch("backend.services.youtube_service.get_settings")
def test_load_credentials_exception(self, mock_get_settings):
"""Test handling of exceptions during credential loading."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_settings.get_secret.side_effect = Exception("Secret Manager error")
mock_get_settings.return_value = mock_settings
service = YouTubeService()
result = service.load_credentials()
assert result is False
assert service._loaded is True
@patch("backend.services.youtube_service.get_settings")
def test_load_credentials_cached(self, mock_get_settings):
"""Test that credentials are cached after first load."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"token_uri": "uri",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = YouTubeService()
# First call
result1 = service.load_credentials()
# Second call
result2 = service.load_credentials()
assert result1 is True
assert result2 is True
# Secret Manager should only be called once
assert mock_settings.get_secret.call_count == 1
class TestGetCredentialsDict:
"""Test get_credentials_dict method."""
@patch("backend.services.youtube_service.get_settings")
def test_get_credentials_dict_loads_if_needed(self, mock_get_settings):
"""Test get_credentials_dict loads credentials if not already loaded."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"token_uri": "uri",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = YouTubeService()
# Don't explicitly call load_credentials
result = service.get_credentials_dict()
assert result is not None
assert result["refresh_token"] == "token"
@patch("backend.services.youtube_service.get_settings")
def test_get_credentials_dict_returns_none_if_not_configured(
self, mock_get_settings
):
"""Test get_credentials_dict returns None when not configured."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service = YouTubeService()
result = service.get_credentials_dict()
assert result is None
class TestIsConfigured:
"""Test is_configured property."""
@patch("backend.services.youtube_service.get_settings")
def test_is_configured_true(self, mock_get_settings):
"""Test is_configured returns True when credentials exist."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_settings.get_secret.return_value = json.dumps({
"refresh_token": "token",
"token_uri": "uri",
"client_id": "id",
"client_secret": "secret",
})
mock_get_settings.return_value = mock_settings
service = YouTubeService()
assert service.is_configured is True
@patch("backend.services.youtube_service.get_settings")
def test_is_configured_false(self, mock_get_settings):
"""Test is_configured returns False when credentials missing."""
from backend.services.youtube_service import YouTubeService
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service = YouTubeService()
assert service.is_configured is False
class TestGetYouTubeService:
"""Test get_youtube_service singleton."""
@patch("backend.services.youtube_service.get_settings")
def test_get_youtube_service_singleton(self, mock_get_settings):
"""Test get_youtube_service returns singleton instance."""
from backend.services.youtube_service import get_youtube_service
import backend.services.youtube_service as youtube_module
# Reset singleton
youtube_module._youtube_service = None
mock_settings = Mock()
mock_settings.get_secret.return_value = None
mock_get_settings.return_value = mock_settings
service1 = get_youtube_service()
service2 = get_youtube_service()
assert service1 is service2
+119 -86
View File
@@ -7,11 +7,17 @@ Handles the audio processing track of parallel processing:
3. Post-processing: Combine instrumentals, normalize
Re-uses karaoke_gen.audio_processor.AudioProcessor for remote GPU separation.
Observability:
- All operations wrapped in tracing spans for Cloud Trace visibility
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
- Worker start/end timing logged with WORKER_START/WORKER_END markers
"""
import logging
import os
import shutil
import tempfile
import time
from typing import Optional, Dict, Any
from pathlib import Path
@@ -20,6 +26,8 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.config import get_settings
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.services.tracing import job_span, add_span_event, add_span_attribute
from backend.services.metrics import metrics
# Import from karaoke_gen package
from karaoke_gen.audio_processor import AudioProcessor
@@ -215,6 +223,7 @@ async def process_audio_separation(job_id: str) -> bool:
Returns:
True if successful, False otherwise
"""
start_time = time.time()
job_manager = JobManager()
storage = StorageService()
settings = get_settings()
@@ -222,7 +231,8 @@ async def process_audio_separation(job_id: str) -> bool:
# Create job logger for remote debugging FIRST
job_log = create_job_logger(job_id, "audio")
# Log immediately to verify logging is working
# Log with structured markers for easy Cloud Logging queries
logger.info(f"[job:{job_id}] WORKER_START worker=audio")
job_log.info("=== AUDIO WORKER STARTED ===")
job_log.info(f"Job ID: {job_id}")
@@ -232,7 +242,7 @@ async def process_audio_separation(job_id: str) -> bool:
job = job_manager.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
logger.error(f"[job:{job_id}] Job not found in Firestore")
job_log.error(f"Job {job_id} not found in Firestore!")
return False
@@ -241,91 +251,114 @@ async def process_audio_separation(job_id: str) -> bool:
job_log.info(f"Created temp directory: {temp_dir}")
try:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
# This ensures logs from third-party libraries (karaoke_gen.audio_processor) are
# only captured by this job's handler, not handlers from other concurrent jobs
with job_logging_context(job_id):
job_log.info(f"Starting audio separation for {job.artist} - {job.title}")
logger.info(f"Starting audio separation for job {job_id}")
# Ensure AUDIO_SEPARATOR_API_URL is set
api_url = os.environ.get("AUDIO_SEPARATOR_API_URL")
if not api_url:
raise Exception("AUDIO_SEPARATOR_API_URL environment variable not set. "
"Cannot perform audio separation without remote API access.")
job_log.info(f"Audio separator API: {api_url}")
# Download audio file from GCS or URL
job_log.info("Downloading audio file...")
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager_instance=job_manager)
if not audio_path:
raise Exception("Failed to download audio file")
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
# Update progress using state_data (don't change status during parallel processing)
# The status is managed at a higher level - workers just track their progress
job_manager.update_state_data(job_id, 'audio_progress', {
'stage': 'separating_stage1',
'progress': 10,
'message': 'Starting audio separation (Stage 1: Clean instrumental)'
})
# Create AudioProcessor instance (reuses karaoke_gen code)
# Use model configuration from job if provided, otherwise use defaults
job_log.info("Creating AudioProcessor instance...")
if job.clean_instrumental_model:
job_log.info(f" Using clean instrumental model: {job.clean_instrumental_model}")
if job.backing_vocals_models:
job_log.info(f" Using backing vocals models: {job.backing_vocals_models}")
if job.other_stems_models:
job_log.info(f" Using other stems models: {job.other_stems_models}")
audio_processor = create_audio_processor(
temp_dir,
clean_instrumental_model=job.clean_instrumental_model,
backing_vocals_models=job.backing_vocals_models,
other_stems_models=job.other_stems_models
)
# Format artist-title for file naming (matches CLI behavior)
artist_title = f"{job.artist} - {job.title}"
# Run audio separation (calls Modal API internally)
# This returns a dict with paths to all separated stems
job_log.info("Starting audio separation (this may take 5-10 minutes)...")
job_log.info(" Stage 1: Clean instrumental separation (MDX models)")
job_log.info(" Stage 2: Backing vocals separation (Demucs model)")
logger.info(f"Job {job_id}: Calling audio_processor.process_audio_separation()")
separation_result = audio_processor.process_audio_separation(
audio_file=audio_path,
artist_title=artist_title,
track_output_dir=temp_dir
)
job_log.info("Audio separation complete!")
job_log.info(f" Generated {len(separation_result)} stem files")
logger.info(f"Job {job_id}: Audio separation complete, organizing results")
# Update progress using state_data (don't change status during parallel processing)
job_manager.update_state_data(job_id, 'audio_progress', {
'stage': 'audio_complete',
'progress': 45,
'message': 'Audio separation complete, uploading stems'
})
# Upload all stems to GCS
await upload_separation_results(job_id, separation_result, storage, job_manager)
logger.info(f"Job {job_id}: All stems uploaded successfully")
# Mark audio processing complete
# This will check if lyrics are also complete and transition to next stage if so
job_manager.mark_audio_complete(job_id)
return True
# Wrap entire worker in a tracing span
with job_span("audio-worker", job_id, {"artist": job.artist, "title": job.title}) as root_span:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
# This ensures logs from third-party libraries (karaoke_gen.audio_processor) are
# only captured by this job's handler, not handlers from other concurrent jobs
with job_logging_context(job_id):
job_log.info(f"Starting audio separation for {job.artist} - {job.title}")
logger.info(f"[job:{job_id}] Starting audio separation for {job.artist} - {job.title}")
# Ensure AUDIO_SEPARATOR_API_URL is set
api_url = os.environ.get("AUDIO_SEPARATOR_API_URL")
if not api_url:
raise Exception("AUDIO_SEPARATOR_API_URL environment variable not set. "
"Cannot perform audio separation without remote API access.")
job_log.info(f"Audio separator API: {api_url}")
add_span_attribute("audio_separator_api", api_url)
# Download audio file from GCS or URL
with job_span("download-audio", job_id) as download_span:
job_log.info("Downloading audio file...")
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager_instance=job_manager)
if not audio_path:
raise Exception("Failed to download audio file")
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
download_span.set_attribute("audio_file", os.path.basename(audio_path))
download_span.set_attribute("source", "url" if job.url else "gcs")
# Update progress using state_data (don't change status during parallel processing)
# The status is managed at a higher level - workers just track their progress
job_manager.update_state_data(job_id, 'audio_progress', {
'stage': 'separating_stage1',
'progress': 10,
'message': 'Starting audio separation (Stage 1: Clean instrumental)'
})
# Create AudioProcessor instance (reuses karaoke_gen code)
# Use model configuration from job if provided, otherwise use defaults
job_log.info("Creating AudioProcessor instance...")
if job.clean_instrumental_model:
job_log.info(f" Using clean instrumental model: {job.clean_instrumental_model}")
add_span_attribute("clean_model", job.clean_instrumental_model)
if job.backing_vocals_models:
job_log.info(f" Using backing vocals models: {job.backing_vocals_models}")
if job.other_stems_models:
job_log.info(f" Using other stems models: {job.other_stems_models}")
audio_processor = create_audio_processor(
temp_dir,
clean_instrumental_model=job.clean_instrumental_model,
backing_vocals_models=job.backing_vocals_models,
other_stems_models=job.other_stems_models
)
# Format artist-title for file naming (matches CLI behavior)
artist_title = f"{job.artist} - {job.title}"
# Run audio separation (calls Modal API internally)
# This returns a dict with paths to all separated stems
with job_span("modal-separation", job_id) as sep_span:
sep_start = time.time()
job_log.info("Starting audio separation (this may take 5-10 minutes)...")
job_log.info(" Stage 1: Clean instrumental separation (MDX models)")
job_log.info(" Stage 2: Backing vocals separation (Demucs model)")
add_span_event("separation_started")
logger.info(f"[job:{job_id}] Calling Modal API for audio separation")
with metrics.time_external_api("modal", job_id):
separation_result = audio_processor.process_audio_separation(
audio_file=audio_path,
artist_title=artist_title,
track_output_dir=temp_dir
)
sep_duration = time.time() - sep_start
sep_span.set_attribute("duration_seconds", sep_duration)
sep_span.set_attribute("stem_count", len(separation_result))
add_span_event("separation_completed", {"duration_seconds": sep_duration})
job_log.info("Audio separation complete!")
job_log.info(f" Generated {len(separation_result)} stem files")
logger.info(f"[job:{job_id}] Audio separation complete, organizing results")
# Update progress using state_data (don't change status during parallel processing)
job_manager.update_state_data(job_id, 'audio_progress', {
'stage': 'audio_complete',
'progress': 45,
'message': 'Audio separation complete, uploading stems'
})
# Upload all stems to GCS
with job_span("upload-stems", job_id) as upload_span:
await upload_separation_results(job_id, separation_result, storage, job_manager)
upload_span.set_attribute("stem_count", len(separation_result))
logger.info(f"[job:{job_id}] All stems uploaded successfully")
# Mark audio processing complete
# This will check if lyrics are also complete and transition to next stage if so
job_manager.mark_audio_complete(job_id)
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
logger.info(f"[job:{job_id}] WORKER_END worker=audio status=success duration={duration:.1f}s")
return True
except Exception as e:
logger.error(f"Job {job_id}: Audio separation failed: {e}", exc_info=True)
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=audio status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Audio separation failed: {str(e)}",
@@ -344,7 +377,7 @@ async def process_audio_separation(job_id: str) -> bool:
# Cleanup temporary directory
if os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
logger.debug(f"Cleaned up temp directory: {temp_dir}")
logger.debug(f"[job:{job_id}] Cleaned up temp directory: {temp_dir}")
async def download_audio(
+207 -162
View File
@@ -12,12 +12,18 @@ Handles the lyrics processing track of parallel processing:
Re-uses:
- karaoke_gen.lyrics_processor.LyricsProcessor for lyrics fetching and orchestration
- lyrics_transcriber library (submodule) for transcription and correction
Observability:
- All operations wrapped in tracing spans for Cloud Trace visibility
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
- Worker start/end timing logged with WORKER_START/WORKER_END markers
"""
import asyncio
import logging
import os
import shutil
import tempfile
import time
import json
from typing import Optional, Dict, Any
from pathlib import Path
@@ -28,6 +34,8 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.workers.style_helper import load_style_config
from backend.services.tracing import job_span, add_span_event, add_span_attribute
from backend.services.metrics import metrics
# Import from karaoke_gen package
from karaoke_gen.lyrics_processor import LyricsProcessor
@@ -108,13 +116,15 @@ async def process_lyrics_transcription(job_id: str) -> bool:
Returns:
True if successful, False otherwise
"""
start_time = time.time()
job_manager = JobManager()
storage = StorageService()
# Create job logger for remote debugging FIRST
job_log = create_job_logger(job_id, "lyrics")
# Log immediately to verify logging is working
# Log with structured markers for easy Cloud Logging queries
logger.info(f"[job:{job_id}] WORKER_START worker=lyrics")
job_log.info("=== LYRICS WORKER STARTED ===")
job_log.info(f"Job ID: {job_id}")
@@ -127,166 +137,188 @@ async def process_lyrics_transcription(job_id: str) -> bool:
temp_dir = None
try:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
# This ensures logs from third-party libraries (lyrics_transcriber) are only captured
# by this job's handler, not handlers from other concurrent jobs
with job_logging_context(job_id):
job = job_manager.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
job_log.error(f"Job {job_id} not found in Firestore!")
return False
# Create temporary working directory
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_lyrics_{job_id}_")
job_log.info(f"Created temp directory: {temp_dir}")
job_log.info(f"Starting lyrics transcription for {job.artist} - {job.title}")
job_log.info(f"Log capture enabled for: {', '.join(LYRICS_WORKER_LOGGERS)}")
logger.info(f"Starting lyrics transcription for job {job_id}")
# Log environment configuration
job_log.info("Checking API configuration...")
if os.environ.get("GENIUS_API_TOKEN"):
job_log.info(" Genius API: configured")
else:
job_log.warning(" Genius API: NOT configured")
if os.environ.get("SPOTIFY_COOKIE_SP_DC"):
job_log.info(" Spotify: configured")
else:
job_log.warning(" Spotify: NOT configured")
if os.environ.get("RAPIDAPI_KEY"):
job_log.info(" Musixmatch (RapidAPI): configured")
else:
job_log.warning(" Musixmatch (RapidAPI): NOT configured")
# Ensure required environment variables are set
if not os.environ.get("AUDIOSHAKE_API_TOKEN"):
job_log.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
logger.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
# Download audio file from GCS (waits for audio worker if URL job)
job_log.info("Downloading audio file from GCS...")
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager)
if not audio_path:
raise Exception("Failed to download audio file")
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
# Update progress using state_data (don't change status during parallel processing)
# The status is managed at a higher level - workers just track their progress
job_manager.update_state_data(job_id, 'lyrics_progress', {
'stage': 'transcribing',
'progress': 10,
'message': 'Starting lyrics transcription via AudioShake'
})
# Load style configuration (downloads assets from GCS if available)
# This is needed for max_line_length and other style settings in LyricsTranscriber
job_log.info("Loading style configuration for lyrics processing...")
style_config = await load_style_config(job, storage, temp_dir)
style_params_json_path = style_config.get_style_params_path() if style_config.has_custom_styles() else None
if style_params_json_path:
job_log.info(f"Using custom style params: {style_params_json_path}")
# Log the contents of the style params for debugging
try:
with open(style_params_json_path, 'r') as f:
style_content = json.load(f)
job_log.info(f"Style params sections: {list(style_content.keys())}")
if 'karaoke' in style_content:
karaoke_style = style_content['karaoke']
job_log.info(f" karaoke.background_image: {karaoke_style.get('background_image', 'NOT SET')}")
job_log.info(f" karaoke.font_path: {karaoke_style.get('font_path', 'NOT SET')}")
except Exception as e:
job_log.warning(f"Could not read style params for logging: {e}")
else:
job_log.info("No custom style params found, using defaults")
# Download user-provided lyrics file if available
lyrics_file_path = None
if hasattr(job, 'lyrics_file_gcs_path') and job.lyrics_file_gcs_path:
lyrics_file_path = os.path.join(temp_dir, "user_lyrics.txt")
job_log.info(f"Downloading user-provided lyrics file: {job.lyrics_file_gcs_path}")
storage.download_file(job.lyrics_file_gcs_path, lyrics_file_path)
job_log.info(f"User lyrics file downloaded to: {lyrics_file_path}")
# Get lyrics configuration from job
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
if subtitle_offset != 0:
job_log.info(f"Subtitle offset: {subtitle_offset}ms")
# Create LyricsProcessor instance (reuses karaoke_gen code)
job_log.info("Creating LyricsProcessor instance...")
job_log.info(f" style_params_json: {style_params_json_path}")
job_log.info(f" lyrics_file: {lyrics_file_path}")
job_log.info(f" subtitle_offset_ms: {subtitle_offset}")
lyrics_processor = create_lyrics_processor(
style_params_json=style_params_json_path,
lyrics_file=lyrics_file_path,
subtitle_offset_ms=subtitle_offset
)
# Run transcription + correction
# This will:
# 1. Fetch reference lyrics from Genius/Spotify/Musixmatch/LRCLib
# 2. Transcribe audio via AudioShake API
# 3. Run automatic correction with lyrics_transcriber
# 4. Generate corrections JSON (no interactive review)
# 5. Skip video generation (render_video=False)
# Use lyrics_artist/lyrics_title overrides if provided, else fall back to job artist/title
lyrics_search_artist = getattr(job, 'lyrics_artist', None) or job.artist
lyrics_search_title = getattr(job, 'lyrics_title', None) or job.title
job_log.info("Starting LyricsTranscriber processing...")
job_log.info(f" Artist: {job.artist}")
job_log.info(f" Title: {job.title}")
if lyrics_search_artist != job.artist:
job_log.info(f" Lyrics search artist override: {lyrics_search_artist}")
if lyrics_search_title != job.title:
job_log.info(f" Lyrics search title override: {lyrics_search_title}")
logger.info(f"Job {job_id}: Calling lyrics_processor.transcribe_lyrics()")
# Run transcription in thread pool to avoid blocking the event loop
# (transcribe_lyrics is synchronous and takes 1-2 minutes)
# Note: We pass the original artist/title for file naming, but the LyricsProcessor
# will use lyrics_artist/lyrics_title for the search (set via constructor args)
result = await asyncio.to_thread(
lyrics_processor.transcribe_lyrics,
input_audio_wav=audio_path,
artist=job.artist, # Original artist for file naming
title=job.title, # Original title for file naming
track_output_dir=temp_dir,
lyrics_artist=lyrics_search_artist, # Override for lyrics search
lyrics_title=lyrics_search_title # Override for lyrics search
)
job_log.info("Transcription processing complete")
logger.info(f"Job {job_id}: Transcription complete, uploading results")
# Upload lyrics results to GCS
job_log.info("Uploading lyrics results to GCS...")
await upload_lyrics_results(job_id, temp_dir, result, storage, job_manager, job_log)
job_log.info("All lyrics data uploaded successfully")
logger.info(f"Job {job_id}: All lyrics data uploaded successfully")
# Update progress using state_data (don't change status during parallel processing)
job_manager.update_state_data(job_id, 'lyrics_progress', {
'stage': 'lyrics_complete',
'progress': 45,
'message': 'Lyrics transcription complete'
})
# Mark lyrics processing complete
# This will check if audio is also complete and transition to next stage if so
job_log.info("Lyrics worker complete, checking if audio is also done...")
job_manager.mark_lyrics_complete(job_id)
return True
# Wrap entire worker in a tracing span
with job_span("lyrics-worker", job_id) as root_span:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
# This ensures logs from third-party libraries (lyrics_transcriber) are only captured
# by this job's handler, not handlers from other concurrent jobs
with job_logging_context(job_id):
job = job_manager.get_job(job_id)
if not job:
logger.error(f"[job:{job_id}] Job not found in Firestore")
job_log.error(f"Job {job_id} not found in Firestore!")
return False
add_span_attribute("artist", job.artist)
add_span_attribute("title", job.title)
# Create temporary working directory
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_lyrics_{job_id}_")
job_log.info(f"Created temp directory: {temp_dir}")
job_log.info(f"Starting lyrics transcription for {job.artist} - {job.title}")
job_log.info(f"Log capture enabled for: {', '.join(LYRICS_WORKER_LOGGERS)}")
logger.info(f"[job:{job_id}] Starting lyrics transcription for {job.artist} - {job.title}")
# Log environment configuration
with job_span("check-api-config", job_id):
job_log.info("Checking API configuration...")
apis_configured = []
if os.environ.get("GENIUS_API_TOKEN"):
job_log.info(" Genius API: configured")
apis_configured.append("genius")
else:
job_log.warning(" Genius API: NOT configured")
if os.environ.get("SPOTIFY_COOKIE_SP_DC"):
job_log.info(" Spotify: configured")
apis_configured.append("spotify")
else:
job_log.warning(" Spotify: NOT configured")
if os.environ.get("RAPIDAPI_KEY"):
job_log.info(" Musixmatch (RapidAPI): configured")
apis_configured.append("musixmatch")
else:
job_log.warning(" Musixmatch (RapidAPI): NOT configured")
add_span_attribute("lyrics_apis", ",".join(apis_configured))
# Ensure required environment variables are set
if not os.environ.get("AUDIOSHAKE_API_TOKEN"):
job_log.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
logger.warning(f"[job:{job_id}] AUDIOSHAKE_API_TOKEN not set - transcription may fail")
# Download audio file from GCS (waits for audio worker if URL job)
with job_span("download-audio", job_id) as download_span:
job_log.info("Downloading audio file from GCS...")
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager)
if not audio_path:
raise Exception("Failed to download audio file")
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
download_span.set_attribute("audio_file", os.path.basename(audio_path))
# Update progress using state_data (don't change status during parallel processing)
# The status is managed at a higher level - workers just track their progress
job_manager.update_state_data(job_id, 'lyrics_progress', {
'stage': 'transcribing',
'progress': 10,
'message': 'Starting lyrics transcription via AudioShake'
})
# Load style configuration (downloads assets from GCS if available)
# This is needed for max_line_length and other style settings in LyricsTranscriber
with job_span("load-style-config", job_id):
job_log.info("Loading style configuration for lyrics processing...")
style_config = await load_style_config(job, storage, temp_dir)
style_params_json_path = style_config.get_style_params_path() if style_config.has_custom_styles() else None
if style_params_json_path:
job_log.info(f"Using custom style params: {style_params_json_path}")
# Log the contents of the style params for debugging
try:
with open(style_params_json_path, 'r') as f:
style_content = json.load(f)
job_log.info(f"Style params sections: {list(style_content.keys())}")
if 'karaoke' in style_content:
karaoke_style = style_content['karaoke']
job_log.info(f" karaoke.background_image: {karaoke_style.get('background_image', 'NOT SET')}")
job_log.info(f" karaoke.font_path: {karaoke_style.get('font_path', 'NOT SET')}")
except Exception as e:
job_log.warning(f"Could not read style params for logging: {e}")
else:
job_log.info("No custom style params found, using defaults")
# Download user-provided lyrics file if available
lyrics_file_path = None
if hasattr(job, 'lyrics_file_gcs_path') and job.lyrics_file_gcs_path:
with job_span("download-user-lyrics", job_id):
lyrics_file_path = os.path.join(temp_dir, "user_lyrics.txt")
job_log.info(f"Downloading user-provided lyrics file: {job.lyrics_file_gcs_path}")
storage.download_file(job.lyrics_file_gcs_path, lyrics_file_path)
job_log.info(f"User lyrics file downloaded to: {lyrics_file_path}")
# Get lyrics configuration from job
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
if subtitle_offset != 0:
job_log.info(f"Subtitle offset: {subtitle_offset}ms")
add_span_attribute("subtitle_offset_ms", subtitle_offset)
# Create LyricsProcessor instance (reuses karaoke_gen code)
job_log.info("Creating LyricsProcessor instance...")
job_log.info(f" style_params_json: {style_params_json_path}")
job_log.info(f" lyrics_file: {lyrics_file_path}")
job_log.info(f" subtitle_offset_ms: {subtitle_offset}")
lyrics_processor = create_lyrics_processor(
style_params_json=style_params_json_path,
lyrics_file=lyrics_file_path,
subtitle_offset_ms=subtitle_offset
)
# Use lyrics_artist/lyrics_title overrides if provided, else fall back to job artist/title
lyrics_search_artist = getattr(job, 'lyrics_artist', None) or job.artist
lyrics_search_title = getattr(job, 'lyrics_title', None) or job.title
job_log.info("Starting LyricsTranscriber processing...")
job_log.info(f" Artist: {job.artist}")
job_log.info(f" Title: {job.title}")
if lyrics_search_artist != job.artist:
job_log.info(f" Lyrics search artist override: {lyrics_search_artist}")
if lyrics_search_title != job.title:
job_log.info(f" Lyrics search title override: {lyrics_search_title}")
logger.info(f"[job:{job_id}] Calling lyrics_processor.transcribe_lyrics()")
# Run transcription + correction
with job_span("audioshake-transcription", job_id) as trans_span:
trans_start = time.time()
add_span_event("transcription_started")
# Run transcription in thread pool to avoid blocking the event loop
# (transcribe_lyrics is synchronous and takes 1-2 minutes)
# Note: transcribe_lyrics internally calls AudioShake and lyrics APIs
with metrics.time_external_api("audioshake", job_id):
result = await asyncio.to_thread(
lyrics_processor.transcribe_lyrics,
input_audio_wav=audio_path,
artist=job.artist, # Original artist for file naming
title=job.title, # Original title for file naming
track_output_dir=temp_dir,
lyrics_artist=lyrics_search_artist, # Override for lyrics search
lyrics_title=lyrics_search_title # Override for lyrics search
)
trans_duration = time.time() - trans_start
trans_span.set_attribute("duration_seconds", trans_duration)
add_span_event("transcription_completed", {"duration_seconds": trans_duration})
job_log.info("Transcription processing complete")
logger.info(f"[job:{job_id}] Transcription complete, uploading results")
# Upload lyrics results to GCS
with job_span("upload-lyrics-results", job_id):
job_log.info("Uploading lyrics results to GCS...")
await upload_lyrics_results(job_id, temp_dir, result, storage, job_manager, job_log)
job_log.info("All lyrics data uploaded successfully")
logger.info(f"[job:{job_id}] All lyrics data uploaded successfully")
# Update progress using state_data (don't change status during parallel processing)
job_manager.update_state_data(job_id, 'lyrics_progress', {
'stage': 'lyrics_complete',
'progress': 45,
'message': 'Lyrics transcription complete'
})
# Mark lyrics processing complete
# This will check if audio is also complete and transition to next stage if so
job_log.info("Lyrics worker complete, checking if audio is also done...")
job_manager.mark_lyrics_complete(job_id)
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
logger.info(f"[job:{job_id}] WORKER_END worker=lyrics status=success duration={duration:.1f}s")
return True
except Exception as e:
duration = time.time() - start_time
job_log.error(f"Lyrics transcription failed: {str(e)}", exc_info=True)
logger.error(f"Job {job_id}: Lyrics transcription failed: {e}", exc_info=True)
logger.error(f"[job:{job_id}] WORKER_END worker=lyrics status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Lyrics transcription failed: {str(e)}",
@@ -305,7 +337,7 @@ async def process_lyrics_transcription(job_id: str) -> bool:
# Cleanup temporary directory (only if it was created)
if temp_dir and os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
logger.debug(f"Cleaned up temp directory: {temp_dir}")
logger.debug(f"[job:{job_id}] Cleaned up temp directory: {temp_dir}")
async def download_audio(
@@ -447,11 +479,24 @@ async def upload_lyrics_results(
corrections_data = json.load(f)
# Store metadata in state_data
job_manager.update_state_data(job_id, 'lyrics_metadata', {
# Include countdown padding info for instrumental synchronization
lyrics_metadata = {
'segment_count': len(corrections_data.get('corrected_segments', [])),
'has_corrections': True,
'ready_for_review': True
})
'ready_for_review': True,
}
# Add countdown padding info from transcription result
# This is needed by video_worker to pad instrumentals for sync
if transcription_result.get("countdown_padding_added"):
lyrics_metadata['has_countdown_padding'] = True
lyrics_metadata['countdown_padding_seconds'] = transcription_result.get("countdown_padding_seconds", 3.0)
logger.info(f"Job {job_id}: Countdown padding detected: {lyrics_metadata['countdown_padding_seconds']}s")
else:
lyrics_metadata['has_countdown_padding'] = False
lyrics_metadata['countdown_padding_seconds'] = 0.0
job_manager.update_state_data(job_id, 'lyrics_metadata', lyrics_metadata)
except Exception as e:
logger.warning(f"Job {job_id}: Could not parse corrections JSON: {e}")
else:
+40 -21
View File
@@ -14,10 +14,16 @@ This worker:
Key insight: We use OutputGenerator from lyrics_transcriber library
WITHOUT using its blocking ReviewServer. This allows async operation
in Cloud Run.
Observability:
- All operations wrapped in tracing spans for Cloud Trace visibility
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
- Worker start/end timing logged with WORKER_START/WORKER_END markers
"""
import logging
import os
import tempfile
import time
import json
from typing import Optional, Dict, Any
from dataclasses import dataclass
@@ -27,6 +33,7 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.config import get_settings
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.services.tracing import job_span, add_span_event, add_span_attribute
# Import from lyrics_transcriber (submodule)
from lyrics_transcriber.output.generator import OutputGenerator
@@ -62,12 +69,16 @@ async def process_render_video(job_id: str) -> bool:
Returns:
True if successful, False otherwise
"""
start_time = time.time()
job_manager = JobManager()
storage = StorageService()
settings = get_settings()
# Create job logger for remote debugging FIRST
job_log = create_job_logger(job_id, "render_video")
# Log with structured markers for easy Cloud Logging queries
logger.info(f"[job:{job_id}] WORKER_START worker=render-video")
job_log.info("=== RENDER VIDEO WORKER STARTED ===")
job_log.info(f"Job ID: {job_id}")
@@ -77,30 +88,32 @@ async def process_render_video(job_id: str) -> bool:
job = job_manager.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
logger.error(f"[job:{job_id}] Job not found in Firestore")
job_log.error(f"Job {job_id} not found in Firestore!")
return False
job_log.info(f"Starting video render for {job.artist} - {job.title}")
logger.info(f"Job {job_id}: Starting video render (post-review)")
logger.info(f"[job:{job_id}] Starting video render (post-review)")
try:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
# This ensures logs from third-party libraries (lyrics_transcriber.output) are only
# captured by this job's handler, not handlers from other concurrent jobs
with job_logging_context(job_id):
# Transition to RENDERING_VIDEO
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.RENDERING_VIDEO,
progress=75,
message="Rendering karaoke video with corrected lyrics"
)
with tempfile.TemporaryDirectory() as temp_dir:
job_log.info(f"Created temp directory: {temp_dir}")
# 1. Download corrected corrections data
corrections_path = os.path.join(temp_dir, "corrections.json")
# Wrap entire worker in a tracing span
with job_span("render-video-worker", job_id, {"artist": job.artist, "title": job.title}) as root_span:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
# This ensures logs from third-party libraries (lyrics_transcriber.output) are only
# captured by this job's handler, not handlers from other concurrent jobs
with job_logging_context(job_id):
# Transition to RENDERING_VIDEO
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.RENDERING_VIDEO,
progress=75,
message="Rendering karaoke video with corrected lyrics"
)
with tempfile.TemporaryDirectory() as temp_dir:
job_log.info(f"Created temp directory: {temp_dir}")
# 1. Download corrected corrections data
corrections_path = os.path.join(temp_dir, "corrections.json")
# Try updated corrections first (from human review), fall back to original
corrections_gcs_updated = f"jobs/{job_id}/lyrics/corrections_updated.json"
@@ -280,7 +293,10 @@ async def process_render_video(job_id: str) -> bool:
message="Prep phase complete - download outputs to continue locally"
)
job_log.info("=== RENDER VIDEO WORKER COMPLETE (PREP ONLY) ===")
logger.info(f"Job {job_id}: Prep-only video render complete")
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("prep_only", True)
logger.info(f"[job:{job_id}] WORKER_END worker=render-video status=success duration={duration:.1f}s prep_only=true")
else:
# Normal mode: proceed to instrumental selection
job_manager.transition_to_state(
@@ -290,12 +306,15 @@ async def process_render_video(job_id: str) -> bool:
message="Video rendered - select your instrumental"
)
job_log.info("=== RENDER VIDEO WORKER COMPLETE ===")
logger.info(f"Job {job_id}: Video render complete, awaiting instrumental selection")
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
logger.info(f"[job:{job_id}] WORKER_END worker=render-video status=success duration={duration:.1f}s")
return True
except Exception as e:
duration = time.time() - start_time
job_log.error(f"Video render failed: {e}")
logger.error(f"Job {job_id}: Video render failed: {e}", exc_info=True)
logger.error(f"[job:{job_id}] WORKER_END worker=render-video status=error duration={duration:.1f}s error={e}")
job_manager.fail_job(job_id, f"Video render failed: {str(e)}")
return False
+112 -102
View File
@@ -16,17 +16,16 @@ processing complete (via mark_audio_complete/mark_lyrics_complete coordination).
Integrates with karaoke_gen.video_generator.VideoGenerator.
SOLID Principles:
- Single Responsibility: Only generates title/end screens
- Open/Closed: Extensible for different screen styles
- Liskov Substitution: Can swap different video generators
- Interface Segregation: Focused interface for screen generation
- Dependency Inversion: Depends on abstractions (VideoGenerator interface)
Observability:
- All operations wrapped in tracing spans for Cloud Trace visibility
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
- Worker start/end timing logged with WORKER_START/WORKER_END markers
"""
import logging
import os
import shutil
import tempfile
import time
from typing import Optional, Dict, Any
from pathlib import Path
@@ -36,6 +35,7 @@ from backend.services.storage_service import StorageService
from backend.config import get_settings
from backend.workers.style_helper import load_style_config, StyleConfig
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.services.tracing import job_span, add_span_event, add_span_attribute
# Import from karaoke_gen package
from karaoke_gen.video_generator import VideoGenerator
@@ -64,6 +64,7 @@ async def generate_screens(job_id: str) -> bool:
Returns:
True if successful, False otherwise
"""
start_time = time.time()
job_manager = JobManager()
storage = StorageService()
settings = get_settings()
@@ -71,118 +72,127 @@ async def generate_screens(job_id: str) -> bool:
# Create job logger for remote debugging
job_log = create_job_logger(job_id, "screens")
# Log with structured markers for easy Cloud Logging queries
logger.info(f"[job:{job_id}] WORKER_START worker=screens")
# Set up log capture for VideoGenerator and style_helper
log_handler = setup_job_logging(job_id, "screens", *SCREENS_WORKER_LOGGERS)
job = job_manager.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
logger.error(f"[job:{job_id}] Job not found")
return False
# Validate both audio and lyrics are complete
if not _validate_prerequisites(job):
logger.error(f"Job {job_id}: Prerequisites not met for screen generation")
logger.error(f"[job:{job_id}] Prerequisites not met for screen generation")
return False
# Create temporary working directory
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_screens_{job_id}_")
try:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
# This ensures logs from third-party libraries (karaoke_gen.video_generator) are only
# captured by this job's handler, not handlers from other concurrent jobs
with job_logging_context(job_id):
job_log.info(f"Starting screen generation for {job.artist} - {job.title}")
logger.info(f"Starting screen generation for job {job_id}")
# Transition to GENERATING_SCREENS state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.GENERATING_SCREENS,
progress=50,
message="Generating title and end screens"
)
# Log style assets info
style_assets = getattr(job, 'style_assets', {}) or {}
job_log.info(f"Style assets from job: {list(style_assets.keys()) if style_assets else 'None'}")
if style_assets:
for key, path in style_assets.items():
job_log.info(f" {key}: {path}")
# Load style configuration (downloads assets from GCS if available)
job_log.info("Loading style configuration from GCS...")
style_config = await load_style_config(job, storage, temp_dir)
if style_config.has_custom_styles():
job_log.info("Using CUSTOM style configuration")
logger.info(f"Job {job_id}: Using custom style configuration")
else:
job_log.warning("Using DEFAULT style configuration (no custom styles found)")
logger.info(f"Job {job_id}: Using default style configuration")
# Initialize video generator
video_generator = _create_video_generator(temp_dir)
# Generate title screen with style config
job_log.info("Generating title screen...")
title_screen_path = await _generate_title_screen(
job_id=job_id,
job=job,
video_generator=video_generator,
style_config=style_config,
temp_dir=temp_dir,
job_log=job_log
)
if not title_screen_path:
raise Exception("Title screen generation failed")
job_log.info(f"Title screen generated: {title_screen_path}")
# Generate end screen with style config
job_log.info("Generating end screen...")
end_screen_path = await _generate_end_screen(
job_id=job_id,
job=job,
video_generator=video_generator,
style_config=style_config,
temp_dir=temp_dir,
job_log=job_log
)
if not end_screen_path:
raise Exception("End screen generation failed")
job_log.info(f"End screen generated: {end_screen_path}")
# Upload screens to GCS
await _upload_screens(
job_id=job_id,
job_manager=job_manager,
storage=storage,
title_screen_path=title_screen_path,
end_screen_path=end_screen_path
)
# Apply countdown padding if needed
# (This is handled automatically by checking lyrics metadata)
await _apply_countdown_padding_if_needed(job_id, job_manager, job)
# Transition to AWAITING_REVIEW
# Human must review lyrics before video can be rendered
logger.info(f"Job {job_id}: Screens generated, awaiting lyrics review")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.AWAITING_REVIEW,
progress=55,
message="Ready for lyrics review. Please review and correct lyrics."
)
# TODO: Send notification to user that selection is needed
# await notify_user(job_id, "instrumental_selection_ready")
return True
# Wrap entire worker in a tracing span
with job_span("screens-worker", job_id, {"artist": job.artist, "title": job.title}) as root_span:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
with job_logging_context(job_id):
job_log.info(f"Starting screen generation for {job.artist} - {job.title}")
logger.info(f"[job:{job_id}] Starting screen generation for {job.artist} - {job.title}")
# Transition to GENERATING_SCREENS state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.GENERATING_SCREENS,
progress=50,
message="Generating title and end screens"
)
# Log style assets info
style_assets = getattr(job, 'style_assets', {}) or {}
job_log.info(f"Style assets from job: {list(style_assets.keys()) if style_assets else 'None'}")
if style_assets:
for key, path in style_assets.items():
job_log.info(f" {key}: {path}")
# Load style configuration (downloads assets from GCS if available)
with job_span("load-style-config", job_id):
job_log.info("Loading style configuration from GCS...")
style_config = await load_style_config(job, storage, temp_dir)
if style_config.has_custom_styles():
job_log.info("Using CUSTOM style configuration")
logger.info(f"[job:{job_id}] Using custom style configuration")
add_span_attribute("style_type", "custom")
else:
job_log.warning("Using DEFAULT style configuration (no custom styles found)")
logger.info(f"[job:{job_id}] Using default style configuration")
add_span_attribute("style_type", "default")
# Initialize video generator
video_generator = _create_video_generator(temp_dir)
# Generate title screen with style config
with job_span("generate-title-screen", job_id):
job_log.info("Generating title screen...")
title_screen_path = await _generate_title_screen(
job_id=job_id,
job=job,
video_generator=video_generator,
style_config=style_config,
temp_dir=temp_dir,
job_log=job_log
)
if not title_screen_path:
raise Exception("Title screen generation failed")
job_log.info(f"Title screen generated: {title_screen_path}")
# Generate end screen with style config
with job_span("generate-end-screen", job_id):
job_log.info("Generating end screen...")
end_screen_path = await _generate_end_screen(
job_id=job_id,
job=job,
video_generator=video_generator,
style_config=style_config,
temp_dir=temp_dir,
job_log=job_log
)
if not end_screen_path:
raise Exception("End screen generation failed")
job_log.info(f"End screen generated: {end_screen_path}")
# Upload screens to GCS
with job_span("upload-screens", job_id):
await _upload_screens(
job_id=job_id,
job_manager=job_manager,
storage=storage,
title_screen_path=title_screen_path,
end_screen_path=end_screen_path
)
# Apply countdown padding if needed
await _apply_countdown_padding_if_needed(job_id, job_manager, job)
# Transition to AWAITING_REVIEW
# Human must review lyrics before video can be rendered
logger.info(f"[job:{job_id}] Screens generated, awaiting lyrics review")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.AWAITING_REVIEW,
progress=55,
message="Ready for lyrics review. Please review and correct lyrics."
)
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
logger.info(f"[job:{job_id}] WORKER_END worker=screens status=success duration={duration:.1f}s")
return True
except Exception as e:
logger.error(f"Job {job_id}: Screen generation failed: {e}", exc_info=True)
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=screens status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Screen generation failed: {str(e)}",
@@ -201,7 +211,7 @@ async def generate_screens(job_id: str) -> bool:
# Cleanup temporary directory
if os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
logger.debug(f"Cleaned up temp directory: {temp_dir}")
logger.debug(f"[job:{job_id}] Cleaned up temp directory: {temp_dir}")
def _validate_prerequisites(job) -> bool:
+67 -27
View File
@@ -15,11 +15,17 @@ The worker's job is simply to:
4. Upload results back to GCS
This reuses 100% of the existing karaoke_gen functionality.
Observability:
- All operations wrapped in tracing spans for Cloud Trace visibility
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
- Worker start/end timing logged with WORKER_START/WORKER_END markers
"""
import logging
import os
import shutil
import tempfile
import time
import json
from typing import Optional, Dict, Any
from pathlib import Path
@@ -32,6 +38,7 @@ from backend.services.youtube_service import get_youtube_service
from backend.config import get_settings
from backend.workers.style_helper import load_style_config
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.services.tracing import job_span, add_span_event, add_span_attribute
# Import from karaoke_gen package - reuse existing implementation
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
@@ -61,6 +68,7 @@ async def generate_video(job_id: str) -> bool:
Returns:
True if successful, False otherwise
"""
start_time = time.time()
job_manager = JobManager()
storage = StorageService()
settings = get_settings()
@@ -68,17 +76,20 @@ async def generate_video(job_id: str) -> bool:
# Create job logger for remote debugging
job_log = create_job_logger(job_id, "video")
# Log with structured markers for easy Cloud Logging queries
logger.info(f"[job:{job_id}] WORKER_START worker=video")
# Set up log capture for KaraokeFinalise
log_handler = setup_job_logging(job_id, "video", *VIDEO_WORKER_LOGGERS)
job = job_manager.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
logger.error(f"[job:{job_id}] Job not found")
return False
# Validate prerequisites
if not _validate_prerequisites(job):
logger.error(f"Job {job_id}: Prerequisites not met for video generation")
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
return False
# Create temporary working directory
@@ -105,12 +116,14 @@ async def generate_video(job_id: str) -> bool:
job_log.warning("YouTube credentials not available - upload will be skipped")
try:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
# This ensures logs from third-party libraries (karaoke_gen.karaoke_finalise) are only
# captured by this job's handler, not handlers from other concurrent jobs
with job_logging_context(job_id):
job_log.info(f"Starting video finalization for {job.artist} - {job.title}")
logger.info(f"Starting video generation for job {job_id}")
# Wrap entire worker in a tracing span
with job_span("video-worker", job_id, {"artist": job.artist, "title": job.title}) as root_span:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
# This ensures logs from third-party libraries (karaoke_gen.karaoke_finalise) are only
# captured by this job's handler, not handlers from other concurrent jobs
with job_logging_context(job_id):
job_log.info(f"Starting video finalization for {job.artist} - {job.title}")
logger.info(f"[job:{job_id}] Starting video generation for {job.artist} - {job.title}")
# Transition to GENERATING_VIDEO state
job_manager.transition_to_state(
@@ -122,9 +135,10 @@ async def generate_video(job_id: str) -> bool:
# Download and set up files in the format KaraokeFinalise expects
base_name = f"{job.artist} - {job.title}"
job_log.info("Downloading files from GCS (this may take a few minutes for large files)...")
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
job_log.info("All files downloaded successfully")
with job_span("download-files", job_id):
job_log.info("Downloading files from GCS (this may take a few minutes for large files)...")
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
job_log.info("All files downloaded successfully")
# Load style config for CDG styles
job_log.info("Loading CDG style configuration...")
@@ -162,6 +176,14 @@ async def generate_video(job_id: str) -> bool:
message="Encoding videos (15-20 min)"
)
# Get countdown padding info from lyrics metadata
# This ensures instrumental is padded to match vocals if countdown was added
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
job_log.info(f"Countdown padding detected: {countdown_padding_seconds}s - instrumental will be padded if needed")
# Log finalization parameters
job_log.info("Creating KaraokeFinalise with parameters:")
job_log.info(f" enable_cdg: {getattr(job, 'enable_cdg', False)}")
@@ -169,6 +191,7 @@ async def generate_video(job_id: str) -> bool:
job_log.info(f" brand_prefix: {getattr(job, 'brand_prefix', None)}")
job_log.info(f" discord_webhook: {'configured' if getattr(job, 'discord_webhook_url', None) else 'not configured'}")
job_log.info(f" instrumental source: {instrumental_source}")
job_log.info(f" countdown_padding_seconds: {countdown_padding_seconds}")
if existing_instrumental_path:
job_log.info(f" using user-provided instrumental (selection was: {instrumental_selection})")
@@ -209,6 +232,8 @@ async def generate_video(job_id: str) -> bool:
non_interactive=True,
server_side_mode=True,
selected_instrumental_file=instrumental_file,
# Audio synchronization - ensure instrumental matches vocal padding
countdown_padding_seconds=countdown_padding_seconds,
)
# Call process() - this does ALL the work:
@@ -216,25 +241,34 @@ async def generate_video(job_id: str) -> bool:
# - Generates CDG/TXT packages if enabled
# - Posts Discord notification if configured
# - Handles brand code generation
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
logger.info(f"Job {job_id}: Starting KaraokeFinalise.process()")
result = finalise.process()
with job_span("karaoke-finalise", job_id) as finalise_span:
finalise_start = time.time()
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
logger.info(f"[job:{job_id}] Starting KaraokeFinalise.process()")
add_span_event("finalise_started")
result = finalise.process()
finalise_duration = time.time() - finalise_start
finalise_span.set_attribute("duration_seconds", finalise_duration)
add_span_event("finalise_completed", {"duration_seconds": finalise_duration})
job_log.info("KaraokeFinalise.process() complete!")
if result.get('brand_code'):
job_log.info(f"Brand code: {result.get('brand_code')}")
logger.info(f"Job {job_id}: KaraokeFinalise.process() complete")
logger.info(f"Job {job_id}: Brand code: {result.get('brand_code')}")
logger.info(f"[job:{job_id}] KaraokeFinalise.process() complete in {finalise_duration:.1f}s")
logger.info(f"[job:{job_id}] Brand code: {result.get('brand_code')}")
# Native API distribution uploads (used by remote CLI instead of rclone)
await _handle_native_distribution(
job_id=job_id,
job=job,
job_log=job_log,
job_manager=job_manager,
temp_dir=temp_dir,
result=result,
)
with job_span("distribution", job_id):
await _handle_native_distribution(
job_id=job_id,
job=job,
job_log=job_log,
job_manager=job_manager,
temp_dir=temp_dir,
result=result,
)
# Upload generated files to GCS
job_manager.transition_to_state(
@@ -244,10 +278,11 @@ async def generate_video(job_id: str) -> bool:
message="Uploading final files"
)
await _upload_results(job_id, job_manager, storage, temp_dir, result)
with job_span("upload-results", job_id):
await _upload_results(job_id, job_manager, storage, temp_dir, result)
# Mark job as complete
logger.info(f"Job {job_id}: Video generation complete")
logger.info(f"[job:{job_id}] Video generation complete")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
@@ -264,10 +299,15 @@ async def generate_video(job_id: str) -> bool:
}
})
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.get('brand_code', ''))
logger.info(f"[job:{job_id}] WORKER_END worker=video status=success duration={duration:.1f}s")
return True
except Exception as e:
logger.error(f"Job {job_id}: Video generation failed: {e}", exc_info=True)
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=video status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Video generation failed: {str(e)}",
+9 -8
View File
@@ -1,22 +1,23 @@
# Cloud Build configuration for base image
#
# This builds the base image with all Python dependencies pre-installed.
# Run this manually when poetry.lock changes:
# Run this manually when poetry.lock OR Dockerfile.base changes:
# gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
#
# The base image is used by the main Dockerfile for fast app builds.
# The poetry.lock hash is stored as a label for change detection.
# A combined hash of poetry.lock + Dockerfile.base is stored as a label for change detection.
steps:
# Calculate poetry.lock hash for change detection
# Calculate combined hash for change detection (poetry.lock + Dockerfile.base)
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-c'
- |
POETRY_LOCK_HASH=$$(sha256sum poetry.lock | cut -d' ' -f1)
echo "Poetry.lock hash: $$POETRY_LOCK_HASH"
echo "$$POETRY_LOCK_HASH" > /workspace/POETRY_LOCK_HASH
# Hash both files to detect changes in either
CONTENT_HASH=$$(cat poetry.lock backend/Dockerfile.base | sha256sum | cut -d' ' -f1)
echo "Combined content hash (poetry.lock + Dockerfile.base): $$CONTENT_HASH"
echo "$$CONTENT_HASH" > /workspace/CONTENT_HASH
# Build base image with all dependencies
- name: 'gcr.io/cloud-builders/docker'
@@ -24,11 +25,11 @@ steps:
args:
- '-c'
- |
POETRY_LOCK_HASH=$$(cat /workspace/POETRY_LOCK_HASH)
CONTENT_HASH=$$(cat /workspace/CONTENT_HASH)
docker build \
--build-arg BUILD_DATE=$BUILD_ID \
--build-arg BUILD_ID=$BUILD_ID \
--build-arg POETRY_LOCK_HASH=$$POETRY_LOCK_HASH \
--build-arg BASE_CONTENT_HASH=$$CONTENT_HASH \
-t us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend-base:latest \
-t us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend-base:$BUILD_ID \
-f backend/Dockerfile.base \
+1 -1
View File
@@ -76,7 +76,7 @@ steps:
--max-instances 50 \
--min-instances 0 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest"
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
images:
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
+36
View File
@@ -0,0 +1,36 @@
# Codecov Configuration
# Docs: https://docs.codecov.com/docs/codecovyml-reference
coverage:
# Coverage value precision (2 decimal places)
precision: 2
round: down
# Green range starts at 70% (our target)
range: "70...100"
status:
# Project-level coverage status check
project:
default:
target: 70% # Pass if coverage >= 70%
threshold: 2% # Allow 2% drop without failing
if_ci_failed: error
# Patch coverage (new/changed lines only)
patch:
default:
target: 70% # New code should also meet 70%
threshold: 5% # More lenient for patches
only_pulls: true # Only run on PRs, not on main branch pushes
# Disable PR comments entirely since we're satisfied with 70%+ coverage
# Status checks will still appear on the PR
comment: false
# Alternative: Uncomment below to only comment when coverage changes
# comment:
# layout: "diff, flags, files"
# behavior: default
# require_changes: true # Only comment if coverage actually changes
# require_base: true # Require base report to exist
@@ -681,6 +681,10 @@ The karaoke-gen system supports multiple interfaces to the same core functionali
| `dropbox-oauth-credentials` | ✅ | Dropbox API OAuth tokens |
| `gdrive-service-account` | ✅ | Google Drive service account |
| `discord-webhook-url` | ✅ | Default Discord webhook |
| `red-api-key` | ⏳ | RED tracker API key (flacfetch audio search) |
| `red-api-url` | ⏳ | RED tracker base URL (flacfetch audio search) |
| `ops-api-key` | ⏳ | OPS tracker API key (flacfetch audio search) |
| `ops-api-url` | ⏳ | OPS tracker base URL (flacfetch audio search) |
### Default Distribution Settings
+608
View File
@@ -0,0 +1,608 @@
# Observability & Tracing Plan for Karaoke Generator
**Created:** 2025-12-22
**Status:** Planning
**Priority:** High
## Executive Summary
The karaoke-gen backend has **partial observability infrastructure** but **no rich tracing implemented**. While the code includes OpenTelemetry setup and tracing utilities, actual job processing is minimally instrumented. This means debugging job failures requires manually querying Cloud Logging with grep-style searches rather than clicking through a trace waterfall.
### Current State: 🟡 Partial
| Capability | Status | Notes |
|------------|--------|-------|
| **Logs exported to Cloud Logging** | ✅ Working | Standard Python logs visible |
| **OpenTelemetry SDK installed** | ✅ Working | Dependencies in pyproject.toml |
| **Cloud Trace exporter configured** | 🟡 Broken | 403 permission errors (fixed in infra today) |
| **FastAPI auto-instrumentation** | ✅ Working | HTTP requests create spans |
| **HTTPX instrumentation** | ✅ Working | Outgoing HTTP calls traced |
| **Job ID correlation in logs** | ❌ Missing | Can't filter logs by job ID |
| **Worker tracing spans** | ❌ Missing | No spans in audio/lyrics workers |
| **Custom metrics** | ❌ Missing | No Cloud Monitoring metrics |
| **Dashboard** | ❌ Missing | No pre-built observability view |
| **Alerting** | ❌ Missing | No alerts for job failures |
### What You Can Do Today
```bash
# Query Cloud Logging for a specific job (manual grep)
gcloud logging read 'resource.type="cloud_run_revision" AND textPayload:"JOB_ID"' \
--project=nomadkaraoke --limit=100
# View traces (once permission is fixed)
# Go to: https://console.cloud.google.com/traces/list?project=nomadkaraoke
```
### What You Should Be Able To Do
1. Go to a dashboard, enter job ID → see entire job lifecycle
2. Click trace → see waterfall of audio worker, lyrics worker, Cloud Tasks
3. See metrics: jobs/hour, success rate, p95 latency by stage
4. Get alerted when job failure rate exceeds threshold
---
## Current Implementation Analysis
### 1. Tracing Infrastructure (Exists, Partially Working)
**File: `backend/services/tracing.py`**
```python
# Good: SDK is set up, exporter is configured
setup_tracing(service_name="karaoke-backend", service_version=VERSION)
# Good: Utility functions exist
@traced("my-function") # Decorator for automatic spans
with create_span("operation", {"job_id": job_id}): # Context manager
add_span_attribute("key", "value") # Add to current span
```
**Problem:** These utilities are barely used! Only `backend/api/routes/review.py` actually creates spans for business logic. The critical workers (`audio_worker.py`, `lyrics_worker.py`, etc.) have zero tracing.
### 2. Logging Infrastructure (Exists, Not Correlated)
**File: `backend/workers/worker_logging.py`**
```python
# Good: Logs go to Firestore for CLI streaming
class JobLogHandler(logging.Handler):
"""Forwards logs to Firestore job document"""
# Good: Context vars prevent log mixing in concurrent jobs
with job_logging_context(job_id):
# logs isolated per job
```
**Problem:** Logs go to Firestore for CLI display, but Cloud Logging has no job_id label. You can't filter Cloud Logging by job ID.
### 3. Permission Issues (Fixed Today)
The Cloud Trace exporter was failing with 403 errors because the Cloud Run service account lacked `roles/cloudtrace.agent`. This was just fixed in `infrastructure/__main__.py`:
```python
# NEW: Added today
cloud_trace_iam = gcp.projects.IAMMember(
"karaoke-backend-cloudtrace-agent",
role="roles/cloudtrace.agent",
...
)
```
---
## Gap Analysis: What's Missing
### Gap 1: No Trace Context Propagation to Workers
When Cloud Tasks triggers a worker (audio, lyrics), a new trace is started. The parent trace from the initial API call is lost.
```
Current:
POST /uploads-complete → Trace A
↓ (Cloud Task)
POST /internal/workers/audio → NEW Trace B (no link to A!)
```
**Should be:**
```
POST /uploads-complete → Trace A
↓ (Cloud Task - propagates trace context)
POST /internal/workers/audio → Trace A, Span B (child of A)
```
### Gap 2: No Job ID on Traces
Even when traces exist, they don't have `job_id` as a searchable attribute. You can't search Cloud Trace for "all spans for job 8f03f0ac".
### Gap 3: No Custom Spans in Workers
The workers just log text. There are no spans for:
- Audio download duration
- Modal API call duration
- GCS upload duration
- Lyrics transcription duration
- Each processing stage
### Gap 4: No Structured Logging with Job Context
Cloud Logging receives unstructured text logs. Should be structured JSON with `job_id`, `worker`, `stage` fields for filtering.
### Gap 5: No Metrics
No custom Cloud Monitoring metrics for:
- Jobs started/completed/failed per hour
- Processing time by stage (p50, p95, p99)
- Queue depth
- Error rates
### Gap 6: No Dashboard
No Cloud Console dashboard showing:
- Active jobs
- Success/failure rates
- Latency distribution
- Worker health
### Gap 7: No Alerting
No alerts for:
- Job stuck in status > X minutes
- Error rate spike
- Worker health degradation
---
## Implementation Plan
### Phase 1: Fix Core Tracing (2-3 hours) ⭐ High Priority
**Goal:** Every job should have a single trace that spans its entire lifecycle.
#### 1.1 Propagate Trace Context Through Cloud Tasks
**File: `backend/services/worker_service.py`**
```python
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry import trace
def _enqueue_cloud_task(self, queue: str, endpoint: str, payload: Dict, ...):
# Get current trace context
propagator = TraceContextTextMapPropagator()
carrier = {}
propagator.inject(carrier) # Injects traceparent header
# Add trace context to task headers
headers = {
"Authorization": f"Bearer {admin_token}",
"Content-Type": "application/json",
**carrier, # Add traceparent, tracestate
}
task = {
"http_request": {
"http_method": "POST",
"url": url,
"headers": headers, # Now includes trace context
"body": base64.b64encode(json.dumps(payload).encode()).decode(),
},
...
}
```
**File: `backend/api/routes/internal.py`**
```python
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry import trace
@router.post("/workers/audio")
async def trigger_audio_worker(request: Request, ...):
# Extract trace context from headers
propagator = TraceContextTextMapPropagator()
context = propagator.extract(request.headers)
# Create span as child of propagated context
with tracer.start_as_current_span(
"audio-worker",
context=context,
attributes={"job_id": payload.job_id}
):
await process_audio_separation(payload.job_id)
```
#### 1.2 Add Job ID to All Spans
Create a helper that adds job_id as a span attribute:
```python
# backend/services/tracing.py
def job_span(name: str, job_id: str, **extra_attributes):
"""Create a span with job_id attribute for easy filtering."""
return create_span(name, {
"job_id": job_id,
"service.operation": name,
**extra_attributes
})
```
Usage in workers:
```python
# backend/workers/audio_worker.py
async def process_audio_separation(job_id: str):
with job_span("audio-worker", job_id) as span:
span.set_attribute("worker", "audio")
with job_span("download-audio", job_id):
# download logic
pass
with job_span("modal-separation", job_id):
# Modal API call
pass
with job_span("upload-stems", job_id):
# GCS upload
pass
```
### Phase 2: Instrument All Workers (3-4 hours) ⭐ High Priority
Add spans to every worker following this pattern:
#### Audio Worker (`audio_worker.py`)
```python
async def process_audio_separation(job_id: str):
with job_span("audio-worker", job_id) as root:
# Stage 1: Download/Get Audio
with job_span("get-audio-file", job_id) as span:
if job.audio_url:
span.set_attribute("source", "url")
with job_span("download-from-url", job_id):
audio_path = await download_from_url(...)
else:
span.set_attribute("source", "gcs")
with job_span("download-from-gcs", job_id):
audio_path = storage.download_file(...)
# Stage 2: Audio Separation
with job_span("audio-separation", job_id) as span:
with job_span("clean-instrumental", job_id):
# First separation pass
pass
with job_span("backing-vocals", job_id):
# Second separation pass
pass
# Stage 3: Upload Results
with job_span("upload-stems", job_id) as span:
span.set_attribute("stem_count", len(stems))
for stem_name, stem_path in stems.items():
with job_span("upload-stem", job_id, stem_name=stem_name):
storage.upload_file(...)
```
#### Lyrics Worker (`lyrics_worker.py`)
```python
async def process_lyrics_transcription(job_id: str):
with job_span("lyrics-worker", job_id) as root:
with job_span("get-vocals", job_id):
# Download vocals from GCS
pass
with job_span("audioshake-transcription", job_id):
# AudioShake API call
pass
with job_span("fetch-genius-lyrics", job_id) as span:
span.set_attribute("artist", artist)
span.set_attribute("title", title)
# Genius API call
pass
with job_span("auto-correction", job_id):
# LyricsTranscriber correction
pass
with job_span("upload-corrections", job_id):
# Upload corrections.json
pass
```
### Phase 3: Structured Logging with Job Context (1-2 hours)
Replace text logs with structured JSON that includes job context.
**File: `backend/main.py`**
```python
import json
import logging
class StructuredFormatter(logging.Formatter):
"""JSON formatter with trace correlation."""
def format(self, record):
from backend.services.tracing import get_current_trace_id, get_current_span_id
log_entry = {
"timestamp": self.formatTime(record),
"severity": record.levelname,
"message": record.getMessage(),
"logger": record.name,
# Trace correlation
"logging.googleapis.com/trace": f"projects/nomadkaraoke/traces/{get_current_trace_id()}" if get_current_trace_id() else None,
"logging.googleapis.com/spanId": get_current_span_id(),
# Custom fields (if available in log record)
"job_id": getattr(record, 'job_id', None),
"worker": getattr(record, 'worker', None),
}
return json.dumps({k: v for k, v in log_entry.items() if v is not None})
# Configure in Cloud Run
if os.environ.get("K_SERVICE"):
handler = logging.StreamHandler()
handler.setFormatter(StructuredFormatter())
logging.root.handlers = [handler]
```
Now logs in Cloud Logging can be filtered:
```
jsonPayload.job_id="8f03f0ac"
```
### Phase 4: Custom Metrics (2-3 hours) 🟡 Medium Priority
Add Cloud Monitoring metrics for key indicators.
**File: `backend/services/metrics.py`**
```python
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.cloud_monitoring import CloudMonitoringMetricsExporter
def setup_metrics():
exporter = CloudMonitoringMetricsExporter(project_id="nomadkaraoke")
provider = MeterProvider(metric_readers=[...])
metrics.set_meter_provider(provider)
meter = metrics.get_meter("karaoke-backend")
# Counter: Jobs by status
jobs_counter = meter.create_counter(
"jobs_total",
description="Total jobs by status",
unit="1"
)
# Histogram: Processing duration by stage
duration_histogram = meter.create_histogram(
"job_stage_duration_seconds",
description="Duration of job processing stages",
unit="s"
)
return {
"jobs": jobs_counter,
"duration": duration_histogram,
}
# Usage
metrics["jobs"].add(1, {"status": "completed", "worker": "audio"})
metrics["duration"].record(45.2, {"stage": "audio-separation", "job_id": job_id})
```
**Metrics to track:**
| Metric | Type | Labels |
|--------|------|--------|
| `jobs_total` | Counter | status, source (upload/url/search) |
| `job_stage_duration_seconds` | Histogram | stage, worker |
| `worker_invocations_total` | Counter | worker, success/failure |
| `gcs_operations_total` | Counter | operation (upload/download), bucket |
| `external_api_duration_seconds` | Histogram | api (modal/audioshake/genius) |
### Phase 5: Dashboard & Alerting (2-3 hours) 🟡 Medium Priority
#### Create Cloud Monitoring Dashboard
**File: `infrastructure/monitoring/dashboard.json`**
```json
{
"displayName": "Karaoke Backend Overview",
"dashboardFilters": [],
"mosaicLayout": {
"columns": 12,
"tiles": [
{
"title": "Active Jobs by Status",
"scorecard": {
"timeSeriesQuery": {
"prometheusQuery": "sum by (status) (jobs_total)"
}
}
},
{
"title": "Job Processing Time (p95)",
"xyChart": {
"dataSets": [{
"timeSeriesQuery": {
"prometheusQuery": "histogram_quantile(0.95, rate(job_stage_duration_seconds_bucket[5m]))"
}
}]
}
},
{
"title": "Error Rate",
"xyChart": {
"dataSets": [{
"timeSeriesQuery": {
"prometheusQuery": "rate(worker_invocations_total{success=\"false\"}[5m]) / rate(worker_invocations_total[5m])"
}
}]
}
},
{
"title": "Cloud Tasks Queue Depth",
"xyChart": {
"dataSets": [{
"timeSeriesQuery": {
"prometheusQuery": "cloud_tasks_queue_depth"
}
}]
}
}
]
}
}
```
#### Create Alerts
**File: `infrastructure/__main__.py`** (Pulumi)
```python
# Alert: Job failure rate > 10%
job_failure_alert = gcp.monitoring.AlertPolicy(
"job-failure-rate-alert",
display_name="High Job Failure Rate",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Failure rate > 10%",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='metric.type="custom.googleapis.com/karaoke/job_failure_rate"',
comparison="COMPARISON_GT",
threshold_value=0.1,
duration="300s",
),
),
],
notification_channels=[...], # Slack, email, etc.
)
# Alert: Job stuck in processing > 30 min
stuck_job_alert = gcp.monitoring.AlertPolicy(
"stuck-job-alert",
display_name="Job Stuck in Processing",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Job > 30 min in processing state",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='metric.type="custom.googleapis.com/karaoke/job_age_seconds" AND metric.labels.status!="complete"',
comparison="COMPARISON_GT",
threshold_value=1800,
duration="60s",
),
),
],
)
```
---
## Quick Wins (Do First)
These give the most debugging value with minimal effort:
### 1. Add Job ID to Worker Logs (30 min)
Simplest possible fix - add job_id to every log message:
```python
# backend/workers/audio_worker.py
logger.info(f"[job:{job_id}] Starting audio separation")
logger.info(f"[job:{job_id}] Downloaded audio file: {path}")
logger.info(f"[job:{job_id}] Modal separation complete")
```
Now you can grep:
```bash
gcloud logging read 'textPayload:"[job:8f03f0ac]"' --project=nomadkaraoke
```
### 2. Log Worker Start/End with Timing (30 min)
```python
import time
async def process_audio_separation(job_id: str):
start = time.time()
logger.info(f"[job:{job_id}] WORKER_START worker=audio")
try:
# ... processing ...
duration = time.time() - start
logger.info(f"[job:{job_id}] WORKER_END worker=audio status=success duration={duration:.1f}s")
except Exception as e:
duration = time.time() - start
logger.error(f"[job:{job_id}] WORKER_END worker=audio status=error duration={duration:.1f}s error={e}")
raise
```
Now you can easily find failed workers:
```bash
gcloud logging read 'textPayload:"WORKER_END" AND textPayload:"status=error"' --project=nomadkaraoke
```
### 3. Enable Cloud Trace Viewing (0 min - just fixed!)
With the IAM permission fix deployed today, Cloud Trace should start working. Visit:
https://console.cloud.google.com/traces/list?project=nomadkaraoke
---
## Summary: Implementation Priority
| Phase | Effort | Value | Priority |
|-------|--------|-------|----------|
| Quick Win #1: Job ID in logs | 30 min | High | ⭐ Do now |
| Quick Win #2: Worker timing | 30 min | High | ⭐ Do now |
| Phase 1: Trace propagation | 2-3 hrs | Very High | ⭐ Do soon |
| Phase 2: Instrument workers | 3-4 hrs | Very High | ⭐ Do soon |
| Phase 3: Structured logging | 1-2 hrs | Medium | 🟡 Nice to have |
| Phase 4: Custom metrics | 2-3 hrs | Medium | 🟡 Nice to have |
| Phase 5: Dashboard & alerts | 2-3 hrs | Medium | 🟡 Nice to have |
**Total for full observability: ~12-15 hours**
**Minimum viable observability (Quick Wins + Phase 1-2): ~5-6 hours**
---
## Appendix: GCP Observability Resources
- **Cloud Trace:** https://console.cloud.google.com/traces/list?project=nomadkaraoke
- **Cloud Logging:** https://console.cloud.google.com/logs?project=nomadkaraoke
- **Cloud Monitoring:** https://console.cloud.google.com/monitoring?project=nomadkaraoke
- **OpenTelemetry Python Docs:** https://opentelemetry.io/docs/instrumentation/python/
- **Cloud Trace Python:** https://cloud.google.com/trace/docs/setup/python-ot
### Useful Log Queries
```
# All logs for a specific job
resource.type="cloud_run_revision" AND textPayload:"JOB_ID"
# All worker starts
resource.type="cloud_run_revision" AND textPayload:"WORKER_START"
# All errors
resource.type="cloud_run_revision" AND severity>=ERROR
# Cloud Tasks failures
resource.type="cloud_tasks_queue"
# Trace-correlated logs (after structured logging)
jsonPayload.job_id="JOB_ID"
```
@@ -0,0 +1,143 @@
# Refactor: Unify Audio Fetcher Code
**Created:** 2025-12-20
**Completed:** 2025-12-20
**Priority:** MEDIUM (prevents future bugs like YouTubeProvider -> YoutubeProvider)
**Status:** ✅ Completed
## Problem
The codebase has **two separate implementations** of flacfetch integration:
1. `karaoke_gen/audio_fetcher.py` - Used by local CLI
2. `backend/services/audio_search_service.py` - Used by cloud backend
This violates DRY and caused a bug where the backend's import wasn't updated when flacfetch renamed `YouTubeProvider` to `YoutubeProvider`.
## Current State
### Duplicated Code
| Component | Local (`audio_fetcher.py`) | Backend (`audio_search_service.py`) |
|-----------|---------------------------|-------------------------------------|
| `_get_manager()` | ✅ 40 lines | ❌ 35 lines (copy) |
| Provider setup | ✅ RED, OPS, YouTube | ❌ Same, diverged |
| `search()` | ✅ Full implementation | ❌ Similar, diverged |
| `download()` | ✅ With downloaders | ⚠️ Different, simpler |
| `AudioSearchResult` | ✅ Dataclass | ❌ Duplicate, different fields |
| `AudioDownloadResult` | ✅ (`AudioFetchResult`) | ❌ Duplicate |
| Exceptions | ✅ 4 types | ❌ 3 types, different names |
### Drift Examples
- Backend missing `raw_result` field on `AudioSearchResult`
- Backend has `index` field, local doesn't
- Backend exceptions inherit from `AudioSearchError`, local from `AudioFetcherError`
- Backend missing `UserCancelledError`
- Backend missing downloader registration (TorrentDownloader, YoutubeDownloader)
## Proposed Solution
### Option 1: Backend imports from karaoke_gen (Recommended)
The backend already depends on `karaoke_gen` (installed via `pip install -e .` in Dockerfile).
```python
# backend/services/audio_search_service.py - REFACTORED
from karaoke_gen.audio_fetcher import (
FlacFetcher,
AudioSearchResult,
AudioFetchResult,
AudioFetcherError,
NoResultsError,
DownloadError,
)
class AudioSearchService:
"""
Thin wrapper around karaoke_gen.audio_fetcher.FlacFetcher.
Adds backend-specific functionality:
- Firestore serialization (to_dict/from_dict)
- GCS integration for downloads
- Job state management
"""
def __init__(
self,
red_api_key: Optional[str] = None,
red_api_url: Optional[str] = None,
ops_api_key: Optional[str] = None,
):
self._fetcher = FlacFetcher(
red_api_key=red_api_key,
red_api_url=red_api_url,
ops_api_key=ops_api_key,
)
def search(self, artist: str, title: str) -> List[AudioSearchResult]:
"""Delegate to FlacFetcher.search()."""
return self._fetcher.search(artist, title)
def select_best(self, results: List[AudioSearchResult]) -> int:
"""Delegate to FlacFetcher.select_best()."""
return self._fetcher.select_best(results)
def download(
self,
result: AudioSearchResult,
output_dir: str,
output_filename: Optional[str] = None,
) -> AudioFetchResult:
"""Delegate to FlacFetcher.download()."""
return self._fetcher.download(result, output_dir, output_filename)
```
### Changes Required to karaoke_gen.audio_fetcher
1. **Add `to_dict()` method to `AudioSearchResult`** - For Firestore serialization
2. **Add `from_dict()` classmethod** - For deserialization
3. **Ensure `select_best()` is a standalone method** - Currently buried in interactive flow
4. **Export all public classes from `__init__.py`**
### Changes Required to backend
1. **Remove duplicate dataclasses** - Use imports from karaoke_gen
2. **Remove duplicate `_get_manager()`** - Delegate to FlacFetcher
3. **Remove duplicate exception classes** - Use karaoke_gen exceptions
4. **Update tests** - Point to real classes, not mocks
## Benefits
1. **Single source of truth** - Bug fixes apply everywhere
2. **No more drift** - Dataclasses, exceptions stay in sync
3. **Better test coverage** - karaoke_gen has integration tests for flacfetch imports
4. **Less code to maintain** - Backend becomes thin wrapper
## Migration Steps
1. [x] Add `to_dict()` / `from_dict()` to `AudioSearchResult` in karaoke_gen
2. [x] Extract `select_best()` as standalone method in FlacFetcher
3. [x] Update karaoke_gen `__init__.py` exports (with lazy imports)
4. [x] Refactor backend to import from karaoke_gen
5. [x] Update backend tests
6. [x] Delete duplicate code from backend (~150 lines removed)
7. [x] Add integration test that catches import divergence
## Related Files
- `karaoke_gen/audio_fetcher.py` - Source of truth (keep)
- `backend/services/audio_search_service.py` - Refactor to thin wrapper
- `backend/api/routes/audio_search.py` - Update imports
- `backend/tests/test_audio_search.py` - Update tests
## Estimated Effort
- **Time:** 2-4 hours
- **Risk:** Low (well-defined changes, good test coverage)
- **Dependencies:** None
## Related Architecture Docs
- `docs/01-reference/ARCHITECTURE.md` - "LyricsTranscriber Integration" section shows similar pattern
- `docs/00-current-plan/SCALABLE-ARCHITECTURE-PLAN.md` - "Shared pipeline architecture" goal
+240
View File
@@ -0,0 +1,240 @@
# Backend Testing Gaps Analysis
**Created:** 2025-12-22
**Updated:** 2025-12-22
**Status:** Mostly Complete ✅
**Priority:** High
## Background
During end-to-end testing of the remote karaoke-gen CLI, multiple bugs were discovered:
### Bug #1: Missing `extra_info` Attribute (Fixed 2025-12-22)
```
Error: 'AudioSearchResult' object has no attribute 'extra_info'
```
This bug existed in production code but wasn't caught by automated tests because the remote download code path had no test coverage.
### Bug #2: Missing `gcs_path` Parameter in API Route (Fixed 2025-12-22)
```
Error handling audio selection: Error selecting audio: {'detail': "Download failed: [Errno 2] No such file or directory: '/var/lib/transmission-daemon/downloads/Avril Lavigne - Unwanted (Album, 2002, CD, 16bit, RED).flac'"}
```
The backend API route `_download_and_start_processing` in `audio_search.py` was not passing the `gcs_path` parameter to `audio_search_service.download()`. Without this parameter, the remote `flacfetch` VM would download the file locally but not upload it to GCS. The backend then tried to open the VM's local path (`/var/lib/transmission-daemon/downloads/...`) which doesn't exist on Cloud Run.
**Why this wasn't caught:**
- Service-level tests for `AudioSearchService.download()` existed and tested `gcs_path` parameter
- But the **API route caller** (`_download_and_start_processing`) that invokes this service was never tested
- Tests validated the service layer but not the route layer's usage of the service
## Root Cause Analysis
1. **Missing remote code path tests**: The download routing logic for `provider in ["RED", "OPS"]` was never exercised
2. **State not set up for remote scenario**: Tests never configured `_remote_search_id` and `_remote_client` together
3. **`FlacfetchClient` had zero tests**: The entire HTTP client was untested
4. **Tests mocked at wrong level**: Tests mocked `service._fetcher` completely, bypassing the routing logic
5. **API route orchestration untested**: The `_download_and_start_processing` function in `api/routes/audio_search.py` orchestrates multiple services but had no dedicated tests verifying correct parameter passing
## Services Testing Status
### ✅ Fully Tested
| Service | Test File | Test Count | Notes |
|---------|-----------|------------|-------|
| `credential_manager.py` | `test_credential_manager.py` | 27 | Good coverage |
| `job_manager.py` | `test_job_manager.py` | 15 | Good coverage |
| `flacfetch_client.py` | `test_flacfetch_client.py` | 26 | **Added 2025-12-22** |
| `audio_search_service.py` | `test_audio_search.py` | 43 | **+11 tests 2025-12-22** |
| `api/routes/audio_search.py` | `test_audio_search.py` | 4 | **API route tests added 2025-12-22** |
| `storage_service.py` | `test_storage_service.py` | 17 | **Added 2025-12-22** |
| `audio_analysis_service.py` | `test_audio_analysis_service.py` | 9 | **Added 2025-12-22** |
| `audio_editing_service.py` | `test_audio_editing_service.py` | 13 | **Added 2025-12-22** |
| `youtube_service.py` | `test_youtube_service.py` | 12 | **Added 2025-12-22** |
| `dropbox_service.py` | `test_dropbox_service.py` | 14 | **Added 2025-12-22** |
| `gdrive_service.py` | `test_gdrive_service.py` | 13 | **Added 2025-12-22** |
### ❌ No Dedicated Test File (Lower Priority)
| Service | Priority | Complexity | Notes |
|---------|----------|------------|-------|
| `rclone_service.py` | Low | Low | Wrapper around rclone CLI |
| `firestore_service.py` | Low | Medium | Firestore operations |
| `auth_service.py` | Low | Low | Auth utilities |
| `job_logging.py` | Low | Low | Logging utilities |
| `tracing.py` | Low | Low | OpenTelemetry setup |
| `worker_service.py` | Medium | High | Worker orchestration |
## Implementation Plan
### Phase 1: Core Audio Services (High Priority) ✅
- [x] `flacfetch_client.py` - Remote audio download client
- [x] `audio_search_service.py` - Remote download path tests
### Phase 2: Audio Processing Services (High Priority) ✅
- [x] `audio_analysis_service.py` - Waveform analysis, peak detection
- [x] `audio_editing_service.py` - Mute regions, audio manipulation
### Phase 3: Distribution Services (Medium Priority) ✅
- [x] `youtube_service.py` - YouTube credential management
- [x] `dropbox_service.py` - Dropbox upload API
- [x] `gdrive_service.py` - Google Drive upload API
- [x] `storage_service.py` - GCS operations
### Phase 4: Infrastructure Services (Low Priority)
- [ ] `worker_service.py` - Worker orchestration
- [ ] `firestore_service.py` - Firestore operations
- [ ] `rclone_service.py` - Rclone wrapper
- [ ] `auth_service.py` - Auth utilities
- [ ] `job_logging.py` - Logging
- [ ] `tracing.py` - Tracing
## Test Coverage Goals
- **Target:** 80%+ line coverage for high-priority services
- **Minimum:** Key code paths tested (happy path + error handling)
- **Integration:** Mock external APIs (YouTube, Dropbox, GCS, etc.)
## Testing Patterns to Follow
### 1. Mock External Services
```python
@patch("backend.services.youtube_service.build")
def test_upload_video(self, mock_build):
mock_service = Mock()
mock_build.return_value = mock_service
# ...
```
### 2. Test Error Handling
```python
def test_upload_handles_network_error(self):
with pytest.raises(ServiceError) as exc_info:
service.upload(...)
assert "network" in str(exc_info.value).lower()
```
### 3. Test State Transitions
```python
def test_download_routes_to_remote_for_torrent_providers(self):
service._remote_client = Mock()
service._remote_search_id = "search_123"
service._cached_results = [result_with_provider_red]
service.download(0, "/tmp")
service._download_remote.assert_called_once()
```
### 4. Verify Request/Response Format
```python
def test_search_sends_correct_payload(self):
# ...
call_args = mock_client.post.call_args
assert call_args.kwargs["json"] == {"artist": "ABBA", "title": "Waterloo"}
```
### 5. Test API Route Orchestration (CRITICAL)
**Key Lesson from Bug #2:** Service-level tests are not sufficient. API routes that orchestrate multiple services must also be tested to verify correct parameter passing.
```python
class TestAudioSearchApiRouteDownload:
"""Tests for the _download_and_start_processing function in audio_search.py."""
@pytest.mark.asyncio
async def test_remote_download_passes_gcs_path(self):
"""Verify that gcs_path is passed to audio_search_service.download for torrent providers."""
# Mock the audio_search_service.download to capture the call
mock_audio_search_service = Mock()
mock_audio_search_service.download.return_value = AudioDownloadResult(
filepath="gs://bucket/path.flac", artist="A", title="T", provider="RED", quality="FLAC"
)
result = await _download_and_start_processing(job_id="job1", ...)
# Assert gcs_path was passed
call_kwargs = mock_audio_search_service.download.call_args.kwargs
assert call_kwargs.get('gcs_path') is not None # CRITICAL check!
assert call_kwargs['gcs_path'].startswith('uploads/')
```
This pattern ensures that when service APIs change or new required parameters are added, API route tests will catch missing parameter passing.
## Files Changed
**Bug Fixes:**
- `backend/services/audio_search_service.py` - Removed `extra_info` attribute access (Bug #1)
- `backend/api/routes/audio_search.py` - Added `gcs_path` parameter to `audio_search_service.download()` call (Bug #2)
**Tests Added (2025-12-22):**
- `backend/tests/test_audio_search.py` - +11 tests for service + 4 tests for API route orchestration
- `backend/tests/test_flacfetch_client.py` - 26 new tests (new file)
- `backend/tests/test_storage_service.py` - 17 new tests (new file)
- `backend/tests/test_audio_analysis_service.py` - 9 new tests (new file)
- `backend/tests/test_audio_editing_service.py` - 13 new tests (new file)
- `backend/tests/test_youtube_service.py` - 12 new tests (new file)
- `backend/tests/test_dropbox_service.py` - 14 new tests (new file)
- `backend/tests/test_gdrive_service.py` - 13 new tests (new file)
**Total New Tests:** 119 tests added
**Version:** 0.74.3
## Running Tests
```bash
# All backend tests
cd backend && python -m pytest tests/ -v
# Specific service tests
python -m pytest tests/test_flacfetch_client.py -v
python -m pytest tests/test_audio_search.py -v
# With coverage
python -m pytest tests/ -v --cov=services --cov-report=term-missing
```
## Notes for Future Sessions
When continuing this work:
1. **Test API route orchestration, not just services**: Bug #2 showed that service tests alone don't catch parameter passing bugs at the route level
2. Distribution services need OAuth mocking - check existing patterns in `test_credential_manager.py`
3. Use `pytest.mark.asyncio` for async service methods
4. Follow existing test patterns in `test_flacfetch_client.py` for HTTP client mocking
5. **For remote operations**: Always verify that `gcs_path` parameters are passed when code paths involve remote VM operations
## Lessons Learned
### Testing Layers Matter
```
┌─────────────────────────────────────────────────────────────────┐
│ API Routes (api/routes/*.py) │
│ └─> These orchestrate services and MUST be tested separately │
│ to verify correct parameter passing │
├─────────────────────────────────────────────────────────────────┤
│ Services (services/*.py) │
│ └─> Business logic with proper unit tests │
├─────────────────────────────────────────────────────────────────┤
│ Clients (FlacfetchClient, etc.) │
│ └─> HTTP/external API interactions │
└─────────────────────────────────────────────────────────────────┘
```
Bug #2 was missed because:
- `FlacfetchClient` tests ✅ verified `gcs_path` is sent in HTTP requests
- `AudioSearchService` tests ✅ verified `gcs_path` is passed to client
- **API route tests ❌ didn't exist** - so nobody verified the route passes `gcs_path` to the service
Now we have `TestAudioSearchApiRouteDownload` to catch this class of bugs.
@@ -0,0 +1,837 @@
# Flacfetch HTTP API Service Implementation Plan
**Created:** 2025-12-20
**Updated:** 2025-12-22
**Status:** Complete ✅
**Priority:** Medium
**Estimated Effort:** 3-4 days (actual: ~3 days)
## Problem Statement
BitTorrent downloads don't work reliably in Cloud Run due to:
- No inbound peer connections (NAT/firewall)
- Ephemeral containers with no stable IP
- DHT peer discovery blocked
- Container timeout limits (max 60 min)
This prevents downloading lossless audio from private trackers (RED, OPS) via the remote backend.
## Solution Overview
Add an HTTP API mode to **flacfetch** (our audio fetching library) and deploy it on a dedicated GCP VM with:
- Full BitTorrent peer connectivity via Transmission daemon
- Static IP for tracker whitelist (seedbox-like)
- Indefinite seeding of downloaded torrents
- Automatic disk cleanup when space runs low
- YouTube downloads also handled (single service for all audio acquisition)
This approach provides clean separation of concerns:
- **flacfetch**: Handles ALL audio acquisition (torrents + YouTube)
- **karaoke-gen backend**: Orchestrates karaoke generation pipeline, calls flacfetch API
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ FLACFETCH SERVICE ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Cloud Run Backend Flacfetch VM (e2-small) │
│ ┌────────────────────┐ ┌─────────────────────────┐ │
│ │ │ │ │ │
│ │ karaoke-gen │ 1. Search │ flacfetch HTTP API │ │
│ │ AudioSearchService │ ────────────▶ │ POST /search │ │
│ │ │ │ │ │
│ │ │ 2. Results │ Providers: │ │
│ │ │ ◀──────────── │ • RED (FLAC) │ │
│ │ │ │ • OPS (FLAC) │ │
│ │ │ 3. Download │ • YouTube (fallback) │ │
│ │ │ ────────────▶ │ │ │
│ │ │ │ POST /download │ │
│ │ │ 4. Status │ GET /download/{id} │ │
│ │ │ ◀──────────── │ │ │
│ │ │ (polling) │ Transmission Daemon │ │
│ │ │ │ • Full peer connect │ │
│ │ │ 5. Complete │ • DHT enabled │ │
│ │ │ ◀──────────── │ • Indefinite seeding │ │
│ │ │ (file path) │ │ │
│ └────────────────────┘ └───────────┬─────────────┘ │
│ │ │ │
│ │ │ 6. Upload to GCS │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Google Cloud Storage │ │
│ │ │ │
│ │ uploads/{job_id}/audio/{filename}.flac │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
## Key Design Decisions
### 1. HTTP API in flacfetch (not separate service)
- **Why**: Clean separation of concerns, reusable by future projects, single source of truth
- **How**: Add `flacfetch/api/` module with FastAPI, new CLI mode `flacfetch serve`
### 2. Indefinite Seeding
- **Why**: VM is always-on anyway, helps maintain tracker ratio, acts like a seedbox
- **Current behavior**: Torrent removed after download (line 317-321 in torrent.py)
- **New behavior**: Torrent kept seeding, only removed by cleanup or explicit delete
### 3. Static IP
- **Why**: Required for tracker account IP whitelist, consistent for firewall rules
- **Cost**: ~$3/month additional
### 4. Disk Space Management
- **Strategy**: Auto-cleanup oldest torrents when free space < 5GB threshold
- **API**: Manual cleanup endpoints + automatic background cleanup
### 5. Instance Type: e2-small
- **Specs**: 0.5 vCPU, 2GB RAM, ~$12/month
- **Why**: Sufficient for I/O-bound workload, newer C3/N4 require 2+ vCPUs (overkill)
- **Disk**: 30GB SSD (20GB for torrents + headroom)
### 6. Authentication
- **Method**: Static API key in `X-API-Key` header
- **Firewall**: Permissive (allows future web UI), auth enforced at app level
## Implementation Phases
### Phase 1: Add HTTP API to flacfetch
**Location:** `./flacfetch/flacfetch/api/`
**New Directory Structure:**
```
flacfetch/
├── flacfetch/
│ ├── api/ # NEW
│ │ ├── __init__.py
│ │ ├── main.py # FastAPI app entry point
│ │ ├── auth.py # API key authentication
│ │ ├── models.py # Pydantic request/response models
│ │ ├── routes/
│ │ │ ├── __init__.py
│ │ │ ├── search.py # POST /search
│ │ │ ├── download.py # POST /download, GET /download/{id}/status
│ │ │ ├── torrents.py # GET /torrents, DELETE /torrents/{id}
│ │ │ └── health.py # GET /health, GET /disk-usage
│ │ └── services/
│ │ ├── __init__.py
│ │ ├── download_manager.py # Track active downloads
│ │ └── disk_manager.py # Disk cleanup logic
│ ├── core/
│ ├── downloaders/
│ │ └── torrent.py # MODIFIED: optional keep-seeding mode
│ ├── interface/
│ │ └── cli.py # MODIFIED: add 'serve' subcommand
│ └── providers/
└── pyproject.toml # MODIFIED: add fastapi, uvicorn deps
```
**API Endpoints:**
```yaml
# All endpoints require X-API-Key header
# Search for audio
POST /search
Request:
{ "artist": "ABBA", "title": "Waterloo" }
Response:
{
"search_id": "abc123",
"results": [
{
"index": 0,
"title": "Waterloo",
"artist": "ABBA",
"provider": "RED",
"quality": "FLAC 16bit CD",
"seeders": 45,
"size_bytes": 31457280,
"target_file": "02. Waterloo.flac",
...
},
...
]
}
# Start download
POST /download
Request:
{
"search_id": "abc123", # From previous search
"result_index": 0, # Which result to download
"output_filename": "ABBA - Waterloo", # Optional
"upload_to_gcs": true, # Upload to GCS when complete
"gcs_path": "uploads/job123/audio/" # Required if upload_to_gcs
}
Response:
{
"download_id": "dl_xyz789",
"status": "queued"
}
# Check download status
GET /download/{download_id}/status
Response:
{
"download_id": "dl_xyz789",
"status": "downloading", # queued, downloading, uploading, seeding, complete, failed
"progress": 45.2,
"peers": 3,
"download_speed_kbps": 1250.5,
"eta_seconds": 120,
"provider": "RED",
"title": "ABBA - Waterloo",
"output_path": null, # Set when complete (local path)
"gcs_path": null, # Set when uploaded to GCS
"error": null
}
# List all torrents (for management)
GET /torrents
Response:
{
"torrents": [
{
"id": 1,
"name": "ABBA - Waterloo [FLAC]",
"status": "seeding",
"progress": 100.0,
"size_bytes": 314572800,
"uploaded_bytes": 628145600, # Seeding stats
"ratio": 2.0,
"added_at": "2025-12-20T10:30:00Z"
},
...
],
"total_size_bytes": 5368709120,
"count": 15
}
# Delete a torrent (manual cleanup)
DELETE /torrents/{torrent_id}
Query params: ?delete_data=true # Also delete downloaded files
Response:
{ "status": "deleted" }
# Trigger cleanup (free disk space)
POST /torrents/cleanup
Request:
{
"strategy": "oldest", # oldest, largest, lowest_ratio
"target_free_gb": 10 # Keep removing until this much free
}
Response:
{
"removed_count": 3,
"freed_bytes": 1073741824,
"free_space_gb": 10.5
}
# Health check
GET /health
Response:
{
"status": "healthy",
"transmission": {
"available": true,
"version": "4.0.5",
"active_torrents": 5
},
"disk": {
"total_gb": 30,
"used_gb": 18,
"free_gb": 12
},
"providers": {
"red": true,
"ops": true,
"youtube": true
}
}
```
**Torrent Downloader Changes:**
```python
# flacfetch/downloaders/torrent.py
class TorrentDownloader(Downloader):
def __init__(self, ..., keep_seeding: bool = False):
"""
Args:
keep_seeding: If True, don't remove torrent after download.
Used in server mode for indefinite seeding.
"""
self.keep_seeding = keep_seeding
def download(self, release, output_path, output_filename=None):
# ... existing download logic ...
# After download complete:
if self.keep_seeding:
# Don't remove, just return the file path
logger.info(f"Download complete, keeping torrent for seeding: {torrent.name}")
else:
# Original behavior: stop and remove
self.client.stop_torrent(torrent.id)
self.client.remove_torrent(torrent.id, delete_data=True)
```
**CLI Changes:**
```bash
# New serve subcommand
flacfetch serve --port 8080 --host 0.0.0.0
# Or via environment
FLACFETCH_API_PORT=8080 flacfetch serve
```
### Phase 2: Infrastructure (Pulumi)
**Location:** `./infrastructure/__main__.py`
**New Resources:**
```python
# Static IP for flacfetch service (for tracker whitelist)
flacfetch_ip = gcp.compute.Address(
"flacfetch-ip",
name="flacfetch-static-ip",
region="us-central1",
)
# Service account for flacfetch VM
flacfetch_sa = gcp.serviceaccount.Account(
"flacfetch-sa",
account_id="flacfetch-service",
display_name="Flacfetch Service Account",
)
# GCS write permissions for flacfetch
gcp.storage.BucketIAMMember(
"flacfetch-gcs-writer",
bucket=bucket.name,
role="roles/storage.objectCreator",
member=pulumi.Output.concat("serviceAccount:", flacfetch_sa.email),
)
# Flacfetch VM
flacfetch_vm = gcp.compute.Instance(
"flacfetch-service",
machine_type="e2-small",
zone="us-central1-a",
boot_disk=gcp.compute.InstanceBootDiskArgs(
initialize_params=gcp.compute.InstanceBootDiskInitializeParamsArgs(
image="debian-cloud/debian-12",
size=30, # GB
type="pd-ssd",
),
),
network_interfaces=[gcp.compute.InstanceNetworkInterfaceArgs(
network="default",
access_configs=[gcp.compute.InstanceNetworkInterfaceAccessConfigArgs(
nat_ip=flacfetch_ip.address, # Static IP
)],
)],
service_account=gcp.compute.InstanceServiceAccountArgs(
email=flacfetch_sa.email,
scopes=["cloud-platform"],
),
metadata_startup_script=FLACFETCH_STARTUP_SCRIPT,
tags=["flacfetch-service"],
)
# Firewall for flacfetch
flacfetch_firewall = gcp.compute.Firewall(
"flacfetch-firewall",
network="default",
allows=[
# BitTorrent peer connections
gcp.compute.FirewallAllowArgs(protocol="tcp", ports=["51413"]),
gcp.compute.FirewallAllowArgs(protocol="udp", ports=["51413"]),
# Flacfetch HTTP API (auth required at app level)
gcp.compute.FirewallAllowArgs(protocol="tcp", ports=["8080"]),
],
source_ranges=["0.0.0.0/0"], # Permissive, auth at app level
target_tags=["flacfetch-service"],
)
# Store API key in Secret Manager
flacfetch_api_key = gcp.secretmanager.Secret(
"flacfetch-api-key",
secret_id="flacfetch-api-key",
replication=gcp.secretmanager.SecretReplicationArgs(
auto=gcp.secretmanager.SecretReplicationAutoArgs(),
),
)
# Export for use
pulumi.export("flacfetch_static_ip", flacfetch_ip.address)
pulumi.export("flacfetch_service_url", pulumi.Output.concat("http://", flacfetch_ip.address, ":8080"))
```
**Startup Script:**
```bash
#!/bin/bash
set -e
# Install dependencies
apt-get update
apt-get install -y python3-pip python3-venv transmission-daemon ffmpeg git
# Configure Transmission
systemctl stop transmission-daemon
cat > /etc/transmission-daemon/settings.json << 'EOF'
{
"download-dir": "/var/lib/transmission-daemon/downloads",
"incomplete-dir": "/var/lib/transmission-daemon/.incomplete",
"incomplete-dir-enabled": true,
"rpc-authentication-required": false,
"rpc-bind-address": "127.0.0.1",
"rpc-enabled": true,
"rpc-port": 9091,
"rpc-whitelist-enabled": false,
"peer-port": 51413,
"port-forwarding-enabled": false,
"speed-limit-down": 0,
"speed-limit-down-enabled": false,
"speed-limit-up": 0,
"speed-limit-up-enabled": false,
"ratio-limit-enabled": false,
"umask": 2
}
EOF
systemctl start transmission-daemon
# Install flacfetch
cd /opt
git clone https://github.com/nomadkaraoke/flacfetch.git
cd flacfetch
python3 -m venv venv
source venv/bin/activate
pip install -e ".[api]"
# Get secrets from metadata/Secret Manager
FLACFETCH_API_KEY=$(gcloud secrets versions access latest --secret=flacfetch-api-key)
RED_API_KEY=$(gcloud secrets versions access latest --secret=red-api-key)
RED_API_URL=$(gcloud secrets versions access latest --secret=red-api-url)
OPS_API_KEY=$(gcloud secrets versions access latest --secret=ops-api-key)
GCS_BUCKET=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/attributes/gcs-bucket" -H "Metadata-Flavor: Google")
# Create systemd service
cat > /etc/systemd/system/flacfetch.service << EOF
[Unit]
Description=Flacfetch HTTP API Service
After=network.target transmission-daemon.service
[Service]
Type=simple
User=root
WorkingDirectory=/opt/flacfetch
Environment="FLACFETCH_API_KEY=${FLACFETCH_API_KEY}"
Environment="RED_API_KEY=${RED_API_KEY}"
Environment="RED_API_URL=${RED_API_URL}"
Environment="OPS_API_KEY=${OPS_API_KEY}"
Environment="GCS_BUCKET=${GCS_BUCKET}"
Environment="FLACFETCH_KEEP_SEEDING=true"
Environment="FLACFETCH_MIN_FREE_GB=5"
ExecStart=/opt/flacfetch/venv/bin/flacfetch serve --host 0.0.0.0 --port 8080
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable flacfetch
systemctl start flacfetch
```
### Phase 3: Backend Integration
**New File:** `backend/services/flacfetch_client.py`
```python
"""
Client for the remote flacfetch HTTP API service.
Used when FLACFETCH_API_URL is configured, otherwise falls back to local flacfetch.
"""
import asyncio
import logging
from typing import Optional, List, Dict, Any
import httpx
logger = logging.getLogger(__name__)
class FlacfetchServiceError(Exception):
"""Error communicating with flacfetch service."""
pass
class FlacfetchClient:
"""Client for remote flacfetch HTTP API."""
def __init__(self, base_url: str, api_key: str, timeout: int = 30):
self.base_url = base_url.rstrip('/')
self.api_key = api_key
self.timeout = timeout
def _headers(self) -> Dict[str, str]:
return {"X-API-Key": self.api_key}
async def health_check(self) -> Dict[str, Any]:
"""Check if service is healthy."""
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/health",
headers=self._headers(),
timeout=10,
)
resp.raise_for_status()
return resp.json()
async def search(self, artist: str, title: str) -> Dict[str, Any]:
"""Search for audio."""
async with httpx.AsyncClient() as client:
resp = await client.post(
f"{self.base_url}/search",
headers=self._headers(),
json={"artist": artist, "title": title},
timeout=self.timeout,
)
resp.raise_for_status()
return resp.json()
async def download(
self,
search_id: str,
result_index: int,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""
Start download and return download_id.
If gcs_path is provided, file will be uploaded to GCS.
"""
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
return resp.json()["download_id"]
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
"""Get download status."""
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{self.base_url}/download/{download_id}/status",
headers=self._headers(),
timeout=10,
)
resp.raise_for_status()
return resp.json()
async def wait_for_download(
self,
download_id: str,
timeout: int = 600,
poll_interval: int = 2,
) -> Dict[str, Any]:
"""
Wait for download to complete.
Returns final status dict with output_path or gcs_path.
Raises FlacfetchServiceError on failure or timeout.
"""
elapsed = 0
while elapsed < timeout:
status = await self.get_download_status(download_id)
if status["status"] == "complete":
return status
elif status["status"] == "failed":
raise FlacfetchServiceError(f"Download failed: {status.get('error')}")
await asyncio.sleep(poll_interval)
elapsed += poll_interval
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
# Singleton
_client: Optional[FlacfetchClient] = None
def get_flacfetch_client() -> Optional[FlacfetchClient]:
"""Get flacfetch client if configured."""
global _client
if _client is None:
from backend.config import get_settings
settings = get_settings()
if settings.flacfetch_api_url and settings.flacfetch_api_key:
_client = FlacfetchClient(
base_url=settings.flacfetch_api_url,
api_key=settings.flacfetch_api_key,
)
return _client
```
**Modify:** `backend/services/audio_search_service.py` to use remote client when available.
**Modify:** `backend/config.py` to add new settings:
```python
flacfetch_api_url: Optional[str] = None # e.g., http://10.0.0.5:8080
flacfetch_api_key: Optional[str] = None
```
### Phase 4: Monitoring & Reliability
**Tasks:**
- [ ] Add Cloud Monitoring alert for VM CPU/memory
- [ ] Add alert for disk space < 5GB
- [ ] Add alert for flacfetch service down (health check fails)
- [ ] Configure Cloud Logging export from VM
- [ ] Add Grafana dashboard (optional)
**Automatic Disk Cleanup:**
The disk manager runs as a background task in the flacfetch API service:
```python
# Check every 5 minutes, cleanup if free space < threshold
async def background_cleanup_task():
while True:
await asyncio.sleep(300)
free_gb = get_free_space_gb()
if free_gb < MIN_FREE_GB:
cleanup_oldest_torrents(target_free_gb=MIN_FREE_GB + 5)
```
### Phase 5: Testing & Documentation
**Testing:**
- [ ] Unit tests for flacfetch API endpoints
- [ ] Unit tests for disk manager
- [ ] Integration test: search → download → GCS upload
- [ ] Mock tests for karaoke-gen backend client
**Documentation:**
- [ ] Update `docs/01-reference/ARCHITECTURE.md` with flacfetch service
- [ ] Add `flacfetch/README.md` HTTP API section
- [ ] Document deployment process
- [ ] Document environment variables
## Environment Variables
### Flacfetch Service (VM)
```bash
# API Configuration
FLACFETCH_API_KEY=<secret> # Required for API auth
FLACFETCH_API_PORT=8080 # Default: 8080
FLACFETCH_API_HOST=0.0.0.0 # Default: 0.0.0.0
# Tracker API Keys
RED_API_KEY=<key> # For RED provider
RED_API_URL=<url> # For RED provider base URL
OPS_API_KEY=<key> # For OPS provider
# GCS Upload (optional)
GCS_BUCKET=karaoke-uploads # For uploading completed downloads
GOOGLE_CLOUD_PROJECT=nomadkaraoke
# Transmission
TRANSMISSION_HOST=localhost # Default: localhost
TRANSMISSION_PORT=9091 # Default: 9091
# Behavior
FLACFETCH_KEEP_SEEDING=true # Keep torrents seeding after download
FLACFETCH_MIN_FREE_GB=5 # Minimum free disk space before cleanup
FLACFETCH_DOWNLOAD_DIR=/var/lib/transmission-daemon/downloads
```
### Cloud Run Backend
```bash
# New variables
FLACFETCH_API_URL=http://10.128.0.5:8080 # Internal IP of flacfetch VM
FLACFETCH_API_KEY=<same-secret>
```
## Security Considerations
1. **API Authentication**: Static API key in `X-API-Key` header
2. **Firewall**: Permissive for future web UI, auth enforced at application level
3. **GCS**: Service account with `storage.objectCreator` role only
4. **Transmission**: RPC bound to localhost only, no external access
5. **Static IP**: Can be whitelisted on tracker accounts
## Rollback Plan
If flacfetch service fails:
1. Set `FLACFETCH_API_URL=""` in Cloud Run (disables remote service)
2. Backend falls back to local flacfetch (YouTube only in Cloud Run)
3. Users see warning that lossless downloads may be slower/unavailable
## Cost Estimate
| Resource | Monthly Cost |
|----------|--------------|
| e2-small VM | ~$12 |
| Static IP | ~$3 |
| 30GB SSD | ~$5 |
| Network egress | ~$2-5 |
| **Total** | **~$22-25/month** |
## Success Criteria
- [x] Flacfetch HTTP API accepts search requests and returns results
- [x] Downloads complete successfully for RED/OPS sources
- [x] Downloaded files upload to GCS automatically
- [x] Torrents continue seeding after download
- [x] Disk cleanup triggers automatically when space is low
- [x] Remote CLI shows download progress from flacfetch service
- [x] Fallback to local flacfetch works when service unavailable
- [x] Static IP can be whitelisted on tracker accounts
## Timeline
| Phase | Duration | Dependencies |
|-------|----------|--------------|
| Phase 1: Flacfetch HTTP API | 1.5 days | None |
| Phase 2: Infrastructure | 0.5 day | Phase 1 |
| Phase 3: Backend Integration | 0.5 day | Phase 1, 2 |
| Phase 4: Monitoring | 0.5 day | Phase 2 |
| Phase 5: Testing & Docs | 0.5 day | Phase 1-3 |
| **Total** | **3.5 days** | |
## Implementation Status (2025-12-22)
### Completed Work
#### Phase 1: Flacfetch HTTP API ✅
- Full HTTP API implemented in `flacfetch/flacfetch/api/`
- Endpoints: `/search`, `/download`, `/download/{id}/status`, `/download/{id}/file`, `/torrents`, `/health`
- API key authentication via `X-API-Key` header
- Disk manager with automatic cleanup when space < 5GB
- Download manager with search caching and progress tracking
- Dynamic version reporting via `importlib.metadata`
- **Current Version:** 0.7.3
#### Phase 2: Infrastructure ✅
- Pulumi infrastructure in `infrastructure/__main__.py`
- GCP VM: `flacfetch-service` (e2-small, us-central1-a)
- Static IP: `104.198.214.26` (whitelisted with trackers)
- Firewall rules: TCP/UDP 51413 (BitTorrent), TCP 8080 (API)
- Secrets: `flacfetch-api-key`, `red-api-key`, `red-api-url`, `ops-api-key`, `ops-api-url`, `flacfetch-api-url`
- GitHub Actions deployer has `compute.admin` and `compute.osLogin` roles
#### Phase 3: Backend Integration ✅
- `FlacfetchClient` in `backend/services/flacfetch_client.py`
- `AudioSearchService` updated to use remote flacfetch when configured
- Cloud Run configured with `FLACFETCH_API_URL` and `FLACFETCH_API_KEY` from Secret Manager
- `--loop asyncio` added to uvicorn CMD (required for `nest_asyncio` compatibility)
- `nest_asyncio` applied only when async loop is running (not at import time)
#### Phase 4: Monitoring ✅
- Health endpoint reports Transmission status, disk usage, provider availability
- Version verification in deployment workflows
- Automatic deployment on release via GitHub Actions
#### Phase 5: CLI & Testing ✅
- `flacfetch-remote` CLI added for testing remote API
- Downloads file locally with descriptive filenames (e.g., `Artist - Title (Album, 2022, CD, 16bit, RED).flac`)
- 70%+ unit test coverage requirement enforced
- Integration tests for API routes
### Deployment Automation
#### Flacfetch VM (automatic)
- `.github/workflows/release-on-version-bump.yml` in flacfetch repo
- Triggers on version bump in `pyproject.toml`
- Runs tests → Creates GitHub release → SSHs to VM → Pulls latest → Restarts service
- Verifies deployed version matches release version
#### Karaoke-gen Backend (version-gated)
- `.github/workflows/ci.yml` deploy-backend job
- Only deploys if version tag doesn't exist (requires version bump)
- Verifies deployed version via `/api/health/detailed` endpoint
- Retries version check with backoff for Cloud Run routing delays
### Key Files Modified
**Flacfetch repo (`./flacfetch/`):**
- `flacfetch/api/` - New HTTP API module
- `flacfetch/interface/cli_remote.py` - New remote CLI
- `flacfetch/core/manager.py` - Improved sorting (artist match, release type, seeders)
- `flacfetch/downloaders/torrent.py` - Keep-seeding mode, file permissions fix
- `.github/workflows/release-on-version-bump.yml` - Auto-deploy to VM
**Karaoke-gen repo:**
- `backend/services/flacfetch_client.py` - Remote API client
- `backend/services/audio_search_service.py` - Remote integration with nest_asyncio
- `backend/Dockerfile` - `--loop asyncio` for uvicorn
- `infrastructure/__main__.py` - VM, firewall, secrets, WIF bindings
- `cloudbuild.yaml` - FLACFETCH_API_URL/KEY from Secret Manager
- `.github/workflows/ci.yml` - Version-gated deployments
### Testing the Integration
```bash
# Test flacfetch-remote CLI (requires env vars)
export FLACFETCH_API_URL=http://104.198.214.26:8080
export FLACFETCH_API_KEY=<from-secret-manager>
flacfetch-remote "Avril Lavigne" "Unwanted"
# Test backend health
curl -s https://api.nomadkaraoke.com/api/health/detailed | jq '.dependencies.flacfetch_remote'
# Test backend search
curl -s -X POST https://api.nomadkaraoke.com/api/audio-search/search \
-H "Content-Type: application/json" \
-d '{"artist": "Avril Lavigne", "title": "Unwanted"}' | jq '.results | length'
```
### Known Issues / Future Work
1. **Transmission removal from Cloud Run**: Transmission daemon and startup script removed from Docker images as torrent downloads are now handled by dedicated VM.
2. **URL field in search results**: Made optional in `AudioSearchResultResponse` since remote results don't have direct download URLs.
3. **Sorting improvements**: Results now prioritize:
- Lossless first
- Artist name match (exact match wins)
- Release type (Album > Single > EP > Compilation > Anthology > Remix)
- Seeders (more = higher)
- Quality (24bit > 16bit, FLAC > MP3)
4. **Future enhancements**:
- Cloud Monitoring alerts for disk/CPU/memory
- Grafana dashboard for torrent stats
- Web UI for manual torrent management
## References
- [Transmission RPC Spec](https://github.com/transmission/transmission/blob/main/docs/rpc-spec.md)
- [transmission-rpc Python library](https://github.com/Trim21/transmission-rpc)
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
- [flacfetch source code](./flacfetch/)
+481
View File
@@ -0,0 +1,481 @@
# Frontend Guide - Karaoke Generator Web UI
## Overview
The karaoke-gen frontend is a Next.js-based static web application deployed to GitHub Pages at **https://gen.nomadkaraoke.com**. It provides a web interface for submitting jobs to the backend API, monitoring job progress, and managing karaoke video generation.
## Architecture
### Deployment Topology
```
┌─────────────────────────────────────────────────────────┐
│ gen.nomadkaraoke.com (GitHub Pages) │
│ - Next.js static export │
│ - Pure client-side React application │
│ - No server-side rendering │
└────────────────┬────────────────────────────────────────┘
│
│ API calls (HTTPS)
▼
┌─────────────────────────────────────────────────────────┐
│ api.nomadkaraoke.com (Cloud Run) │
│ - FastAPI backend │
│ - Requires Bearer token authentication │
│ - Manages jobs, storage, workers │
└─────────────────────────────────────────────────────────┘
```
### Key Components
- **Frontend Domain**: `gen.nomadkaraoke.com` → GitHub Pages (static HTML/CSS/JS)
- **Backend API**: `api.nomadkaraoke.com` → Google Cloud Run
- **Framework**: Next.js 16.0.7 with Turbopack (static export mode)
- **UI Library**: React 19 + shadcn/ui components
- **State Management**: React hooks (useState, useEffect)
- **Storage**: localStorage for auth tokens (client-side only)
## Authentication
### How It Works
The backend API requires **Bearer token authentication** for all requests. The frontend implements client-side token management:
1. **Token Storage**: Tokens are stored in browser localStorage
2. **Token Input**: Users enter tokens via the AuthBanner component
3. **API Requests**: All API calls include `Authorization: Bearer <token>` header
4. **Token Persistence**: Tokens persist across browser sessions until cleared
### Getting a Token
Tokens are managed by administrators. Users need to contact an admin to receive an access token. There is no self-service signup.
**Admin token management**: Admin tokens are stored in Google Cloud Secret Manager (`ADMIN_TOKENS` secret).
### Using Authentication in the UI
```typescript
// Import auth functions
import { setAccessToken, getAccessToken, clearAccessToken } from '@/lib/api';
// Set token (after user input)
setAccessToken('your-token-here');
// Get current token
const token = getAccessToken();
// Clear token (logout)
clearAccessToken();
```
## Features
### 1. Job Submission
Three methods for submitting karaoke generation jobs:
#### File Upload
- Upload audio files directly (FLAC, WAV, MP3, etc.)
- Provide artist and title metadata
- Automatic processing starts immediately
#### YouTube URL
- Submit a YouTube video URL
- Backend downloads audio automatically
- Provide artist and title metadata
#### Audio Search
- Search by artist and title
- Backend finds high-quality audio sources
- Select from multiple results
- Automatic download and processing
### 2. Job Monitoring
Real-time job status tracking with:
- **Status badges**: Visual indicators (Pending, Processing, Awaiting Review, etc.)
- **Progress bars**: Visual progress for active jobs
- **Auto-refresh**: Updates every 10 seconds
- **Manual refresh**: Click refresh button anytime
**Job Statuses**:
- `PENDING` - Job created, waiting to start
- `DOWNLOADING` - Fetching audio
- `PROCESSING_AUDIO` - Separating vocals/instrumental
- `PROCESSING_LYRICS` - Transcribing lyrics
- `AWAITING_REVIEW` - Ready for human review
- `AWAITING_INSTRUMENTAL_SELECTION` - User needs to select instrumental
- `RENDERING_VIDEO` - Creating final video
- `UPLOADING` - Distributing to YouTube/Dropbox
- `COMPLETE` - Finished successfully
- `FAILED` - Error occurred
- `CANCELLED` - User cancelled
### 3. Interactive Elements
#### Lyrics Review
- Launch lyrics correction UI in new tab
- Make timing and text corrections
- Submit corrections to trigger video rendering
#### Instrumental Selection
- Preview multiple instrumental options
- Select best quality instrumental
- Audio player for A/B comparison
#### Job Actions
- **Cancel**: Stop a running job
- **Retry**: Restart a failed job
- **Delete**: Remove job and associated files
- **View Logs**: See detailed processing logs
### 4. Output Access
When jobs complete, access outputs via:
- **YouTube**: Direct link to uploaded video
- **Dropbox**: Link to Dropbox folder with files
- **Direct Downloads**: Download MP4, CDG, TXT files
## Component Structure
### Modular Architecture
```
frontend/
├── app/
│ ├── page.tsx # Main application page
│ ├── layout.tsx # Root layout with metadata
│ └── globals.css # Global styles and CSS variables
│
├── components/
│ ├── auth-banner.tsx # Authentication UI
│ │
│ ├── job/ # Job-related components
│ │ ├── JobCard.tsx # Individual job display
│ │ ├── JobSubmission.tsx # Job creation forms
│ │ ├── JobActions.tsx # Job management buttons
│ │ ├── JobLogs.tsx # Log viewer
│ │ ├── OutputLinks.tsx # Download/distribution links
│ │ └── InstrumentalSelector.tsx # Instrumental picker
│ │
│ ├── audio-search/ # Audio search feature
│ │ └── AudioSearchDialog.tsx
│ │
│ └── ui/ # Reusable UI primitives (shadcn/ui)
│ ├── button.tsx
│ ├── card.tsx
│ ├── dialog.tsx
│ ├── tabs.tsx
│ └── ... (more)
│
└── lib/
├── api.ts # API client (all backend calls)
├── types.ts # TypeScript type definitions
└── utils.ts # Utility functions
```
### Key Files
#### `lib/api.ts` - API Client
The central hub for all backend communication:
```typescript
const API_BASE_URL = process.env.NEXT_PUBLIC_API_URL || 'https://api.nomadkaraoke.com';
export const api = {
// Authentication
setAccessToken(token: string): void
getAccessToken(): string | null
clearAccessToken(): void
// Job Management
listJobs(params?: { status?: string; limit?: number }): Promise<Job[]>
getJob(jobId: string): Promise<Job>
uploadJob(file: File, artist: string, title: string): Promise<JobResponse>
createJobFromUrl(body: CreateJobFromUrlRequest): Promise<JobResponse>
cancelJob(jobId: string, reason?: string): Promise<void>
retryJob(jobId: string): Promise<void>
deleteJob(jobId: string): Promise<void>
// Audio Search
searchAudio(request: AudioSearchRequest): Promise<AudioSearchResponse>
getAudioSearchResults(jobId: string): Promise<AudioSearchResult[]>
selectAudioResult(jobId: string, resultIndex: number): Promise<void>
// Lyrics Review
getReviewData(jobId: string): Promise<ReviewData>
completeReview(jobId: string, corrections: any): Promise<void>
// Instrumental Selection
getInstrumentalOptions(jobId: string): Promise<InstrumentalOption[]>
selectInstrumental(jobId: string, sourceIndex: number): Promise<void>
// Outputs
getDownloadUrls(jobId: string): Promise<DownloadUrls>
}
```
#### `components/auth-banner.tsx` - Authentication UI
Prominent banner at top of page for token management:
- Shows "Authentication Required" when no token
- Displays truncated token when authenticated
- Password-masked input field
- Persistent across page reloads
#### `components/job/JobCard.tsx` - Job Display
Main component for displaying individual jobs:
- Collapsible details view
- Status badges and progress bars
- Conditional rendering based on job state
- Integrates all sub-components (actions, logs, outputs, etc.)
## Styling
### Color Scheme
Matches the instrumental review UI for consistency:
```css
/* Background colors */
--bg: #0f0f0f; /* Very dark, neutral */
--card: #1a1a1a; /* Card backgrounds */
/* Text colors */
--text: #e5e5e5; /* Primary text */
--text-muted: #999999; /* Secondary text */
/* UI colors */
--primary: #3b82f6; /* Blue (actions, links) */
--success: #10b981; /* Green (success states) */
--warning: #f59e0b; /* Orange (warnings) */
--error: #ef4444; /* Red (errors) */
--border: #2a2a2a; /* Borders */
```
### Typography
System font stack for fast loading and native feel:
```css
--font-sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
--font-mono: "SF Mono", Monaco, "Cascadia Code", "Courier New", monospace;
```
## Local Development
### Setup
```bash
cd frontend
# Install dependencies
npm ci --legacy-peer-deps
# Start dev server with hot-reload
npm run dev
# → http://localhost:3000
# Build static export
npm run build
# → Output: frontend/out/
# Run tests
npm test
```
### Environment Variables
Create `.env.local` for local development:
```bash
# Optional: Override API URL for local backend testing
NEXT_PUBLIC_API_URL=http://localhost:8080
```
### Testing Against Production API
The default configuration connects to the production API at `api.nomadkaraoke.com`, so you can develop locally while using the real backend.
**Important**: You'll need a valid access token to make API requests.
## Deployment
### Automatic Deployment (CI/CD)
Changes to `main` branch automatically deploy via GitHub Actions:
1. **Trigger**: Push to `main` or merge PR
2. **Build**: `npm ci --legacy-peer-deps && npm run build`
3. **Deploy**: Upload `frontend/out/` to GitHub Pages
4. **Live**: Available at `gen.nomadkaraoke.com` within ~2 minutes
### CI Workflow
`.github/workflows/ci.yml`:
```yaml
deploy-frontend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Build
working-directory: frontend
run: npm run build
- name: Upload artifact
uses: actions/upload-pages-artifact@v3
with:
path: frontend/out
```
### Manual Deployment
If needed, manually deploy via GitHub Pages settings:
1. Build: `cd frontend && npm run build`
2. Upload `frontend/out/` contents to GitHub Pages
3. Wait for GitHub Pages to rebuild
## Configuration Files
### `next.config.mjs`
```javascript
/** @type {import('next').NextConfig} */
const nextConfig = {
output: 'export', // Static export for GitHub Pages
trailingSlash: true, // Ensures paths work on static hosts
typescript: {
ignoreBuildErrors: false // Enforce type safety
}
};
```
### `package.json`
Key dependencies:
- `next@16.0.7` - Framework
- `react@19.0.0` - UI library
- `lucide-react` - Icon library
- `tailwindcss` - Utility-first CSS
- `@testing-library/react` - Testing
## Common Issues
### Issue: API requests return 404
**Cause**: Frontend trying to call `gen.nomadkaraoke.com/api` instead of `api.nomadkaraoke.com/api`
**Solution**: Verify `API_BASE_URL` in `lib/api.ts` is set to `https://api.nomadkaraoke.com`
### Issue: API requests return 401 Unauthorized
**Cause**: No token or invalid token
**Solution**: Enter a valid access token via the AuthBanner
### Issue: Changes not appearing after deployment
**Cause**: GitHub Pages caching or CDN propagation delay
**Solutions**:
- Wait 2-5 minutes for CDN to update
- Hard refresh browser (Cmd/Ctrl + Shift + R)
- Check GitHub Actions for deployment status
### Issue: Build fails in CI with dependency conflicts
**Cause**: React 19 peer dependency mismatch with testing libraries
**Solution**: Use `npm ci --legacy-peer-deps` (already configured in CI)
## Testing
### Running Tests
```bash
cd frontend
# Run all tests
npm test
# Run with coverage
npm run test:coverage
# Run in CI mode (non-interactive)
npm run test:ci
```
### Test Structure
```
frontend/
├── components/__tests__/
│ ├── JobCard.test.tsx
│ └── ...
└── lib/__tests__/
├── api.test.ts
└── ...
```
### Coverage Requirements
Minimum 70% code coverage enforced in CI (though frontend tests are still being expanded).
## API Integration
### Request Flow
1. User action in UI (e.g., click "Create Job")
2. Component calls API function from `lib/api.ts`
3. API function adds auth header if token exists
4. Fetch request to `api.nomadkaraoke.com`
5. Response parsed and returned
6. Component updates UI with response data
### Error Handling
All API errors are wrapped in `ApiError` class:
```typescript
try {
const jobs = await api.listJobs({ limit: 20 });
setJobs(jobs);
} catch (error) {
if (error instanceof ApiError) {
console.error(`API Error ${error.status}: ${error.message}`);
// error.data contains full response body
}
}
```
## Future Enhancements
Potential improvements for future iterations:
1. **OAuth Integration**: Replace manual token entry with OAuth flow
2. **Real-time Updates**: WebSocket connection for live job updates
3. **Admin Dashboard**: User management, token generation, system stats
4. **Job History**: Pagination and filtering for large job lists
5. **Batch Operations**: Select and manage multiple jobs at once
6. **Mobile Optimization**: Responsive design improvements for mobile devices
7. **Offline Support**: Service worker for offline functionality
8. **Enhanced Testing**: Increase test coverage to 90%+
## Related Documentation
- [Backend API Reference](../README.md#backend-api-reference)
- [Cloud Run Deployment](../03-deployment/CLOUD-RUN-DEPLOYMENT.md)
- [Architecture Overview](./ARCHITECTURE.md)
- [GitHub Pages Setup](.github/workflows/ci.yml)
---
**Last Updated**: December 2025
**Frontend Version**: Deployed from `main` branch commit `91f51d6`
@@ -0,0 +1,106 @@
# Instrumental Review: Original Audio Tab Support
**Date:** 2025-12-22
**Version:** 0.75.12
## Summary
Added support for an "Original Audio" tab in the Instrumental Review UI, allowing users to compare the original audio (with lead vocals) alongside the backing vocals, clean instrumental, and instrumental with backing vocals options. This helps users make better decisions about whether certain audio regions are actually backing vocals or need to be muted.
## Problem
When reviewing backing vocals in the Instrumental Review UI, users couldn't hear the original full song to compare and determine whether detected regions were actually backing vocals or if they needed to be muted. This made it difficult to make informed decisions about the instrumental selection.
## Solution
The Instrumental Review HTML UI already had built-in support for displaying an "Original Audio" tab when `hasOriginal` flag was set, but this feature was not being activated because the `original_audio_path` parameter was not being passed to the `InstrumentalReviewServer`.
### Changes Made
1. **`karaoke_gen/utils/gen_cli.py`** (lines 203-217, 239):
- Added code to locate the original audio file from the track data (`track.get("input_audio_wav")`)
- Resolved the path relative to the track directory
- Passed `original_audio_path` parameter when initializing `InstrumentalReviewServer`
2. **`scripts/test-instrumental-review.py`**:
- Added optional `--original` command-line argument
- Added validation for the original audio file path
- Passed `original_audio_path` parameter to `InstrumentalReviewServer`
3. **`tests/unit/test_instrumental_review/test_server.py`**:
- Added `test_server_initialization_with_original_audio()` to verify server properly stores the original audio path
- Added `test_get_analysis_with_original_audio()` to verify the API properly exposes `has_original` flag and original audio URL
4. **`pyproject.toml`**:
- Bumped version from 0.75.11 to 0.75.12
## How It Works
### Backend Flow
1. When the instrumental review is triggered, the system:
- Locates the original audio file (`input_audio_wav` from the track data)
- Resolves the path relative to the working directory
- Passes it to `InstrumentalReviewServer` as `original_audio_path`
2. The server exposes the original audio through:
- `/api/audio/original` endpoint (serves the audio file)
- `has_original: true` flag in the analysis response
### Frontend Flow
The HTML UI (already implemented in `karaoke_gen/instrumental_review/static/index.html`):
- Checks `hasOriginal` flag from the API
- Conditionally shows "Original" tab in the audio toggle group (line 697-699)
- Provides audio player that switches between:
- **Original Audio** - Full song with lead vocals
- **Backing Vocals Only** - Isolated backing vocals
- **Pure Instrumental** - Clean instrumental (no vocals)
- **Instrumental + Backing** - Instrumental with backing vocals included
- **Custom** - User-created muted regions
### Usage
When running the local CLI:
```bash
karaoke-gen "Artist" "Song Title"
```
During the instrumental review phase, the UI will now automatically show the "Original Audio" tab if the original audio file is available.
For testing:
```bash
python scripts/test-instrumental-review.py \
backing_vocals.flac \
clean_instrumental.flac \
--with_backing instrumental_with_backing.flac \
--original original.wav
```
## Files Modified
- `karaoke_gen/utils/gen_cli.py` - Added original audio path detection and passing
- `scripts/test-instrumental-review.py` - Added --original argument support
- `tests/unit/test_instrumental_review/test_server.py` - Added tests for original audio functionality
- `pyproject.toml` - Version bump
## Testing
All 139 tests in `tests/unit/test_instrumental_review/` pass, including:
- `TestServerInitialization::test_server_initialization_with_original_audio` - Verifies server stores original audio path
- `TestServerAPIEndpoints::test_get_analysis_with_original_audio` - Verifies API exposes original audio properly
## Benefits
1. **Better Decision Making**: Users can hear the full original song to compare against backing vocals
2. **Improved Accuracy**: Easier to identify if detected regions are truly backing vocals or artifacts
3. **No UI Changes Required**: Leveraged existing UI capabilities that were already built-in
4. **Backward Compatible**: If no original audio is provided, the tab simply doesn't appear
## Implementation Notes
- The original audio feature is optional - if the file is not available, the tab won't appear
- The server already had the route handler for serving original audio (`/api/audio/original`)
- The HTML UI already had the conditional rendering logic for the "Original" tab
- The main change was ensuring the `original_audio_path` parameter was properly populated and passed through
+145
View File
@@ -0,0 +1,145 @@
# Backend API Authentication - Remaining Work
## Overview
This document tracks the progress of adding authentication requirements to ALL backend API endpoints. Currently, many endpoints are publicly accessible without any authentication, which is a **major security vulnerability**.
## Completed Endpoints
### ✅ jobs.py (20 endpoints) - DONE
- All job management endpoints now require `Depends(require_auth)`
- Includes: create, get, list, delete, bulk_delete, review endpoints, instrumental selection, download, cancel, retry, logs
### ✅ file_upload.py (6 endpoints) - DONE
- upload_and_create_job
- create_job_with_upload_urls
- mark_uploads_complete
- create_job_from_url
- create_finalise_only_job
- mark_finalise_uploads_complete
### ✅ audio_search.py (3 endpoints) - DONE
- search_audio
- get_audio_search_results
- select_audio_source
### ⚠️ review.py (12 endpoints) - PARTIAL (4/12 done)
**Completed:**
- get_correction_data ✅
- complete_review ✅
- add_lyrics ✅
- generate_preview_video ✅
**Remaining - Need Auth:**
- ❌ ping - Health check (maybe OK without auth?)
- ❌ get_audio_with_hash - Audio streaming for review UI
- ❌ get_audio_no_hash - Audio streaming for review UI
- ❌ update_handlers - Correction handlers config
- ❌ get_preview_video - Preview video streaming
- ❌ submit_annotation - ML training data
- ❌ get_annotation_stats - ML stats
- ❌ (1 more endpoint not yet reviewed)
**Special Considerations:**
- Audio streaming endpoints (`get_audio_*`) are used by the review UI frontend. May need token passed via query param or special handling.
- `ping` endpoint is just a health check - possibly OK without auth
## Remaining Routes
### ❌ auth.py (8 endpoints) - TODO
These are auth-related endpoints themselves, so they have special requirements:
- `/auth/status` - Get OAuth credentials status (probably needs auth)
- `/auth/status/{service}` - Get service credential status (probably needs auth)
- `/auth/validate` - Validate credentials (probably needs auth)
- `/auth/youtube/device` - Start YouTube device auth (admin only)
- `/auth/youtube/device/{device_code}` - Poll YouTube auth (admin only)
- `/auth/gdrive/device` - Start GDrive device auth (admin only)
- `/auth/gdrive/device/{device_code}` - Poll GDrive auth (admin only)
- (1 more endpoint)
**Recommendation:** Most/all of these should require admin auth (`Depends(require_admin)`)
### ❌ health.py (3 endpoints) - TODO
- `/health` - Basic health check (probably OK without auth for k8s/monitoring)
- `/health/detailed` - Detailed health (should require auth)
- `/readiness` - Readiness check for Cloud Run (probably OK without auth)
**Recommendation:**
- Keep `/health` and `/readiness` unauthenticated for infrastructure monitoring
- Require auth for `/health/detailed` as it may expose sensitive info
### ❌ internal.py (6 endpoints) - CRITICAL
**All worker trigger endpoints - MUST have auth!** Currently using `X-Admin-Token` header but needs formal validation:
- `/internal/workers/audio` - Trigger audio worker
- `/internal/workers/lyrics` - Trigger lyrics worker
- `/internal/workers/screens` - Trigger screens worker
- `/internal/workers/video` - Trigger video worker
- `/internal/workers/render-video` - Trigger render worker
- `/internal/health` - Internal health (already has `require_admin`)
**Current State:** These endpoints check for `X-Admin-Token` in the request, but this should use the formal `require_admin` dependency.
**Recommendation:** ALL internal endpoints should use `Depends(require_admin)`
## Implementation Strategy
### Phase 1: Critical Security Fixes (HIGH PRIORITY)
1. ✅ Complete jobs.py, file_upload.py, audio_search.py
2. ❌ **Secure internal.py worker triggers** - These are the most critical as they allow arbitrary job manipulation
3. ❌ **Secure auth.py admin endpoints** - Prevent unauthorized OAuth credential management
### Phase 2: User-Facing Endpoints
1. ❌ Complete review.py endpoints (with special handling for streaming)
2. ❌ health.py detailed endpoint
### Phase 3: Testing & Documentation
1. ❌ Update frontend to always include auth token
2. ❌ Test all endpoints with/without auth
3. ❌ Update API documentation
4. ❌ Add integration tests for auth
## Testing Checklist
After adding auth to all endpoints:
- [ ] Test job creation without token → should return 401
- [ ] Test job listing without token → should return 401
- [ ] Test worker triggers without admin token → should return 401 or 403
- [ ] Test OAuth endpoints without admin token → should return 401 or 403
- [ ] Test health endpoint without token → should work (by design)
- [ ] Test authenticated requests with valid user token → should work
- [ ] Test authenticated requests with valid admin token → should work
- [ ] Update frontend auth flow to handle 401 errors gracefully
## Notes
- The authentication system is already implemented in `backend/api/dependencies.py`
- Functions available: `require_auth()`, `require_admin()`, `optional_auth()`
- Tokens are validated against `ADMIN_TOKENS` env var or Firestore user tokens
- Frontend already has token storage in localStorage via `AuthBanner` component
## Migration Path for Frontend
Once all endpoints require auth:
1. Frontend must obtain a token (either admin token or user token from a signup flow)
2. Token must be included in Authorization header: `Authorization: Bearer <token>`
3. Alternatively, some endpoints support `?token=<token>` query parameter
4. Update frontend API client to always include auth
##Frontend Update Required
Current state: Frontend has AuthBanner that prompts for token, but some endpoints worked without auth (security issue).
New state: ALL endpoints require auth, so users MUST provide a token before using the app.
Update `frontend/lib/api.ts` to:
- Throw user-friendly error if token is missing when calling authenticated endpoints
- Handle 401 responses by prompting user to re-authenticate
- Consider implementing a "logged out" state in the UI
+140
View File
@@ -0,0 +1,140 @@
# Test Fixes Needed for Authentication
## Problem
After adding `Depends(require_auth)` to 33+ backend endpoints, all unit and integration tests are failing because they don't include authentication headers in their requests.
## Solution
All test requests to authenticated endpoints need to include:
```python
headers={"Authorization": "Bearer test-admin-token"}
```
## Files Need Updating
### 1. Global Fixture (DONE)
✅ `backend/tests/conftest.py` - Added mock_auth_service and auth_headers fixtures
### 2. Test Files to Update
Each test file needs:
1. Mock auth service in the client fixture
2. Add `auth_headers` parameter to test methods
3. Pass `headers=auth_headers` to all client requests (get, post, put, delete)
Files:
- `backend/tests/test_jobs_api.py` - ~15 test methods
- `backend/tests/test_upload_api.py` - ~10 test methods
- `backend/tests/test_audio_search.py` - ~5 test methods
- `backend/tests/test_api_routes.py` - ~10 test methods
- `backend/tests/test_instrumental_api.py` - ~5 test methods
- `backend/tests/emulator/test_emulator_integration.py` - ~20 test methods
## Quick Fix Script
Rather than manually updating 60+ test methods, here's a Python script to auto-fix them:
```python
#!/usr/bin/env python3
"""
Auto-fix test files to include authentication headers.
"""
import re
from pathlib import Path
def fix_test_file(filepath):
"""Add auth_headers to client requests in test file."""
with open(filepath, 'r') as f:
content = f.read()
# Pattern: client.METHOD(
# Replace with: client.METHOD(..., headers=auth_headers)
patterns = [
# client.get("/path")
(r'client\.(get|post|put|delete|patch)\((["\'][^"\']+["\'])\)',
r'client.\1(\2, headers=auth_headers)'),
# client.get("/path", json={...})
(r'client\.(get|post|put|delete|patch)\((["\'][^"\']+["\']),\s*json=',
r'client.\1(\2, headers=auth_headers, json='),
# client.post("/path", data={...})
(r'client\.(get|post|put|delete|patch)\((["\'][^"\']+["\']),\s*data=',
r'client.\1(\2, headers=auth_headers, data='),
# client.post("/path", files={...})
(r'client\.(post)\((["\'][^"\']+["\']),\s*files=',
r'client.\1(\2, headers=auth_headers, files='),
]
modified = content
for pattern, replacement in patterns:
modified = re.sub(pattern, replacement, modified)
# Add auth_headers parameter to test methods that don't have it
# def test_something(self, client, mock_job):
# -> def test_something(self, client, mock_job, auth_headers):
modified = re.sub(
r'def (test_\w+)\(self,\s*client([^)]*)\):',
r'def \1(self, client\2, auth_headers):',
modified
)
with open(filepath, 'w') as f:
f.write(modified)
print(f"Fixed: {filepath}")
# Find all test files
test_dir = Path("backend/tests")
for test_file in test_dir.rglob("test_*.py"):
if 'conftest' not in str(test_file):
fix_test_file(test_file)
print("Done!")
```
## Manual Approach
For each test method:
**Before:**
```python
def test_get_job(self, client, mock_job_manager):
response = client.get("/api/jobs/test123")
assert response.status_code == 200
```
**After:**
```python
def test_get_job(self, client, mock_job_manager, auth_headers):
response = client.get("/api/jobs/test123", headers=auth_headers)
assert response.status_code == 200
```
## Emulator Tests
Emulator tests already have `auth_headers` fixture in `backend/tests/emulator/conftest.py`.
Just need to:
1. Add `auth_headers` parameter to test methods
2. Pass `headers=auth_headers` to client requests
## Health Endpoints
Health endpoints (`/api/health`, `/api/readiness`) should remain unauthenticated, so those tests don't need auth headers.
## Internal API Tests
Internal API tests in `test_internal_api.py` already have auth handling via `X-Admin-Token` header. May need updating if we migrate to formal `require_admin`.
## Test Coverage
After fixes, run:
```bash
cd backend
python -m pytest tests/ -v --ignore=tests/integration
```
Should see all tests passing.
-348
View File
@@ -1,348 +0,0 @@
"use client"
import { useState } from "react"
import { useJobs } from "@/lib/jobs"
import type { JobStatus } from "@/lib/types"
import { AppHeader } from "@/components/app-header"
import { ProtectedRoute } from "@/components/protected-route"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Label } from "@/components/ui/label"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"
import { StatusBadge } from "@/components/status-badge"
import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table"
import { BarChart3, FileVideo, Ticket, TrendingUp, Activity, ChevronRight } from "lucide-react"
import Link from "next/link"
function AdminContent() {
const { jobs } = useJobs()
const [promoCode, setPromoCode] = useState("")
const [promoCredits, setPromoCredits] = useState("5")
const [generatedCode, setGeneratedCode] = useState("")
const totalJobs = jobs.length
const completedJobs = jobs.filter((j) => j.status === "completed").length
const processingJobs = jobs.filter((j) => j.status === "processing" || j.status === "queued").length
const failedJobs = jobs.filter((j) => j.status === "failed").length
const handleGeneratePromo = () => {
const code = `NOMAD${Math.random().toString(36).substr(2, 6).toUpperCase()}`
setGeneratedCode(code)
setPromoCode("")
}
const statusCounts = {
queued: jobs.filter((j) => j.status === "queued").length,
processing: jobs.filter((j) => j.status === "processing").length,
awaiting_review: jobs.filter((j) => j.status === "awaiting_review").length,
awaiting_instrumental: jobs.filter((j) => j.status === "awaiting_instrumental").length,
completed: completedJobs,
failed: failedJobs,
}
const recentJobs = [...jobs]
.sort((a, b) => new Date(b.createdAt).getTime() - new Date(a.createdAt).getTime())
.slice(0, 10)
return (
<div className="min-h-screen bg-background">
<AppHeader />
<main className="container mx-auto px-4 py-8">
<div className="mb-8">
<h1 className="text-4xl font-bold mb-2">Admin Dashboard</h1>
<p className="text-muted-foreground">Manage users, jobs, and system analytics</p>
</div>
{/* Stats Grid */}
<div className="grid gap-4 md:grid-cols-2 lg:grid-cols-4 mb-8">
<Card className="bg-gradient-to-br from-primary/10 to-transparent border-primary/20">
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium text-muted-foreground">Total Jobs</CardTitle>
<FileVideo className="w-4 h-4 text-primary" />
</CardHeader>
<CardContent>
<div className="text-3xl font-bold text-primary">{totalJobs}</div>
<p className="text-xs text-muted-foreground mt-1">All time</p>
</CardContent>
</Card>
<Card>
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium text-muted-foreground">Completed</CardTitle>
<TrendingUp className="w-4 h-4 text-green-400" />
</CardHeader>
<CardContent>
<div className="text-3xl font-bold text-green-400">{completedJobs}</div>
<p className="text-xs text-muted-foreground mt-1">
{((completedJobs / totalJobs) * 100).toFixed(1)}% success rate
</p>
</CardContent>
</Card>
<Card>
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium text-muted-foreground">Processing</CardTitle>
<Activity className="w-4 h-4 text-blue-400" />
</CardHeader>
<CardContent>
<div className="text-3xl font-bold text-blue-400">{processingJobs}</div>
<p className="text-xs text-muted-foreground mt-1">Active jobs</p>
</CardContent>
</Card>
<Card>
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium text-muted-foreground">Failed</CardTitle>
<BarChart3 className="w-4 h-4 text-destructive" />
</CardHeader>
<CardContent>
<div className="text-3xl font-bold text-destructive">{failedJobs}</div>
<p className="text-xs text-muted-foreground mt-1">
{((failedJobs / totalJobs) * 100).toFixed(1)}% failure rate
</p>
</CardContent>
</Card>
</div>
{/* Tabs */}
<Tabs defaultValue="overview" className="space-y-6">
<TabsList className="grid w-full max-w-md grid-cols-3">
<TabsTrigger value="overview">Overview</TabsTrigger>
<TabsTrigger value="jobs">All Jobs</TabsTrigger>
<TabsTrigger value="promo">Promo Codes</TabsTrigger>
</TabsList>
<TabsContent value="overview" className="space-y-6">
{/* Status Breakdown */}
<Card>
<CardHeader>
<CardTitle>Job Status Breakdown</CardTitle>
<CardDescription>Current distribution of job statuses</CardDescription>
</CardHeader>
<CardContent>
<div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-3">
{Object.entries(statusCounts).map(([status, count]) => (
<div
key={status}
className="flex items-center justify-between p-3 rounded-lg border border-border bg-card"
>
<div className="flex items-center gap-3">
<StatusBadge status={status as JobStatus} size="sm" />
<span className="text-sm text-muted-foreground">Jobs</span>
</div>
<span className="text-lg font-bold">{count}</span>
</div>
))}
</div>
</CardContent>
</Card>
{/* Recent Activity */}
<Card>
<CardHeader className="flex flex-row items-center justify-between">
<div>
<CardTitle>Recent Activity</CardTitle>
<CardDescription>Latest jobs across all users</CardDescription>
</div>
<Button variant="outline" size="sm" asChild>
<Link href="#" onClick={(e) => e.preventDefault()}>
View All
</Link>
</Button>
</CardHeader>
<CardContent>
<div className="space-y-2">
{recentJobs.slice(0, 5).map((job) => (
<Link
key={job.id}
href={`/jobs/${job.id}`}
className="flex items-center justify-between p-3 rounded-lg border border-border hover:border-primary/50 hover:bg-card/50 transition-all group"
>
<div className="flex-1 min-w-0">
<p className="font-medium text-sm truncate group-hover:text-primary transition-colors">
{job.artist} - {job.title}
</p>
<p className="text-xs text-muted-foreground">{new Date(job.createdAt).toLocaleString()}</p>
</div>
<div className="flex items-center gap-2">
<StatusBadge status={job.status} size="sm" />
<ChevronRight className="w-4 h-4 text-muted-foreground group-hover:text-primary transition-colors" />
</div>
</Link>
))}
</div>
</CardContent>
</Card>
</TabsContent>
<TabsContent value="jobs" className="space-y-6">
<Card>
<CardHeader>
<CardTitle>All Jobs</CardTitle>
<CardDescription>Complete list of all karaoke video jobs in the system</CardDescription>
</CardHeader>
<CardContent>
<div className="rounded-md border border-border overflow-auto">
<Table>
<TableHeader>
<TableRow>
<TableHead>Artist - Title</TableHead>
<TableHead>User ID</TableHead>
<TableHead>Status</TableHead>
<TableHead>Created</TableHead>
<TableHead>Progress</TableHead>
<TableHead></TableHead>
</TableRow>
</TableHeader>
<TableBody>
{recentJobs.map((job) => (
<TableRow key={job.id} className="hover:bg-muted/50">
<TableCell className="font-medium">
{job.artist} - {job.title}
</TableCell>
<TableCell className="text-muted-foreground text-sm">{job.userId}</TableCell>
<TableCell>
<StatusBadge status={job.status} size="sm" />
</TableCell>
<TableCell className="text-sm text-muted-foreground">
{new Date(job.createdAt).toLocaleDateString()}
</TableCell>
<TableCell className="text-sm">{job.progress}%</TableCell>
<TableCell>
<Button variant="ghost" size="sm" asChild>
<Link href={`/jobs/${job.id}`}>
<ChevronRight className="w-4 h-4" />
</Link>
</Button>
</TableCell>
</TableRow>
))}
</TableBody>
</Table>
</div>
</CardContent>
</Card>
</TabsContent>
<TabsContent value="promo" className="space-y-6">
<div className="grid gap-6 lg:grid-cols-2">
{/* Generate Promo Code */}
<Card>
<CardHeader>
<CardTitle className="flex items-center gap-2">
<Ticket className="w-5 h-5 text-primary" />
Generate Promo Code
</CardTitle>
<CardDescription>Create promotional codes for users to redeem credits</CardDescription>
</CardHeader>
<CardContent className="space-y-4">
<div className="space-y-2">
<Label htmlFor="promo-credits">Credit Amount</Label>
<Input
id="promo-credits"
type="number"
min="1"
max="100"
value={promoCredits}
onChange={(e) => setPromoCredits(e.target.value)}
/>
<p className="text-xs text-muted-foreground">Number of credits this code will provide</p>
</div>
<Button onClick={handleGeneratePromo} className="w-full bg-primary hover:bg-primary/90">
Generate Code
</Button>
{generatedCode && (
<div className="p-4 rounded-lg bg-primary/10 border border-primary/20">
<p className="text-sm text-muted-foreground mb-2">Generated Promo Code:</p>
<p className="text-2xl font-mono font-bold text-primary">{generatedCode}</p>
<p className="text-xs text-muted-foreground mt-2">Worth {promoCredits} credits</p>
</div>
)}
</CardContent>
</Card>
{/* Active Promo Codes */}
<Card>
<CardHeader>
<CardTitle>Active Promo Codes</CardTitle>
<CardDescription>Currently available promotional codes</CardDescription>
</CardHeader>
<CardContent>
<div className="space-y-3">
<div className="flex items-center justify-between p-3 rounded-lg border border-border bg-card">
<div>
<p className="font-mono font-medium">NOMADWELCOME</p>
<p className="text-xs text-muted-foreground">5 credits • Used 23 times</p>
</div>
<Button variant="ghost" size="sm" className="text-destructive">
Disable
</Button>
</div>
<div className="flex items-center justify-between p-3 rounded-lg border border-border bg-card">
<div>
<p className="font-mono font-medium">LAUNCH2024</p>
<p className="text-xs text-muted-foreground">10 credits • Used 8 times</p>
</div>
<Button variant="ghost" size="sm" className="text-destructive">
Disable
</Button>
</div>
{generatedCode && (
<div className="flex items-center justify-between p-3 rounded-lg border border-primary/20 bg-primary/5">
<div>
<p className="font-mono font-medium text-primary">{generatedCode}</p>
<p className="text-xs text-muted-foreground">{promoCredits} credits • Not used yet</p>
</div>
<Button variant="ghost" size="sm" className="text-destructive">
Disable
</Button>
</div>
)}
</div>
</CardContent>
</Card>
</div>
{/* Usage Statistics */}
<Card>
<CardHeader>
<CardTitle>Promo Code Statistics</CardTitle>
<CardDescription>Usage analytics and performance metrics</CardDescription>
</CardHeader>
<CardContent>
<div className="grid gap-4 sm:grid-cols-3">
<div className="space-y-2">
<p className="text-sm text-muted-foreground">Total Codes</p>
<p className="text-3xl font-bold">12</p>
</div>
<div className="space-y-2">
<p className="text-sm text-muted-foreground">Total Redemptions</p>
<p className="text-3xl font-bold">156</p>
</div>
<div className="space-y-2">
<p className="text-sm text-muted-foreground">Credits Distributed</p>
<p className="text-3xl font-bold text-primary">892</p>
</div>
</div>
</CardContent>
</Card>
</TabsContent>
</Tabs>
</main>
</div>
)
}
export default function AdminPage() {
return (
<ProtectedRoute requireAdmin>
<AdminContent />
</ProtectedRoute>
)
}
-171
View File
@@ -1,171 +0,0 @@
"use client"
import { useRouter } from "next/navigation"
import { useAuth } from "@/lib/auth"
import { useJobs } from "@/lib/jobs"
import { AppHeader } from "@/components/app-header"
import { ProtectedRoute } from "@/components/protected-route"
import { Button } from "@/components/ui/button"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import { StatusBadge } from "@/components/status-badge"
import { Plus, FileVideo, Clock, CheckCircle2, Loader2 } from "lucide-react"
import Link from "next/link"
function DashboardContent() {
const { user } = useAuth()
const { jobs, getUserJobs } = useJobs()
const router = useRouter()
if (!user) return null
const userJobs = getUserJobs(user.token.includes("demo") ? "demo" : user.token)
const completedJobs = userJobs.filter((j) => j.status === "completed").length
const inProgressJobs = userJobs.filter((j) => j.status === "processing" || j.status === "queued").length
const totalJobs = userJobs.length
const recentJobs = userJobs.slice(0, 5)
return (
<div className="min-h-screen bg-background">
<AppHeader />
<main className="container mx-auto px-4 py-8">
{/* Welcome Section */}
<div className="mb-8">
<h1 className="text-4xl font-bold mb-2">
Welcome back<span className="text-primary">!</span>
</h1>
<p className="text-muted-foreground">Manage your karaoke video projects</p>
</div>
{/* Stats Grid */}
<div className="grid gap-4 md:grid-cols-4 mb-8">
<Card>
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium text-muted-foreground">Total Jobs</CardTitle>
<FileVideo className="w-4 h-4 text-muted-foreground" />
</CardHeader>
<CardContent>
<div className="text-3xl font-bold">{totalJobs}</div>
</CardContent>
</Card>
<Card>
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium text-muted-foreground">Completed</CardTitle>
<CheckCircle2 className="w-4 h-4 text-green-400" />
</CardHeader>
<CardContent>
<div className="text-3xl font-bold text-green-400">{completedJobs}</div>
</CardContent>
</Card>
<Card>
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium text-muted-foreground">In Progress</CardTitle>
<Loader2 className="w-4 h-4 text-blue-400 animate-spin" />
</CardHeader>
<CardContent>
<div className="text-3xl font-bold text-blue-400">{inProgressJobs}</div>
</CardContent>
</Card>
<Card className="bg-gradient-to-br from-primary/10 to-secondary/10 border-primary/20">
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium">Credits Remaining</CardTitle>
<div className="w-2 h-2 rounded-full bg-primary animate-pulse" />
</CardHeader>
<CardContent>
<div className="text-3xl font-bold text-primary">{user.credits}</div>
<p className="text-xs text-muted-foreground mt-1">
<Link href="#" className="hover:text-primary transition-colors">
Purchase more
</Link>
</p>
</CardContent>
</Card>
</div>
{/* Create New Job CTA */}
<Card className="mb-8 bg-gradient-to-r from-primary/5 via-card to-secondary/5 border-primary/20">
<CardHeader>
<CardTitle className="text-2xl">Create New Karaoke Video</CardTitle>
<CardDescription>
Transform any song into a professional karaoke video with synchronized lyrics
</CardDescription>
</CardHeader>
<CardContent>
<Button size="lg" className="bg-primary hover:bg-primary/90" onClick={() => router.push("/jobs/new")}>
<Plus className="w-5 h-5 mr-2" />
Create New Job
</Button>
</CardContent>
</Card>
{/* Recent Jobs */}
<Card>
<CardHeader className="flex flex-row items-center justify-between">
<div>
<CardTitle>Recent Jobs</CardTitle>
<CardDescription>Your latest karaoke video projects</CardDescription>
</div>
<Button variant="outline" size="sm" asChild>
<Link href="/jobs">View All</Link>
</Button>
</CardHeader>
<CardContent>
{recentJobs.length === 0 ? (
<div className="text-center py-12">
<FileVideo className="w-12 h-12 mx-auto mb-4 text-muted-foreground" />
<p className="text-muted-foreground mb-4">No jobs yet</p>
<Button onClick={() => router.push("/jobs/new")}>
<Plus className="w-4 h-4 mr-2" />
Create Your First Job
</Button>
</div>
) : (
<div className="space-y-3">
{recentJobs.map((job) => (
<Link
key={job.id}
href={`/jobs/${job.id}`}
className="flex items-center justify-between p-4 rounded-lg border border-border hover:border-primary/50 hover:bg-card/50 transition-all cursor-pointer group"
>
<div className="flex-1">
<div className="flex items-center gap-3 mb-1">
<h3 className="font-medium group-hover:text-primary transition-colors">
{job.artist} - {job.title}
</h3>
<StatusBadge status={job.status} size="sm" />
</div>
<div className="flex items-center gap-4 text-sm text-muted-foreground">
<span className="flex items-center gap-1">
<Clock className="w-3 h-3" />
{new Date(job.createdAt).toLocaleDateString()}
</span>
{job.status === "processing" && <span>{job.progress}% complete</span>}
</div>
</div>
{job.status === "completed" && (
<Button size="sm" variant="outline" className="ml-4 bg-transparent">
Download
</Button>
)}
</Link>
))}
</div>
)}
</CardContent>
</Card>
</main>
</div>
)
}
export default function DashboardPage() {
return (
<ProtectedRoute>
<DashboardContent />
</ProtectedRoute>
)
}
+36 -48
View File
@@ -4,45 +4,46 @@
@custom-variant dark (&:is(.dark *));
:root {
/* Nomad Karaoke brand colors - matching nomadkaraoke.com */
--background: oklch(0.22 0.04 260); /* Dark slate blue #2b3544 */
--foreground: oklch(0.95 0.01 0); /* White text */
--card: oklch(0.26 0.04 260); /* Slightly lighter slate for cards */
--card-foreground: oklch(0.95 0.01 0);
--popover: oklch(0.26 0.04 260);
--popover-foreground: oklch(0.95 0.01 0);
--primary: oklch(0.68 0.22 340); /* Pink #ff7acc */
--primary-foreground: oklch(0.98 0.01 0);
--secondary: oklch(0.78 0.15 85); /* Gold/yellow #ffc857 */
--secondary-foreground: oklch(0.15 0.01 270);
--muted: oklch(0.3 0.04 260);
--muted-foreground: oklch(0.65 0.01 260);
--accent: oklch(0.68 0.22 340);
--accent-foreground: oklch(0.98 0.01 0);
--destructive: oklch(0.55 0.22 25);
--destructive-foreground: oklch(0.98 0.01 0);
--border: oklch(0.32 0.04 260);
--input: oklch(0.32 0.04 260);
--ring: oklch(0.68 0.22 340);
--chart-1: oklch(0.68 0.22 340);
--chart-2: oklch(0.78 0.15 85);
--chart-3: oklch(0.55 0.2 280);
--chart-4: oklch(0.6 0.18 180);
--chart-5: oklch(0.65 0.2 60);
/* Matching instrumental review UI color scheme */
--bg: #0f0f0f;
--card: #1a1a1a;
--card-border: #2a2a2a;
--waveform-bg: #0d1117;
--text: #e5e5e5;
--text-muted: #888;
--primary: #3b82f6;
--primary-hover: #2563eb;
--secondary: #252525;
--secondary-hover: #333;
--success: #22c55e;
--warning: #f59e0b;
--danger: #ef4444;
--pink: #ec4899;
--blue: #60a5fa;
/* Tailwind compatibility mappings */
--background: var(--bg);
--foreground: var(--text);
--card-foreground: var(--text);
--popover: var(--card);
--popover-foreground: var(--text);
--primary-foreground: #ffffff;
--secondary-foreground: var(--text);
--muted: var(--secondary);
--muted-foreground: var(--text-muted);
--accent: var(--primary);
--accent-foreground: #ffffff;
--destructive: var(--danger);
--destructive-foreground: #ffffff;
--border: var(--card-border);
--input: var(--secondary);
--ring: var(--primary);
--radius: 0.5rem;
--sidebar: oklch(0.26 0.04 260);
--sidebar-foreground: oklch(0.95 0.01 0);
--sidebar-primary: oklch(0.68 0.22 340);
--sidebar-primary-foreground: oklch(0.98 0.01 0);
--sidebar-accent: oklch(0.3 0.04 260);
--sidebar-accent-foreground: oklch(0.95 0.01 0);
--sidebar-border: oklch(0.32 0.04 260);
--sidebar-ring: oklch(0.68 0.22 340);
}
@theme inline {
--font-sans: "Geist", "Geist Fallback";
--font-mono: "Geist Mono", "Geist Mono Fallback";
--font-sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
--font-mono: "SF Mono", Monaco, "Cascadia Code", monospace;
--color-background: var(--background);
--color-foreground: var(--foreground);
--color-card: var(--card);
@@ -62,23 +63,10 @@
--color-border: var(--border);
--color-input: var(--input);
--color-ring: var(--ring);
--color-chart-1: var(--chart-1);
--color-chart-2: var(--chart-2);
--color-chart-3: var(--chart-3);
--color-chart-4: var(--chart-4);
--color-chart-5: var(--chart-5);
--radius-sm: calc(var(--radius) - 4px);
--radius-md: calc(var(--radius) - 2px);
--radius-lg: var(--radius);
--radius-xl: calc(var(--radius) + 4px);
--color-sidebar: var(--sidebar);
--color-sidebar-foreground: var(--sidebar-foreground);
--color-sidebar-primary: var(--sidebar-primary);
--color-sidebar-primary-foreground: var(--sidebar-primary-foreground);
--color-sidebar-accent: var(--sidebar-accent);
--color-sidebar-accent-foreground: var(--sidebar-accent-foreground);
--color-sidebar-border: var(--sidebar-border);
--color-sidebar-ring: var(--sidebar-ring);
}
@layer base {
@@ -1,603 +0,0 @@
"use client"
import { useParams, useRouter } from "next/navigation"
import { useState, useEffect, useCallback } from "react"
import { AppHeader } from "@/components/app-header"
import { ProtectedRoute } from "@/components/protected-route"
import { Button } from "@/components/ui/button"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import { Alert, AlertDescription } from "@/components/ui/alert"
import { Badge } from "@/components/ui/badge"
import { RadioGroup, RadioGroupItem } from "@/components/ui/radio-group"
import { Label } from "@/components/ui/label"
import { Skeleton } from "@/components/ui/skeleton"
import {
WaveformDisplay,
AudioPlayer,
RegionSelector,
} from "@/components/instrumental-review"
import {
ArrowLeft,
AlertCircle,
CheckCircle2,
Loader2,
Music,
VolumeX,
Volume2,
} from "lucide-react"
interface AudibleSegment {
start_seconds: number
end_seconds: number
duration_seconds: number
avg_amplitude_db: number
peak_amplitude_db?: number
}
interface MuteRegion {
start_seconds: number
end_seconds: number
}
interface AnalysisData {
has_audible_content: boolean
total_duration_seconds: number
audible_segments: AudibleSegment[]
recommended_selection: string
total_audible_duration_seconds: number
audible_percentage: number
silence_threshold_db: number
}
interface InstrumentalAnalysisResponse {
job_id: string
artist: string
title: string
status: string
analysis: AnalysisData
audio_urls: {
clean_instrumental: string | null
backing_vocals: string | null
with_backing: string | null
custom_instrumental: string | null
}
waveform_url: string | null
has_custom_instrumental: boolean
}
// Get API base URL from environment or default to relative path
const API_BASE = process.env.NEXT_PUBLIC_API_URL || "/api"
function InstrumentalReviewContent() {
const params = useParams()
const router = useRouter()
const jobId = params.id as string
// State
const [isLoading, setIsLoading] = useState(true)
const [error, setError] = useState<string | null>(null)
const [analysisData, setAnalysisData] = useState<InstrumentalAnalysisResponse | null>(null)
const [waveformData, setWaveformData] = useState<{ amplitudes: number[]; duration: number } | null>(null)
const [selectedOption, setSelectedOption] = useState<string>("clean")
const [muteRegions, setMuteRegions] = useState<MuteRegion[]>([])
const [isSelectionMode, setIsSelectionMode] = useState(false)
const [currentTime, setCurrentTime] = useState(0)
const [seekTo, setSeekTo] = useState<number | undefined>(undefined)
const [isCreatingCustom, setIsCreatingCustom] = useState(false)
const [isSubmitting, setIsSubmitting] = useState(false)
const [activeAudio, setActiveAudio] = useState<"clean" | "backing" | "with_backing" | "custom">("backing")
// Fetch analysis data
const fetchAnalysis = useCallback(async () => {
try {
setIsLoading(true)
setError(null)
const response = await fetch(`${API_BASE}/jobs/${jobId}/instrumental-analysis`)
if (!response.ok) {
const errorData = await response.json().catch(() => ({}))
throw new Error(errorData.detail || `Failed to fetch analysis: ${response.status}`)
}
const data: InstrumentalAnalysisResponse = await response.json()
setAnalysisData(data)
// Set initial selection based on recommendation
const recommendation = data.analysis.recommended_selection
if (recommendation === "clean") {
setSelectedOption("clean")
} else if (recommendation === "with_backing") {
setSelectedOption("with_backing")
} else {
// "review_needed" - default to with_backing but user should review
setSelectedOption("with_backing")
}
// Fetch waveform data for interactive display
const waveformResponse = await fetch(`${API_BASE}/jobs/${jobId}/waveform-data?num_points=600`)
if (waveformResponse.ok) {
const waveform = await waveformResponse.json()
setWaveformData(waveform)
}
} catch (err) {
setError(err instanceof Error ? err.message : "Failed to load analysis data")
} finally {
setIsLoading(false)
}
}, [jobId])
useEffect(() => {
fetchAnalysis()
}, [fetchAnalysis])
// Handle seeking from waveform click
const handleSeek = (time: number) => {
setSeekTo(time)
// Reset seekTo after a short delay
setTimeout(() => setSeekTo(undefined), 100)
}
// Handle adding a mute region
const handleAddRegion = (region: MuteRegion) => {
setMuteRegions((prev) => {
// Check for overlaps and merge if needed
const newRegions = [...prev, region].sort((a, b) => a.start_seconds - b.start_seconds)
// Merge overlapping regions
const merged: MuteRegion[] = []
for (const r of newRegions) {
if (merged.length === 0) {
merged.push(r)
} else {
const last = merged[merged.length - 1]
if (r.start_seconds <= last.end_seconds) {
last.end_seconds = Math.max(last.end_seconds, r.end_seconds)
} else {
merged.push(r)
}
}
}
return merged
})
setIsSelectionMode(false)
}
// Handle removing a mute region
const handleRemoveRegion = (index: number) => {
setMuteRegions((prev) => prev.filter((_, i) => i !== index))
}
// Handle clearing all mute regions
const handleClearAll = () => {
setMuteRegions([])
}
// Create custom instrumental
const handleCreateCustom = async () => {
if (muteRegions.length === 0) {
setError("Please select at least one region to mute")
return
}
try {
setIsCreatingCustom(true)
setError(null)
const response = await fetch(`${API_BASE}/jobs/${jobId}/create-custom-instrumental`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
mute_regions: muteRegions.map((r) => ({
start_seconds: r.start_seconds,
end_seconds: r.end_seconds,
})),
}),
})
if (!response.ok) {
const errorData = await response.json().catch(() => ({}))
throw new Error(errorData.detail || "Failed to create custom instrumental")
}
// Refresh analysis to get updated URLs
await fetchAnalysis()
setSelectedOption("custom")
setActiveAudio("custom")
} catch (err) {
setError(err instanceof Error ? err.message : "Failed to create custom instrumental")
} finally {
setIsCreatingCustom(false)
}
}
// Submit selection
const handleSubmit = async () => {
let selection = selectedOption
// If "review" is selected, determine based on mute regions
if (selectedOption === "review") {
if (muteRegions.length > 0 && analysisData?.has_custom_instrumental) {
selection = "custom"
} else {
selection = "with_backing"
}
}
try {
setIsSubmitting(true)
setError(null)
const response = await fetch(`${API_BASE}/jobs/${jobId}/select-instrumental`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ selection }),
})
if (!response.ok) {
const errorData = await response.json().catch(() => ({}))
throw new Error(errorData.detail || "Failed to submit selection")
}
// Navigate back to job page
router.push(`/jobs/${jobId}`)
} catch (err) {
setError(err instanceof Error ? err.message : "Failed to submit selection")
} finally {
setIsSubmitting(false)
}
}
// Get current audio URL based on active selection
const getCurrentAudioUrl = () => {
if (!analysisData) return null
switch (activeAudio) {
case "clean":
return analysisData.audio_urls.clean_instrumental
case "backing":
return analysisData.audio_urls.backing_vocals
case "with_backing":
return analysisData.audio_urls.with_backing
case "custom":
return analysisData.audio_urls.custom_instrumental
default:
return null
}
}
// Loading state
if (isLoading) {
return (
<div className="min-h-screen bg-background">
<AppHeader />
<main className="container mx-auto px-4 py-8 max-w-5xl">
<Skeleton className="h-8 w-48 mb-4" />
<Skeleton className="h-4 w-96 mb-8" />
<Skeleton className="h-[200px] w-full mb-4" />
<Skeleton className="h-32 w-full" />
</main>
</div>
)
}
// Error state
if (error && !analysisData) {
return (
<div className="min-h-screen bg-background">
<AppHeader />
<main className="container mx-auto px-4 py-8 max-w-5xl">
<Alert variant="destructive">
<AlertCircle className="h-4 w-4" />
<AlertDescription>{error}</AlertDescription>
</Alert>
<Button variant="outline" className="mt-4" onClick={() => router.push(`/jobs/${jobId}`)}>
<ArrowLeft className="w-4 h-4 mr-2" />
Back to Job
</Button>
</main>
</div>
)
}
const analysis = analysisData?.analysis
return (
<div className="min-h-screen bg-background">
<AppHeader />
<main className="container mx-auto px-4 py-8 max-w-6xl">
{/* Header */}
<Button variant="ghost" className="mb-6" onClick={() => router.push(`/jobs/${jobId}`)}>
<ArrowLeft className="w-4 h-4 mr-2" />
Back to Job
</Button>
<div className="mb-8">
<h1 className="text-3xl font-bold mb-2">
Select Instrumental Track
</h1>
<p className="text-muted-foreground">
{analysisData?.artist} - {analysisData?.title}
</p>
</div>
{/* Error Alert */}
{error && (
<Alert variant="destructive" className="mb-6">
<AlertCircle className="h-4 w-4" />
<AlertDescription>{error}</AlertDescription>
</Alert>
)}
{/* Analysis Summary */}
{analysis && (
<Card className="mb-6">
<CardHeader>
<CardTitle className="text-lg flex items-center gap-2">
<Music className="h-5 w-5" />
Backing Vocals Analysis
</CardTitle>
<CardDescription>
Automated analysis of the backing vocals stem
</CardDescription>
</CardHeader>
<CardContent>
<div className="flex flex-wrap gap-4">
<div className="flex items-center gap-2">
{analysis.has_audible_content ? (
<Volume2 className="h-4 w-4 text-yellow-500" />
) : (
<VolumeX className="h-4 w-4 text-green-500" />
)}
<span className="text-sm">
{analysis.has_audible_content
? "Backing vocals detected"
: "No backing vocals detected"}
</span>
</div>
{analysis.has_audible_content && (
<>
<Badge variant="outline">
{analysis.audible_segments.length} audible segments
</Badge>
<Badge variant="outline">
{analysis.audible_percentage.toFixed(1)}% of track
</Badge>
<Badge variant="outline">
{analysis.total_audible_duration_seconds.toFixed(1)}s total
</Badge>
</>
)}
<Badge
variant={analysis.recommended_selection === "clean" ? "default" : "secondary"}
>
Recommended: {analysis.recommended_selection === "clean" ? "Clean Instrumental" : "Review Needed"}
</Badge>
</div>
</CardContent>
</Card>
)}
{/* Waveform Display */}
{waveformData && analysis && (
<Card className="mb-6">
<CardHeader>
<CardTitle className="text-lg">Backing Vocals Waveform</CardTitle>
<CardDescription>
Click to seek • {isSelectionMode ? "Drag to select mute region" : "Click 'Add Region' to start selecting"}
</CardDescription>
</CardHeader>
<CardContent>
<WaveformDisplay
amplitudes={waveformData.amplitudes}
duration={waveformData.duration}
currentTime={currentTime}
segments={analysis.audible_segments}
muteRegions={muteRegions}
onSeek={handleSeek}
onRegionSelect={handleAddRegion}
isSelecting={isSelectionMode}
width={900}
height={150}
className="w-full overflow-x-auto"
/>
</CardContent>
</Card>
)}
<div className="grid lg:grid-cols-2 gap-6 mb-6">
{/* Audio Players */}
<Card>
<CardHeader>
<CardTitle className="text-lg">Audio Preview</CardTitle>
<CardDescription>
Listen to different instrumental options
</CardDescription>
</CardHeader>
<CardContent className="space-y-4">
{/* Audio source selector */}
<div className="flex flex-wrap gap-2">
<Button
variant={activeAudio === "backing" ? "default" : "outline"}
size="sm"
onClick={() => setActiveAudio("backing")}
>
Backing Vocals
</Button>
<Button
variant={activeAudio === "clean" ? "default" : "outline"}
size="sm"
onClick={() => setActiveAudio("clean")}
>
Clean Instrumental
</Button>
<Button
variant={activeAudio === "with_backing" ? "default" : "outline"}
size="sm"
onClick={() => setActiveAudio("with_backing")}
>
With Backing
</Button>
{analysisData?.has_custom_instrumental && (
<Button
variant={activeAudio === "custom" ? "default" : "outline"}
size="sm"
onClick={() => setActiveAudio("custom")}
>
Custom
</Button>
)}
</div>
{/* Audio player */}
{getCurrentAudioUrl() && (
<AudioPlayer
src={getCurrentAudioUrl()!}
onTimeUpdate={setCurrentTime}
seekTo={seekTo}
label={
activeAudio === "backing"
? "Backing Vocals Only"
: activeAudio === "clean"
? "Clean Instrumental"
: activeAudio === "custom"
? "Custom Instrumental"
: "Instrumental + Backing Vocals"
}
/>
)}
</CardContent>
</Card>
{/* Region Selector */}
<RegionSelector
muteRegions={muteRegions}
audibleSegments={analysis?.audible_segments}
onAddRegion={handleAddRegion}
onRemoveRegion={handleRemoveRegion}
onClearAll={handleClearAll}
onSeekTo={handleSeek}
isSelectionMode={isSelectionMode}
onToggleSelectionMode={() => setIsSelectionMode(!isSelectionMode)}
/>
</div>
{/* Create Custom Button */}
{muteRegions.length > 0 && !analysisData?.has_custom_instrumental && (
<Card className="mb-6">
<CardContent className="py-4">
<div className="flex items-center justify-between">
<div>
<p className="font-medium">Ready to create custom instrumental</p>
<p className="text-sm text-muted-foreground">
{muteRegions.length} region{muteRegions.length > 1 ? "s" : ""} will be muted
</p>
</div>
<Button onClick={handleCreateCustom} disabled={isCreatingCustom}>
{isCreatingCustom ? (
<>
<Loader2 className="h-4 w-4 mr-2 animate-spin" />
Creating...
</>
) : (
"Create Custom Instrumental"
)}
</Button>
</div>
</CardContent>
</Card>
)}
{/* Selection Options */}
<Card className="mb-6">
<CardHeader>
<CardTitle className="text-lg">Final Selection</CardTitle>
<CardDescription>
Choose which instrumental to use for your karaoke video
</CardDescription>
</CardHeader>
<CardContent>
<RadioGroup value={selectedOption} onValueChange={setSelectedOption}>
<div className="space-y-4">
<div className="flex items-center space-x-3">
<RadioGroupItem value="clean" id="clean" />
<Label htmlFor="clean" className="flex-1 cursor-pointer">
<div className="font-medium">Clean Instrumental</div>
<div className="text-sm text-muted-foreground">
Use the instrumental with no backing vocals at all
</div>
</Label>
{analysis?.recommended_selection === "clean" && (
<Badge variant="default" className="bg-green-600">
<CheckCircle2 className="h-3 w-3 mr-1" />
Recommended
</Badge>
)}
</div>
<div className="flex items-center space-x-3">
<RadioGroupItem value="with_backing" id="with_backing" />
<Label htmlFor="with_backing" className="flex-1 cursor-pointer">
<div className="font-medium">Instrumental with Backing Vocals</div>
<div className="text-sm text-muted-foreground">
Use the instrumental with all backing vocals included
</div>
</Label>
</div>
{analysisData?.has_custom_instrumental && (
<div className="flex items-center space-x-3">
<RadioGroupItem value="custom" id="custom" />
<Label htmlFor="custom" className="flex-1 cursor-pointer">
<div className="font-medium">Custom Instrumental</div>
<div className="text-sm text-muted-foreground">
Use your custom instrumental with {muteRegions.length} muted region{muteRegions.length > 1 ? "s" : ""}
</div>
</Label>
<Badge variant="secondary">Custom</Badge>
</div>
)}
</div>
</RadioGroup>
</CardContent>
</Card>
{/* Submit Button */}
<div className="flex gap-4">
<Button
size="lg"
onClick={handleSubmit}
disabled={isSubmitting}
className="flex-1 max-w-md"
>
{isSubmitting ? (
<>
<Loader2 className="h-4 w-4 mr-2 animate-spin" />
Submitting...
</>
) : (
<>
<CheckCircle2 className="h-4 w-4 mr-2" />
Confirm Selection & Continue
</>
)}
</Button>
</div>
</main>
</div>
)
}
export default function InstrumentalReviewPage() {
return (
<ProtectedRoute>
<InstrumentalReviewContent />
</ProtectedRoute>
)
}
-216
View File
@@ -1,216 +0,0 @@
"use client"
import { useParams, useRouter } from "next/navigation"
import { useJobs } from "@/lib/jobs"
import { AppHeader } from "@/components/app-header"
import { ProtectedRoute } from "@/components/protected-route"
import { Button } from "@/components/ui/button"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import { Progress } from "@/components/ui/progress"
import { StatusBadge } from "@/components/status-badge"
import { Alert, AlertDescription } from "@/components/ui/alert"
import {
ArrowLeft,
Download,
Youtube,
FileAudio,
Calendar,
Clock,
CheckCircle2,
Loader2,
XCircle,
AlertCircle,
} from "lucide-react"
function JobDetailContent() {
const params = useParams()
const router = useRouter()
const { getJobById } = useJobs()
const job = getJobById(params.id as string)
if (!job) {
return (
<div className="min-h-screen bg-background">
<AppHeader />
<main className="container mx-auto px-4 py-8">
<Alert variant="destructive">
<AlertCircle className="h-4 w-4" />
<AlertDescription>Job not found</AlertDescription>
</Alert>
<Button variant="outline" className="mt-4 bg-transparent" onClick={() => router.push("/jobs")}>
<ArrowLeft className="w-4 h-4 mr-2" />
Back to Jobs
</Button>
</main>
</div>
)
}
const getStageIcon = (status: string) => {
switch (status) {
case "completed":
return <CheckCircle2 className="w-5 h-5 text-green-400" />
case "in_progress":
return <Loader2 className="w-5 h-5 text-blue-400 animate-spin" />
case "failed":
return <XCircle className="w-5 h-5 text-destructive" />
default:
return <div className="w-5 h-5 rounded-full border-2 border-muted" />
}
}
return (
<div className="min-h-screen bg-background">
<AppHeader />
<main className="container mx-auto px-4 py-8 max-w-5xl">
<Button variant="ghost" className="mb-6" onClick={() => router.push("/jobs")}>
<ArrowLeft className="w-4 h-4 mr-2" />
Back to Jobs
</Button>
{/* Header */}
<div className="mb-8">
<div className="flex items-start justify-between gap-4 mb-4">
<div className="flex-1">
<h1 className="text-4xl font-bold mb-2 text-balance">
{job.artist} - {job.title}
</h1>
<div className="flex flex-wrap items-center gap-4 text-muted-foreground">
<span className="flex items-center gap-1.5">
<Calendar className="w-4 h-4" />
Created {new Date(job.createdAt).toLocaleDateString()}
</span>
<span className="flex items-center gap-1.5">
<Clock className="w-4 h-4" />
Updated {new Date(job.updatedAt).toLocaleDateString()}
</span>
</div>
</div>
<StatusBadge status={job.status} size="lg" />
</div>
{job.status === "completed" && (
<Button size="lg" className="bg-primary hover:bg-primary/90">
<Download className="w-5 h-5 mr-2" />
Download Karaoke Video
</Button>
)}
</div>
{/* Source Info */}
<Card className="mb-6">
<CardHeader>
<CardTitle className="text-lg">Source Information</CardTitle>
</CardHeader>
<CardContent>
<div className="flex items-center gap-3">
{job.sourceType === "youtube" ? (
<>
<div className="w-10 h-10 rounded-lg bg-red-500/10 flex items-center justify-center">
<Youtube className="w-5 h-5 text-red-500" />
</div>
<div className="flex-1 min-w-0">
<p className="text-sm font-medium">YouTube Video</p>
<p className="text-sm text-muted-foreground truncate">{job.sourceUrl}</p>
</div>
</>
) : (
<>
<div className="w-10 h-10 rounded-lg bg-primary/10 flex items-center justify-center">
<FileAudio className="w-5 h-5 text-primary" />
</div>
<div className="flex-1 min-w-0">
<p className="text-sm font-medium">Uploaded Audio File</p>
<p className="text-sm text-muted-foreground truncate">{job.fileName}</p>
</div>
</>
)}
</div>
</CardContent>
</Card>
{/* Progress Section */}
<Card className="mb-6">
<CardHeader>
<CardTitle className="text-lg">Processing Pipeline</CardTitle>
<CardDescription>Track the progress of your karaoke video generation</CardDescription>
</CardHeader>
<CardContent>
<div className="space-y-6">
{job.stages.map((stage, index) => (
<div key={index} className="flex items-start gap-4">
<div className="pt-1">{getStageIcon(stage.status)}</div>
<div className="flex-1">
<div className="flex items-center justify-between mb-2">
<h3 className="font-medium">{stage.name}</h3>
{stage.status === "in_progress" && (
<span className="text-sm text-muted-foreground">{stage.progress}%</span>
)}
</div>
{stage.status === "in_progress" && <Progress value={stage.progress} className="h-2 bg-muted" />}
{stage.message && <p className="text-sm text-muted-foreground mt-1">{stage.message}</p>}
</div>
</div>
))}
</div>
{job.status === "processing" && (
<div className="mt-6 pt-6 border-t border-border">
<div className="flex items-center justify-between mb-2">
<p className="text-sm font-medium">Overall Progress</p>
<span className="text-sm text-muted-foreground">{job.progress}%</span>
</div>
<Progress value={job.progress} className="h-3" />
</div>
)}
</CardContent>
</Card>
{/* Error Message */}
{job.status === "failed" && job.errorMessage && (
<Alert variant="destructive">
<AlertCircle className="h-4 w-4" />
<AlertDescription>{job.errorMessage}</AlertDescription>
</Alert>
)}
{/* Action Alerts */}
{job.status === "awaiting_review" && (
<Alert className="bg-secondary/10 border-secondary/30">
<AlertCircle className="h-4 w-4 text-secondary" />
<AlertDescription className="text-foreground">
Your karaoke video is ready for review. Please check the lyrics synchronization and approve or request
changes.
</AlertDescription>
</Alert>
)}
{job.status === "awaiting_instrumental" && (
<Alert className="bg-secondary/10 border-secondary/30">
<AlertCircle className="h-4 w-4 text-secondary" />
<AlertDescription className="text-foreground flex items-center justify-between">
<span>Multiple instrumental versions were detected. Please select your preferred version to continue.</span>
<Button
size="sm"
onClick={() => router.push(`/jobs/${job.id}/instrumental-review`)}
className="ml-4"
>
Select Instrumental
</Button>
</AlertDescription>
</Alert>
)}
</main>
</div>
)
}
export default function JobDetailPage() {
return (
<ProtectedRoute>
<JobDetailContent />
</ProtectedRoute>
)
}
-3
View File
@@ -1,3 +0,0 @@
export default function Loading() {
return null
}
-366
View File
@@ -1,366 +0,0 @@
"use client"
import type React from "react"
import { useState } from "react"
import { useRouter } from "next/navigation"
import { useAuth } from "@/lib/auth"
import { useJobs } from "@/lib/jobs"
import { AppHeader } from "@/components/app-header"
import { ProtectedRoute } from "@/components/protected-route"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Label } from "@/components/ui/label"
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import { Alert, AlertDescription } from "@/components/ui/alert"
import { Youtube, Upload, AlertCircle, Sparkles } from "lucide-react"
function NewJobContent() {
const { user, updateCredits } = useAuth()
const { addJob } = useJobs()
const router = useRouter()
const [activeTab, setActiveTab] = useState("youtube")
// YouTube form state
const [youtubeUrl, setYoutubeUrl] = useState("")
const [youtubeArtist, setYoutubeArtist] = useState("")
const [youtubeTitle, setYoutubeTitle] = useState("")
// Upload form state
const [uploadFile, setUploadFile] = useState<File | null>(null)
const [uploadArtist, setUploadArtist] = useState("")
const [uploadTitle, setUploadTitle] = useState("")
const [isSubmitting, setIsSubmitting] = useState(false)
const [error, setError] = useState("")
if (!user) return null
const insufficientCredits = user.credits < 1
const handleYoutubeSubmit = async (e: React.FormEvent) => {
e.preventDefault()
setError("")
if (!youtubeUrl.trim()) {
setError("Please enter a YouTube URL")
return
}
if (!youtubeArtist.trim() || !youtubeTitle.trim()) {
setError("Please enter both artist and title")
return
}
if (insufficientCredits) {
setError("Insufficient credits. Please purchase more credits to continue.")
return
}
setIsSubmitting(true)
// Simulate processing
setTimeout(() => {
addJob({
userId: user.token.includes("demo") ? "demo" : user.token,
artist: youtubeArtist,
title: youtubeTitle,
status: "queued",
sourceType: "youtube",
sourceUrl: youtubeUrl,
})
updateCredits(user.credits - 1)
setIsSubmitting(false)
router.push("/jobs")
}, 1000)
}
const handleUploadSubmit = async (e: React.FormEvent) => {
e.preventDefault()
setError("")
if (!uploadFile) {
setError("Please select an audio file")
return
}
if (!uploadArtist.trim() || !uploadTitle.trim()) {
setError("Please enter both artist and title")
return
}
if (insufficientCredits) {
setError("Insufficient credits. Please purchase more credits to continue.")
return
}
setIsSubmitting(true)
// Simulate processing
setTimeout(() => {
addJob({
userId: user.token.includes("demo") ? "demo" : user.token,
artist: uploadArtist,
title: uploadTitle,
status: "queued",
sourceType: "upload",
fileName: uploadFile.name,
})
updateCredits(user.credits - 1)
setIsSubmitting(false)
router.push("/jobs")
}, 1000)
}
const handleFileChange = (e: React.ChangeEvent<HTMLInputElement>) => {
const file = e.target.files?.[0]
if (file) {
const validTypes = ["audio/mpeg", "audio/wav", "audio/flac", "audio/mp4", "audio/ogg"]
if (!validTypes.includes(file.type) && !file.name.match(/\.(mp3|wav|flac|m4a|ogg)$/i)) {
setError("Invalid file type. Please upload MP3, WAV, FLAC, M4A, or OGG files.")
return
}
if (file.size > 50 * 1024 * 1024) {
// 50MB limit
setError("File too large. Maximum size is 50MB.")
return
}
setUploadFile(file)
setError("")
}
}
return (
<div className="min-h-screen bg-background">
<AppHeader />
<main className="container mx-auto px-4 py-8 max-w-3xl">
<div className="mb-8">
<h1 className="text-4xl font-bold mb-2">Create New Karaoke Video</h1>
<p className="text-muted-foreground">Start from a YouTube URL or upload your own audio file</p>
</div>
{insufficientCredits && (
<Alert variant="destructive" className="mb-6">
<AlertCircle className="h-4 w-4" />
<AlertDescription>
You don't have enough credits to create a new job.{" "}
<a href="#" className="underline font-medium">
Purchase more credits
</a>{" "}
to continue.
</AlertDescription>
</Alert>
)}
<Card className="border-border/50">
<CardHeader>
<div className="flex items-center justify-between">
<div>
<CardTitle>Source Selection</CardTitle>
<CardDescription>Choose how you want to provide the audio</CardDescription>
</div>
<div className="flex items-center gap-2 px-3 py-1.5 rounded-lg bg-primary/10 border border-primary/20">
<Sparkles className="w-4 h-4 text-primary" />
<span className="text-sm font-medium text-primary">Cost: 1 credit</span>
</div>
</div>
</CardHeader>
<CardContent>
<Tabs value={activeTab} onValueChange={setActiveTab}>
<TabsList className="grid w-full grid-cols-2">
<TabsTrigger value="youtube" className="gap-2">
<Youtube className="w-4 h-4" />
From YouTube URL
</TabsTrigger>
<TabsTrigger value="upload" className="gap-2">
<Upload className="w-4 h-4" />
Upload Audio File
</TabsTrigger>
</TabsList>
<TabsContent value="youtube" className="mt-6">
<form onSubmit={handleYoutubeSubmit} className="space-y-6">
<div className="space-y-2">
<Label htmlFor="youtube-url">YouTube URL</Label>
<div className="relative">
<Youtube className="absolute left-3 top-1/2 -translate-y-1/2 w-4 h-4 text-muted-foreground" />
<Input
id="youtube-url"
type="url"
placeholder="https://youtube.com/watch?v=..."
value={youtubeUrl}
onChange={(e) => setYoutubeUrl(e.target.value)}
className="pl-10"
disabled={isSubmitting}
/>
</div>
<p className="text-xs text-muted-foreground">
Paste the URL of the YouTube video you want to convert
</p>
</div>
<div className="grid md:grid-cols-2 gap-4">
<div className="space-y-2">
<Label htmlFor="youtube-artist">Artist Name</Label>
<Input
id="youtube-artist"
type="text"
placeholder="Artist name"
value={youtubeArtist}
onChange={(e) => setYoutubeArtist(e.target.value)}
disabled={isSubmitting}
/>
</div>
<div className="space-y-2">
<Label htmlFor="youtube-title">Song Title</Label>
<Input
id="youtube-title"
type="text"
placeholder="Song title"
value={youtubeTitle}
onChange={(e) => setYoutubeTitle(e.target.value)}
disabled={isSubmitting}
/>
</div>
</div>
{error && (
<Alert variant="destructive">
<AlertCircle className="h-4 w-4" />
<AlertDescription>{error}</AlertDescription>
</Alert>
)}
<div className="flex gap-3">
<Button
type="submit"
className="flex-1 bg-primary hover:bg-primary/90"
disabled={isSubmitting || insufficientCredits}
>
{isSubmitting ? "Creating Job..." : "Create Karaoke Video"}
</Button>
<Button
type="button"
variant="outline"
onClick={() => router.push("/dashboard")}
disabled={isSubmitting}
>
Cancel
</Button>
</div>
</form>
</TabsContent>
<TabsContent value="upload" className="mt-6">
<form onSubmit={handleUploadSubmit} className="space-y-6">
<div className="space-y-2">
<Label htmlFor="audio-file">Audio File</Label>
<div
className={`relative border-2 border-dashed rounded-lg p-8 text-center transition-colors ${
uploadFile
? "border-primary bg-primary/5"
: "border-border hover:border-primary/50 hover:bg-card/50"
}`}
>
<Input
id="audio-file"
type="file"
accept=".mp3,.wav,.flac,.m4a,.ogg,audio/*"
onChange={handleFileChange}
className="absolute inset-0 w-full h-full opacity-0 cursor-pointer"
disabled={isSubmitting}
/>
<Upload className="w-8 h-8 mx-auto mb-2 text-muted-foreground" />
{uploadFile ? (
<div>
<p className="font-medium text-foreground">{uploadFile.name}</p>
<p className="text-sm text-muted-foreground">
{(uploadFile.size / 1024 / 1024).toFixed(2)} MB
</p>
</div>
) : (
<div>
<p className="font-medium text-foreground mb-1">Click to upload or drag and drop</p>
<p className="text-sm text-muted-foreground">MP3, WAV, FLAC, M4A, or OGG (max 50MB)</p>
</div>
)}
</div>
</div>
<div className="grid md:grid-cols-2 gap-4">
<div className="space-y-2">
<Label htmlFor="upload-artist">
Artist Name <span className="text-destructive">*</span>
</Label>
<Input
id="upload-artist"
type="text"
placeholder="Artist name"
value={uploadArtist}
onChange={(e) => setUploadArtist(e.target.value)}
required
disabled={isSubmitting}
/>
</div>
<div className="space-y-2">
<Label htmlFor="upload-title">
Song Title <span className="text-destructive">*</span>
</Label>
<Input
id="upload-title"
type="text"
placeholder="Song title"
value={uploadTitle}
onChange={(e) => setUploadTitle(e.target.value)}
required
disabled={isSubmitting}
/>
</div>
</div>
{error && (
<Alert variant="destructive">
<AlertCircle className="h-4 w-4" />
<AlertDescription>{error}</AlertDescription>
</Alert>
)}
<div className="flex gap-3">
<Button
type="submit"
className="flex-1 bg-primary hover:bg-primary/90"
disabled={isSubmitting || insufficientCredits}
>
{isSubmitting ? "Creating Job..." : "Create Karaoke Video"}
</Button>
<Button
type="button"
variant="outline"
onClick={() => router.push("/dashboard")}
disabled={isSubmitting}
>
Cancel
</Button>
</div>
</form>
</TabsContent>
</Tabs>
</CardContent>
</Card>
</main>
</div>
)
}
export default function NewJobPage() {
return (
<ProtectedRoute>
<NewJobContent />
</ProtectedRoute>
)
}
-193
View File
@@ -1,193 +0,0 @@
"use client"
import { useState } from "react"
import { useAuth } from "@/lib/auth"
import { useJobs } from "@/lib/jobs"
import type { JobStatus } from "@/lib/types"
import { AppHeader } from "@/components/app-header"
import { ProtectedRoute } from "@/components/protected-route"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Card, CardContent } from "@/components/ui/card"
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select"
import { StatusBadge } from "@/components/status-badge"
import { Plus, Search, Clock, Download, ChevronRight } from "lucide-react"
import Link from "next/link"
function JobsListContent() {
const { user } = useAuth()
const { jobs, getUserJobs } = useJobs()
const [searchQuery, setSearchQuery] = useState("")
const [statusFilter, setStatusFilter] = useState<JobStatus | "all">("all")
if (!user) return null
const userJobs = getUserJobs(user.token.includes("demo") ? "demo" : user.token)
const filteredJobs = userJobs.filter((job) => {
const matchesSearch =
searchQuery === "" ||
job.artist.toLowerCase().includes(searchQuery.toLowerCase()) ||
job.title.toLowerCase().includes(searchQuery.toLowerCase())
const matchesStatus = statusFilter === "all" || job.status === statusFilter
return matchesSearch && matchesStatus
})
return (
<div className="min-h-screen bg-background">
<AppHeader />
<main className="container mx-auto px-4 py-8">
<div className="flex items-center justify-between mb-8">
<div>
<h1 className="text-4xl font-bold mb-2">My Jobs</h1>
<p className="text-muted-foreground">Manage and track your karaoke video projects</p>
</div>
<Button asChild className="bg-primary hover:bg-primary/90">
<Link href="/jobs/new">
<Plus className="w-4 h-4 mr-2" />
New Job
</Link>
</Button>
</div>
{/* Filters */}
<Card className="mb-6">
<CardContent className="pt-6">
<div className="flex flex-col sm:flex-row gap-4">
<div className="flex-1 relative">
<Search className="absolute left-3 top-1/2 -translate-y-1/2 w-4 h-4 text-muted-foreground" />
<Input
placeholder="Search by artist or title..."
value={searchQuery}
onChange={(e) => setSearchQuery(e.target.value)}
className="pl-10"
/>
</div>
<Select value={statusFilter} onValueChange={(value) => setStatusFilter(value as JobStatus | "all")}>
<SelectTrigger className="w-full sm:w-[200px]">
<SelectValue placeholder="Filter by status" />
</SelectTrigger>
<SelectContent>
<SelectItem value="all">All Status</SelectItem>
<SelectItem value="queued">Queued</SelectItem>
<SelectItem value="processing">Processing</SelectItem>
<SelectItem value="awaiting_review">Awaiting Review</SelectItem>
<SelectItem value="awaiting_instrumental">Select Instrumental</SelectItem>
<SelectItem value="completed">Completed</SelectItem>
<SelectItem value="failed">Failed</SelectItem>
</SelectContent>
</Select>
</div>
</CardContent>
</Card>
{/* Results count */}
<div className="mb-4">
<p className="text-sm text-muted-foreground">
Showing {filteredJobs.length} of {userJobs.length} jobs
</p>
</div>
{/* Jobs List */}
{filteredJobs.length === 0 ? (
<Card>
<CardContent className="text-center py-12">
<div className="w-16 h-16 mx-auto mb-4 rounded-full bg-muted flex items-center justify-center">
<Search className="w-8 h-8 text-muted-foreground" />
</div>
<h3 className="font-semibold mb-2">No jobs found</h3>
<p className="text-muted-foreground mb-4">
{searchQuery || statusFilter !== "all"
? "Try adjusting your filters"
: "Create your first karaoke video to get started"}
</p>
{!searchQuery && statusFilter === "all" && (
<Button asChild>
<Link href="/jobs/new">
<Plus className="w-4 h-4 mr-2" />
Create Job
</Link>
</Button>
)}
</CardContent>
</Card>
) : (
<div className="space-y-3">
{filteredJobs.map((job) => (
<Link key={job.id} href={`/jobs/${job.id}`}>
<Card className="hover:border-primary/50 hover:bg-card/80 transition-all group cursor-pointer">
<CardContent className="p-6">
<div className="flex items-center justify-between gap-4">
<div className="flex-1 min-w-0">
<div className="flex items-center gap-3 mb-2">
<h3 className="font-semibold text-lg group-hover:text-primary transition-colors truncate">
{job.artist} - {job.title}
</h3>
<StatusBadge status={job.status} />
</div>
<div className="flex flex-wrap items-center gap-4 text-sm text-muted-foreground">
<span className="flex items-center gap-1.5">
<Clock className="w-3.5 h-3.5" />
{new Date(job.createdAt).toLocaleDateString("en-US", {
month: "short",
day: "numeric",
year: "numeric",
})}
</span>
<span className="text-xs">
{job.sourceType === "youtube" ? "From YouTube" : "Uploaded File"}
</span>
{(job.status === "processing" || job.status === "queued") && (
<span className="text-xs font-medium">{job.progress}% complete</span>
)}
</div>
{(job.status === "processing" || job.status === "queued") && (
<div className="mt-3 w-full max-w-xs">
<div className="h-1.5 bg-muted rounded-full overflow-hidden">
<div
className="h-full bg-primary transition-all duration-300"
style={{ width: `${job.progress}%` }}
/>
</div>
</div>
)}
</div>
<div className="flex items-center gap-2">
{job.status === "completed" && (
<Button size="sm" variant="outline" onClick={(e) => e.preventDefault()}>
<Download className="w-4 h-4 mr-2" />
Download
</Button>
)}
<ChevronRight className="w-5 h-5 text-muted-foreground group-hover:text-primary transition-colors" />
</div>
</div>
</CardContent>
</Card>
</Link>
))}
</div>
)}
</main>
</div>
)
}
export default function JobsPage() {
return (
<ProtectedRoute>
<JobsListContent />
</ProtectedRoute>
)
}
+4 -15
View File
@@ -1,16 +1,10 @@
import type React from "react"
import type { Metadata } from "next"
import { Geist, Geist_Mono } from "next/font/google"
import { Analytics } from "@vercel/analytics/next"
import "./globals.css"
const _geist = Geist({ subsets: ["latin"] })
const _geistMono = Geist_Mono({ subsets: ["latin"] })
export const metadata: Metadata = {
title: "Nomad Karaoke - Wherever you are, sing!",
description: "Cloud-based karaoke video generation service",
generator: "v0.app",
title: "Karaoke Generator - Nomad Karaoke",
description: "Generate professional karaoke videos with AI-powered vocal separation and synchronized lyrics",
icons: {
icon: [
{
@@ -21,10 +15,6 @@ export const metadata: Metadata = {
url: "/icon-dark-32x32.png",
media: "(prefers-color-scheme: dark)",
},
{
url: "/icon.svg",
type: "image/svg+xml",
},
],
apple: "/apple-icon.png",
},
@@ -36,10 +26,9 @@ export default function RootLayout({
children: React.ReactNode
}>) {
return (
<html lang="en">
<body className={`font-sans antialiased`}>
<html lang="en" className="dark">
<body className="font-sans antialiased" style={{ background: 'var(--bg)', color: 'var(--text)' }}>
{children}
<Analytics />
</body>
</html>
)
+102 -125
View File
@@ -1,143 +1,120 @@
"use client"
import type React from "react"
import { useState } from "react"
import { useRouter } from "next/navigation"
import { useAuth } from "@/lib/auth"
import { Logo } from "@/components/logo"
import { useState, useEffect } from "react"
import { api, Job } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Label } from "@/components/ui/label"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
import { Sparkles } from "lucide-react"
import { Music2, RefreshCw, Loader2 } from "lucide-react"
import { JobCard } from "@/components/job"
import { JobSubmission } from "@/components/job/JobSubmission"
import { AuthBanner } from "@/components/auth-banner"
export default function LandingPage() {
const [token, setToken] = useState("")
const [error, setError] = useState("")
const [isLoading, setIsLoading] = useState(false)
const { login } = useAuth()
const router = useRouter()
export default function HomePage() {
const [jobs, setJobs] = useState<Job[]>([])
const [isLoadingJobs, setIsLoadingJobs] = useState(true)
const handleLogin = async (e: React.FormEvent) => {
e.preventDefault()
setError("")
setIsLoading(true)
// Load jobs on mount
useEffect(() => {
loadJobs()
// Poll for updates every 10 seconds
const interval = setInterval(loadJobs, 10000)
return () => clearInterval(interval)
}, [])
if (!token.trim()) {
setError("Please enter your access token")
setIsLoading(false)
return
}
const success = await login(token)
setIsLoading(false)
if (success) {
router.push("/dashboard")
} else {
setError("Invalid token. Please try again.")
async function loadJobs() {
try {
const data = await api.listJobs({ limit: 20 })
setJobs(data)
} catch (err) {
console.error("Failed to load jobs:", err)
} finally {
setIsLoadingJobs(false)
}
}
return (
<div className="min-h-screen bg-gradient-to-b from-background via-background to-card">
{/* Animated background effects */}
<div className="fixed inset-0 -z-10 overflow-hidden">
<div className="absolute top-1/4 left-1/4 w-96 h-96 bg-primary/10 rounded-full blur-3xl animate-pulse" />
<div className="absolute bottom-1/4 right-1/4 w-96 h-96 bg-secondary/10 rounded-full blur-3xl animate-pulse delay-1000" />
</div>
<div className="container mx-auto px-4 py-8 lg:py-12">
{/* Hero Section - horizontal layout on desktop */}
<div className="max-w-6xl mx-auto">
<div className="grid lg:grid-cols-2 gap-8 lg:gap-12 items-center">
{/* Left: Hero Content */}
<div className="text-center lg:text-left">
<Logo size="lg" className="mb-8 justify-center lg:justify-start" />
<div className="inline-flex items-center gap-2 px-4 py-2 rounded-full bg-primary/10 border border-primary/20 mb-6">
<Sparkles className="w-4 h-4 text-primary" />
<span className="text-sm text-primary font-medium">AI-Powered Karaoke Generation</span>
</div>
<h1 className="text-4xl md:text-4xl lg:text-4xl font-bold mb-4 text-balance leading-tight">
Wherever you are, <span className="text-primary">sing!</span>
</h1>
<p className="text-base md:text-lg text-muted-foreground text-pretty leading-relaxed">
Transform any song into professional karaoke videos with AI-powered vocal separation, synchronized
lyrics, and stunning visuals.
</p>
</div>
{/* Right: Login Card */}
<div className="min-h-screen bg-gradient-to-br from-slate-950 via-slate-900 to-slate-950">
{/* Header */}
<header className="border-b border-slate-800 bg-slate-950/50 backdrop-blur-sm sticky top-0 z-10">
<div className="container mx-auto px-4 py-4 flex items-center justify-between">
<div className="flex items-center gap-3">
<Music2 className="w-8 h-8 text-amber-500" />
<div>
<Card className="border-border/50 backdrop-blur">
<CardHeader>
<CardTitle className="text-2xl">Get Started</CardTitle>
<CardDescription>Enter your access token to begin creating karaoke videos</CardDescription>
</CardHeader>
<CardContent>
<form onSubmit={handleLogin} className="space-y-4">
<div className="space-y-2">
<Label htmlFor="token">Access Token</Label>
<Input
id="token"
type="text"
placeholder="Enter your token"
value={token}
onChange={(e) => setToken(e.target.value)}
className="bg-background"
/>
{error && <p className="text-sm text-destructive">{error}</p>}
</div>
<Button type="submit" className="w-full bg-primary hover:bg-primary/90" disabled={isLoading}>
{isLoading ? "Logging in..." : "Login"}
</Button>
<div className="text-center text-sm text-muted-foreground">
Don't have access?{" "}
<a href="#" className="text-primary hover:underline">
Request a token
</a>{" "}
or{" "}
<a href="#" className="text-primary hover:underline">
purchase access
</a>
</div>
<div className="pt-4 border-t border-border">
<p className="text-xs text-muted-foreground mb-2">Demo tokens:</p>
<div className="flex gap-2">
<Button
type="button"
variant="outline"
size="sm"
className="flex-1 text-xs bg-transparent"
onClick={() => setToken("demo-user-token")}
>
User Demo
</Button>
<Button
type="button"
variant="outline"
size="sm"
className="flex-1 text-xs bg-transparent"
onClick={() => setToken("demo-admin-token")}
>
Admin Demo
</Button>
</div>
</div>
</form>
</CardContent>
</Card>
<h1 className="text-xl font-bold text-white">Karaoke Generator</h1>
<p className="text-xs text-slate-400">by Nomad Karaoke</p>
</div>
</div>
<Button
variant="ghost"
size="sm"
onClick={loadJobs}
disabled={isLoadingJobs}
className="text-slate-400 hover:text-white"
>
<RefreshCw className={`w-4 h-4 mr-2 ${isLoadingJobs ? 'animate-spin' : ''}`} />
Refresh
</Button>
</div>
</div>
</header>
<main className="container mx-auto px-4 py-8 max-w-5xl space-y-6">
{/* Authentication Banner */}
<AuthBanner />
<div className="grid lg:grid-cols-2 gap-8">
{/* Submit Job Card */}
<Card className="border-slate-800 bg-slate-900/50 backdrop-blur">
<CardHeader>
<CardTitle className="text-white">Create Karaoke Video</CardTitle>
<CardDescription className="text-slate-400">
Upload an audio file, provide a YouTube URL, or search for audio by artist & title
</CardDescription>
</CardHeader>
<CardContent>
<JobSubmission onJobCreated={loadJobs} />
</CardContent>
</Card>
{/* Jobs List Card */}
<Card className="border-slate-800 bg-slate-900/50 backdrop-blur">
<CardHeader>
<CardTitle className="text-white flex items-center justify-between">
Recent Jobs
{isLoadingJobs && <Loader2 className="w-4 h-4 animate-spin text-slate-400" />}
</CardTitle>
<CardDescription className="text-slate-400">
{jobs.length} job{jobs.length !== 1 ? 's' : ''} found
</CardDescription>
</CardHeader>
<CardContent>
{jobs.length === 0 ? (
<div className="text-center py-8 text-slate-500">
<Music2 className="w-12 h-12 mx-auto mb-3 opacity-50" />
<p>No jobs yet. Create one to get started!</p>
</div>
) : (
<div className="space-y-3 max-h-[500px] overflow-y-auto pr-2">
{jobs.map((job) => (
<JobCard key={job.job_id} job={job} onRefresh={loadJobs} />
))}
</div>
)}
</CardContent>
</Card>
</div>
</main>
{/* Footer */}
<footer className="border-t border-slate-800 mt-12 py-6">
<div className="container mx-auto px-4 text-center text-sm text-slate-500">
<p>
Powered by{" "}
<a href="https://github.com/nomadkaraoke/karaoke-gen" className="text-amber-500 hover:underline">
karaoke-gen
</a>
</p>
</div>
</footer>
</div>
)
}
@@ -0,0 +1,135 @@
/**
* Tests for JobCard component
*/
import { render, screen, fireEvent } from '@testing-library/react'
import { JobCard } from '../job/JobCard'
import { Job } from '@/lib/api'
// Mock the child components
jest.mock('../job/JobActions', () => ({
JobActions: () => <div data-testid="job-actions">Job Actions</div>
}))
jest.mock('../job/JobLogs', () => ({
JobLogs: () => <div data-testid="job-logs">Job Logs</div>
}))
jest.mock('../job/OutputLinks', () => ({
OutputLinks: () => <div data-testid="output-links">Output Links</div>
}))
jest.mock('../job/InstrumentalSelector', () => ({
InstrumentalSelector: () => <div data-testid="instrumental-selector">Instrumental Selector</div>
}))
jest.mock('../audio-search/AudioSearchDialog', () => ({
AudioSearchDialog: () => <div data-testid="audio-search-dialog">Audio Search Dialog</div>
}))
describe('JobCard', () => {
const mockOnRefresh = jest.fn()
const mockJob: Job = {
job_id: '123',
status: 'processing',
progress: 50,
created_at: '2025-01-01T00:00:00Z',
updated_at: '2025-01-01T00:00:00Z',
artist: 'Test Artist',
title: 'Test Song',
}
beforeEach(() => {
jest.clearAllMocks()
})
it('renders job information', () => {
render(<JobCard job={mockJob} onRefresh={mockOnRefresh} />)
expect(screen.getByText('Test Artist - Test Song')).toBeInTheDocument()
expect(screen.getByText(/Job ID:/)).toBeInTheDocument()
})
it('shows progress bar for processing jobs', () => {
render(<JobCard job={mockJob} onRefresh={mockOnRefresh} />)
// Expand details
fireEvent.click(screen.getByText('Test Artist - Test Song'))
const progressBar = document.querySelector('[style*="width: 50%"]')
expect(progressBar).toBeInTheDocument()
})
it('shows error message for failed jobs', () => {
const failedJob = { ...mockJob, status: 'failed', error_message: 'Something went wrong' }
render(<JobCard job={failedJob} onRefresh={mockOnRefresh} />)
// Expand details
fireEvent.click(screen.getByText('Test Artist - Test Song'))
expect(screen.getByText('Something went wrong')).toBeInTheDocument()
})
it('shows lyrics review button for awaiting_review status', () => {
const reviewJob = { ...mockJob, status: 'awaiting_review' }
render(<JobCard job={reviewJob} onRefresh={mockOnRefresh} />)
// Expand details
fireEvent.click(screen.getByText('Test Artist - Test Song'))
const reviewLink = screen.getByText('Review Lyrics')
expect(reviewLink).toBeInTheDocument()
expect(reviewLink.closest('a')).toHaveAttribute('href', expect.stringContaining('lyrics.nomadkaraoke.com'))
})
it('shows audio search button for awaiting_audio_selection status', () => {
const audioSelectionJob = { ...mockJob, status: 'awaiting_audio_selection' }
render(<JobCard job={audioSelectionJob} onRefresh={mockOnRefresh} />)
// Expand details
fireEvent.click(screen.getByText('Test Artist - Test Song'))
expect(screen.getByText('Select Audio Source')).toBeInTheDocument()
})
it('shows instrumental selection button for awaiting_instrumental_selection status', () => {
const instrumentalJob = { ...mockJob, status: 'awaiting_instrumental_selection' }
render(<JobCard job={instrumentalJob} onRefresh={mockOnRefresh} />)
// Expand details
fireEvent.click(screen.getByText('Test Artist - Test Song'))
expect(screen.getByText('Select Instrumental')).toBeInTheDocument()
})
it('shows output links for complete jobs', () => {
const completeJob = { ...mockJob, status: 'complete' }
render(<JobCard job={completeJob} onRefresh={mockOnRefresh} />)
// Expand details
fireEvent.click(screen.getByText('Test Artist - Test Song'))
expect(screen.getByTestId('output-links')).toBeInTheDocument()
})
it('toggles details on click', () => {
render(<JobCard job={mockJob} onRefresh={mockOnRefresh} />)
// Should not show actions initially
expect(screen.queryByTestId('job-actions')).not.toBeInTheDocument()
// Click to expand
fireEvent.click(screen.getByText('Test Artist - Test Song'))
// Should show actions now
expect(screen.getByTestId('job-actions')).toBeInTheDocument()
// Click to collapse
fireEvent.click(screen.getByText('Test Artist - Test Song'))
// Should hide actions again
expect(screen.queryByTestId('job-actions')).not.toBeInTheDocument()
})
})
@@ -0,0 +1,154 @@
"use client"
import { useState } from "react"
import { api, AudioSearchResult } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Dialog, DialogContent, DialogDescription, DialogHeader, DialogTitle } from "@/components/ui/dialog"
import { Loader2, Music2, HardDrive, Calendar, Clock } from "lucide-react"
interface AudioSearchDialogProps {
jobId: string
open: boolean
onClose: () => void
onSelect: () => void
}
export function AudioSearchDialog({ jobId, open, onClose, onSelect }: AudioSearchDialogProps) {
const [results, setResults] = useState<AudioSearchResult[]>([])
const [isLoading, setIsLoading] = useState(true)
const [isSelecting, setIsSelecting] = useState(false)
const [error, setError] = useState("")
useState(() => {
if (open) {
loadResults()
}
})
async function loadResults() {
setIsLoading(true)
setError("")
try {
const data = await api.getAudioSearchResults(jobId)
setResults(data.results || [])
} catch (err: any) {
console.error("Failed to load search results:", err)
setError(err.message || "Failed to load search results")
} finally {
setIsLoading(false)
}
}
async function handleSelect(index: number) {
setIsSelecting(true)
try {
await api.selectAudioResult(jobId, index)
onSelect()
onClose()
} catch (err: any) {
console.error("Failed to select audio:", err)
setError(err.message || "Failed to select audio")
} finally {
setIsSelecting(false)
}
}
function formatDuration(seconds?: number): string {
if (!seconds) return ""
const mins = Math.floor(seconds / 60)
const secs = seconds % 60
return `${mins}:${secs.toString().padStart(2, "0")}`
}
return (
<Dialog open={open} onOpenChange={(open) => !open && onClose()}>
<DialogContent className="max-w-3xl max-h-[80vh] overflow-y-auto bg-slate-900 border-slate-700">
<DialogHeader>
<DialogTitle className="text-white">Select Audio Source</DialogTitle>
<DialogDescription className="text-slate-400">
Choose the best quality audio source for your karaoke video
</DialogDescription>
</DialogHeader>
{isLoading ? (
<div className="flex items-center justify-center py-8">
<Loader2 className="w-6 h-6 animate-spin text-slate-400" />
<span className="ml-2 text-slate-400">Loading search results...</span>
</div>
) : error ? (
<div className="text-center py-8 text-red-400">{error}</div>
) : results.length === 0 ? (
<div className="text-center py-8 text-slate-500">
<Music2 className="w-12 h-12 mx-auto mb-3 opacity-50" />
<p>No audio sources found</p>
</div>
) : (
<div className="space-y-2">
{results.map((result) => (
<div
key={result.index}
className="border border-slate-700 rounded-lg p-3 hover:bg-slate-800/50 transition-colors"
>
<div className="flex items-start justify-between gap-3">
<div className="flex-1 min-w-0">
<p className="font-medium text-white truncate">
{result.artist} - {result.title}
</p>
<div className="flex flex-wrap items-center gap-3 mt-1 text-xs text-slate-400">
<span className="flex items-center gap-1">
<HardDrive className="w-3 h-3" />
{result.provider}
</span>
{result.quality && (
<span className={`px-1.5 py-0.5 rounded ${
result.is_lossless ? "bg-green-600/20 text-green-400" : "bg-slate-600/20 text-slate-400"
}`}>
{result.quality}
</span>
)}
{result.duration && (
<span className="flex items-center gap-1">
<Clock className="w-3 h-3" />
{formatDuration(result.duration)}
</span>
)}
{result.year && (
<span className="flex items-center gap-1">
<Calendar className="w-3 h-3" />
{result.year}
</span>
)}
{result.seeders !== undefined && (
<span className="text-green-400">
{result.seeders} seeders
</span>
)}
</div>
{result.album && (
<p className="text-xs text-slate-500 mt-1 truncate">
Album: {result.album}
</p>
)}
</div>
<Button
size="sm"
onClick={() => handleSelect(result.index)}
disabled={isSelecting}
className="bg-amber-600 hover:bg-amber-500 text-white shrink-0"
>
{isSelecting ? (
<Loader2 className="w-3 h-3 animate-spin" />
) : (
"Select"
)}
</Button>
</div>
</div>
))}
</div>
)}
</DialogContent>
</Dialog>
)
}
@@ -0,0 +1,2 @@
export { AudioSearchDialog } from "./AudioSearchDialog"
+107
View File
@@ -0,0 +1,107 @@
'use client';
import { useState, useEffect } from 'react';
import { X, KeyRound } from 'lucide-react';
import { Button } from '@/components/ui/button';
import { Input } from '@/components/ui/input';
import { setAccessToken, getAccessToken, clearAccessToken } from '@/lib/api';
export function AuthBanner() {
const [token, setToken] = useState('');
const [isAuthenticated, setIsAuthenticated] = useState(false);
const [showInput, setShowInput] = useState(false);
useEffect(() => {
const storedToken = getAccessToken();
if (storedToken) {
setIsAuthenticated(true);
setToken(storedToken.substring(0, 8) + '...');
} else {
setShowInput(true);
}
}, []);
const handleSubmit = (e: React.FormEvent) => {
e.preventDefault();
if (token.trim()) {
setAccessToken(token.trim());
setIsAuthenticated(true);
setShowInput(false);
// Reload to fetch jobs with auth
window.location.reload();
}
};
const handleLogout = () => {
clearAccessToken();
setIsAuthenticated(false);
setToken('');
setShowInput(true);
window.location.reload();
};
if (isAuthenticated && !showInput) {
return (
<div className="bg-green-500/10 border border-green-500/20 text-green-400 px-4 py-2 rounded-lg flex items-center justify-between text-sm">
<div className="flex items-center gap-2">
<KeyRound className="h-4 w-4" />
<span>Authenticated</span>
</div>
<Button
variant="ghost"
size="sm"
onClick={handleLogout}
className="h-6 px-2 text-xs text-green-400 hover:text-green-300"
>
Logout
</Button>
</div>
);
}
return (
<div className="bg-yellow-500/10 border border-yellow-500/20 text-yellow-400 px-4 py-3 rounded-lg">
<div className="flex items-start gap-3">
<KeyRound className="h-5 w-5 mt-0.5 flex-shrink-0" />
<div className="flex-1">
<p className="font-medium mb-2">Authentication Required</p>
{showInput ? (
<form onSubmit={handleSubmit} className="flex gap-2">
<Input
type="password"
placeholder="Enter access token"
value={token}
onChange={(e) => setToken(e.target.value)}
className="flex-1 bg-[var(--secondary)] border-[var(--card-border)] text-sm"
/>
<Button type="submit" size="sm" disabled={!token.trim()}>
Authenticate
</Button>
</form>
) : (
<Button
variant="outline"
size="sm"
onClick={() => setShowInput(true)}
className="mt-1"
>
Enter Token
</Button>
)}
<p className="text-xs mt-2 text-[var(--text-muted)]">
Contact an admin to get an access token
</p>
</div>
{showInput && (
<button
onClick={() => setShowInput(false)}
className="text-[var(--text-muted)] hover:text-[var(--text)]"
>
<X className="h-4 w-4" />
</button>
)}
</div>
</div>
);
}
@@ -0,0 +1,166 @@
"use client"
import { useState, useEffect } from "react"
import { api, InstrumentalOption } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Dialog, DialogContent, DialogDescription, DialogHeader, DialogTitle } from "@/components/ui/dialog"
import { Loader2, Music2, Play, Pause } from "lucide-react"
interface InstrumentalSelectorProps {
jobId: string
open: boolean
onClose: () => void
onSelect: () => void
}
export function InstrumentalSelector({ jobId, open, onClose, onSelect }: InstrumentalSelectorProps) {
const [options, setOptions] = useState<InstrumentalOption[]>([])
const [isLoading, setIsLoading] = useState(true)
const [isSelecting, setIsSelecting] = useState(false)
const [error, setError] = useState("")
const [playingId, setPlayingId] = useState<string | null>(null)
const [audioElements, setAudioElements] = useState<Map<string, HTMLAudioElement>>(new Map())
useEffect(() => {
if (open) {
loadOptions()
}
return () => {
// Cleanup audio elements
audioElements.forEach(audio => {
audio.pause()
audio.src = ""
})
}
}, [open])
async function loadOptions() {
setIsLoading(true)
setError("")
try {
const data = await api.getInstrumentalOptions(jobId)
setOptions(data.options || [])
} catch (err: any) {
console.error("Failed to load instrumental options:", err)
setError(err.message || "Failed to load options")
} finally {
setIsLoading(false)
}
}
async function handleSelect(selection: 'clean' | 'with_backing') {
setIsSelecting(true)
try {
await api.selectInstrumental(jobId, selection)
onSelect()
onClose()
} catch (err: any) {
console.error("Failed to select instrumental:", err)
setError(err.message || "Failed to select instrumental")
} finally {
setIsSelecting(false)
}
}
function togglePlayback(optionId: string, audioUrl: string) {
if (playingId === optionId) {
// Pause current
const audio = audioElements.get(optionId)
if (audio) {
audio.pause()
}
setPlayingId(null)
} else {
// Stop any currently playing audio
audioElements.forEach(audio => audio.pause())
// Play new audio
let audio = audioElements.get(optionId)
if (!audio) {
audio = new Audio(audioUrl)
audio.addEventListener('ended', () => setPlayingId(null))
audioElements.set(optionId, audio)
setAudioElements(new Map(audioElements))
}
audio.play()
setPlayingId(optionId)
}
}
return (
<Dialog open={open} onOpenChange={(open) => !open && onClose()}>
<DialogContent className="max-w-2xl bg-slate-900 border-slate-700">
<DialogHeader>
<DialogTitle className="text-white">Select Instrumental Version</DialogTitle>
<DialogDescription className="text-slate-400">
Choose between a clean instrumental or one with backing vocals
</DialogDescription>
</DialogHeader>
{isLoading ? (
<div className="flex items-center justify-center py-8">
<Loader2 className="w-6 h-6 animate-spin text-slate-400" />
<span className="ml-2 text-slate-400">Loading options...</span>
</div>
) : error ? (
<div className="text-center py-8 text-red-400">{error}</div>
) : options.length === 0 ? (
<div className="text-center py-8 text-slate-500">
<Music2 className="w-12 h-12 mx-auto mb-3 opacity-50" />
<p>No instrumental options available</p>
</div>
) : (
<div className="space-y-3">
{options.map((option) => (
<div
key={option.id}
className="border border-slate-700 rounded-lg p-4 hover:bg-slate-800/50 transition-colors"
>
<div className="flex items-center justify-between gap-3">
<div className="flex-1">
<p className="font-medium text-white">{option.label}</p>
{option.duration_seconds && (
<p className="text-xs text-slate-400 mt-1">
Duration: {Math.floor(option.duration_seconds / 60)}:
{(option.duration_seconds % 60).toString().padStart(2, '0')}
</p>
)}
</div>
<div className="flex items-center gap-2">
{option.audio_url && (
<Button
size="sm"
variant="ghost"
onClick={() => togglePlayback(option.id, option.audio_url)}
className="text-slate-400 hover:text-white"
>
{playingId === option.id ? (
<Pause className="w-4 h-4" />
) : (
<Play className="w-4 h-4" />
)}
</Button>
)}
<Button
size="sm"
onClick={() => handleSelect(option.id as 'clean' | 'with_backing')}
disabled={isSelecting}
className="bg-pink-600 hover:bg-pink-500 text-white"
>
{isSelecting ? (
<Loader2 className="w-3 h-3 animate-spin" />
) : (
"Select"
)}
</Button>
</div>
</div>
</div>
))}
</div>
)}
</DialogContent>
</Dialog>
)
}
+131
View File
@@ -0,0 +1,131 @@
"use client"
import { useState } from "react"
import { Job, api } from "@/lib/api"
import { Button } from "@/components/ui/button"
import {
Loader2, RotateCcw, XCircle, Trash2, FileText
} from "lucide-react"
interface JobActionsProps {
job: Job
onRefresh: () => void
showLogs: boolean
onToggleLogs: () => void
}
export function JobActions({ job, onRefresh, showLogs, onToggleLogs }: JobActionsProps) {
const [isRetrying, setIsRetrying] = useState(false)
const [isCancelling, setIsCancelling] = useState(false)
const [isDeleting, setIsDeleting] = useState(false)
const canRetry = job.status === "failed"
const canCancel = !["complete", "failed", "cancelled"].includes(job.status)
const canDelete = ["complete", "failed", "cancelled"].includes(job.status)
async function handleRetry() {
setIsRetrying(true)
try {
await api.retryJob(job.job_id)
onRefresh()
} catch (error) {
console.error("Failed to retry job:", error)
} finally {
setIsRetrying(false)
}
}
async function handleCancel() {
if (!confirm("Are you sure you want to cancel this job?")) return
setIsCancelling(true)
try {
await api.cancelJob(job.job_id)
onRefresh()
} catch (error) {
console.error("Failed to cancel job:", error)
} finally {
setIsCancelling(false)
}
}
async function handleDelete() {
if (!confirm("Are you sure you want to delete this job? This will remove all associated files.")) return
setIsDeleting(true)
try {
await api.deleteJob(job.job_id, true)
onRefresh()
} catch (error) {
console.error("Failed to delete job:", error)
} finally {
setIsDeleting(false)
}
}
return (
<div className="flex flex-wrap gap-2" onClick={(e) => e.stopPropagation()}>
<Button
size="sm"
variant="ghost"
onClick={onToggleLogs}
className="text-xs text-slate-400 hover:text-white"
>
<FileText className="w-3 h-3 mr-1" />
{showLogs ? "Hide" : "Show"} Logs
</Button>
{canRetry && (
<Button
size="sm"
variant="ghost"
onClick={handleRetry}
disabled={isRetrying}
className="text-xs text-blue-400 hover:text-blue-300"
>
{isRetrying ? (
<Loader2 className="w-3 h-3 mr-1 animate-spin" />
) : (
<RotateCcw className="w-3 h-3 mr-1" />
)}
Retry
</Button>
)}
{canCancel && (
<Button
size="sm"
variant="ghost"
onClick={handleCancel}
disabled={isCancelling}
className="text-xs text-orange-400 hover:text-orange-300"
>
{isCancelling ? (
<Loader2 className="w-3 h-3 mr-1 animate-spin" />
) : (
<XCircle className="w-3 h-3 mr-1" />
)}
Cancel
</Button>
)}
{canDelete && (
<Button
size="sm"
variant="ghost"
onClick={handleDelete}
disabled={isDeleting}
className="text-xs text-red-400 hover:text-red-300"
>
{isDeleting ? (
<Loader2 className="w-3 h-3 mr-1 animate-spin" />
) : (
<Trash2 className="w-3 h-3 mr-1" />
)}
Delete
</Button>
)}
</div>
)
}
+196
View File
@@ -0,0 +1,196 @@
"use client"
import { useState } from "react"
import { Job } from "@/lib/api"
import { Badge } from "@/components/ui/badge"
import { Button } from "@/components/ui/button"
import {
Loader2, CheckCircle2, XCircle, Clock, AlertCircle,
ChevronDown, ChevronUp, ExternalLink
} from "lucide-react"
import { JobActions } from "./JobActions"
import { JobLogs } from "./JobLogs"
import { OutputLinks } from "./OutputLinks"
import { InstrumentalSelector } from "./InstrumentalSelector"
import { AudioSearchDialog } from "../audio-search/AudioSearchDialog"
// Status badge styling
const statusConfig: Record<string, { color: string; icon: React.ReactNode; label: string }> = {
pending: { color: "bg-slate-500", icon: <Clock className="w-3 h-3" />, label: "Pending" },
searching_audio: { color: "bg-blue-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Searching Audio" },
awaiting_audio_selection: { color: "bg-orange-500", icon: <AlertCircle className="w-3 h-3" />, label: "Select Audio" },
downloading: { color: "bg-blue-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Downloading" },
downloading_audio: { color: "bg-blue-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Downloading" },
separating_stage1: { color: "bg-purple-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Separating (1/2)" },
separating_stage2: { color: "bg-purple-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Separating (2/2)" },
audio_complete: { color: "bg-purple-600", icon: <CheckCircle2 className="w-3 h-3" />, label: "Audio Ready" },
transcribing: { color: "bg-amber-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Transcribing" },
correcting: { color: "bg-amber-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Correcting" },
lyrics_complete: { color: "bg-amber-600", icon: <CheckCircle2 className="w-3 h-3" />, label: "Lyrics Ready" },
generating_screens: { color: "bg-cyan-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Generating Screens" },
awaiting_review: { color: "bg-orange-500", icon: <AlertCircle className="w-3 h-3" />, label: "Awaiting Review" },
in_review: { color: "bg-orange-600", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "In Review" },
review_complete: { color: "bg-teal-500", icon: <CheckCircle2 className="w-3 h-3" />, label: "Review Complete" },
rendering_video: { color: "bg-indigo-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Rendering Video" },
awaiting_instrumental_selection: { color: "bg-pink-500", icon: <AlertCircle className="w-3 h-3" />, label: "Select Instrumental" },
instrumental_selected: { color: "bg-pink-600", icon: <CheckCircle2 className="w-3 h-3" />, label: "Instrumental Selected" },
generating_video: { color: "bg-violet-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Generating Video" },
encoding: { color: "bg-violet-600", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Encoding" },
packaging: { color: "bg-violet-700", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Packaging" },
uploading: { color: "bg-green-500", icon: <Loader2 className="w-3 h-3 animate-spin" />, label: "Uploading" },
complete: { color: "bg-green-600", icon: <CheckCircle2 className="w-3 h-3" />, label: "Complete" },
failed: { color: "bg-red-600", icon: <XCircle className="w-3 h-3" />, label: "Failed" },
cancelled: { color: "bg-gray-500", icon: <XCircle className="w-3 h-3" />, label: "Cancelled" },
}
export function StatusBadge({ status }: { status: string }) {
const config = statusConfig[status] || { color: "bg-gray-500", icon: <Clock className="w-3 h-3" />, label: status }
return (
<Badge className={`${config.color} text-white gap-1`}>
{config.icon}
{config.label}
</Badge>
)
}
interface JobCardProps {
job: Job
onRefresh: () => void
}
export function JobCard({ job, onRefresh }: JobCardProps) {
const [showDetails, setShowDetails] = useState(false)
const [showLogs, setShowLogs] = useState(false)
const [showAudioSearch, setShowAudioSearch] = useState(false)
const [showInstrumentalSelector, setShowInstrumentalSelector] = useState(false)
const createdAt = new Date(job.created_at).toLocaleString()
const isInteractive = job.status === "awaiting_review" ||
job.status === "awaiting_instrumental_selection" ||
job.status === "awaiting_audio_selection"
const isComplete = job.status === "complete"
const isFailed = job.status === "failed"
const isProcessing = !isComplete && !isFailed && job.status !== "cancelled"
return (
<div
className={`rounded-lg border p-3 transition-colors
${isInteractive ? "border-orange-500/50 bg-orange-500/5" : "border-slate-700 bg-slate-800/30"}
${isComplete ? "border-green-500/30" : ""}
${isFailed ? "border-red-500/30" : ""}`}
>
<div
className="flex items-start justify-between gap-3 cursor-pointer"
onClick={() => setShowDetails(!showDetails)}
>
<div className="flex-1 min-w-0">
<p className="font-medium text-white truncate">
{job.artist || "Unknown"} - {job.title || "Unknown"}
</p>
<p className="text-xs text-slate-500 mt-1">{createdAt}</p>
{job.progress > 0 && job.progress < 100 && isProcessing && (
<div className="mt-2 h-1 rounded-full bg-slate-700 overflow-hidden">
<div
className="h-full bg-amber-500 transition-all duration-500"
style={{ width: `${job.progress}%` }}
/>
</div>
)}
</div>
<div className="flex items-center gap-2">
<StatusBadge status={job.status} />
{showDetails ? (
<ChevronUp className="w-4 h-4 text-slate-400" />
) : (
<ChevronDown className="w-4 h-4 text-slate-400" />
)}
</div>
</div>
{showDetails && (
<div className="mt-3 pt-3 border-t border-slate-700 space-y-3">
<p className="text-xs text-slate-400">
<span className="text-slate-500">Job ID:</span> {job.job_id}
</p>
{job.error_message && (
<div className="text-xs text-red-400 bg-red-500/10 rounded p-2 break-words">
{job.error_message}
</div>
)}
{isInteractive && (
<div className="flex flex-wrap gap-2">
{job.status === "awaiting_review" && (
<a
href={`https://lyrics.nomadkaraoke.com/?job=${job.job_id}`}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-orange-600 hover:bg-orange-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<ExternalLink className="w-3 h-3" />
Review Lyrics
</a>
)}
{job.status === "awaiting_audio_selection" && (
<Button
size="sm"
onClick={(e) => {
e.stopPropagation()
setShowAudioSearch(true)
}}
className="text-xs bg-blue-600 hover:bg-blue-500"
>
Select Audio Source
</Button>
)}
{job.status === "awaiting_instrumental_selection" && (
<Button
size="sm"
onClick={(e) => {
e.stopPropagation()
setShowInstrumentalSelector(true)
}}
className="text-xs bg-pink-600 hover:bg-pink-500"
>
Select Instrumental
</Button>
)}
</div>
)}
{isComplete && (
<OutputLinks jobId={job.job_id} />
)}
<JobActions
job={job}
onRefresh={onRefresh}
showLogs={showLogs}
onToggleLogs={() => setShowLogs(!showLogs)}
/>
{showLogs && (
<JobLogs jobId={job.job_id} />
)}
</div>
)}
<AudioSearchDialog
jobId={job.job_id}
open={showAudioSearch}
onClose={() => setShowAudioSearch(false)}
onSelect={onRefresh}
/>
<InstrumentalSelector
jobId={job.job_id}
open={showInstrumentalSelector}
onClose={() => setShowInstrumentalSelector(false)}
onSelect={onRefresh}
/>
</div>
)
}
+87
View File
@@ -0,0 +1,87 @@
"use client"
import { useEffect, useState } from "react"
import { api, JobLog } from "@/lib/api"
import { Loader2, AlertCircle } from "lucide-react"
interface JobLogsProps {
jobId: string
limit?: number
}
export function JobLogs({ jobId, limit = 50 }: JobLogsProps) {
const [logs, setLogs] = useState<JobLog[]>([])
const [isLoading, setIsLoading] = useState(true)
const [error, setError] = useState("")
useEffect(() => {
loadLogs()
}, [jobId])
async function loadLogs() {
setIsLoading(true)
setError("")
try {
const data = await api.getJobLogs(jobId, limit)
setLogs(data)
} catch (err) {
console.error("Failed to load logs:", err)
setError("Failed to load logs")
} finally {
setIsLoading(false)
}
}
if (isLoading) {
return (
<div className="flex items-center justify-center py-4 text-slate-400">
<Loader2 className="w-4 h-4 animate-spin mr-2" />
Loading logs...
</div>
)
}
if (error) {
return (
<div className="flex items-center py-4 text-red-400 text-sm">
<AlertCircle className="w-4 h-4 mr-2" />
{error}
</div>
)
}
if (logs.length === 0) {
return (
<div className="text-center py-4 text-slate-500 text-sm">
No logs available
</div>
)
}
return (
<div className="bg-slate-900 rounded border border-slate-700 p-3 max-h-64 overflow-y-auto">
<div className="space-y-1 font-mono text-xs">
{logs.map((log, index) => (
<div
key={index}
className={`flex gap-2 ${
log.level === "ERROR" ? "text-red-400" :
log.level === "WARNING" ? "text-yellow-400" :
log.level === "INFO" ? "text-blue-400" :
"text-slate-400"
}`}
>
<span className="text-slate-600 shrink-0">
{new Date(log.timestamp).toLocaleTimeString()}
</span>
<span className="shrink-0 font-semibold w-12">
{log.level}
</span>
<span className="break-all">{log.message}</span>
</div>
))}
</div>
</div>
)
}
+338
View File
@@ -0,0 +1,338 @@
"use client"
import { useState } from "react"
import { api, ApiError } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Label } from "@/components/ui/label"
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"
import { Upload, Youtube, Music, Loader2 } from "lucide-react"
interface JobSubmissionProps {
onJobCreated: () => void
}
export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
const [activeTab, setActiveTab] = useState("upload")
const [error, setError] = useState("")
const [isSubmitting, setIsSubmitting] = useState(false)
// Upload form
const [uploadFile, setUploadFile] = useState<File | null>(null)
const [uploadArtist, setUploadArtist] = useState("")
const [uploadTitle, setUploadTitle] = useState("")
// URL form
const [youtubeUrl, setYoutubeUrl] = useState("")
const [youtubeArtist, setYoutubeArtist] = useState("")
const [youtubeTitle, setYoutubeTitle] = useState("")
// Audio search form
const [searchArtist, setSearchArtist] = useState("")
const [searchTitle, setSearchTitle] = useState("")
async function handleUploadSubmit(e: React.FormEvent) {
e.preventDefault()
setError("")
if (!uploadFile) {
setError("Please select an audio file")
return
}
if (!uploadArtist.trim() || !uploadTitle.trim()) {
setError("Please enter both artist and title")
return
}
setIsSubmitting(true)
try {
await api.uploadJob(uploadFile, uploadArtist.trim(), uploadTitle.trim())
setUploadFile(null)
setUploadArtist("")
setUploadTitle("")
onJobCreated()
} catch (err) {
if (err instanceof ApiError) {
setError(err.message)
} else {
setError("Failed to create job")
}
} finally {
setIsSubmitting(false)
}
}
async function handleUrlSubmit(e: React.FormEvent) {
e.preventDefault()
setError("")
if (!youtubeUrl.trim()) {
setError("Please enter a URL")
return
}
setIsSubmitting(true)
try {
await api.createJobFromUrl(
youtubeUrl.trim(),
youtubeArtist.trim() || undefined,
youtubeTitle.trim() || undefined
)
setYoutubeUrl("")
setYoutubeArtist("")
setYoutubeTitle("")
onJobCreated()
} catch (err) {
if (err instanceof ApiError) {
setError(err.message)
} else {
setError("Failed to create job")
}
} finally {
setIsSubmitting(false)
}
}
async function handleSearchSubmit(e: React.FormEvent) {
e.preventDefault()
setError("")
if (!searchArtist.trim() || !searchTitle.trim()) {
setError("Please enter both artist and title")
return
}
setIsSubmitting(true)
try {
await api.searchAudio(searchArtist.trim(), searchTitle.trim())
setSearchArtist("")
setSearchTitle("")
onJobCreated()
} catch (err) {
if (err instanceof ApiError) {
setError(err.message)
} else {
setError("Failed to search for audio")
}
} finally {
setIsSubmitting(false)
}
}
function handleFileChange(e: React.ChangeEvent<HTMLInputElement>) {
const file = e.target.files?.[0]
if (file) {
setUploadFile(file)
setError("")
}
}
return (
<Tabs value={activeTab} onValueChange={setActiveTab}>
<TabsList className="grid w-full grid-cols-3 bg-slate-800">
<TabsTrigger value="upload" className="gap-2 data-[state=active]:bg-slate-700">
<Upload className="w-4 h-4" />
Upload
</TabsTrigger>
<TabsTrigger value="url" className="gap-2 data-[state=active]:bg-slate-700">
<Youtube className="w-4 h-4" />
URL
</TabsTrigger>
<TabsTrigger value="search" className="gap-2 data-[state=active]:bg-slate-700">
<Music className="w-4 h-4" />
Search
</TabsTrigger>
</TabsList>
<TabsContent value="upload" className="mt-4">
<form onSubmit={handleUploadSubmit} className="space-y-4">
<div className="space-y-2">
<Label htmlFor="audio-file" className="text-slate-200">Audio File</Label>
<div
className={`relative border-2 border-dashed rounded-lg p-6 text-center transition-colors cursor-pointer
${uploadFile ? "border-amber-500/50 bg-amber-500/5" : "border-slate-700 hover:border-slate-600 bg-slate-800/50"}`}
>
<Input
id="audio-file"
type="file"
accept=".mp3,.wav,.flac,.m4a,.ogg,audio/*"
onChange={handleFileChange}
className="absolute inset-0 w-full h-full opacity-0 cursor-pointer"
disabled={isSubmitting}
/>
<Upload className="w-8 h-8 mx-auto mb-2 text-slate-500" />
{uploadFile ? (
<div>
<p className="font-medium text-white">{uploadFile.name}</p>
<p className="text-sm text-slate-400">{(uploadFile.size / 1024 / 1024).toFixed(2)} MB</p>
</div>
) : (
<div>
<p className="font-medium text-slate-300 mb-1">Click to upload</p>
<p className="text-sm text-slate-500">MP3, WAV, FLAC, M4A, or OGG</p>
</div>
)}
</div>
</div>
<div className="grid grid-cols-2 gap-4">
<div className="space-y-2">
<Label htmlFor="upload-artist" className="text-slate-200">Artist</Label>
<Input
id="upload-artist"
placeholder="Artist name"
value={uploadArtist}
onChange={(e) => setUploadArtist(e.target.value)}
disabled={isSubmitting}
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
/>
</div>
<div className="space-y-2">
<Label htmlFor="upload-title" className="text-slate-200">Title</Label>
<Input
id="upload-title"
placeholder="Song title"
value={uploadTitle}
onChange={(e) => setUploadTitle(e.target.value)}
disabled={isSubmitting}
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
/>
</div>
</div>
{error && activeTab === "upload" && (
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
)}
<Button
type="submit"
className="w-full bg-amber-600 hover:bg-amber-500 text-white"
disabled={isSubmitting}
>
{isSubmitting ? (
<>
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
Creating Job...
</>
) : (
"Create Karaoke Video"
)}
</Button>
</form>
</TabsContent>
<TabsContent value="url" className="mt-4">
<form onSubmit={handleUrlSubmit} className="space-y-4">
<div className="space-y-2">
<Label htmlFor="youtube-url" className="text-slate-200">YouTube URL</Label>
<div className="relative">
<Youtube className="absolute left-3 top-1/2 -translate-y-1/2 w-4 h-4 text-slate-500" />
<Input
id="youtube-url"
type="url"
placeholder="https://youtube.com/watch?v=..."
value={youtubeUrl}
onChange={(e) => setYoutubeUrl(e.target.value)}
className="pl-10 bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
disabled={isSubmitting}
/>
</div>
</div>
<div className="grid grid-cols-2 gap-4">
<div className="space-y-2">
<Label htmlFor="youtube-artist" className="text-slate-200">Artist (optional)</Label>
<Input
id="youtube-artist"
placeholder="Auto-detected"
value={youtubeArtist}
onChange={(e) => setYoutubeArtist(e.target.value)}
disabled={isSubmitting}
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
/>
</div>
<div className="space-y-2">
<Label htmlFor="youtube-title" className="text-slate-200">Title (optional)</Label>
<Input
id="youtube-title"
placeholder="Auto-detected"
value={youtubeTitle}
onChange={(e) => setYoutubeTitle(e.target.value)}
disabled={isSubmitting}
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
/>
</div>
</div>
{error && activeTab === "url" && (
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
)}
<Button
type="submit"
className="w-full bg-amber-600 hover:bg-amber-500 text-white"
disabled={isSubmitting}
>
{isSubmitting ? (
<>
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
Creating Job...
</>
) : (
"Create Karaoke Video"
)}
</Button>
</form>
</TabsContent>
<TabsContent value="search" className="mt-4">
<form onSubmit={handleSearchSubmit} className="space-y-4">
<div className="grid grid-cols-2 gap-4">
<div className="space-y-2">
<Label htmlFor="search-artist" className="text-slate-200">Artist</Label>
<Input
id="search-artist"
placeholder="Artist name"
value={searchArtist}
onChange={(e) => setSearchArtist(e.target.value)}
disabled={isSubmitting}
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
/>
</div>
<div className="space-y-2">
<Label htmlFor="search-title" className="text-slate-200">Title</Label>
<Input
id="search-title"
placeholder="Song title"
value={searchTitle}
onChange={(e) => setSearchTitle(e.target.value)}
disabled={isSubmitting}
className="bg-slate-800 border-slate-700 text-white placeholder:text-slate-500"
/>
</div>
</div>
{error && activeTab === "search" && (
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
)}
<Button
type="submit"
className="w-full bg-amber-600 hover:bg-amber-500 text-white"
disabled={isSubmitting}
>
{isSubmitting ? (
<>
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
Searching...
</>
) : (
"Search & Create Job"
)}
</Button>
</form>
</TabsContent>
</Tabs>
)
}
+142
View File
@@ -0,0 +1,142 @@
"use client"
import { useEffect, useState } from "react"
import { api } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Download, Loader2, ExternalLink, FolderOpen } from "lucide-react"
interface OutputLinksProps {
jobId: string
}
export function OutputLinks({ jobId }: OutputLinksProps) {
const [downloadUrls, setDownloadUrls] = useState<Record<string, any> | null>(null)
const [youtubeUrl, setYoutubeUrl] = useState<string | null>(null)
const [dropboxUrl, setDropboxUrl] = useState<string | null>(null)
const [isLoading, setIsLoading] = useState(false)
useEffect(() => {
loadOutputLinks()
}, [jobId])
async function loadOutputLinks() {
setIsLoading(true)
try {
const result = await api.getDownloadUrls(jobId)
setDownloadUrls(result.download_urls)
// Extract YouTube and Dropbox URLs from state_data if available
const job = await api.getJob(jobId)
if (job.state_data?.youtube_url) {
setYoutubeUrl(job.state_data.youtube_url)
}
if (job.state_data?.dropbox_folder_url) {
setDropboxUrl(job.state_data.dropbox_folder_url)
}
} catch (err) {
console.error("Failed to load output links:", err)
} finally {
setIsLoading(false)
}
}
if (isLoading) {
return (
<div className="flex items-center gap-2 text-xs text-slate-400">
<Loader2 className="w-3 h-3 animate-spin" />
Loading downloads...
</div>
)
}
return (
<div className="space-y-2">
{/* Distribution Links */}
{(youtubeUrl || dropboxUrl) && (
<div className="space-y-2">
<p className="text-xs text-slate-400 font-medium">Distribution:</p>
<div className="flex flex-wrap gap-2">
{youtubeUrl && (
<a
href={youtubeUrl}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-red-600 hover:bg-red-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<ExternalLink className="w-3 h-3" />
View on YouTube
</a>
)}
{dropboxUrl && (
<a
href={dropboxUrl}
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-blue-600 hover:bg-blue-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<FolderOpen className="w-3 h-3" />
Open Dropbox Folder
</a>
)}
</div>
</div>
)}
{/* Download Links */}
{downloadUrls && Object.keys(downloadUrls).length > 0 && (
<div className="space-y-2">
<p className="text-xs text-slate-400 font-medium">Downloads:</p>
<div className="flex flex-wrap gap-2">
{downloadUrls.finals?.lossy_720p && (
<a
href={api.getDownloadUrl(jobId, "finals", "lossy_720p")}
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-green-600 hover:bg-green-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<Download className="w-3 h-3" />
720p Video
</a>
)}
{downloadUrls.finals?.lossy_4k_mp4 && (
<a
href={api.getDownloadUrl(jobId, "finals", "lossy_4k_mp4")}
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-green-600 hover:bg-green-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<Download className="w-3 h-3" />
4K Video
</a>
)}
{downloadUrls.packages?.cdg_zip && (
<a
href={api.getDownloadUrl(jobId, "packages", "cdg_zip")}
className="inline-flex items-center gap-1 text-xs px-2 py-1.5 rounded bg-slate-600 hover:bg-slate-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<Download className="w-3 h-3" />
CDG+MP3
</a>
)}
{downloadUrls.packages?.txt_zip && (
<a
href={api.getDownloadUrl(jobId, "packages", "txt_zip")}
className="inline-flex items-center gap-1 text-xs px-2 py-1.5 rounded bg-slate-600 hover:bg-slate-500 text-white"
onClick={(e) => e.stopPropagation()}
>
<Download className="w-3 h-3" />
TXT+MP3
</a>
)}
</div>
</div>
)}
{!downloadUrls && !youtubeUrl && !dropboxUrl && (
<p className="text-xs text-slate-500">No outputs available yet</p>
)}
</div>
)
}
+6
View File
@@ -0,0 +1,6 @@
export { JobCard, StatusBadge } from "./JobCard"
export { JobActions } from "./JobActions"
export { JobLogs } from "./JobLogs"
export { OutputLinks } from "./OutputLinks"
export { InstrumentalSelector } from "./InstrumentalSelector"
+86 -124
View File
@@ -1,143 +1,105 @@
'use client'
"use client"
import * as React from 'react'
import * as DialogPrimitive from '@radix-ui/react-dialog'
import { XIcon } from 'lucide-react'
import * as React from "react"
import * as DialogPrimitive from "@radix-ui/react-dialog"
import { X } from "lucide-react"
import { cn } from '@/lib/utils'
const Dialog = DialogPrimitive.Root
function Dialog({
...props
}: React.ComponentProps<typeof DialogPrimitive.Root>) {
return <DialogPrimitive.Root data-slot="dialog" {...props} />
}
const DialogTrigger = DialogPrimitive.Trigger
function DialogTrigger({
...props
}: React.ComponentProps<typeof DialogPrimitive.Trigger>) {
return <DialogPrimitive.Trigger data-slot="dialog-trigger" {...props} />
}
const DialogPortal = DialogPrimitive.Portal
function DialogPortal({
...props
}: React.ComponentProps<typeof DialogPrimitive.Portal>) {
return <DialogPrimitive.Portal data-slot="dialog-portal" {...props} />
}
const DialogClose = DialogPrimitive.Close
function DialogClose({
...props
}: React.ComponentProps<typeof DialogPrimitive.Close>) {
return <DialogPrimitive.Close data-slot="dialog-close" {...props} />
}
const DialogOverlay = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Overlay>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Overlay>
>(({ className, ...props }, ref) => (
<DialogPrimitive.Overlay
ref={ref}
className="fixed inset-0 z-50 bg-black/80 data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0"
{...props}
/>
))
DialogOverlay.displayName = DialogPrimitive.Overlay.displayName
function DialogOverlay({
const DialogContent = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Content>
>(({ className, children, ...props }, ref) => (
<DialogPortal>
<DialogOverlay />
<DialogPrimitive.Content
ref={ref}
className="fixed left-[50%] top-[50%] z-50 grid w-full max-w-lg translate-x-[-50%] translate-y-[-50%] gap-4 border bg-background p-6 shadow-lg duration-200 data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[state=closed]:slide-out-to-left-1/2 data-[state=closed]:slide-out-to-top-[48%] data-[state=open]:slide-in-from-left-1/2 data-[state=open]:slide-in-from-top-[48%] sm:rounded-lg"
{...props}
>
{children}
<DialogPrimitive.Close className="absolute right-4 top-4 rounded-sm opacity-70 ring-offset-background transition-opacity hover:opacity-100 focus:outline-none focus:ring-2 focus:ring-ring focus:ring-offset-2 disabled:pointer-events-none data-[state=open]:bg-accent data-[state=open]:text-muted-foreground">
<X className="h-4 w-4" />
<span className="sr-only">Close</span>
</DialogPrimitive.Close>
</DialogPrimitive.Content>
</DialogPortal>
))
DialogContent.displayName = DialogPrimitive.Content.displayName
const DialogHeader = ({
className,
...props
}: React.ComponentProps<typeof DialogPrimitive.Overlay>) {
return (
<DialogPrimitive.Overlay
data-slot="dialog-overlay"
className={cn(
'data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 fixed inset-0 z-50 bg-black/50',
className,
)}
{...props}
/>
)
}
}: React.HTMLAttributes<HTMLDivElement>) => (
<div
className="flex flex-col space-y-1.5 text-center sm:text-left"
{...props}
/>
)
DialogHeader.displayName = "DialogHeader"
function DialogContent({
className,
children,
showCloseButton = true,
...props
}: React.ComponentProps<typeof DialogPrimitive.Content> & {
showCloseButton?: boolean
}) {
return (
<DialogPortal data-slot="dialog-portal">
<DialogOverlay />
<DialogPrimitive.Content
data-slot="dialog-content"
className={cn(
'bg-background data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 fixed top-[50%] left-[50%] z-50 grid w-full max-w-[calc(100%-2rem)] translate-x-[-50%] translate-y-[-50%] gap-4 rounded-lg border p-6 shadow-lg duration-200 sm:max-w-lg',
className,
)}
{...props}
>
{children}
{showCloseButton && (
<DialogPrimitive.Close
data-slot="dialog-close"
className="ring-offset-background focus:ring-ring data-[state=open]:bg-accent data-[state=open]:text-muted-foreground absolute top-4 right-4 rounded-xs opacity-70 transition-opacity hover:opacity-100 focus:ring-2 focus:ring-offset-2 focus:outline-hidden disabled:pointer-events-none [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4"
>
<XIcon />
<span className="sr-only">Close</span>
</DialogPrimitive.Close>
)}
</DialogPrimitive.Content>
</DialogPortal>
)
}
function DialogHeader({ className, ...props }: React.ComponentProps<'div'>) {
return (
<div
data-slot="dialog-header"
className={cn('flex flex-col gap-2 text-center sm:text-left', className)}
{...props}
/>
)
}
function DialogFooter({ className, ...props }: React.ComponentProps<'div'>) {
return (
<div
data-slot="dialog-footer"
className={cn(
'flex flex-col-reverse gap-2 sm:flex-row sm:justify-end',
className,
)}
{...props}
/>
)
}
function DialogTitle({
const DialogFooter = ({
className,
...props
}: React.ComponentProps<typeof DialogPrimitive.Title>) {
return (
<DialogPrimitive.Title
data-slot="dialog-title"
className={cn('text-lg leading-none font-semibold', className)}
{...props}
/>
)
}
}: React.HTMLAttributes<HTMLDivElement>) => (
<div
className="flex flex-col-reverse sm:flex-row sm:justify-end sm:space-x-2"
{...props}
/>
)
DialogFooter.displayName = "DialogFooter"
function DialogDescription({
className,
...props
}: React.ComponentProps<typeof DialogPrimitive.Description>) {
return (
<DialogPrimitive.Description
data-slot="dialog-description"
className={cn('text-muted-foreground text-sm', className)}
{...props}
/>
)
}
const DialogTitle = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Title>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Title>
>(({ className, ...props }, ref) => (
<DialogPrimitive.Title
ref={ref}
className="text-lg font-semibold leading-none tracking-tight"
{...props}
/>
))
DialogTitle.displayName = DialogPrimitive.Title.displayName
const DialogDescription = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Description>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Description>
>(({ className, ...props }, ref) => (
<DialogPrimitive.Description
ref={ref}
className="text-sm text-muted-foreground"
{...props}
/>
))
DialogDescription.displayName = DialogPrimitive.Description.displayName
export {
Dialog,
DialogClose,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogOverlay,
DialogPortal,
DialogTitle,
DialogOverlay,
DialogClose,
DialogTrigger,
DialogContent,
DialogHeader,
DialogFooter,
DialogTitle,
DialogDescription,
}
+32
View File
@@ -0,0 +1,32 @@
const nextJest = require('next/jest')
const createJestConfig = nextJest({
// Provide the path to your Next.js app to load next.config.js and .env files in your test environment
dir: './',
})
// Add any custom config to be passed to Jest
const customJestConfig = {
setupFilesAfterEnv: ['<rootDir>/jest.setup.js'],
testEnvironment: 'jest-environment-jsdom',
moduleNameMapper: {
'^@/(.*)$': '<rootDir>/$1',
},
testMatch: [
'**/__tests__/**/*.test.[jt]s?(x)',
'**/?(*.)+(spec|test).[jt]s?(x)'
],
collectCoverageFrom: [
'components/**/*.{js,jsx,ts,tsx}',
'lib/**/*.{js,jsx,ts,tsx}',
'app/**/*.{js,jsx,ts,tsx}',
'!**/*.d.ts',
'!**/node_modules/**',
'!**/.next/**',
'!**/out/**',
],
}
// createJestConfig is exported this way to ensure that next/jest can load the Next.js config which is async
module.exports = createJestConfig(customJestConfig)
+29
View File
@@ -0,0 +1,29 @@
// Learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom'
// Mock window.matchMedia
Object.defineProperty(window, 'matchMedia', {
writable: true,
value: jest.fn().mockImplementation(query => ({
matches: false,
media: query,
onchange: null,
addListener: jest.fn(),
removeListener: jest.fn(),
addEventListener: jest.fn(),
removeEventListener: jest.fn(),
dispatchEvent: jest.fn(),
})),
})
// Mock IntersectionObserver
global.IntersectionObserver = class IntersectionObserver {
constructor() {}
disconnect() {}
observe() {}
takeRecords() {
return []
}
unobserve() {}
}
+186
View File
@@ -0,0 +1,186 @@
/**
* Tests for API client
*/
import { api, ApiError } from "../api"
// Mock fetch globally
global.fetch = jest.fn()
describe("API Client", () => {
beforeEach(() => {
jest.clearAllMocks()
})
describe("listJobs", () => {
it("should fetch jobs successfully", async () => {
const mockJobs = [
{ job_id: "1", status: "complete", artist: "Test Artist", title: "Test Song" },
]
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: true,
json: async () => mockJobs,
})
const result = await api.listJobs()
expect(result).toEqual(mockJobs)
expect(global.fetch).toHaveBeenCalledWith(
expect.stringContaining("/api/jobs")
)
})
it("should handle errors", async () => {
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: false,
status: 500,
json: async () => ({ detail: "Server error" }),
})
await expect(api.listJobs()).rejects.toThrow(ApiError)
})
it("should include query parameters", async () => {
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: true,
json: async () => [],
})
await api.listJobs({ status: "complete", limit: 10 })
expect(global.fetch).toHaveBeenCalledWith(
expect.stringContaining("status=complete")
)
expect(global.fetch).toHaveBeenCalledWith(
expect.stringContaining("limit=10")
)
})
})
describe("getJob", () => {
it("should fetch a single job", async () => {
const mockJob = { job_id: "123", status: "complete" }
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: true,
json: async () => mockJob,
})
const result = await api.getJob("123")
expect(result).toEqual(mockJob)
expect(global.fetch).toHaveBeenCalledWith(
expect.stringContaining("/api/jobs/123")
)
})
})
describe("uploadJob", () => {
it("should upload a file and create a job", async () => {
const mockFile = new File(["test"], "test.mp3", { type: "audio/mpeg" })
const mockResponse = { status: "success", job_id: "123" }
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: true,
json: async () => mockResponse,
})
const result = await api.uploadJob(mockFile, "Artist", "Title")
expect(result).toEqual(mockResponse)
expect(global.fetch).toHaveBeenCalledWith(
expect.stringContaining("/api/jobs/upload"),
expect.objectContaining({ method: "POST" })
)
})
})
describe("searchAudio", () => {
it("should search for audio", async () => {
const mockResponse = {
status: "success",
job_id: "123",
results: [],
total_results: 0,
}
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: true,
json: async () => mockResponse,
})
const result = await api.searchAudio("Artist", "Title")
expect(result).toEqual(mockResponse)
expect(global.fetch).toHaveBeenCalledWith(
expect.stringContaining("/api/audio-search/search"),
expect.objectContaining({
method: "POST",
headers: { "Content-Type": "application/json" },
})
)
})
})
describe("getJobLogs", () => {
it("should fetch job logs", async () => {
const mockLogs = [
{ timestamp: "2025-01-01T00:00:00Z", level: "INFO", message: "Test log" },
]
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: true,
json: async () => ({ logs: mockLogs }),
})
const result = await api.getJobLogs("123")
expect(result).toEqual(mockLogs)
expect(global.fetch).toHaveBeenCalledWith(
expect.stringContaining("/api/jobs/123/logs")
)
})
it("should handle missing logs field", async () => {
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: true,
json: async () => ({}),
})
const result = await api.getJobLogs("123")
expect(result).toEqual([])
})
})
describe("cancelJob", () => {
it("should cancel a job", async () => {
const mockResponse = { status: "success", message: "Job cancelled" }
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: true,
json: async () => mockResponse,
})
const result = await api.cancelJob("123", "User requested")
expect(result).toEqual(mockResponse)
expect(global.fetch).toHaveBeenCalledWith(
expect.stringContaining("/api/jobs/123/cancel"),
expect.objectContaining({ method: "POST" })
)
})
})
describe("retryJob", () => {
it("should retry a failed job", async () => {
const mockResponse = { status: "success", job_id: "123", message: "Job retried" }
;(global.fetch as jest.Mock).mockResolvedValueOnce({
ok: true,
json: async () => mockResponse,
})
const result = await api.retryJob("123")
expect(result).toEqual(mockResponse)
expect(global.fetch).toHaveBeenCalledWith(
expect.stringContaining("/api/jobs/123/retry"),
expect.objectContaining({ method: "POST" })
)
})
})
})
+407
View File
@@ -0,0 +1,407 @@
/**
* API client for karaoke-gen backend
*/
const API_BASE_URL = process.env.NEXT_PUBLIC_API_URL || 'https://api.nomadkaraoke.com';
// Token management - stored in localStorage (client-side only)
let accessToken: string | null = null;
if (typeof window !== 'undefined') {
accessToken = localStorage.getItem('karaoke_access_token');
}
export function setAccessToken(token: string) {
accessToken = token;
if (typeof window !== 'undefined') {
localStorage.setItem('karaoke_access_token', token);
}
}
export function getAccessToken(): string | null {
return accessToken;
}
export function clearAccessToken() {
accessToken = null;
if (typeof window !== 'undefined') {
localStorage.removeItem('karaoke_access_token');
}
}
function getAuthHeaders(): HeadersInit {
const headers: HeadersInit = {};
if (accessToken) {
headers['Authorization'] = `Bearer ${accessToken}`;
}
return headers;
}
export interface Job {
job_id: string;
status: string;
progress: number;
created_at: string;
updated_at: string;
url?: string;
artist?: string;
title?: string;
filename?: string;
error_message?: string;
file_urls?: Record<string, any>;
state_data?: Record<string, any>;
timeline?: Array<{
status: string;
timestamp: string;
progress?: number;
message?: string;
}>;
}
export interface UploadJobResponse {
status: string;
job_id: string;
message: string;
filename?: string;
style_assets_uploaded?: string[];
server_version?: string;
distribution_services?: Record<string, any>;
}
export interface InstrumentalOption {
id: string;
label: string;
audio_url: string;
duration_seconds?: number;
}
export interface InstrumentalOptionsResponse {
options: InstrumentalOption[];
status: string;
artist?: string;
title?: string;
}
export interface DownloadUrlsResponse {
job_id: string;
artist?: string;
title?: string;
download_urls: Record<string, any>;
}
export interface AudioSearchResult {
index: number;
title: string;
artist: string;
provider: string;
url?: string;
duration?: number;
quality?: string;
source_id?: string;
seeders?: number;
is_lossless?: boolean;
year?: number;
album?: string;
}
export interface AudioSearchResponse {
status: string;
job_id: string;
results: AudioSearchResult[];
total_results: number;
message?: string;
}
export interface JobLog {
timestamp: string;
level: string;
message: string;
details?: Record<string, any>;
}
class ApiError extends Error {
status: number;
data?: any;
constructor(message: string, status: number, data?: any) {
super(message);
this.name = 'ApiError';
this.status = status;
this.data = data;
}
}
async function handleResponse<T>(response: Response): Promise<T> {
if (!response.ok) {
let data;
try {
data = await response.json();
} catch {
data = { detail: response.statusText };
}
throw new ApiError(
data.detail || data.message || 'API request failed',
response.status,
data
);
}
return response.json();
}
export const api = {
/**
* List all jobs
*/
async listJobs(params?: { status?: string; limit?: number }): Promise<Job[]> {
const searchParams = new URLSearchParams();
if (params?.status) searchParams.set('status', params.status);
if (params?.limit) searchParams.set('limit', String(params.limit));
const url = `${API_BASE_URL}/api/jobs${searchParams.toString() ? '?' + searchParams.toString() : ''}`;
const response = await fetch(url, {
headers: getAuthHeaders()
});
return handleResponse<Job[]>(response);
},
/**
* Get a single job by ID
*/
async getJob(jobId: string): Promise<Job> {
const response = await fetch(`${API_BASE_URL}/api/jobs/${jobId}`, {
headers: getAuthHeaders()
});
return handleResponse<Job>(response);
},
/**
* Upload a file and create a new job
*/
async uploadJob(
file: File,
artist: string,
title: string,
options?: {
enable_cdg?: boolean;
enable_txt?: boolean;
brand_prefix?: string;
enable_youtube_upload?: boolean;
}
): Promise<UploadJobResponse> {
const formData = new FormData();
formData.append('file', file);
formData.append('artist', artist);
formData.append('title', title);
if (options?.enable_cdg !== undefined) {
formData.append('enable_cdg', String(options.enable_cdg));
}
if (options?.enable_txt !== undefined) {
formData.append('enable_txt', String(options.enable_txt));
}
if (options?.brand_prefix) {
formData.append('brand_prefix', options.brand_prefix);
}
if (options?.enable_youtube_upload !== undefined) {
formData.append('enable_youtube_upload', String(options.enable_youtube_upload));
}
const response = await fetch(`${API_BASE_URL}/api/jobs/upload`, {
method: 'POST',
headers: getAuthHeaders(),
body: formData,
});
return handleResponse<UploadJobResponse>(response);
},
/**
* Create a job from a URL (YouTube, etc.)
*/
async createJobFromUrl(
url: string,
artist?: string,
title?: string,
options?: {
enable_cdg?: boolean;
enable_txt?: boolean;
brand_prefix?: string;
enable_youtube_upload?: boolean;
}
): Promise<{ status: string; job_id: string; message: string }> {
const body: Record<string, any> = { url };
if (artist) body.artist = artist;
if (title) body.title = title;
if (options?.enable_cdg !== undefined) body.enable_cdg = options.enable_cdg;
if (options?.enable_txt !== undefined) body.enable_txt = options.enable_txt;
if (options?.brand_prefix) body.brand_prefix = options.brand_prefix;
if (options?.enable_youtube_upload !== undefined) body.enable_youtube_upload = options.enable_youtube_upload;
const response = await fetch(`${API_BASE_URL}/api/jobs/create-from-url`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
...getAuthHeaders()
},
body: JSON.stringify(body),
});
return handleResponse(response);
},
/**
* Get review data for lyrics review
*/
async getReviewData(jobId: string): Promise<{
corrections_url: string;
audio_url: string;
status: string;
artist?: string;
title?: string;
}> {
const response = await fetch(`${API_BASE_URL}/api/jobs/${jobId}/review-data`);
return handleResponse(response);
},
/**
* Complete the lyrics review
*/
async completeReview(jobId: string): Promise<{ status: string; job_status: string; message: string }> {
const response = await fetch(`${API_BASE_URL}/api/jobs/${jobId}/complete-review`, {
method: 'POST',
});
return handleResponse(response);
},
/**
* Get instrumental options
*/
async getInstrumentalOptions(jobId: string): Promise<InstrumentalOptionsResponse> {
const response = await fetch(`${API_BASE_URL}/api/jobs/${jobId}/instrumental-options`);
return handleResponse(response);
},
/**
* Select an instrumental
*/
async selectInstrumental(
jobId: string,
selection: 'clean' | 'with_backing' | 'custom'
): Promise<{ status: string; job_status: string; selection: string; message: string }> {
const response = await fetch(`${API_BASE_URL}/api/jobs/${jobId}/select-instrumental`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ selection }),
});
return handleResponse(response);
},
/**
* Get download URLs for completed job
*/
async getDownloadUrls(jobId: string): Promise<DownloadUrlsResponse> {
const response = await fetch(`${API_BASE_URL}/api/jobs/${jobId}/download-urls`);
return handleResponse(response);
},
/**
* Cancel a job
*/
async cancelJob(jobId: string, reason?: string): Promise<{ status: string; message: string }> {
const response = await fetch(`${API_BASE_URL}/api/jobs/${jobId}/cancel`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ reason: reason || 'Cancelled by user' }),
});
return handleResponse(response);
},
/**
* Delete a job
*/
async deleteJob(jobId: string, deleteFiles: boolean = true): Promise<{ status: string; message: string }> {
const response = await fetch(
`${API_BASE_URL}/api/jobs/${jobId}?delete_files=${deleteFiles}`,
{ method: 'DELETE' }
);
return handleResponse(response);
},
/**
* Get the base download URL for a file
*/
getDownloadUrl(jobId: string, category: string, fileKey: string): string {
return `${API_BASE_URL}/api/jobs/${jobId}/download/${category}/${fileKey}`;
},
/**
* Search for audio by artist and title
*/
async searchAudio(
artist: string,
title: string,
autoDownload: boolean = false,
options?: {
enable_cdg?: boolean;
enable_txt?: boolean;
brand_prefix?: string;
enable_youtube_upload?: boolean;
}
): Promise<AudioSearchResponse> {
const body: Record<string, any> = { artist, title, auto_download: autoDownload };
if (options?.enable_cdg !== undefined) body.enable_cdg = options.enable_cdg;
if (options?.enable_txt !== undefined) body.enable_txt = options.enable_txt;
if (options?.brand_prefix) body.brand_prefix = options.brand_prefix;
if (options?.enable_youtube_upload !== undefined) body.enable_youtube_upload = options.enable_youtube_upload;
const response = await fetch(`${API_BASE_URL}/api/audio-search/search`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
return handleResponse(response);
},
/**
* Get audio search results for a job
*/
async getAudioSearchResults(jobId: string): Promise<AudioSearchResponse> {
const response = await fetch(`${API_BASE_URL}/api/audio-search/${jobId}/results`);
return handleResponse(response);
},
/**
* Select an audio search result
*/
async selectAudioResult(jobId: string, index: number): Promise<{ status: string; message: string }> {
const response = await fetch(`${API_BASE_URL}/api/audio-search/${jobId}/select`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ index }),
});
return handleResponse(response);
},
/**
* Get job logs
*/
async getJobLogs(jobId: string, limit: number = 100): Promise<JobLog[]> {
const response = await fetch(`${API_BASE_URL}/api/jobs/${jobId}/logs?limit=${limit}`);
const data = await handleResponse<{ logs: JobLog[] }>(response);
return data.logs || [];
},
/**
* Retry a failed job
*/
async retryJob(jobId: string): Promise<{ status: string; job_id: string; message: string }> {
const response = await fetch(`${API_BASE_URL}/api/jobs/${jobId}/retry`, {
method: 'POST',
});
return handleResponse(response);
},
};
export { ApiError };
+44
View File
@@ -0,0 +1,44 @@
"use client"
import { create } from "zustand"
import { persist } from "zustand/middleware"
import type { User } from "./types"
interface AuthStore {
user: User | null
login: (token: string) => Promise<boolean>
logout: () => void
updateCredits: (credits: number) => void
}
export const useAuth = create<AuthStore>()(
persist(
(set) => ({
user: null,
login: async (token: string) => {
// In a real app, validate token with backend
// For demo, parse token to determine role
try {
const isAdmin = token.toLowerCase().includes("admin")
const user: User = {
token,
role: isAdmin ? "admin" : "user",
credits: isAdmin ? 999 : 3,
}
set({ user })
return true
} catch {
return false
}
},
logout: () => set({ user: null }),
updateCredits: (credits: number) =>
set((state) => ({
user: state.user ? { ...state.user, credits } : null,
})),
}),
{
name: "nomad-karaoke-auth",
},
),
)
+77
View File
@@ -0,0 +1,77 @@
import { create } from "zustand"
import type { Job, JobStatus } from "./types"
interface JobsStore {
jobs: Job[]
addJob: (job: Omit<Job, "id" | "createdAt" | "updatedAt" | "progress" | "stages">) => void
updateJobStatus: (id: string, status: JobStatus) => void
getJobById: (id: string) => Job | undefined
getUserJobs: (userId: string) => Job[]
}
// Mock initial jobs for demo
const mockJobs: Job[] = [
{
id: "1",
userId: "demo",
artist: "Journey",
title: "Don't Stop Believin'",
status: "completed",
sourceType: "youtube",
sourceUrl: "https://youtube.com/watch?v=example",
progress: 100,
createdAt: new Date(Date.now() - 86400000 * 2).toISOString(),
updatedAt: new Date(Date.now() - 86400000).toISOString(),
stages: [
{ name: "Audio Extraction", status: "completed", progress: 100 },
{ name: "Vocal Separation", status: "completed", progress: 100 },
{ name: "Lyrics Sync", status: "completed", progress: 100 },
{ name: "Video Generation", status: "completed", progress: 100 },
],
resultUrl: "/downloads/journey-dont-stop-believin.mp4",
},
{
id: "2",
userId: "demo",
artist: "Queen",
title: "Bohemian Rhapsody",
status: "processing",
sourceType: "upload",
fileName: "bohemian-rhapsody.mp3",
progress: 65,
createdAt: new Date(Date.now() - 3600000).toISOString(),
updatedAt: new Date(Date.now() - 600000).toISOString(),
stages: [
{ name: "Audio Extraction", status: "completed", progress: 100 },
{ name: "Vocal Separation", status: "completed", progress: 100 },
{ name: "Lyrics Sync", status: "in_progress", progress: 60 },
{ name: "Video Generation", status: "pending", progress: 0 },
],
},
]
export const useJobs = create<JobsStore>((set, get) => ({
jobs: mockJobs,
addJob: (jobData) => {
const newJob: Job = {
...jobData,
id: Math.random().toString(36).substr(2, 9),
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
progress: 0,
stages: [
{ name: "Audio Extraction", status: "pending", progress: 0 },
{ name: "Vocal Separation", status: "pending", progress: 0 },
{ name: "Lyrics Sync", status: "pending", progress: 0 },
{ name: "Video Generation", status: "pending", progress: 0 },
],
}
set((state) => ({ jobs: [newJob, ...state.jobs] }))
},
updateJobStatus: (id, status) =>
set((state) => ({
jobs: state.jobs.map((job) => (job.id === id ? { ...job, status, updatedAt: new Date().toISOString() } : job)),
})),
getJobById: (id) => get().jobs.find((job) => job.id === id),
getUserJobs: (userId) => get().jobs.filter((job) => job.userId === userId),
}))
+34
View File
@@ -0,0 +1,34 @@
export type UserRole = "user" | "admin"
export interface User {
token: string
role: UserRole
credits: number
email?: string
}
export type JobStatus = "queued" | "processing" | "awaiting_review" | "awaiting_instrumental" | "completed" | "failed"
export interface Job {
id: string
userId: string
artist: string
title: string
status: JobStatus
sourceType: "youtube" | "upload"
sourceUrl?: string
fileName?: string
progress: number
createdAt: string
updatedAt: string
stages: JobStage[]
resultUrl?: string
errorMessage?: string
}
export interface JobStage {
name: string
status: "pending" | "in_progress" | "completed" | "failed"
progress: number
message?: string
}
+6
View File
@@ -0,0 +1,6 @@
import { clsx, type ClassValue } from 'clsx'
import { twMerge } from 'tailwind-merge'
export function cn(...inputs: ClassValue[]) {
return twMerge(clsx(inputs))
}
+4
View File
@@ -1,11 +1,15 @@
/** @type {import('next').NextConfig} */
const nextConfig = {
output: 'export',
trailingSlash: true,
typescript: {
ignoreBuildErrors: true,
},
images: {
unoptimized: true,
},
// GitHub Pages serves from subdomain, so no basePath needed
// gen.nomadkaraoke.com -> nomadkaraoke.github.io/karaoke-gen
}
export default nextConfig
+5515 -16
View File
File diff suppressed because it is too large. Load diff
+10 -1
View File
@@ -6,7 +6,10 @@
"build": "next build",
"dev": "next dev",
"lint": "eslint .",
"start": "next start"
"start": "next start",
"test": "jest --watch",
"test:ci": "jest --ci --coverage --maxWorkers=2",
"test:coverage": "jest --coverage"
},
"dependencies": {
"@hookform/resolvers": "^3.10.0",
@@ -65,9 +68,15 @@
},
"devDependencies": {
"@tailwindcss/postcss": "^4.1.9",
"@testing-library/jest-dom": "^6.1.5",
"@testing-library/react": "^14.1.2",
"@testing-library/user-event": "^14.5.1",
"@types/jest": "^29.5.11",
"@types/node": "^22",
"@types/react": "^19",
"@types/react-dom": "^19",
"jest": "^29.7.0",
"jest-environment-jsdom": "^29.7.0",
"postcss": "^8.5",
"tailwindcss": "^4.1.9",
"tw-animate-css": "1.3.3",
+12
View File
@@ -94,6 +94,18 @@ echo -n "your-genius-key" | \
# Add Audio Separator API URL
echo -n "https://your-modal-url" | \
gcloud secrets versions add audio-separator-api-url --data-file=-
# Add RED API key (for flacfetch audio search - private tracker)
echo -n "your-red-api-key" | \
gcloud secrets versions add red-api-key --data-file=-
# Add RED API URL (for flacfetch audio search - private tracker)
echo -n "https://your.red.url" | \
gcloud secrets versions add red-api-url --data-file=-
# Add OPS API key (for flacfetch audio search - private tracker)
echo -n "your-ops-api-key" | \
gcloud secrets versions add ops-api-key --data-file=-
```
Or use Pulumi config secrets:
+494 -3
View File
@@ -98,6 +98,14 @@ secrets_write_iam = gcp.projects.IAMMember(
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Cloud Trace write permissions (for OpenTelemetry tracing)
cloud_trace_iam = gcp.projects.IAMMember(
"karaoke-backend-cloudtrace-agent",
project=project_id,
role="roles/cloudtrace.agent",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Cloud Build service accounts access to Artifact Registry
# Cloud Build uses multiple service accounts depending on the context
cloudbuild_service_accounts = [
@@ -123,6 +131,16 @@ cloudbuild_logging_iam = gcp.projects.IAMMember(
member=f"serviceAccount:{project.number}-compute@developer.gserviceaccount.com",
)
# Grant Cloud Build compute service account permission to deploy as karaoke-backend SA
# This is required for Cloud Build to deploy Cloud Run services that use the backend service account
# Without this, gcloud run deploy fails with: PERMISSION_DENIED: Permission 'iam.serviceaccounts.actAs' denied
cloudbuild_deploy_as_backend = gcp.serviceaccount.IAMMember(
"cloudbuild-deploy-as-backend",
service_account_id=service_account.name,
role="roles/iam.serviceAccountUser",
member=f"serviceAccount:{project.number}-compute@developer.gserviceaccount.com",
)
# ==================== GitHub Actions Deployer Service Account ====================
# This service account is used by GitHub Actions CI/CD via Workload Identity Federation
@@ -228,13 +246,44 @@ workload_identity_provider = gcp.iam.WorkloadIdentityPoolProvider(
)
# Allow GitHub Actions to impersonate the service account
# Both karaoke-gen and flacfetch repos need access
github_actions_wif_binding = gcp.serviceaccount.IAMBinding(
"github-actions-wif-binding",
service_account_id=github_actions_sa.name,
role="roles/iam.workloadIdentityUser",
members=[workload_identity_pool.name.apply(
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/karaoke-gen"
)],
members=[
workload_identity_pool.name.apply(
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/karaoke-gen"
),
workload_identity_pool.name.apply(
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/flacfetch"
),
],
)
# Grant Compute Engine permissions for flacfetch VM deployment
# Needed for gcloud compute ssh to update the VM
github_actions_compute_admin = gcp.projects.IAMMember(
"github-actions-compute-admin",
project=project_id,
role="roles/compute.instanceAdmin.v1",
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant OS Login permissions for SSH access
github_actions_os_login = gcp.projects.IAMMember(
"github-actions-os-login",
project=project_id,
role="roles/compute.osLogin",
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant IAP tunnel user for secure SSH (used by gcloud compute ssh)
github_actions_iap_tunnel = gcp.projects.IAMMember(
"github-actions-iap-tunnel",
project=project_id,
role="roles/iap.tunnelResourceAccessor",
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# ==================== Cloud Tasks Queues (Phase 1 Scalability) ====================
@@ -348,6 +397,15 @@ cloud_tasks_enqueuer = gcp.projects.IAMMember(
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant backend service account permission to "act as" itself for Cloud Tasks OIDC authentication
# This is required when creating Cloud Tasks with oidc_token that specifies this service account
backend_sa_act_as_self = gcp.serviceaccount.IAMBinding(
"karaoke-backend-act-as-self",
service_account_id=service_account.name,
role="roles/iam.serviceAccountUser",
members=[service_account.email.apply(lambda email: f"serviceAccount:{email}")],
)
# ==================== Cloud Run Job for Video Encoding (Phase 2 Scalability) ====================
# Cloud Run Jobs provide up to 24-hour execution time for long-running video encoding.
# This is used when video encoding might exceed the 30-minute Cloud Tasks timeout.
@@ -428,6 +486,42 @@ audio_separator_secret = secretmanager.Secret(
),
)
# RED API Key Secret (for flacfetch audio search)
red_api_key_secret = secretmanager.Secret(
"red-api-key",
secret_id="red-api-key",
replication=secretmanager.SecretReplicationArgs(
auto=secretmanager.SecretReplicationAutoArgs(),
),
)
# RED API URL Secret (for flacfetch audio search)
red_api_url_secret = secretmanager.Secret(
"red-api-url",
secret_id="red-api-url",
replication=secretmanager.SecretReplicationArgs(
auto=secretmanager.SecretReplicationAutoArgs(),
),
)
# OPS API Key Secret (for flacfetch audio search)
ops_api_key_secret = secretmanager.Secret(
"ops-api-key",
secret_id="ops-api-key",
replication=secretmanager.SecretReplicationArgs(
auto=secretmanager.SecretReplicationAutoArgs(),
),
)
# OPS API URL Secret (for flacfetch audio search)
ops_api_url_secret = secretmanager.Secret(
"ops-api-url",
secret_id="ops-api-url",
replication=secretmanager.SecretReplicationArgs(
auto=secretmanager.SecretReplicationAutoArgs(),
),
)
# Create Cloud Run Domain Mapping
# Maps api.nomadkaraoke.com to the karaoke-backend service
domain_mapping = cloudrun.DomainMapping(
@@ -479,3 +573,400 @@ domain_mapping.statuses.apply(
})
)
# ==================== Flacfetch Service (Torrent Downloads) ====================
# A dedicated VM running flacfetch with Transmission daemon for BitTorrent downloads.
# This is required because Cloud Run doesn't support inbound peer connections for torrents.
# The VM has a static IP for tracker whitelist and runs indefinitely for seeding.
# Static IP for flacfetch (for tracker account whitelist)
flacfetch_ip = gcp.compute.Address(
"flacfetch-ip",
name="flacfetch-static-ip",
region="us-central1",
address_type="EXTERNAL",
description="Static IP for flacfetch service (torrent downloads)",
)
# Service account for flacfetch VM
flacfetch_sa = serviceaccount.Account(
"flacfetch-sa",
account_id="flacfetch-service",
display_name="Flacfetch Service Account",
description="Service account for flacfetch torrent/audio download VM",
)
# Grant flacfetch service account GCS write permissions
flacfetch_storage_iam = gcp.storage.BucketIAMMember(
"flacfetch-storage-writer",
bucket=bucket.name,
role="roles/storage.objectCreator",
member=flacfetch_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant flacfetch service account GCS read permissions (for uploads folder)
flacfetch_storage_reader_iam = gcp.storage.BucketIAMMember(
"flacfetch-storage-reader",
bucket=bucket.name,
role="roles/storage.objectViewer",
member=flacfetch_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant flacfetch service account Secret Manager access
flacfetch_secrets_iam = gcp.projects.IAMMember(
"flacfetch-secrets-access",
project=project_id,
role="roles/secretmanager.secretAccessor",
member=flacfetch_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Flacfetch API key secret (for authenticating API requests)
flacfetch_api_key_secret = secretmanager.Secret(
"flacfetch-api-key",
secret_id="flacfetch-api-key",
replication=secretmanager.SecretReplicationArgs(
auto=secretmanager.SecretReplicationAutoArgs(),
),
)
# Flacfetch API URL secret (the static IP address of the flacfetch VM)
# This is stored as a secret to avoid committing infrastructure IPs to the repo
flacfetch_api_url_secret = secretmanager.Secret(
"flacfetch-api-url",
secret_id="flacfetch-api-url",
replication=secretmanager.SecretReplicationArgs(
auto=secretmanager.SecretReplicationAutoArgs(),
),
)
# Startup script for the flacfetch VM
# This installs and configures Transmission daemon and the flacfetch service
FLACFETCH_STARTUP_SCRIPT = """#!/bin/bash
set -e
# Log to a file for debugging
exec > >(tee /var/log/flacfetch-startup.log) 2>&1
echo "Starting flacfetch setup at $(date)"
# Install dependencies
apt-get update
apt-get install -y python3-pip python3-venv transmission-daemon ffmpeg git curl
# Configure Transmission
echo "Configuring Transmission daemon..."
systemctl stop transmission-daemon || true
# Create transmission directories
mkdir -p /var/lib/transmission-daemon/downloads
mkdir -p /var/lib/transmission-daemon/.config/transmission-daemon
chown -R debian-transmission:debian-transmission /var/lib/transmission-daemon
# Configure transmission settings
cat > /var/lib/transmission-daemon/.config/transmission-daemon/settings.json << 'SETTINGS'
{
"download-dir": "/var/lib/transmission-daemon/downloads",
"incomplete-dir": "/var/lib/transmission-daemon/.incomplete",
"incomplete-dir-enabled": true,
"rpc-authentication-required": false,
"rpc-bind-address": "127.0.0.1",
"rpc-enabled": true,
"rpc-port": 9091,
"rpc-whitelist-enabled": false,
"peer-port": 51413,
"port-forwarding-enabled": false,
"speed-limit-down": 0,
"speed-limit-down-enabled": false,
"speed-limit-up": 0,
"speed-limit-up-enabled": false,
"ratio-limit-enabled": false,
"umask": 2,
"encryption": 1
}
SETTINGS
chown debian-transmission:debian-transmission /var/lib/transmission-daemon/.config/transmission-daemon/settings.json
systemctl start transmission-daemon
echo "Transmission daemon started"
# Install flacfetch
echo "Installing flacfetch..."
cd /opt
# Clone flacfetch (we'll use the repo from GitHub)
if [ ! -d "flacfetch" ]; then
git clone https://github.com/nomadkaraoke/flacfetch.git
fi
cd flacfetch
# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate
# Install flacfetch with API dependencies
pip install -e ".[api]"
# Get secrets from Secret Manager
echo "Fetching secrets from Secret Manager..."
FLACFETCH_API_KEY=$(gcloud secrets versions access latest --secret=flacfetch-api-key 2>/dev/null || echo "")
RED_API_KEY=$(gcloud secrets versions access latest --secret=red-api-key 2>/dev/null || echo "")
RED_API_URL=$(gcloud secrets versions access latest --secret=red-api-url 2>/dev/null || echo "")
OPS_API_KEY=$(gcloud secrets versions access latest --secret=ops-api-key 2>/dev/null || echo "")
OPS_API_URL=$(gcloud secrets versions access latest --secret=ops-api-url 2>/dev/null || echo "")
# Get bucket name from project metadata
GCS_BUCKET=$(gcloud compute project-info describe --format='value(commonInstanceMetadata.items.gcs-bucket)' 2>/dev/null || echo "")
if [ -z "$GCS_BUCKET" ]; then
# Fallback: construct from project ID
PROJECT_ID=$(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google")
GCS_BUCKET="karaoke-gen-storage-${PROJECT_ID}"
fi
echo "Using GCS bucket: $GCS_BUCKET"
# Create systemd service for flacfetch
echo "Creating systemd service..."
cat > /etc/systemd/system/flacfetch.service << SYSTEMD
[Unit]
Description=Flacfetch HTTP API Service
After=network.target transmission-daemon.service
Requires=transmission-daemon.service
[Service]
Type=simple
User=root
WorkingDirectory=/opt/flacfetch
Environment="FLACFETCH_API_KEY=${FLACFETCH_API_KEY}"
Environment="RED_API_KEY=${RED_API_KEY}"
Environment="RED_API_URL=${RED_API_URL}"
Environment="OPS_API_KEY=${OPS_API_KEY}"
Environment="OPS_API_URL=${OPS_API_URL}"
Environment="GCS_BUCKET=${GCS_BUCKET}"
Environment="FLACFETCH_KEEP_SEEDING=true"
Environment="FLACFETCH_MIN_FREE_GB=5"
Environment="FLACFETCH_DOWNLOAD_DIR=/var/lib/transmission-daemon/downloads"
Environment="TRANSMISSION_HOST=localhost"
Environment="TRANSMISSION_PORT=9091"
ExecStart=/opt/flacfetch/venv/bin/flacfetch serve --host 0.0.0.0 --port 8080
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
SYSTEMD
systemctl daemon-reload
systemctl enable flacfetch
systemctl start flacfetch
echo "Flacfetch service started at $(date)"
echo "Setup complete!"
"""
# Flacfetch VM instance
flacfetch_vm = gcp.compute.Instance(
"flacfetch-service",
name="flacfetch-service",
machine_type="e2-small", # 0.5 vCPU, 2GB RAM - sufficient for I/O-bound workload
zone="us-central1-a",
boot_disk=gcp.compute.InstanceBootDiskArgs(
initialize_params=gcp.compute.InstanceBootDiskInitializeParamsArgs(
image="debian-cloud/debian-12",
size=30, # GB - for torrent storage + headroom
type="pd-ssd", # SSD for faster I/O
),
),
network_interfaces=[gcp.compute.InstanceNetworkInterfaceArgs(
network="default",
access_configs=[gcp.compute.InstanceNetworkInterfaceAccessConfigArgs(
nat_ip=flacfetch_ip.address, # Static IP
)],
)],
service_account=gcp.compute.InstanceServiceAccountArgs(
email=flacfetch_sa.email,
scopes=["cloud-platform"],
),
metadata_startup_script=FLACFETCH_STARTUP_SCRIPT,
metadata={
"gcs-bucket": bucket.name, # Pass bucket name to startup script
},
tags=["flacfetch-service"],
allow_stopping_for_update=True,
)
# Firewall rule for flacfetch
# - Port 51413 TCP/UDP: BitTorrent peer connections
# - Port 8080 TCP: Flacfetch HTTP API (auth required at app level)
flacfetch_firewall = gcp.compute.Firewall(
"flacfetch-firewall",
name="flacfetch-firewall",
network="default",
allows=[
# BitTorrent peer connections
gcp.compute.FirewallAllowArgs(protocol="tcp", ports=["51413"]),
gcp.compute.FirewallAllowArgs(protocol="udp", ports=["51413"]),
# Flacfetch HTTP API (auth required at app level)
gcp.compute.FirewallAllowArgs(protocol="tcp", ports=["8080"]),
],
source_ranges=["0.0.0.0/0"], # Permissive, auth enforced at app level
target_tags=["flacfetch-service"],
description="Firewall rules for flacfetch torrent/API service",
)
# Export flacfetch values
pulumi.export("flacfetch_static_ip", flacfetch_ip.address)
pulumi.export("flacfetch_service_url", flacfetch_ip.address.apply(
lambda ip: f"http://{ip}:8080"
))
pulumi.export("flacfetch_service_account", flacfetch_sa.email)
# ==================== Cloud Monitoring Alerting ====================
# Alert policies for proactive monitoring and incident response
# Discord notification channel (webhook-based)
# Note: You'll need to manually create this in Cloud Console and reference it,
# or use the Discord webhook URL via Cloud Functions
# Alert: High Error Rate (>10% of requests returning errors)
error_rate_alert = gcp.monitoring.AlertPolicy(
"high-error-rate-alert",
display_name="Karaoke Backend - High Error Rate",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Error rate > 10%",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/request_count" AND metric.labels.response_code_class!="2xx"',
comparison="COMPARISON_GT",
threshold_value=0.1,
duration="300s", # Must be above threshold for 5 minutes
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_RATE",
cross_series_reducer="REDUCE_SUM",
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="3600s", # Auto-close after 1 hour if resolved
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="High error rate detected on karaoke-backend. Check Cloud Run logs for details.\n\nDashboard: https://console.cloud.google.com/run/detail/us-central1/karaoke-backend/logs?project=nomadkaraoke",
mime_type="text/markdown",
),
enabled=True,
)
# Alert: Cloud Tasks Queue Backlog (tasks piling up)
queue_backlog_alert = gcp.monitoring.AlertPolicy(
"queue-backlog-alert",
display_name="Karaoke Backend - Queue Backlog",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Queue depth > 50 tasks",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='resource.type="cloud_tasks_queue" AND metric.type="cloudtasks.googleapis.com/queue/depth"',
comparison="COMPARISON_GT",
threshold_value=50,
duration="600s", # Must be above threshold for 10 minutes
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_MEAN",
cross_series_reducer="REDUCE_MAX",
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="1800s", # Auto-close after 30 min if resolved
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="Cloud Tasks queue backlog detected. Tasks are piling up faster than workers can process.\n\nPossible causes:\n- Worker errors (check logs)\n- External API issues (Modal, AudioShake)\n- Cloud Run scaling limits\n\nDashboard: https://console.cloud.google.com/cloudtasks?project=nomadkaraoke",
mime_type="text/markdown",
),
enabled=True,
)
# Alert: High Memory Utilization (workers may be crashing)
memory_alert = gcp.monitoring.AlertPolicy(
"high-memory-alert",
display_name="Karaoke Backend - High Memory Usage",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Memory > 85%",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/container/memory/utilizations"',
comparison="COMPARISON_GT",
threshold_value=0.85,
duration="300s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_PERCENTILE_95",
cross_series_reducer="REDUCE_MAX",
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="1800s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="High memory utilization on karaoke-backend. Workers may be running out of memory.\n\nActions:\n- Check for memory leaks in worker code\n- Consider increasing Cloud Run memory limits\n- Review temp file cleanup",
mime_type="text/markdown",
),
enabled=True,
)
# Alert: Cloud Run Service Unavailable
service_unavailable_alert = gcp.monitoring.AlertPolicy(
"service-unavailable-alert",
display_name="Karaoke Backend - Service Unavailable",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="No healthy instances",
condition_absent=gcp.monitoring.AlertPolicyConditionConditionAbsentArgs(
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/container/instance_count"',
duration="300s", # Alert if no instances for 5 minutes
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionAbsentAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_MEAN",
cross_series_reducer="REDUCE_SUM",
),
],
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="1800s", # Minimum 30 minutes required by GCP
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="Karaoke backend service appears to be down - no healthy Cloud Run instances detected.\n\nImmediate actions:\n1. Check Cloud Run logs for crash reasons\n2. Verify recent deployments\n3. Check external dependencies (Modal, AudioShake)\n\nService URL: https://api.nomadkaraoke.com/api/health",
mime_type="text/markdown",
),
enabled=True,
)
# Export alert policy IDs
pulumi.export("error_rate_alert_id", error_rate_alert.name)
pulumi.export("queue_backlog_alert_id", queue_backlog_alert.name)
pulumi.export("memory_alert_id", memory_alert.name)
pulumi.export("service_unavailable_alert_id", service_unavailable_alert.name)
+207
View File
@@ -0,0 +1,207 @@
# Monitoring Infrastructure
This directory contains Cloud Monitoring dashboard and alerting configurations for the karaoke-gen backend.
---
## 🎯 Quick Start: Debug a Failed Job
**Bookmark this URL:**
```
https://console.cloud.google.com/logs/query;query=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22karaoke-backend%22;duration=PT6H?project=nomadkaraoke
```
### How to investigate a job:
1. **Open the bookmarked URL** (Logs Explorer)
2. **Add your job ID to the query.** In the query box, add a third line with your job ID:
```
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
"[job:YOUR_JOB_ID"
```
(Replace `YOUR_JOB_ID` with the actual ID, e.g. `"[job:8f03f0ac"`)
3. **Click "Run query"** or press `Shift+Enter`
4. **Enable "Stream logs"** (toggle at top right) for real-time updates
### What you'll see:
- `WORKER_START` - When each worker (audio/lyrics/screens/video) begins
- `WORKER_END status=success/error` - When it finishes with duration
- All log messages from that job's processing
- Any error tracebacks
### Example queries:
**All logs for job `8f03f0ac`:**
```
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
"[job:8f03f0ac"
```
**Only errors for a job:**
```
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
"[job:8f03f0ac"
severity>=ERROR
```
**Only worker start/end events:**
```
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
"[job:8f03f0ac"
("WORKER_START" OR "WORKER_END")
```
---
## Dashboard
The `dashboard.json` file defines a Cloud Monitoring dashboard with the following panels:
### Overview Metrics
- **Jobs Created (Last Hour)**: Count of jobs created
- **Cloud Run Request Count**: Request rate to the backend
- **Cloud Run Error Rate**: Percentage of non-2xx responses
### Performance Metrics
- **Request Latency (p95)**: 95th percentile response time
- **Cloud Run Instance Count**: Active container instances
- **CPU Utilization**: CPU usage across instances
- **Memory Utilization**: Memory usage across instances
### Cloud Tasks
- **Queue Depth**: Number of tasks waiting in each queue
- **Execution Rate**: Task processing rate per queue
### Logs
- **Worker Logs**: Searchable by `job_id`
- **Error Logs**: All ERROR-level logs from the backend
## Importing the Dashboard
To import the dashboard into Cloud Monitoring:
```bash
# Using gcloud CLI
gcloud monitoring dashboards create --config-from-file=infrastructure/monitoring/dashboard.json --project=nomadkaraoke
```
Or manually:
1. Go to Cloud Console → Monitoring → Dashboards
2. Click "Create Dashboard"
3. Click "JSON Editor" and paste contents of `dashboard.json`
## Alert Policies
Alert policies are defined in `infrastructure/__main__.py` as Pulumi resources:
### High Error Rate
- **Condition**: >10% of requests return errors for 5+ minutes
- **Severity**: Warning
- **Auto-close**: 1 hour after resolution
### Queue Backlog
- **Condition**: >50 tasks in any queue for 10+ minutes
- **Severity**: Warning
- **Auto-close**: 30 minutes after resolution
### High Memory Usage
- **Condition**: >85% memory utilization for 5+ minutes
- **Severity**: Warning
- **Auto-close**: 30 minutes after resolution
### Service Unavailable
- **Condition**: No healthy Cloud Run instances for 5+ minutes
- **Severity**: Critical
- **Auto-close**: 10 minutes after resolution
## Adding Notification Channels
To receive alerts, you need to create notification channels:
### Discord Webhook
1. Go to Cloud Console → Monitoring → Alerting → Notification channels
2. Click "Add New" → Webhook
3. Enter your Discord webhook URL
4. Save and note the channel ID
Then update the alert policies in `__main__.py`:
```python
error_rate_alert = gcp.monitoring.AlertPolicy(
# ...existing config...
notification_channels=[discord_webhook_channel.name],
)
```
### Email
1. Go to Cloud Console → Monitoring → Alerting → Notification channels
2. Click "Add New" → Email
3. Enter email addresses to notify
## Log-Based Metrics
The structured logging in `backend/services/structured_logging.py` outputs metrics as JSON log entries. You can create log-based metrics to track:
1. Go to Cloud Console → Logging → Log-based metrics
2. Create new metric with filter:
```
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
jsonPayload.metric_name="jobs_total"
```
3. Set extraction field to `jsonPayload.metric_value`
Repeat for other metrics:
- `worker_invocations_total`
- `job_stage_duration_seconds`
- `external_api_calls_total`
- `external_api_duration_seconds`
- `gcs_operations_total`
## Useful Log Queries
### Find all logs for a specific job:
```
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
jsonPayload.job_id="YOUR_JOB_ID"
```
### Find worker errors:
```
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
textPayload:"WORKER_END" AND textPayload:"status=error"
```
### Find slow workers (>5 min):
```
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
textPayload:"WORKER_END" AND textPayload:"duration="
```
Then add duration filter in the UI.
## Cloud Trace Integration
Structured logs include trace correlation fields:
- `logging.googleapis.com/trace`: Links to Cloud Trace
- `logging.googleapis.com/spanId`: Current span ID
To view correlated traces:
1. Find a log entry in Cloud Logging
2. Click the "View in Trace" link in the log entry
3. See the full distributed trace across services
+351
View File
@@ -0,0 +1,351 @@
{
"displayName": "Karaoke Backend Overview",
"mosaicLayout": {
"columns": 12,
"tiles": [
{
"xPos": 0,
"yPos": 0,
"width": 4,
"height": 4,
"widget": {
"title": "Jobs Created (Last Hour)",
"scorecard": {
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_run_revision\" AND metric.type=\"logging.googleapis.com/log_entry_count\"",
"aggregation": {
"alignmentPeriod": "3600s",
"perSeriesAligner": "ALIGN_SUM",
"crossSeriesReducer": "REDUCE_SUM"
}
}
},
"sparkChartView": {
"sparkChartType": "SPARK_LINE"
}
}
}
},
{
"xPos": 4,
"yPos": 0,
"width": 4,
"height": 4,
"widget": {
"title": "Cloud Run Request Count",
"scorecard": {
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"karaoke-backend\" AND metric.type=\"run.googleapis.com/request_count\"",
"aggregation": {
"alignmentPeriod": "3600s",
"perSeriesAligner": "ALIGN_RATE",
"crossSeriesReducer": "REDUCE_SUM"
}
}
},
"sparkChartView": {
"sparkChartType": "SPARK_LINE"
}
}
}
},
{
"xPos": 8,
"yPos": 0,
"width": 4,
"height": 4,
"widget": {
"title": "Cloud Run Error Rate",
"scorecard": {
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"karaoke-backend\" AND metric.type=\"run.googleapis.com/request_count\" AND metric.labels.response_code_class!=\"2xx\"",
"aggregation": {
"alignmentPeriod": "3600s",
"perSeriesAligner": "ALIGN_RATE",
"crossSeriesReducer": "REDUCE_SUM"
}
}
},
"sparkChartView": {
"sparkChartType": "SPARK_LINE"
},
"thresholds": [
{
"value": 0.05,
"color": "YELLOW",
"direction": "ABOVE"
},
{
"value": 0.1,
"color": "RED",
"direction": "ABOVE"
}
]
}
}
},
{
"xPos": 0,
"yPos": 4,
"width": 6,
"height": 4,
"widget": {
"title": "Cloud Run Request Latency (p95)",
"xyChart": {
"dataSets": [
{
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"karaoke-backend\" AND metric.type=\"run.googleapis.com/request_latencies\"",
"aggregation": {
"alignmentPeriod": "60s",
"perSeriesAligner": "ALIGN_PERCENTILE_95",
"crossSeriesReducer": "REDUCE_MEAN"
}
}
},
"plotType": "LINE",
"legendTemplate": "p95 latency"
}
],
"yAxis": {
"label": "Latency (ms)",
"scale": "LINEAR"
},
"chartOptions": {
"mode": "COLOR"
}
}
}
},
{
"xPos": 6,
"yPos": 4,
"width": 6,
"height": 4,
"widget": {
"title": "Cloud Run Instance Count",
"xyChart": {
"dataSets": [
{
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"karaoke-backend\" AND metric.type=\"run.googleapis.com/container/instance_count\"",
"aggregation": {
"alignmentPeriod": "60s",
"perSeriesAligner": "ALIGN_MEAN",
"crossSeriesReducer": "REDUCE_SUM"
}
}
},
"plotType": "LINE",
"legendTemplate": "instances"
}
],
"yAxis": {
"label": "Instances",
"scale": "LINEAR"
},
"chartOptions": {
"mode": "COLOR"
}
}
}
},
{
"xPos": 0,
"yPos": 8,
"width": 6,
"height": 4,
"widget": {
"title": "Cloud Tasks Queue Depth",
"xyChart": {
"dataSets": [
{
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_tasks_queue\" AND metric.type=\"cloudtasks.googleapis.com/queue/depth\"",
"aggregation": {
"alignmentPeriod": "60s",
"perSeriesAligner": "ALIGN_MEAN",
"groupByFields": ["resource.labels.queue_id"]
}
}
},
"plotType": "STACKED_AREA",
"legendTemplate": "${resource.labels.queue_id}"
}
],
"yAxis": {
"label": "Tasks in Queue",
"scale": "LINEAR"
},
"chartOptions": {
"mode": "COLOR"
}
}
}
},
{
"xPos": 6,
"yPos": 8,
"width": 6,
"height": 4,
"widget": {
"title": "Cloud Tasks Execution Rate",
"xyChart": {
"dataSets": [
{
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_tasks_queue\" AND metric.type=\"cloudtasks.googleapis.com/queue/task_attempt_count\"",
"aggregation": {
"alignmentPeriod": "60s",
"perSeriesAligner": "ALIGN_RATE",
"groupByFields": ["resource.labels.queue_id"]
}
}
},
"plotType": "STACKED_BAR",
"legendTemplate": "${resource.labels.queue_id}"
}
],
"yAxis": {
"label": "Tasks/sec",
"scale": "LINEAR"
},
"chartOptions": {
"mode": "COLOR"
}
}
}
},
{
"xPos": 0,
"yPos": 12,
"width": 12,
"height": 4,
"widget": {
"title": "Worker Logs (Search by job_id)",
"logsPanel": {
"filter": "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"karaoke-backend\" AND (textPayload=~\"\\[job:\" OR jsonPayload.job_id!=\"\")",
"resourceNames": []
}
}
},
{
"xPos": 0,
"yPos": 16,
"width": 6,
"height": 4,
"widget": {
"title": "CPU Utilization",
"xyChart": {
"dataSets": [
{
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"karaoke-backend\" AND metric.type=\"run.googleapis.com/container/cpu/utilizations\"",
"aggregation": {
"alignmentPeriod": "60s",
"perSeriesAligner": "ALIGN_PERCENTILE_50",
"crossSeriesReducer": "REDUCE_MEAN"
}
}
},
"plotType": "LINE",
"legendTemplate": "CPU (p50)"
},
{
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"karaoke-backend\" AND metric.type=\"run.googleapis.com/container/cpu/utilizations\"",
"aggregation": {
"alignmentPeriod": "60s",
"perSeriesAligner": "ALIGN_PERCENTILE_95",
"crossSeriesReducer": "REDUCE_MEAN"
}
}
},
"plotType": "LINE",
"legendTemplate": "CPU (p95)"
}
],
"yAxis": {
"label": "CPU Utilization",
"scale": "LINEAR"
},
"chartOptions": {
"mode": "COLOR"
}
}
}
},
{
"xPos": 6,
"yPos": 16,
"width": 6,
"height": 4,
"widget": {
"title": "Memory Utilization",
"xyChart": {
"dataSets": [
{
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"karaoke-backend\" AND metric.type=\"run.googleapis.com/container/memory/utilizations\"",
"aggregation": {
"alignmentPeriod": "60s",
"perSeriesAligner": "ALIGN_PERCENTILE_50",
"crossSeriesReducer": "REDUCE_MEAN"
}
}
},
"plotType": "LINE",
"legendTemplate": "Memory (p50)"
},
{
"timeSeriesQuery": {
"timeSeriesFilter": {
"filter": "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"karaoke-backend\" AND metric.type=\"run.googleapis.com/container/memory/utilizations\"",
"aggregation": {
"alignmentPeriod": "60s",
"perSeriesAligner": "ALIGN_PERCENTILE_95",
"crossSeriesReducer": "REDUCE_MEAN"
}
}
},
"plotType": "LINE",
"legendTemplate": "Memory (p95)"
}
],
"yAxis": {
"label": "Memory Utilization",
"scale": "LINEAR"
},
"chartOptions": {
"mode": "COLOR"
}
}
}
},
{
"xPos": 0,
"yPos": 20,
"width": 12,
"height": 4,
"widget": {
"title": "Error Logs",
"logsPanel": {
"filter": "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"karaoke-backend\" AND severity>=ERROR",
"resourceNames": []
}
}
}
]
}
}
@@ -0,0 +1,676 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Karaoke Job Inspector</title>
<style>
* {
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, sans-serif;
background: linear-gradient(135deg, #1a1a2e 0%, #16213e 100%);
color: #eee;
min-height: 100vh;
margin: 0;
padding: 20px;
}
.container {
max-width: 800px;
margin: 0 auto;
padding-top: 30px;
}
h1 {
text-align: center;
font-size: 2rem;
margin-bottom: 0.5rem;
background: linear-gradient(90deg, #00d4ff, #7b2cbf);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
background-clip: text;
}
.subtitle {
text-align: center;
color: #888;
margin-bottom: 30px;
}
.input-group {
background: rgba(255,255,255,0.05);
border-radius: 16px;
padding: 30px;
margin-bottom: 20px;
}
label {
display: block;
font-size: 0.9rem;
color: #aaa;
margin-bottom: 10px;
text-transform: uppercase;
letter-spacing: 1px;
}
input[type="text"] {
width: 100%;
padding: 16px 20px;
font-size: 1.5rem;
font-family: 'Monaco', 'Menlo', monospace;
border: 2px solid #333;
border-radius: 10px;
background: #0d0d1a;
color: #00d4ff;
outline: none;
transition: border-color 0.2s;
}
input[type="text"]:focus {
border-color: #00d4ff;
}
input[type="text"]::placeholder {
color: #444;
}
.buttons {
display: flex;
gap: 12px;
margin-top: 20px;
}
button, .btn {
flex: 1;
padding: 16px 24px;
font-size: 1rem;
font-weight: 600;
border: none;
border-radius: 10px;
cursor: pointer;
transition: transform 0.1s, box-shadow 0.2s;
text-align: center;
text-decoration: none;
}
button:hover, .btn:hover {
transform: translateY(-2px);
}
button:active, .btn:active {
transform: translateY(0);
}
.btn-primary {
background: linear-gradient(135deg, #00d4ff, #0099cc);
color: #000;
}
.btn-primary:hover {
box-shadow: 0 4px 20px rgba(0, 212, 255, 0.4);
}
.btn-secondary {
background: rgba(255,255,255,0.1);
color: #fff;
border: 1px solid #444;
}
.btn-secondary:hover {
background: rgba(255,255,255,0.15);
box-shadow: 0 4px 20px rgba(255,255,255,0.1);
}
.settings-row {
display: flex;
align-items: center;
gap: 12px;
margin-bottom: 20px;
padding-bottom: 20px;
border-bottom: 1px solid #333;
}
.settings-row label {
margin: 0;
flex-shrink: 0;
}
select {
flex: 1;
padding: 10px 14px;
font-size: 0.95rem;
border: 2px solid #333;
border-radius: 8px;
background: #0d0d1a;
color: #00d4ff;
outline: none;
cursor: pointer;
}
select:focus {
border-color: #00d4ff;
}
.quick-links {
background: rgba(255,255,255,0.03);
border-radius: 16px;
padding: 20px 30px;
margin-bottom: 20px;
}
.quick-links h3 {
font-size: 0.8rem;
text-transform: uppercase;
letter-spacing: 1px;
color: #666;
margin: 0 0 15px 0;
}
.quick-links a {
display: block;
color: #00d4ff;
text-decoration: none;
padding: 10px 0;
border-bottom: 1px solid #222;
transition: color 0.2s;
cursor: pointer;
}
.quick-links a:last-child {
border-bottom: none;
}
.quick-links a:hover {
color: #fff;
}
/* Guide section styles */
.guide-section {
background: rgba(255,255,255,0.03);
border-radius: 16px;
margin-bottom: 20px;
overflow: hidden;
}
.guide-header {
padding: 20px 30px;
cursor: pointer;
display: flex;
justify-content: space-between;
align-items: center;
transition: background 0.2s;
}
.guide-header:hover {
background: rgba(255,255,255,0.02);
}
.guide-header h3 {
font-size: 0.8rem;
text-transform: uppercase;
letter-spacing: 1px;
color: #666;
margin: 0;
}
.guide-header .toggle {
color: #666;
transition: transform 0.3s;
}
.guide-header.open .toggle {
transform: rotate(180deg);
}
.guide-content {
display: none;
padding: 0 30px 25px;
}
.guide-content.open {
display: block;
}
.guide-content h4 {
color: #00d4ff;
font-size: 1rem;
margin: 20px 0 10px 0;
padding-bottom: 5px;
border-bottom: 1px solid #333;
}
.guide-content h4:first-child {
margin-top: 0;
}
.guide-content p {
color: #aaa;
line-height: 1.6;
margin: 10px 0;
}
.guide-content ul, .guide-content ol {
color: #aaa;
line-height: 1.8;
padding-left: 20px;
}
.guide-content li {
margin: 8px 0;
}
.guide-content code {
background: #0d0d1a;
color: #00d4ff;
padding: 2px 8px;
border-radius: 4px;
font-family: 'Monaco', 'Menlo', monospace;
font-size: 0.85rem;
}
.guide-content pre {
background: #0d0d1a;
border: 1px solid #333;
border-radius: 8px;
padding: 15px;
overflow-x: auto;
margin: 15px 0;
}
.guide-content pre code {
background: none;
padding: 0;
color: #7fdbca;
font-size: 0.8rem;
line-height: 1.5;
}
.status-badge {
display: inline-block;
padding: 3px 10px;
border-radius: 12px;
font-size: 0.75rem;
font-weight: 600;
text-transform: uppercase;
}
.status-badge.error { background: #ff4757; color: #fff; }
.status-badge.warning { background: #ffa502; color: #000; }
.status-badge.success { background: #2ed573; color: #000; }
.status-badge.info { background: #1e90ff; color: #fff; }
.pipeline-flow {
display: flex;
flex-wrap: wrap;
gap: 8px;
align-items: center;
margin: 15px 0;
padding: 15px;
background: rgba(0,0,0,0.2);
border-radius: 8px;
}
.pipeline-step {
background: #1e3a5f;
color: #7fdbca;
padding: 8px 14px;
border-radius: 6px;
font-size: 0.8rem;
font-weight: 500;
}
.pipeline-arrow {
color: #444;
font-size: 1.2rem;
}
.hint {
text-align: center;
color: #555;
font-size: 0.85rem;
margin-top: 20px;
}
.copy-btn {
background: rgba(255,255,255,0.1);
border: none;
color: #888;
padding: 4px 10px;
border-radius: 4px;
cursor: pointer;
font-size: 0.75rem;
margin-left: 10px;
transition: all 0.2s;
}
.copy-btn:hover {
background: rgba(255,255,255,0.2);
color: #fff;
transform: none;
}
.copy-btn.copied {
background: #2ed573;
color: #000;
}
</style>
</head>
<body>
<div class="container">
<h1>🎤 Karaoke Job Inspector</h1>
<p class="subtitle">Debug failed jobs in seconds</p>
<div class="input-group">
<div class="settings-row">
<label for="authUser">Google Account</label>
<select id="authUser" onchange="saveAuthUser()">
<option value="0">Account 0 (default)</option>
<option value="1" selected>Account 1</option>
<option value="2">Account 2</option>
<option value="3">Account 3</option>
<option value="4">Account 4</option>
<option value="5">Account 5</option>
</select>
</div>
<label for="jobId">Enter Job ID</label>
<input
type="text"
id="jobId"
placeholder="e.g. 8f03f0ac-1234-5678-abcd"
autofocus
autocomplete="off"
>
<div class="buttons">
<button class="btn-primary" onclick="openLogs()">
📋 View Logs
</button>
<button class="btn-secondary" onclick="openTrace()">
🔍 View Trace
</button>
</div>
</div>
<div class="quick-links">
<h3>Quick Links</h3>
<a onclick="openLink('monitoring/dashboards/builder/0ea8ed34-409c-4fa4-b663-d8e282f3b3a2')">
📊 System Dashboard
</a>
<a onclick="openLink('logs/query;query=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22karaoke-backend%22%0Aseverity%3E%3DERROR;duration=PT1H')">
🚨 Recent Errors (last hour)
</a>
<a onclick="openLink('cloudtasks')">
📥 Cloud Tasks Queues
</a>
<a onclick="openLink('run/detail/us-central1/karaoke-backend/metrics')">
☁️ Cloud Run Service
</a>
</div>
<!-- Job Pipeline Guide -->
<div class="guide-section">
<div class="guide-header" onclick="toggleGuide('pipeline')">
<h3>📚 Understanding the Job Pipeline</h3>
<span class="toggle">▼</span>
</div>
<div class="guide-content" id="guide-pipeline">
<h4>Job Processing Flow</h4>
<p>When a job is submitted, it goes through these stages:</p>
<div class="pipeline-flow">
<span class="pipeline-step">Job Created</span>
<span class="pipeline-arrow">→</span>
<span class="pipeline-step">Audio Worker</span>
<span class="pipeline-arrow">→</span>
<span class="pipeline-step">Lyrics Worker</span>
<span class="pipeline-arrow">→</span>
<span class="pipeline-step">Screens Worker</span>
<span class="pipeline-arrow">→</span>
<span class="pipeline-step">Human Review</span>
<span class="pipeline-arrow">→</span>
<span class="pipeline-step">Render Video</span>
<span class="pipeline-arrow">→</span>
<span class="pipeline-step">Video Worker</span>
</div>
<h4>Worker Descriptions</h4>
<ul>
<li><strong>Audio Worker:</strong> Downloads audio, separates vocals/instrumental using Modal API (5-10 min)</li>
<li><strong>Lyrics Worker:</strong> Transcribes lyrics via AudioShake, fetches from Genius, auto-corrects (2-5 min)</li>
<li><strong>Screens Worker:</strong> Generates title and end screen images</li>
<li><strong>Render Video Worker:</strong> After human review, renders the karaoke video with corrections</li>
<li><strong>Video Worker:</strong> Final encoding, thumbnail generation, distribution (YouTube/Dropbox/GDrive)</li>
</ul>
<h4>Job Statuses</h4>
<ul>
<li><span class="status-badge info">searching_audio</span> Looking for audio source</li>
<li><span class="status-badge info">downloading</span> Downloading/preparing input files</li>
<li><span class="status-badge info">separating</span> Audio separation in progress (Modal)</li>
<li><span class="status-badge info">transcribing</span> Lyrics transcription (AudioShake)</li>
<li><span class="status-badge warning">awaiting_review</span> Ready for human review</li>
<li><span class="status-badge info">rendering</span> Video generation in progress</li>
<li><span class="status-badge success">complete</span> Job finished successfully</li>
<li><span class="status-badge error">failed</span> Job encountered an error</li>
</ul>
</div>
</div>
<!-- Debugging Guide -->
<div class="guide-section">
<div class="guide-header" onclick="toggleGuide('debugging')">
<h3>🔍 How to Debug a Failed Job</h3>
<span class="toggle">▼</span>
</div>
<div class="guide-content" id="guide-debugging">
<h4>Step 1: Check the Logs</h4>
<p>Enter the job ID above and click "View Logs". Look for:</p>
<ul>
<li><span class="status-badge error">ERROR</span> severity entries - these show what went wrong</li>
<li>Messages like <code>Created Cloud Task for audio worker</code> - confirms task was queued</li>
<li>Messages like <code>AUDIO WORKER STARTED</code> - confirms worker began processing</li>
<li>The last log entry before failure - often reveals the cause</li>
</ul>
<h4>Step 2: Check Cloud Tasks</h4>
<p>Go to <strong>Cloud Tasks Queues</strong> and look for:</p>
<ul>
<li><strong>Queue depth</strong> - if > 0, tasks are waiting to be processed</li>
<li><strong>Failed tasks</strong> - click into the queue to see failed task details</li>
<li><strong>Retry count</strong> - tasks retry up to 5 times before failing permanently</li>
</ul>
<h4>Step 3: Check Trace (for timing issues)</h4>
<p>Click "View Trace" to see the distributed trace. Look for:</p>
<ul>
<li>Spans with long durations - indicates slow operations</li>
<li>Missing spans - worker may have crashed before creating span</li>
<li>Error annotations on spans</li>
</ul>
<h4>Step 4: Common Issues</h4>
<table style="width:100%; color:#aaa; font-size:0.9rem; border-collapse:collapse;">
<tr style="border-bottom:1px solid #333;">
<td style="padding:10px 0;"><strong>401 on worker endpoints</strong></td>
<td>Auth token issue - check ADMIN_TOKENS secret is set</td>
</tr>
<tr style="border-bottom:1px solid #333;">
<td style="padding:10px 0;"><strong>Job stuck at "downloading"</strong></td>
<td>Audio download failed - check if source URL is valid</td>
</tr>
<tr style="border-bottom:1px solid #333;">
<td style="padding:10px 0;"><strong>No worker logs appear</strong></td>
<td>Cloud Task failed to trigger - check Cloud Tasks queue</td>
</tr>
<tr style="border-bottom:1px solid #333;">
<td style="padding:10px 0;"><strong>Modal API timeout</strong></td>
<td>Audio separation took > 30min - try a shorter track</td>
</tr>
<tr>
<td style="padding:10px 0;"><strong>AudioShake error</strong></td>
<td>Transcription failed - check API key and credits</td>
</tr>
</table>
</div>
</div>
<!-- CLI Commands Guide -->
<div class="guide-section">
<div class="guide-header" onclick="toggleGuide('commands')">
<h3>💻 Useful gcloud Commands</h3>
<span class="toggle">▼</span>
</div>
<div class="guide-content" id="guide-commands">
<h4>View logs for a specific job</h4>
<pre><code id="cmd1">gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND "YOUR_JOB_ID"' \
--project=nomadkaraoke --limit=50 --format="table(timestamp,severity,textPayload)"</code></pre>
<button class="copy-btn" onclick="copyCommand('cmd1')">📋 Copy</button>
<h4>View recent errors only</h4>
<pre><code id="cmd2">gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND severity>=ERROR' \
--project=nomadkaraoke --limit=20 --freshness=1h --format="table(timestamp,jsonPayload.message,textPayload)"</code></pre>
<button class="copy-btn" onclick="copyCommand('cmd2')">📋 Copy</button>
<h4>Check Cloud Tasks queue status</h4>
<pre><code id="cmd3">gcloud tasks queues list --location=us-central1 --project=nomadkaraoke</code></pre>
<button class="copy-btn" onclick="copyCommand('cmd3')">📋 Copy</button>
<h4>View pending tasks in a queue</h4>
<pre><code id="cmd4">gcloud tasks list --queue=audio-worker-queue --location=us-central1 --project=nomadkaraoke</code></pre>
<button class="copy-btn" onclick="copyCommand('cmd4')">📋 Copy</button>
<h4>Check Cloud Run service status</h4>
<pre><code id="cmd5">gcloud run services describe karaoke-backend --region=us-central1 --project=nomadkaraoke \
--format="yaml(status.conditions)"</code></pre>
<button class="copy-btn" onclick="copyCommand('cmd5')">📋 Copy</button>
<h4>Get job data from Firestore (via API)</h4>
<pre><code id="cmd6">curl -H "Authorization: Bearer YOUR_TOKEN" \
"https://karaoke-backend-723537321261.us-central1.run.app/api/jobs/YOUR_JOB_ID"</code></pre>
<button class="copy-btn" onclick="copyCommand('cmd6')">📋 Copy</button>
<h4>Force redeploy Cloud Run</h4>
<pre><code id="cmd7">gcloud run services update karaoke-backend --region=us-central1 --project=nomadkaraoke \
--update-labels=force-redeploy=$(date +%s)</code></pre>
<button class="copy-btn" onclick="copyCommand('cmd7')">📋 Copy</button>
</div>
</div>
<!-- Log Patterns Guide -->
<div class="guide-section">
<div class="guide-header" onclick="toggleGuide('patterns')">
<h3>📝 Log Patterns to Look For</h3>
<span class="toggle">▼</span>
</div>
<div class="guide-content" id="guide-patterns">
<h4>Successful Worker Execution</h4>
<pre><code>[job:abc123] Created Cloud Task for audio worker: projects/.../tasks/123
[job:abc123] === AUDIO WORKER STARTED ===
[job:abc123] Downloading audio file...
[job:abc123] Audio downloaded: track.flac
[job:abc123] Starting audio separation...
[job:abc123] Audio separation complete!
[job:abc123] Uploaded stems to GCS</code></pre>
<h4>Authentication Failure (401)</h4>
<pre><code>POST /api/internal/workers/audio HTTP/1.1" 401
Authentication failed: Invalid token. Token provided: True, Token length: 820</code></pre>
<p><strong>Cause:</strong> Cloud Tasks OIDC token overwriting admin token (fixed in v0.75.3+)</p>
<h4>Worker Timeout</h4>
<pre><code>DeadlineExceeded: 504 Deadline Exceeded
The request deadline was exceeded.</code></pre>
<p><strong>Cause:</strong> Worker took longer than Cloud Run timeout (default 30min)</p>
<h4>External API Failure</h4>
<pre><code>Modal API error: Connection refused
AudioShake error: 429 Too Many Requests
Genius API: 401 Unauthorized</code></pre>
<p><strong>Cause:</strong> External service issue - check API keys and rate limits</p>
<h4>File Not Found</h4>
<pre><code>Failed to download audio file
GCS object not found: gs://bucket/path/file.flac</code></pre>
<p><strong>Cause:</strong> Input file missing or path incorrect</p>
</div>
</div>
<p class="hint">
Tip: Press Enter after typing the job ID to view logs. All guides above are collapsible - click to expand.
</p>
</div>
<script>
const PROJECT = 'nomadkaraoke';
const SERVICE = 'karaoke-backend';
const STORAGE_KEY = 'karaoke-inspector-authuser';
const GUIDE_STATE_KEY = 'karaoke-inspector-guides';
// Load saved authUser and guide states on page load
document.addEventListener('DOMContentLoaded', function() {
// Load auth user
const savedAuth = localStorage.getItem(STORAGE_KEY);
if (savedAuth !== null) {
document.getElementById('authUser').value = savedAuth;
}
// Load guide states (which ones are open)
const savedGuides = JSON.parse(localStorage.getItem(GUIDE_STATE_KEY) || '{}');
Object.keys(savedGuides).forEach(guideId => {
if (savedGuides[guideId]) {
const content = document.getElementById('guide-' + guideId);
const header = content.previousElementSibling;
content.classList.add('open');
header.classList.add('open');
}
});
});
function saveAuthUser() {
const value = document.getElementById('authUser').value;
localStorage.setItem(STORAGE_KEY, value);
}
function getAuthUser() {
return document.getElementById('authUser').value;
}
function toggleGuide(guideId) {
const content = document.getElementById('guide-' + guideId);
const header = content.previousElementSibling;
content.classList.toggle('open');
header.classList.toggle('open');
// Save state
const savedGuides = JSON.parse(localStorage.getItem(GUIDE_STATE_KEY) || '{}');
savedGuides[guideId] = content.classList.contains('open');
localStorage.setItem(GUIDE_STATE_KEY, JSON.stringify(savedGuides));
}
function buildUrl(path) {
const authUser = getAuthUser();
return `https://console.cloud.google.com/${path}?project=${PROJECT}&authuser=${authUser}`;
}
function openLink(path) {
window.open(buildUrl(path), '_blank');
}
function getJobId() {
const jobId = document.getElementById('jobId').value.trim();
if (!jobId) {
alert('Please enter a Job ID');
return null;
}
return jobId;
}
function openLogs() {
const jobId = getJobId();
if (!jobId) return;
const authUser = getAuthUser();
// Build the Logs Explorer query - search both structured (jsonPayload) and text formats
const query = `resource.type="cloud_run_revision"
resource.labels.service_name="${SERVICE}"
("${jobId}" OR jsonPayload.job_id="${jobId}")`;
const encodedQuery = encodeURIComponent(query);
const url = `https://console.cloud.google.com/logs/query;query=${encodedQuery};duration=PT6H?project=${PROJECT}&authuser=${authUser}`;
window.open(url, '_blank');
}
function openTrace() {
const jobId = getJobId();
if (!jobId) return;
const authUser = getAuthUser();
// Cloud Trace search - filter spans containing job_id in the name or attributes
const filter = encodeURIComponent(`+span_name:${jobId}`);
const url = `https://console.cloud.google.com/traces/list?project=${PROJECT}&filter=${filter}&authuser=${authUser}`;
window.open(url, '_blank');
}
function copyCommand(elementId) {
const code = document.getElementById(elementId).textContent;
navigator.clipboard.writeText(code).then(() => {
const btn = document.querySelector(`button[onclick="copyCommand('${elementId}')"]`);
btn.textContent = '✓ Copied!';
btn.classList.add('copied');
setTimeout(() => {
btn.textContent = '📋 Copy';
btn.classList.remove('copied');
}, 2000);
});
}
// Handle Enter key
document.getElementById('jobId').addEventListener('keypress', function(e) {
if (e.key === 'Enter') {
openLogs();
}
});
</script>
</body>
</html>
+32 -1
View File
@@ -4,4 +4,35 @@ import warnings
warnings.filterwarnings("ignore", category=SyntaxWarning, module="pydub.*")
warnings.filterwarnings("ignore", category=SyntaxWarning, module="syrics.*")
from .karaoke_gen import KaraokePrep
# Lazy imports to avoid loading heavy dependencies when not needed.
# KaraokePrep has many dependencies (lyrics_transcriber, etc.) which may not
# be available in all contexts (e.g., backend which only needs audio_fetcher).
#
# Use explicit imports instead:
# from karaoke_gen.karaoke_gen import KaraokePrep
# from karaoke_gen.audio_fetcher import FlacFetcher, AudioSearchResult, ...
__all__ = [
"KaraokePrep",
# Audio fetcher
"FlacFetcher",
"AudioSearchResult",
"AudioFetchResult",
"AudioFetcherError",
"NoResultsError",
"DownloadError",
"UserCancelledError",
]
def __getattr__(name):
"""Lazy import for heavy modules."""
if name == "KaraokePrep":
from .karaoke_gen import KaraokePrep
return KaraokePrep
elif name in ("FlacFetcher", "AudioSearchResult", "AudioFetchResult",
"AudioFetcherError", "NoResultsError", "DownloadError",
"UserCancelledError"):
from . import audio_fetcher
return getattr(audio_fetcher, name)
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+339 -63
View File
@@ -13,8 +13,8 @@ import tempfile
import threading
from abc import ABC, abstractmethod
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
from dataclasses import dataclass
from typing import List, Optional
from dataclasses import dataclass, asdict, field
from typing import List, Optional, Dict, Any
# Global flag to track if user requested cancellation via Ctrl+C
_interrupt_requested = False
@@ -22,7 +22,14 @@ _interrupt_requested = False
@dataclass
class AudioSearchResult:
"""Represents a single search result for audio."""
"""Represents a single search result for audio.
Used by both local CLI and cloud backend. Supports serialization
for Firestore storage via to_dict()/from_dict().
For rich display, this class can serialize the full flacfetch Release
data so remote CLIs can use flacfetch's shared display functions.
"""
title: str
artist: str
@@ -31,13 +38,75 @@ class AudioSearchResult:
duration: Optional[int] = None # Duration in seconds
quality: Optional[str] = None # e.g., "FLAC", "320kbps", etc.
source_id: Optional[str] = None # Unique ID from the source
# Raw result object from the provider (for download)
raw_result: Optional[object] = None
index: int = 0 # Index in the results list (for API selection)
seeders: Optional[int] = None # Number of seeders (for torrent sources)
target_file: Optional[str] = None # Target filename in the release
# Raw result object from the provider (for download) - not serialized
raw_result: Optional[object] = field(default=None, repr=False)
def to_dict(self) -> Dict[str, Any]:
"""Convert to dict for JSON/Firestore serialization.
Includes full flacfetch Release data if available, enabling
remote CLIs to use flacfetch's shared display functions.
"""
result = {
"title": self.title,
"artist": self.artist,
"url": self.url,
"provider": self.provider,
"duration": self.duration,
"quality": self.quality,
"source_id": self.source_id,
"index": self.index,
"seeders": self.seeders,
"target_file": self.target_file,
}
# If we have a raw_result (flacfetch Release), include its full data
# This enables rich display on the remote CLI
if self.raw_result:
try:
release_dict = self.raw_result.to_dict()
# Merge Release fields into result (they may override basic fields)
for key in ['year', 'label', 'edition_info', 'release_type', 'channel',
'view_count', 'size_bytes', 'target_file_size', 'track_pattern',
'match_score', 'formatted_size', 'formatted_duration',
'formatted_views', 'is_lossless', 'quality_str', 'quality']:
if key in release_dict:
# Use 'quality_data' for the quality dict to avoid confusion with quality string
result_key = 'quality_data' if key == 'quality' else key
result[result_key] = release_dict[key]
except AttributeError:
pass # raw_result doesn't have to_dict() method
return result
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioSearchResult":
"""Create from dict (e.g., from Firestore)."""
return cls(
title=data.get("title", ""),
artist=data.get("artist", ""),
url=data.get("url", ""),
provider=data.get("provider", "Unknown"),
duration=data.get("duration"),
quality=data.get("quality"),
source_id=data.get("source_id"),
index=data.get("index", 0),
seeders=data.get("seeders"),
target_file=data.get("target_file"),
raw_result=None, # Not stored in serialized form
)
@dataclass
class AudioFetchResult:
"""Result of an audio fetch operation."""
"""Result of an audio fetch operation.
Used by both local CLI and cloud backend. Supports serialization
for Firestore storage via to_dict()/from_dict().
"""
filepath: str
artist: str
@@ -45,6 +114,22 @@ class AudioFetchResult:
provider: str
duration: Optional[int] = None
quality: Optional[str] = None
def to_dict(self) -> Dict[str, Any]:
"""Convert to dict for JSON/Firestore serialization."""
return asdict(self)
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioFetchResult":
"""Create from dict (e.g., from Firestore)."""
return cls(
filepath=data.get("filepath", ""),
artist=data.get("artist", ""),
title=data.get("title", ""),
provider=data.get("provider", "Unknown"),
duration=data.get("duration"),
quality=data.get("quality"),
)
class AudioFetcherError(Exception):
@@ -160,13 +245,17 @@ class FlacFetchAudioFetcher(AudioFetcher):
This provides access to multiple audio sources including private music trackers
and YouTube, with intelligent prioritization of high-quality sources.
Also exported as FlacFetcher for shorter name.
"""
def __init__(
self,
logger: Optional[logging.Logger] = None,
redacted_api_key: Optional[str] = None,
red_api_key: Optional[str] = None,
red_api_url: Optional[str] = None,
ops_api_key: Optional[str] = None,
ops_api_url: Optional[str] = None,
provider_priority: Optional[List[str]] = None,
):
"""
@@ -174,15 +263,60 @@ class FlacFetchAudioFetcher(AudioFetcher):
Args:
logger: Logger instance for output
redacted_api_key: API key for Redacted tracker (optional)
red_api_key: API key for RED tracker (optional)
red_api_url: Base URL for RED tracker API (optional, required if using RED)
ops_api_key: API key for OPS tracker (optional)
ops_api_url: Base URL for OPS tracker API (optional, required if using OPS)
provider_priority: Custom provider priority order (optional)
"""
self.logger = logger or logging.getLogger(__name__)
self._redacted_api_key = redacted_api_key or os.environ.get("REDACTED_API_KEY")
self._red_api_key = red_api_key or os.environ.get("RED_API_KEY")
self._red_api_url = red_api_url or os.environ.get("RED_API_URL")
self._ops_api_key = ops_api_key or os.environ.get("OPS_API_KEY")
self._ops_api_url = ops_api_url or os.environ.get("OPS_API_URL")
self._provider_priority = provider_priority
self._manager = None
self._transmission_available = None # Cached result of Transmission check
def _check_transmission_available(self) -> bool:
"""
Check if Transmission daemon is available for torrent downloads.
This prevents adding tracker providers (RED/OPS) when Transmission
isn't running, which would result in search results that can't be downloaded.
Returns:
True if Transmission is available and responsive, False otherwise.
"""
if self._transmission_available is not None:
self.logger.info(f"[Transmission] Using cached status: available={self._transmission_available}")
return self._transmission_available
host = os.environ.get("TRANSMISSION_HOST", "localhost")
port = int(os.environ.get("TRANSMISSION_PORT", "9091"))
self.logger.info(f"[Transmission] Checking availability at {host}:{port}")
try:
import transmission_rpc
self.logger.info(f"[Transmission] transmission_rpc imported successfully")
client = transmission_rpc.Client(host=host, port=port, timeout=5)
self.logger.info(f"[Transmission] Client created, calling session_stats()...")
# Simple test to verify connection works
stats = client.session_stats()
self.logger.info(f"[Transmission] Connected! Download dir: {getattr(stats, 'download_dir', 'unknown')}")
self._transmission_available = True
except ImportError as e:
self._transmission_available = False
self.logger.warning(f"[Transmission] transmission_rpc not installed: {e}")
except Exception as e:
self._transmission_available = False
self.logger.warning(f"[Transmission] Connection failed to {host}:{port}: {type(e).__name__}: {e}")
self.logger.info(f"[Transmission] Final status: available={self._transmission_available}")
return self._transmission_available
def _get_manager(self):
"""Lazily initialize and return the FetchManager."""
@@ -193,30 +327,50 @@ class FlacFetchAudioFetcher(AudioFetcher):
from flacfetch.downloaders.youtube import YoutubeDownloader
# Try to import TorrentDownloader (has optional dependencies)
TorrentDownloader = None
try:
from flacfetch.downloaders.torrent import TorrentDownloader
except ImportError:
TorrentDownloader = None
self.logger.debug("TorrentDownloader not available (missing dependencies)")
self._manager = FetchManager()
# Add providers and downloaders based on available API keys
if self._redacted_api_key:
from flacfetch.providers.redacted import RedactedProvider
# Only add tracker providers if we can actually download from them
# This requires both TorrentDownloader and a running Transmission daemon
has_torrent_downloader = TorrentDownloader is not None
transmission_available = self._check_transmission_available()
can_use_trackers = has_torrent_downloader and transmission_available
self.logger.info(
f"[FlacFetcher] Provider setup: TorrentDownloader={has_torrent_downloader}, "
f"Transmission={transmission_available}, can_use_trackers={can_use_trackers}"
)
if not can_use_trackers and (self._red_api_key or self._ops_api_key):
self.logger.warning(
"[FlacFetcher] Tracker providers (RED/OPS) DISABLED: "
f"TorrentDownloader={has_torrent_downloader}, Transmission={transmission_available}. "
"Only YouTube sources will be used."
)
self._manager.add_provider(RedactedProvider(api_key=self._redacted_api_key))
if TorrentDownloader:
self._manager.register_downloader("Redacted", TorrentDownloader())
self.logger.debug("Added Redacted provider")
# Add providers and downloaders based on available API keys and URLs
if self._red_api_key and self._red_api_url and can_use_trackers:
from flacfetch.providers.red import REDProvider
if self._ops_api_key:
self._manager.add_provider(REDProvider(api_key=self._red_api_key, base_url=self._red_api_url))
self._manager.register_downloader("RED", TorrentDownloader())
self.logger.info("[FlacFetcher] Added RED provider with TorrentDownloader")
elif self._red_api_key and not self._red_api_url:
self.logger.warning("[FlacFetcher] RED_API_KEY set but RED_API_URL not set - RED provider disabled")
if self._ops_api_key and self._ops_api_url and can_use_trackers:
from flacfetch.providers.ops import OPSProvider
self._manager.add_provider(OPSProvider(api_key=self._ops_api_key))
if TorrentDownloader:
self._manager.register_downloader("OPS", TorrentDownloader())
self.logger.debug("Added OPS provider")
self._manager.add_provider(OPSProvider(api_key=self._ops_api_key, base_url=self._ops_api_url))
self._manager.register_downloader("OPS", TorrentDownloader())
self.logger.info("[FlacFetcher] Added OPS provider with TorrentDownloader")
elif self._ops_api_key and not self._ops_api_url:
self.logger.warning("[FlacFetcher] OPS_API_KEY set but OPS_API_URL not set - OPS provider disabled")
# Always add YouTube as a fallback provider with its downloader
self._manager.add_provider(YoutubeProvider())
@@ -252,7 +406,7 @@ class FlacFetchAudioFetcher(AudioFetcher):
# Convert to our AudioSearchResult format
search_results = []
for result in results:
for i, result in enumerate(results):
# Get quality as string if it's a Quality object
quality = getattr(result, "quality", None)
quality_str = str(quality) if quality else None
@@ -266,6 +420,9 @@ class FlacFetchAudioFetcher(AudioFetcher):
duration=getattr(result, "duration_seconds", None),
quality=quality_str,
source_id=getattr(result, "info_hash", None),
index=i, # Set index for API selection
seeders=getattr(result, "seeders", None),
target_file=getattr(result, "target_file", None),
raw_result=result,
)
)
@@ -329,6 +486,40 @@ class FlacFetchAudioFetcher(AudioFetcher):
except Exception as e:
raise DownloadError(f"Failed to download {result.artist} - {result.title}: {e}") from e
def select_best(self, results: List[AudioSearchResult]) -> int:
"""
Select the best result from a list of search results.
Uses flacfetch's built-in quality ranking to determine the best source.
This is useful for automated/non-interactive usage.
Args:
results: List of AudioSearchResult objects from search()
Returns:
Index of the best result in the list
"""
if not results:
return 0
manager = self._get_manager()
# Get raw results that have raw_result set
raw_results = [r.raw_result for r in results if r.raw_result is not None]
if raw_results:
try:
best = manager.select_best(raw_results)
# Find index of best result
for i, r in enumerate(results):
if r.raw_result == best:
return i
except Exception as e:
self.logger.warning(f"select_best failed, using first result: {e}")
# Fallback: return first result
return 0
def search_and_download(
self,
artist: str,
@@ -396,13 +587,15 @@ class FlacFetchAudioFetcher(AudioFetcher):
self.logger.info(f"Downloading from {getattr(selected, 'source_name', 'Unknown')}...")
try:
filepath = manager.download(
# Use interruptible download so Ctrl+C works during torrent downloads
filepath = self._interruptible_download(
manager,
selected,
output_path=output_dir,
output_filename=output_filename,
)
if filepath is None:
if not filepath:
raise DownloadError(f"Download returned no file path for: {artist} - {title}")
self.logger.info(f"Downloaded to: {filepath}")
@@ -488,6 +681,111 @@ class FlacFetchAudioFetcher(AudioFetcher):
signal.signal(signal.SIGINT, original_handler)
_interrupt_requested = False
def _interruptible_download(self, manager, selected, output_path: str, output_filename: str) -> str:
"""
Run download in a way that can be interrupted by Ctrl+C.
The flacfetch/transmission download is a blocking operation that doesn't
respond to SIGINT while running (especially for torrent downloads).
This method runs it in a background thread and periodically checks for interrupts.
Args:
manager: The FetchManager instance
selected: The selected result to download
output_path: Directory to save the file
output_filename: Filename to save as
Returns:
Path to the downloaded file
Raises:
UserCancelledError: If user presses Ctrl+C during download
DownloadError: If download fails
"""
global _interrupt_requested
_interrupt_requested = False
result_container = {"filepath": None, "error": None}
was_cancelled = False
def do_download():
try:
result_container["filepath"] = manager.download(
selected,
output_path=output_path,
output_filename=output_filename,
)
except Exception as e:
result_container["error"] = e
# Set up signal handler for immediate response to Ctrl+C
original_handler = signal.getsignal(signal.SIGINT)
def interrupt_handler(signum, frame):
global _interrupt_requested
_interrupt_requested = True
# Print immediately so user knows it was received
print("\nCancelling download... please wait (may take a few seconds)", file=sys.stderr)
signal.signal(signal.SIGINT, interrupt_handler)
try:
# Start download in background thread
download_thread = threading.Thread(target=do_download, daemon=True)
download_thread.start()
# Wait for completion with periodic interrupt checks
while download_thread.is_alive():
download_thread.join(timeout=0.2) # Check every 200ms
if _interrupt_requested:
was_cancelled = True
# Clean up any pending torrents before raising
self._cleanup_transmission_torrents(selected)
raise UserCancelledError("Download cancelled by user (Ctrl+C)")
# Check for errors from the download
if result_container["error"] is not None:
raise result_container["error"]
return result_container["filepath"]
finally:
# Restore original signal handler
signal.signal(signal.SIGINT, original_handler)
_interrupt_requested = False
def _cleanup_transmission_torrents(self, selected) -> None:
"""
Clean up any torrents in Transmission that were started for this download.
Called when a download is cancelled to remove incomplete torrents and their data.
Args:
selected: The selected result that was being downloaded
"""
try:
import transmission_rpc
host = os.environ.get("TRANSMISSION_HOST", "localhost")
port = int(os.environ.get("TRANSMISSION_PORT", "9091"))
client = transmission_rpc.Client(host=host, port=port, timeout=5)
# Get the release name to match against torrents
release_name = getattr(selected, 'name', None) or getattr(selected, 'title', None)
if not release_name:
self.logger.debug("[Transmission] No release name to match for cleanup")
return
# Find and remove matching incomplete torrents
torrents = client.get_torrents()
for torrent in torrents:
# Match by name similarity and incomplete status
if torrent.progress < 100 and release_name.lower() in torrent.name.lower():
self.logger.info(f"[Transmission] Removing cancelled torrent: {torrent.name}")
client.remove_torrent(torrent.id, delete_data=True)
except Exception as e:
# Don't fail the cancellation if cleanup fails
self.logger.debug(f"[Transmission] Cleanup failed (non-fatal): {e}")
def _interactive_select(self, results: list, artist: str, title: str) -> object:
"""
Present search results to the user for interactive selection.
@@ -539,41 +837,9 @@ class FlacFetchAudioFetcher(AudioFetcher):
Raises:
UserCancelledError: If user cancels selection
"""
print(f"\n{'=' * 60}")
print(f"Search Results for: {artist} - {title}")
print(f"{'=' * 60}\n")
for i, result in enumerate(results, 1):
# Use correct flacfetch attribute names
source_name = getattr(result, "source_name", "Unknown")
result_title = getattr(result, "title", "Unknown")
result_artist = getattr(result, "artist", "Unknown")
quality = getattr(result, "quality", None)
duration_seconds = getattr(result, "duration_seconds", None)
seeders = getattr(result, "seeders", None)
target_file = getattr(result, "target_file", None)
# Format duration if available
duration_str = ""
if duration_seconds:
minutes = duration_seconds // 60
seconds = duration_seconds % 60
duration_str = f" [{minutes}:{seconds:02d}]"
# Format quality - it's a Quality object with __str__
quality_str = f" ({quality})" if quality else ""
# Format seeders for torrent sources
seeders_str = f" Seeders: {seeders}" if seeders is not None else ""
# Format target file
file_str = f' "{target_file}"' if target_file else ""
print(f" {i}. [{source_name}] {result_artist} - {result_title}{quality_str}{duration_str}{seeders_str}{file_str}")
print()
print(" 0. Cancel")
print()
# Use flacfetch's shared display function
from flacfetch import print_releases
print_releases(results, target_artist=artist, use_colors=True)
while True:
try:
@@ -598,24 +864,34 @@ class FlacFetchAudioFetcher(AudioFetcher):
raise UserCancelledError("Selection cancelled by user (Ctrl+C)")
# Alias for shorter name - used by backend and other consumers
FlacFetcher = FlacFetchAudioFetcher
def create_audio_fetcher(
logger: Optional[logging.Logger] = None,
redacted_api_key: Optional[str] = None,
red_api_key: Optional[str] = None,
red_api_url: Optional[str] = None,
ops_api_key: Optional[str] = None,
ops_api_url: Optional[str] = None,
) -> AudioFetcher:
"""
Factory function to create an appropriate AudioFetcher instance.
Args:
logger: Logger instance for output
redacted_api_key: API key for Redacted tracker (optional)
red_api_key: API key for RED tracker (optional)
red_api_url: Base URL for RED tracker API (optional, required if using RED)
ops_api_key: API key for OPS tracker (optional)
ops_api_url: Base URL for OPS tracker API (optional, required if using OPS)
Returns:
An AudioFetcher instance
"""
return FlacFetchAudioFetcher(
logger=logger,
redacted_api_key=redacted_api_key,
red_api_key=red_api_key,
red_api_url=red_api_url,
ops_api_key=ops_api_key,
ops_api_url=ops_api_url,
)
+11 -3
View File
@@ -771,8 +771,16 @@ class AudioProcessor:
padded_result["other_stems"] = separation_result.get("other_stems", {})
padded_result["backing_vocals"] = separation_result.get("backing_vocals", {})
self.logger.info(
f"✓ Countdown padding applied to {len(padded_result['combined_instrumentals']) + 1} instrumental file(s)"
)
# Count actual padded files (don't assume clean instrumental was padded)
padded_count = (1 if padded_result["clean_instrumental"].get("instrumental") else 0) + len(padded_result["combined_instrumentals"])
if padded_count > 0:
self.logger.info(
f"✓ Countdown padding applied to {padded_count} instrumental file(s)"
)
else:
self.logger.warning(
"No instrumental files found to pad. Check that audio separation completed successfully."
)
return padded_result
+56 -4
View File
@@ -11,6 +11,7 @@ Similar pattern to LyricsTranscriber's ReviewServer.
import logging
import os
from pathlib import Path
import socket
import threading
import webbrowser
from typing import List, Optional
@@ -358,23 +359,74 @@ class InstrumentalReviewServer:
</body>
</html>"""
@staticmethod
def _is_port_available(host: str, port: int) -> bool:
"""Check if a port is available for binding."""
try:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((host, port))
return True
except OSError:
return False
@staticmethod
def _find_available_port(host: str, preferred_port: int, max_attempts: int = 100) -> int:
"""
Find an available port, starting with the preferred port.
Args:
host: Host to bind to
preferred_port: The preferred port to try first
max_attempts: Maximum number of ports to try
Returns:
An available port number
Raises:
RuntimeError: If no available port could be found
"""
# Try the preferred port first
if InstrumentalReviewServer._is_port_available(host, preferred_port):
return preferred_port
# Try subsequent ports
for offset in range(1, max_attempts):
port = preferred_port + offset
if port > 65535:
break
if InstrumentalReviewServer._is_port_available(host, port):
return port
# Last resort: let the OS assign a port
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.bind((host, 0))
return sock.getsockname()[1]
def start_and_open_browser(self, port: int = 8765) -> str:
"""
Start server, open browser, and block until selection is submitted.
Args:
port: Port to run the server on
port: Preferred port to run the server on. If unavailable, will
automatically find an available port.
Returns:
The user's selection ("clean", "with_backing", or "custom")
"""
self._app = self._create_app()
# Find an available port (handles concurrent CLI instances)
host = "127.0.0.1"
actual_port = self._find_available_port(host, port)
if actual_port != port:
logger.info(f"Port {port} in use, using port {actual_port} instead")
# Run uvicorn in a separate thread
config = uvicorn.Config(
self._app,
host="127.0.0.1",
port=port,
host=host,
port=actual_port,
log_level="warning",
)
server = uvicorn.Server(config)
@@ -389,7 +441,7 @@ class InstrumentalReviewServer:
import time
time.sleep(0.5)
url = f"http://localhost:{port}/"
url = f"http://localhost:{actual_port}/"
logger.info(f"Instrumental review server started at {url}")
# Open browser
@@ -598,7 +598,22 @@
let animationFrameId = null;
let currentAudioElement = null; // Track audio element reference for listener management
const API_BASE = '/api/jobs/local';
// Parse URL parameters for cloud mode
const urlParams = new URLSearchParams(window.location.search);
const encodedBaseApiUrl = urlParams.get('baseApiUrl');
const instrumentalToken = urlParams.get('instrumentalToken');
// Determine API base URL - cloud mode uses provided URL, local mode uses default
const API_BASE = encodedBaseApiUrl
? decodeURIComponent(encodedBaseApiUrl)
: '/api/jobs/local';
// Helper to add token to URL if available
function addTokenToUrl(url) {
if (!instrumentalToken) return url;
const separator = url.includes('?') ? '&' : '?';
return `${url}${separator}instrumental_token=${encodeURIComponent(instrumentalToken)}`;
}
// HTML escape helper to prevent XSS
function escapeHtml(str) {
@@ -617,8 +632,8 @@
async function init() {
try {
const [analysisRes, waveformRes] = await Promise.all([
fetch(`${API_BASE}/instrumental-analysis`),
fetch(`${API_BASE}/waveform-data?num_points=1000`)
fetch(addTokenToUrl(`${API_BASE}/instrumental-analysis`)),
fetch(addTokenToUrl(`${API_BASE}/waveform-data?num_points=1000`))
]);
if (!analysisRes.ok) throw new Error('Failed to load analysis');
@@ -1120,15 +1135,23 @@
}
function getAudioUrl() {
const urls = {
original: '/api/audio/original',
backing: '/api/audio/backing_vocals',
clean: '/api/audio/clean_instrumental',
with_backing: '/api/audio/with_backing',
custom: '/api/audio/custom_instrumental',
uploaded: '/api/audio/uploaded_instrumental'
const stemTypes = {
original: 'original',
backing: 'backing_vocals',
clean: 'clean_instrumental',
with_backing: 'with_backing',
custom: 'custom_instrumental',
uploaded: 'uploaded_instrumental'
};
return urls[activeAudio] || urls.backing;
const stemType = stemTypes[activeAudio] || stemTypes.backing;
// Cloud mode uses /audio-stream/{stem_type}, local mode uses /api/audio/{stem_type}
const isCloudMode = !!encodedBaseApiUrl;
const url = isCloudMode
? `${API_BASE}/audio-stream/${stemType}`
: `/api/audio/${stemType}`;
return addTokenToUrl(url);
}
function formatTime(seconds) {
@@ -1295,7 +1318,7 @@
const formData = new FormData();
formData.append('file', file);
const response = await fetch(`${API_BASE}/upload-instrumental`, {
const response = await fetch(addTokenToUrl(`${API_BASE}/upload-instrumental`), {
method: 'POST',
body: formData
});
@@ -1354,7 +1377,7 @@
}
try {
const response = await fetch(`${API_BASE}/create-custom-instrumental`, {
const response = await fetch(addTokenToUrl(`${API_BASE}/create-custom-instrumental`), {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ mute_regions: muteRegions })
@@ -1404,7 +1427,7 @@
}
try {
const response = await fetch(`${API_BASE}/select-instrumental`, {
const response = await fetch(addTokenToUrl(`${API_BASE}/select-instrumental`), {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ selection: selectedOption })
@@ -1428,9 +1451,25 @@
<div class="success-screen">
<h2>✓ Selection Submitted</h2>
<p>You selected: <strong>${escapeHtml(selectionLabel)}</strong></p>
<p style="color: var(--text-muted);">You can close this window now.</p>
<p id="close-msg" style="color: var(--text-muted);">Closing in <span id="countdown">2</span>s...</p>
</div>
`;
// Auto-close window after 2 seconds
let countdown = 2;
const countdownEl = document.getElementById('countdown');
const countdownInterval = setInterval(() => {
countdown--;
if (countdownEl) countdownEl.textContent = countdown;
if (countdown <= 0) {
clearInterval(countdownInterval);
// Try to close the window (works for windows opened by script)
window.close();
// If window.close() didn't work, update message
const msg = document.getElementById('close-msg');
if (msg) msg.textContent = 'You can close this window now.';
}
}, 1000);
} catch (error) {
showError(error.message);
if (btn) {
@@ -47,6 +47,7 @@ class KaraokeFinalise:
user_youtube_credentials=None, # Add support for pre-stored credentials
server_side_mode=False, # New parameter for server-side deployment
selected_instrumental_file=None, # Add support for pre-selected instrumental file
countdown_padding_seconds=None, # Padding applied to vocals; instrumental must match
):
self.log_level = log_level
self.log_formatter = log_formatter
@@ -108,6 +109,7 @@ class KaraokeFinalise:
self.user_youtube_credentials = user_youtube_credentials
self.server_side_mode = server_side_mode
self.selected_instrumental_file = selected_instrumental_file
self.countdown_padding_seconds = countdown_padding_seconds
self.suffixes = {
"title_mov": " (Title).mov",
@@ -732,6 +734,32 @@ class KaraokeFinalise:
artist, title = base_name.split(" - ", 1)
return base_name, artist, title
def _pad_audio_file(self, input_audio, output_audio, padding_seconds):
"""
Pad an audio file by prepending silence at the beginning.
Uses the same ffmpeg approach as LyricsTranscriber's CountdownProcessor
to ensure consistent padding behavior.
Args:
input_audio: Path to input audio file
output_audio: Path for the padded output file
padding_seconds: Amount of silence to prepend (in seconds)
"""
self.logger.info(f"Padding audio file with {padding_seconds}s of silence")
# Use ffmpeg to prepend silence - this matches the approach in audio_processor.py
# adelay filter adds delay in milliseconds
delay_ms = int(padding_seconds * 1000)
ffmpeg_command = (
f'{self.ffmpeg_base_command} -i "{input_audio}" '
f'-af "adelay={delay_ms}|{delay_ms}" '
f'"{output_audio}"'
)
self.execute_command(ffmpeg_command, f"Padding audio with {padding_seconds}s silence")
def execute_command(self, command, description):
"""Execute a shell command and log the output. For general commands (rclone, etc.)"""
self.logger.info(f"{description}")
@@ -776,11 +804,32 @@ class KaraokeFinalise:
def remux_with_instrumental(self, with_vocals_file, instrumental_audio, output_file):
"""Remux the video with instrumental audio to create karaoke version"""
# Safety net: If countdown padding was applied to vocals, ensure instrumental is padded too
actual_instrumental = instrumental_audio
if self.countdown_padding_seconds and self.countdown_padding_seconds > 0:
# Check if the instrumental file is already padded (has "(Padded)" in name)
if "(Padded)" not in instrumental_audio:
self.logger.warning(
f"Countdown padding ({self.countdown_padding_seconds}s) was applied to vocals, "
f"but instrumental doesn't appear to be padded. Creating padded version..."
)
# Create a padded version of the instrumental
base, ext = os.path.splitext(instrumental_audio)
padded_instrumental = f"{base} (Padded){ext}"
if not os.path.exists(padded_instrumental):
self._pad_audio_file(instrumental_audio, padded_instrumental, self.countdown_padding_seconds)
self.logger.info(f"Created padded instrumental: {padded_instrumental}")
actual_instrumental = padded_instrumental
else:
self.logger.info(f"Using already-padded instrumental: {instrumental_audio}")
# This operation is primarily I/O bound (remuxing), so hardware acceleration doesn't provide significant benefit
# Keep the existing approach but use the new execute method
ffmpeg_command = (
f'{self.ffmpeg_base_command} -an -i "{with_vocals_file}" '
f'-vn -i "{instrumental_audio}" -c:v copy -c:a pcm_s16le "{output_file}"'
f'-vn -i "{actual_instrumental}" -c:v copy -c:a pcm_s16le "{output_file}"'
)
self.execute_command(ffmpeg_command, "Remuxing video with instrumental audio")
Loaded 100 of 114 files, more files were not shown because too many files have changed in this diff. Show more