Compare commits

..
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.6 cfeb30819e Merge main, resolve version conflict (0.119.5)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 14:53:00 -05:00
Andrew BeveridgeandClaude Opus 4.6 85af3c08d5 Fix brand code race condition with Firestore atomic counter
Two concurrent jobs received the same brand code (NOMADNP-0012) because
get_next_brand_code() scanned Dropbox folders to determine the next
number — a TOCTOU race condition. Replace with BrandCodeService that
uses Firestore transactions for atomic allocation. Counter initializes
from Dropbox on first use, then allocates atomically. E2E cleanup
recycles brand codes for reuse. Adds input validation for recycled
numbers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 14:45:21 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 b671b83f19 Fix custom instrumental ignored during encoding (#410)
## Summary
- Fix critical bug: uploaded custom instrumentals were silently replaced
with clean instrumental during encoding
- GCE encoding worker had no `"custom"` case — fell through to clean
instrumental patterns
- Orchestrator config constructed wrong path (`Backing` instead of
`Custom`) for custom selection
- Upload endpoint now converts all formats to FLAC for pipeline
consistency

## Root Cause
When `instrumental_selection == "custom"`, two downstream components
didn't handle it:
1. **GCE encoding worker** (`gce_encoding/main.py`): only had
`with_backing` and default (clean) cases
2. **Orchestrator** (`video_worker_orchestrator.py`): `"custom" !=
"clean"` evaluated to `"Backing"` suffix

The uploaded `.mp3` file also wouldn't match `*.flac` glob patterns even
with the wildcard fallback.

## Changes
- **backend/services/gce_encoding/main.py**: Add `"custom"` case to
instrumental file search
- **backend/workers/video_worker_orchestrator.py**: Add `"custom"` case
for path construction
- **backend/api/routes/jobs.py**: Convert uploaded audio to FLAC before
storing
- **backend/tests/test_custom_instrumental_route.py**: 10 new regression
tests (43 total)
- **pyproject.toml**: Bump 0.119.3 → 0.119.4

## Test plan
- [x] 43 tests passing (10 new: 5 GCE resolution + 5 orchestrator path)
- [x] Regression test: custom selection does NOT fall back to clean
- [x] Regression test: custom selection does NOT produce Backing path
- [x] Non-FLAC upload triggers FLAC conversion
- [ ] Manual: re-test upload + render for affected jobs

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 01:02:05 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d7e12e3443 Add custom instrumental upload endpoint with duration validation (#409)
## Summary
- Add missing `POST /api/jobs/{job_id}/upload-instrumental` endpoint
(was returning 404 in production)
- Validate uploaded audio duration matches original ±0.5s using ffprobe
on GCS signed URLs
- Fix frontend bug: map "uploaded" → "custom" selection type at submit
time (API only accepts "custom")

## Changes
- **backend/api/routes/jobs.py**: New upload endpoint +
`_get_audio_duration_ffprobe_signed` helper
- **frontend/components/instrumental-review/InstrumentalSelector.tsx**:
Selection type mapping fix
- **backend/tests/test_custom_instrumental_route.py**: 22 new tests
(upload endpoint, ffprobe helper, contract)
- **docs/API.md**: Document new endpoint
- **docs/README.md**: Add to recent changes
- **pyproject.toml**: Bump version 0.119.2 → 0.119.3

## Test plan
- [x] 22 new tests covering success, auth, state validation, duration
mismatch, boundary cases
- [x] All 33 existing tests continue to pass
- [ ] Manual verification: upload custom instrumental in review UI

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 23:59:59 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.6 4c55e61cb8 docs: add lessons from private tracks production bugs (#408)
## Summary
- Documents two bugs found during production verification of the private
tracks feature
- **Field mapping gaps**: `is_private` omitted from manual `Job()`
constructor in `create_job()`
- **Return value contract mismatches**: void `update_job()` treated as
boolean, causing all admin PATCH updates to 500

Docs-only change.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 15:04:21 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 6d44089be0 Fix: admin job update always returning 500 (#407)
@coderabbitai ignore

## Summary

Fix admin PATCH endpoint always returning 500 error. Found during
production verification of private tracks feature.

## Root cause

`job_manager.update_job()` returns `None` (no return value), but the
admin PATCH handler checked:
```python
success = job_manager.update_job(job_id, updates)
if not success:  # None is falsy, so this always triggers
    raise HTTPException(status_code=500, ...)
```

## Fix

Changed to `try/except` which properly handles actual errors from the
Firestore update.

## Test plan

- [x] All 25 private tracks tests pass
- [ ] After deploy: verify admin PATCH works for is_private toggle

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 14:22:49 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 b8a107228b Fix: persist is_private flag in job creation (#405)
@coderabbitai ignore

## Summary

- Fix bug where `is_private` was accepted by the API but not persisted
to Firestore
- The `job_manager.create_job()` method was missing
`is_private=job_create.is_private` when constructing the `Job` object

## Root cause

The `create_job` method in `job_manager.py` manually maps fields from
`JobCreate` to `Job`. The `is_private` field was added to both models in
PR #403 but was not added to this mapping, so all jobs were created with
`is_private=False` regardless of the request.

## Changes

- `backend/services/job_manager.py`: Add
`is_private=job_create.is_private` to `Job()` constructor
- `backend/tests/test_private_tracks.py`: Add 2 regression tests
verifying `create_job` persists `is_private`
- `pyproject.toml`: Bump version 0.119.0 → 0.119.1

## Test plan

- [x] New unit tests pass: `test_create_job_persists_is_private`,
`test_create_job_defaults_is_private_false`
- [x] All 25 private tracks tests pass
- [ ] After deploy: create private job via API, verify `is_private=True`
in Firestore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 13:51:13 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 374ce78951 Fix 402 error propagation for credit enforcement (#406)
## Summary

- Fix InsufficientCreditsError being swallowed by catch-all `except
Exception` blocks in route handlers
- Routes were converting 402 errors to 500 because the global exception
handler never saw them

## Root Cause

Route handlers like `search_audio()` wrap `create_job()` in `try/except
Exception` blocks that convert all exceptions to HTTP 500. Our
`InsufficientCreditsError` inherits from `Exception`, so it was caught
and converted before reaching the global
`@app.exception_handler(InsufficientCreditsError)`.

## Fix

Add explicit re-raise for `InsufficientCreditsError` and
`RateLimitExceededError` in all route handlers that call `create_job()`:
- `audio_search.py` - audio search endpoint
- `jobs.py` - direct job creation
- `file_upload.py` - file upload, upload URLs, URL import, finalise-only
(4 endpoints)

## Verified in Production

- Created test user → got 2 welcome credits (confirmed)
- Created job → credit deducted (confirmed)
- Zero credits → got 500 instead of 402 (this bug)
- After this fix → should return 402

## Test plan

- [x] All 13 credit enforcement tests pass
- [ ] Full CI suite
- [ ] Retest 402 in production after deploy

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 13:41:04 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 20453a877c Enforce credit system for karaoke video generation (#404)
## Summary

- Wire up the existing credits system to actually gate job creation -
previously, credits were tracked but never enforced
- Add credit check before job creation, atomic deduction after, and
auto-refund on job failure
- New users now receive 2 welcome credits (was 1)
- Frontend shows zero-credit warning banner, disables submit, and
handles 402 errors with "Buy Credits" link
- Clean up dead credit deduction code from auth_service

## Changes

**Backend:**
- `InsufficientCreditsError` exception with 402 handler
- Credit check + deduction in `JobManager.create_job()` (admin bypass,
atomic deduction, rollback on failure)
- Auto-refund in `JobManager.fail_job()` (skipped for admins,
error-isolated)
- Welcome credits bumped 1 → 2
- Removed dead credit deduction path from
`auth_service.increment_token_usage()`

**Frontend:**
- Zero-credit warning banner at top of job submission form
- Submit buttons disabled when credits = 0
- 402 error handling with "Buy Credits" link
- Boolean state `isCreditError` (not string matching) for credit error
display

**Tests:**
- 13 new backend unit tests covering enforcement, refunds, admin bypass,
atomicity
- 3 new Playwright E2E tests for frontend credit UX
- Tests for 402 exception handler and auth_service cleanup

## Test plan

- [x] Backend unit tests: `pytest tests/test_credit_enforcement.py -v`
(13 tests)
- [x] 402 handler test in `test_main.py`
- [x] Auth service test in `test_auth_firestore.py`
- [x] Frontend E2E tests: `npx playwright test
e2e/regression/credit-enforcement.spec.ts`
- [x] Full suite: `make test` (1928 tests passed)
- [ ] Manual: Create test user with 0 credits → verify 402
- [ ] Manual: Create test user with 1 credit → verify deduction
- [ ] Manual: Fail a job → verify credit refund

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 01:00:06 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 fa9bc8eeeb Add private (non-published) tracks option (#403)
@coderabbitai ignore

## Summary

- Add `is_private` flag to jobs that changes distribution behavior for
non-published tracks
- Private jobs upload to Dropbox `Tracks-NonPublished` folder with
`NOMADNP` brand prefix
- Private jobs skip YouTube and Google Drive uploads entirely
- Completion emails omit YouTube section for private jobs
- Available to all users at job creation; admin can toggle on existing
jobs
- When admin toggles completed job to private, existing outputs are
auto-deleted

## Changes

### Backend
- **Job model**: Added `is_private: bool = False` to `Job` and
`JobCreate`
- **Config**: Added `DEFAULT_PRIVATE_DROPBOX_PATH` and
`DEFAULT_PRIVATE_BRAND_PREFIX` settings
- **Distribution helper**: New `get_effective_distribution_for_job()` in
`job_defaults_service.py` applies private overrides at runtime
- **Orchestrator + native paths**: Both distribution code paths use the
helper (per "Fix Both Sides" lesson)
- **Email template**: YouTube section conditionally removed when
`is_private=True`
- **Credential pre-validation**: Skip YouTube/GDrive credential checks
for private jobs
- **API endpoints**: All 4 job creation endpoints accept `is_private`;
admin can PATCH it
- **Admin auto-delete**: Toggling completed job to private triggers
output deletion

### Frontend
- **Job creation form**: "Private (Non-Published)" checkbox on all 3
tabs (upload, URL, search)
- **Admin dashboard**: Private toggle in job configuration section
- **API interfaces**: Updated TypeScript interfaces and API methods

### Tests
- 23 unit tests covering model, distribution overrides, email template,
notification service, request models, admin toggle, and orchestrator
config
- Fixed MagicMock truthiness issue in existing tests

## Test plan

- [x] All 23 private tracks tests pass
- [x] All 1794 existing backend tests pass (1 flaky unrelated test)
- [x] CodeRabbit review: credential pre-validation issue found and fixed
- [ ] Manual: Create private job end-to-end, verify Dropbox folder
- [ ] Manual: Verify completion email has Dropbox link but no YouTube
section
- [ ] Manual: Admin toggle on completed job deletes outputs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 18:07:57 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 0cee5b8ae3 feat: add beveradb/cryptotrader to WIF and SA IAM binding (#402)
## Summary
- Add `beveradb` org to Workload Identity Federation provider attribute
condition
- Add `beveradb/cryptotrader` repo to the service account IAM binding
for `roles/iam.workloadIdentityUser`
- Syncs Pulumi IaC with GCP drift (changes already applied in GCP)

## Test plan
- [ ] `pulumi preview` shows no drift after merge
- [ ] GitHub Actions from beveradb/cryptotrader can authenticate to GCP

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 19:20:38 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 9726192de0 fix: persist uploaded audio, fix custom instrumental selection (#401)
## Summary
- **Uploaded audio persistence**: Files uploaded to `uploads/` are
auto-deleted by a 7-day GCS lifecycle rule. After processing, copies are
now saved to `jobs/{job_id}/input/` (mirroring URL job behavior) so
review playback works indefinitely.
- **Defensive cache fallback**: When the transcoding service can't find
the source file, it now checks for an existing OGG cache from earlier
eager transcoding before falling back to FLAC.
- **Custom instrumental selection**: The review completion endpoint
rejected `"custom"` instrumental selection for mute-region edits because
it only checked for an uploaded instrumental file. Now also checks for
the `custom_instrumental` stem created by the mute-region editor.
- **Video worker custom stem**: When `"custom"` is selected and there's
no uploaded instrumental, the video worker now downloads the
`custom_instrumental` stem instead of incorrectly falling through to
`instrumental_with_backing`.

## Test plan
- [x] Unit tests for upload persistence (happy path, no-op for jobs/
prefix, JobManager fallback, persistence failure doesn't block download)
- [x] Unit tests for defensive cache fallback (cache exists when source
deleted, file_exists throws in fallback)
- [x] Unit tests for custom instrumental validation (allowed with stem,
allowed with upload, rejected without either)
- [x] Tightened existing test assertions for new code paths
- [x] All 30 new/modified tests passing

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 02:43:11 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d22f1893f8 fix: add missing create-custom-instrumental endpoint and prevent oversized Firestore docs (#400)
## Summary
- Add missing `POST /api/jobs/{id}/create-custom-instrumental` endpoint
that was lost during the unified review UI migration. The endpoint
creates a custom instrumental by muting selected backing vocal regions
and returns a signed audio URL for playback.
- Strip `reference_lyrics` from corrections data before saving to
Firestore `state_data`, preventing 1MB document size limit errors when
Genius fetches non-lyrics content (e.g. screenplays). Full data is
preserved in GCS.
- Update frontend `InstrumentalSelector` to use the signed audio URL
returned from the create endpoint instead of a non-existent stream
endpoint.

## Test plan
- [x] 11 new backend tests covering both fixes (happy path, auth,
validation, error handling, Firestore stripping, GCS preservation)
- [x] Full test suite passes (1912 backend, 168 frontend)
- [ ] Manual test: create custom instrumental on a job in review
- [ ] Manual test: submit corrections for a job with oversized Genius
lyrics

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:25:02 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 1a56ae09c5 feat: change default jobs list filter from 5 to 10 (#399)
## Summary
- Change the default job display limit from 5 to 10 for new users (or
users who haven't customized the setting)
- Users with a saved preference in localStorage are unaffected

## Test plan
- [x] Frontend unit tests pass (323/323)
- [x] Backend unit tests pass (1920/1920)
- [ ] Verify in production: new/incognito browser shows 10 jobs by
default

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 11:19:38 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 f29fa8b5cd fix: filter exclude_statuses in Python for hide_completed (#398)
## Summary
- Fix `hide_completed=true` query param failing with Firestore composite
index error
- Move `exclude_statuses` filtering from Firestore `not-in` to
Python-side filtering
- Documents are tiny after `select()` projection, so Python filtering is
fast

## Context
Follow-up to #397. The `not-in` filter on `status` combined with
`order_by(created_at)` requires a composite index that doesn't exist.
This moves the filtering to Python after the projected query returns.

## Test plan
- [x] All 33 backend route tests pass
- [ ] Verify `hide_completed=true` works in production (no Firestore
index error)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 09:32:34 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 3ea48e487f feat: server-side jobs filtering with field projection (#397)
## Summary
- Add `fields=summary` query param to `GET /api/jobs` — uses Firestore
`select()` to return only ~12 dashboard-required fields instead of 50+
full Job fields, reducing payload from ~16MB to <500KB
- Add `hide_completed=true` query param for server-side filtering of
terminal statuses (complete, prep_complete, failed, cancelled)
- Safety-net prune helpers strip extra keys from `state_data` and
`file_urls` nested dicts
- Backward compatible: without `fields=summary`, endpoint returns full
`List[Job]` as before (admin pages unaffected)

## Test plan
- [x] 8 backend unit tests covering prune helpers, hide_completed,
exclude_test with dicts, backward compatibility
- [x] 5 frontend Jest tests covering query param serialization and
partial summary response handling
- [x] All 2412 existing tests pass (`make test`)
- [ ] Deploy to staging, verify network tab shows reduced payload size
- [ ] Verify JobCard renders correctly with summary data (title, status,
progress, download links, error)
- [ ] Toggle hide-completed — verify server re-fetch with
`hide_completed=true`
- [ ] Verify admin jobs page (`/admin/jobs`) still works (doesn't send
`fields=summary`)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 09:06:23 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 4fe960481b feat: show all incomplete jobs first + hide-completed toggle (#393)
## Summary
- **Priority sorting**: Blocking + processing jobs always appear first
in Recent Jobs, regardless of creation date
- **Hide completed toggle**: Eye/EyeOff button filters out completed and
failed jobs, showing only incomplete ones
- **Persistent preferences**: Both the display limit dropdown and
hide-completed toggle persist in localStorage across page reloads
- **Oversized lyrics truncation fix**: Corrected screen generation for
songs with unusually long lyric lines

## Changes
- `frontend/lib/job-status.ts`: Added `hideCompleted` parameter to
`getDisplayJobs()` — filters by job priority before applying display
limit
- `frontend/app/app/page.tsx`: Added Eye/EyeOff toggle with tooltip,
localStorage persistence for both `jobLimit` and `hideCompleted`
- `frontend/__tests__/job-status.test.ts`: 5 new tests covering
hideCompleted filtering behavior

## Test plan
- [x] All 168 tests pass (`make test`)
- [x] 85 job-status unit tests pass including 5 new hideCompleted tests
- [ ] Visual verification: toggle button appears next to limit dropdown,
state persists on reload

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 16:22:37 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 fe51369d30 Fix test infrastructure: timeouts, IPv6 hang, and 21 stale E2E tests (#396)
## Summary
- Fix `make test` hanging indefinitely by adding Makefile timeouts and
resolving Playwright IPv6 binding issue
- Update 21 stale Playwright E2E regression tests to match current UI
(buttons renamed, auth mock structure, navigation flow changes)
- Remove dead code: unused admin endpoints, audio transcoding service,
screens worker clear functionality

## Changes

### Test Infrastructure
- **Makefile**: Added 10-min timeout for backend tests, 15-min for
frontend, with proper cleanup on failure
- **Playwright config**: Force IPv4 binding (`127.0.0.1:3000`) to
prevent IPv6 connection hang on macOS

### E2E Test Fixes (21 tests across 4 files)
- **admin-job-detail.spec.ts**: Updated toolbar button selectors
(`Start`→`Audio`, `Lyrics`→`Review`), removed 3 obsolete Clear Workers
tests
- **cloud-mode-review.spec.ts**: Fixed auth mock to match
`UserProfileResponse` nested structure (`{user: {...}, has_session:
true}`), added instrumental-analysis and waveform-data mocks
- **combined-review-corrections.spec.ts**: Fixed auth mock structure,
string path for mock routes, missing auth token parameter
- **lyrics-review.spec.ts**: Updated button text (`Complete
Review`→`Proceed to Instrumental Review`), added `.first()` for
multi-element label selectors, rewrote submit tests to verify navigation
behavior

### Component/Library Fixes
- **TimeInput.tsx**: Added `htmlFor`/`id` association for accessibility
and Playwright `getByLabel()` support
- **local-mode.ts**: Broadened hash check (`hash.startsWith('#/')`) so
invalid hash routes correctly detect cloud mode

### Dead Code Removal
- Removed unused `clear_workers` admin endpoint and related worker code
- Removed unused `audio_transcoding_service.py` and its 353-line test
file

## Testing
- [x] All 168 Playwright tests pass (3 intentionally skipped)
- [x] All backend tests pass
- [x] `make test` completes end-to-end without hanging
- [x] CI passed

## Review
- [x] CodeRabbit CLI review completed locally - clean pass

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 15:19:14 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 0cecc86b79 feat: eager audio transcoding for review UI (#395)
## Summary
- Add `AudioTranscodingService` that transcodes FLAC audio to OGG Opus
128kbps (~3 MB vs 35+ MB), cached in GCS at `review-audio/*.ogg`
- Eager transcoding in `screens_worker` before `AWAITING_REVIEW` so
files are cached before the user opens the review UI
- All 3 review audio paths (`_stream_audio`, `get_correction_data`,
`get_instrumental_analysis`) now serve transcoded OGG URLs with FLAC
fallback
- Admin endpoint `POST /admin/jobs/{job_id}/prepare-review-audio` for
backfilling existing jobs

## Test plan
- [x] 21 unit tests at 100% coverage on `AudioTranscodingService`
- [x] 229 existing review tests pass with 0 regressions
- [ ] Deploy and use admin endpoint to backfill job `4bed6f80`
- [ ] Open review UI — verify audio loads fast, Network tab shows ~3 MB
OGG
- [ ] Check GCS for cached files at `jobs/4bed6f80/review-audio/*.ogg`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 13:24:59 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 87e696556d fix: use signed URL redirect for review audio to bypass 32 MiB limit (#394)
## Summary
- Review audio endpoint was proxying files through Cloud Run via
`FileResponse`, hitting the Google Frontend load balancer's 32 MiB
response body size limit
- Files >32 MiB (like the 35 MiB FLAC for job `4bed6f80`) caused 500
errors — app logged 200 internally but LB rejected the response
- Switched to a 302 redirect to a GCS signed URL, matching the pattern
already used for instrumental stems in the same file

## Test plan
- [x] All 28 review route tests pass
- [ ] Verify audio loads in review UI:
https://gen.nomadkaraoke.com/app/jobs/#/4bed6f80/review

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 22:37:37 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 7ac0e07c95 fix: migrate AudioShake from legacy /upload/ to /assets endpoint (#392)
@coderabbitai ignore

## Summary
- AudioShake removed their legacy `/upload/` endpoint, causing all jobs
to fail with `401 Unauthorized`
- Migrated to the new Assets API (`POST /assets`) which returns an
`assetId`
- Updated task creation to use `assetId` instead of `url` per new API
contract

## Test plan
- [x] Verified new `/assets` endpoint works with existing API token
(returns 200 + asset ID)
- [x] Backend unit tests pass (1081 passed, 8 skipped)
- [ ] CI will run full test suite
- [ ] After merge, verify jobs complete lyrics transcription
successfully in production

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 14:26:56 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 6f08a684d1 fix: close audio search dialog immediately after selection (#390)
## Summary
- The `/api/audio-search/{job_id}/select` endpoint was downloading audio
synchronously before returning, keeping the dialog open for 10-60+
seconds
- Refactored into `_validate_and_prepare_selection()` (returns
instantly) and `_download_audio_and_trigger_workers()` (runs as FastAPI
background task)
- Dialog now closes immediately; download proceeds in background with
job status updates visible in the job card

## Test plan
- [x] All 57 backend tests pass (6 new tests added for the refactored
functions)
- [x] Error paths tested: invalid index, job not found, no results,
download failures
- [x] Background task gracefully handles missing jobs and generic
exceptions
- [x] Worker triggering verified after successful download
- [ ] Manual: select audio in popup, verify it closes immediately and
job transitions to "Downloading audio..."

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 12:00:16 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 2f15cfc985 feat(lyrics-review): Add Replace Segment Lyrics mode (#389)
## Summary

- Adds a third mode ("Replace Segment Lyrics") to the Edit All Lyrics
modal, enabling users to swap custom lyrics line-by-line while
preserving existing segment timing
- Extracts `createWordsWithDistributedTiming` shared utility from
EditModal for reuse
- Fixes Makefile `build-frontend` to clean stale output before copying
(was causing old JS chunks to persist)
- Restores accidentally deleted `nextjs_frontend/__init__.py`

## Test plan

- [x] Unit tests for `createWordsWithDistributedTiming` (8 cases)
- [x] Component tests for Replace Segments mode (17 cases)
- [x] E2E Playwright tests (8 cases) 
- [x] All 306 Jest tests pass
- [x] Manual local testing with real audio file via `karaoke-gen` CLI

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 15:39:22 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 9cca4db147 fix(lyrics-review): Enable and improve mode selector buttons (#388)
## Summary
- Fixed non-functional highlight and delete mode buttons in lyrics
review
- Improved mode selector UX with better spacing, icons, and active state
visibility
- Added test infrastructure support for Radix UI components

## Changes
- **ModeSelector.tsx**: Removed `disabled` attributes and fixed onChange
handler to support all modes
- **ModeSelector.tsx**: Enhanced visual design with proper active states
(blue background on selected mode)
- **ModeSelector.tsx**: Improved spacing consistency and changed Edit
icon to Pencil for better visual balance
- **ModeSelector.test.tsx**: Added comprehensive unit tests (100%
coverage)
- **jest.setup.js**: Added ResizeObserver mock for Radix UI ToggleGroup
component
- **pyproject.toml**: Bumped version to 0.116.3

## Testing
- [x] All tests pass (281 frontend + 103 backend tests)
- [x] Unit test coverage: 100% on ModeSelector component
- [x] Manual testing completed via Chrome DevTools MCP
- [x] E2E regression tests pass (existing lyrics review tests)

## Review
- [x] Manual code review completed (CodeRabbit CLI couldn't run due to
447 built files)
- [x] No security, bug, or quality issues found
- [x] Documentation reviewed (no updates needed)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-11 21:41:49 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 1852df9273 fix(gce-encoding): Fix wheel version sorting regression (0.99.9 > 0.116.1) (#387)
## Summary

Critical regression fix from PR #384: Production encoder was running
v0.99.9 instead of v0.116.1 due to alphabetical sorting instead of
semantic version comparison.

**Impact:** GCE encoding worker was using an extremely old version
(0.99.9 from months ago) instead of the latest release (0.116.1),
potentially causing encoding issues or missing recent bug fixes.

## Root Cause

PR #384 fixed invalid wheel filtering but exposed a latent version
sorting bug:
- `sorted(wheels)[-1]` uses string comparison, not semantic versioning
- String comparison: `"0.99.9"` > `"0.116.1"` (because '9' > '1' in
position 2)
- Result: Old version selected over new version

## Changes

**Code (14 lines):**
- Added imports: `re`, `packaging.version.Version`
- Implemented `extract_version()` helper to parse version from filename
- Changed sorting from `sorted(wheels)[-1]` to semantic version sort
- **File:** `backend/services/gce_encoding/main.py`

**Tests (159 lines, 10 new tests):**
- Test exact regression case (0.99.9 vs 0.116.1)
- Demonstrate alphabetical sorting failure
- Test version extraction and edge cases
- Test complex version comparisons (major/minor/patch)
- **File:** `backend/tests/test_gce_encoding_worker.py`

**Documentation:**
- Updated `docs/README.md` with Recent Changes entry
- Added lesson to `docs/LESSONS-LEARNED.md` about semantic version
sorting
- Pattern documented for future reference

**Version:**
- Bumped to v0.116.2 (patch release for bug fix)

## Testing

- [x] All tests pass (24 total, 10 new)
- [x] Test-to-code ratio: 11.4:1
- [x] Edge cases covered (empty list, invalid filenames, complex
versions)
- [x] CodeRabbit review: 0 issues found

## Review

- [x] Local CodeRabbit CLI review completed
- [x] Documentation updated
- [x] All tests passing

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-11 12:38:44 -05:00
Andrew Beveridge ☄️ 30e55a0031 docs: Add GCP cost optimization analysis (#386)
Comprehensive cost analysis document covering:

- Current spend breakdown (~k/month) 
- Implemented GitHub runner auto-scaling (,390/month saved)
- Additional optimization opportunities (Cloud Run, encoding worker)
- Actionable next steps

Document URL will be: `docs/GCP-COST-OPTIMIZATION.md`

<!-- @coderabbitai ignore -->
2026-02-11 17:28:55 +00:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 8b88f43a96 fix(gce-encoding): Fix wheel installation crash by filtering invalid filenames (#384)
## Summary

Fixes GCE encoding worker crash caused by attempting to install
`karaoke_gen-current.whl`, which is not a valid PEP 427 wheel filename.

## Problem

Job 749141f8 failed with "Cannot connect to host" errors. Investigation
revealed:
- GCE encoding worker was crashing and restarting
- Root cause: pip rejected `karaoke_gen-current.whl` with "Invalid wheel
filename (wrong number of parts)"
- The `ensure_latest_wheel()` function downloaded ALL wheels including
the `-current` convenience file
- This file lacks required PEP 427 naming (version, python, abi,
platform tags)

## Solution

Filter out wheels containing '-current' before selecting which to
install:
```python
wheels = glob.glob("/tmp/karaoke_gen-*.whl")
# Filter out karaoke_gen-current.whl (not a valid PEP 427 wheel name)
wheels = [w for w in wheels if '-current' not in w]
```

## Changes

- **Fix** (`backend/services/gce_encoding/main.py`): Add 3-line filter
for valid wheel names
- **Tests** (`backend/tests/test_gce_encoding_worker.py`): Add 5
comprehensive unit tests
  - Current wheel exclusion
  - Empty result handling
  - Version sorting after filtering
  - Pass-through when no current wheel
  - Path-agnostic filtering
- **Docs** (`docs/README.md`): Add to Recent Changes
- **Docs** (`docs/LESSONS-LEARNED.md`): Add pip wheel filename
validation lesson
- **Version**: Bump to 0.116.1 (patch release)

## Testing

- ✅ All 14 GCE encoding worker tests pass (9 existing + 5 new)
- ✅ Backend tests pass
- ✅ Test-to-code ratio: 26:1 (78 test lines for 3 code lines)
- ✅ Tests cover all edge cases and are isolated/fast
- ⚠️ Frontend E2E tests have pre-existing failures (unrelated to backend
changes)

## Review

- ✅ CodeRabbit CLI review completed locally (zero issues found)
- ✅ Code follows project conventions
- ✅ Comprehensive test coverage
- ✅ Documentation updated

## Impact

- Fixes job 749141f8 and prevents future encoding failures
- No breaking changes
- No API changes
- Production-critical bug fix

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-10 16:48:25 -05:00
Andrew Beveridge ☄️ f313fc4208 fix(infra): Update runner manager memory to 512M in config (#385)
Aligns code with deployed infrastructure. Function was seeing OOM with
256M.

<!-- @coderabbitai ignore -->
2026-02-10 16:41:24 -05:00
Andrew Beveridge ☄️ 15d5ed8db1 chore(infra): Trigger deployment for runner autoscaling (#383)
Trigger infrastructure deployment with Cloud Functions admin permission
fixed.

<!-- @coderabbitai ignore -->
2026-02-10 20:52:58 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 fb40b4ac05 feat(infra): Add auto-scaling GitHub runners with Spot VMs (#376)
## Summary

Implement cost optimization for GitHub Actions self-hosted runners:

- Reduce runners from 20 → 3 (auto-scaling handles demand)
- Switch to Spot/Preemptible instances (60-91% cheaper)
- Add Cloud NAT for outbound internet (no external IPs needed)
- Add runner_manager Cloud Function for auto-start/stop

## Cost Impact

| Item | Before | After | Savings |
|------|--------|-------|---------|
| GitHub Runners | 20 VMs × $72/mo = $1,440 | 3 Spot VMs × ~$25 × ~20%
util = ~$50-100 | **~$1,300-1,400/mo** |
| External IPs | 20 × $3 = $60 | $0 (Cloud NAT) | $50/mo |

## How It Works

1. **GitHub sends webhook** → `workflow_job.queued` event
2. **Cloud Function** receives webhook, starts any TERMINATED runner VMs
3. **VMs boot**, register with GitHub, pick up jobs
4. **Cloud Scheduler** triggers idle check every 15 minutes
5. **If no pending jobs and idle > 1 hour** → VMs are stopped

## Manual Setup Required (Post-Deploy)

1. **Create webhook secret:**
```bash
python -c "import secrets; print(secrets.token_hex(32))" | \
  gcloud secrets versions add github-webhook-secret --data-file=- --project=nomadkaraoke
```

2. **Configure GitHub org webhook:**
   - Go to: https://github.com/organizations/nomadkaraoke/settings/hooks
   - Payload URL: `pulumi stack output runner_manager_function_url`
   - Content type: `application/json`
   - Secret: (value from step 1)
   - Events: Select "Workflow jobs"

## Test Plan

- [ ] Pulumi preview succeeds
- [ ] Deploy to prod
- [ ] Configure webhook secret
- [ ] Configure GitHub webhook
- [ ] Trigger a CI workflow, verify runners start
- [ ] Wait 1+ hour with no activity, verify runners stop

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-10 15:43:30 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 716936c8a9 Add retry logic for flacfetch download failures and audio search recovery (#382)
## Summary

Adds comprehensive retry logic to handle transient flacfetch network
failures, preventing permanent job failures from temporary service
issues. Includes state persistence for retry capability, enhanced retry
endpoint for audio search jobs, and user-facing error feedback.

## Key Changes

**Retry Logic (backend/services/flacfetch_client.py):**
- Manual exponential backoff: 9 attempts over ~6 minutes (10s → 20s →
40s → 60s capped)
- Selective retry: retries on connection errors and 5xx, immediate fail
on 4xx
- Runtime-configurable via environment variables
(FLACFETCH_RETRY_MAX_ATTEMPTS, etc.)

**State Persistence (backend/api/routes/audio_search.py):**
- Save download parameters (audio_source_type, source_name, source_id,
target_file, download_url) BEFORE attempting download
- Enables retry even if download fails completely
- Added 5 new fields to Job model (backend/models/job.py)

**Enhanced Retry Endpoint (backend/api/routes/jobs.py):**
- Added ~150 lines handling audio search retry cases
- Re-downloads using saved parameters for YouTube, RED, OPS, Spotify
sources
- Clears error state and transitions to DOWNLOADING before triggering
workers

**User Feedback (frontend/components/job/JobActions.tsx):**
- Error toast shows actual API error message on retry failure
- Success toast confirms retry started
- No more silent failures

**Configuration (backend/config.py):**
- Added retry config: max_attempts (9), min_wait (10s), max_wait (60s)

## Testing

- [x] Added 5 comprehensive retry tests in
`backend/tests/test_flacfetch_client.py`
  - Network error retry with exponential backoff
  - No retry on 4xx client errors
  - Retry on 5xx server errors
  - Max retries exhausted
  - Wrapper consistency
- [x] All backend tests pass: 103 passed
- [x] Frontend E2E tests: 142 passed (21 pre-existing failures unrelated
to changes)

## Review

- [x] CodeRabbit CLI review completed locally
- [x] Fixed unused import identified in review
- [x] Documentation updated (README.md Recent Changes,
LESSONS-LEARNED.md retry patterns)
- [x] Implementation plan archived to
docs/archive/2026-02-10-flacfetch-retry-logic.md

## Root Cause

Job 749141f8 failed because flacfetch network call failed during audio
search, and no source info was saved, making retry impossible. This PR
fixes all four related issues identified during investigation.

## Version

Bumped 0.115.5 → 0.116.0 (minor version for new feature)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-10 14:21:14 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 89f49ffd48 Fix CDG sync issue: strip countdown padding for CDG output (#380)
## Summary

- Fix CDG+MP3 package sync issue where lyrics appeared 3 seconds early
- The CDG was using padded LRC timestamps with unpadded instrumental
audio
- CDGGenerator now strips countdown segment and adjusts timestamps when
needed

## The Bug

The MP4 video has a 3-second countdown ("3... 2... 1...") prepended, and
LRC timestamps are shifted forward by 3 seconds to match. However, the
CDG+MP3 package uses the original instrumental audio (no countdown), but
was using the shifted timestamps.

Result: CDG lyrics appeared 3 seconds too early.

## The Fix

1. Added `lrc_has_countdown_padding` flag to `OrchestratorConfig`
2. Flag is read from job's `lyrics_metadata.has_countdown_padding`
3. Flag is passed through `PackagingService` to `CDGGenerator`
4. When True, `CDGGenerator` strips the countdown segment from LRC and
shifts all timestamps back by 3 seconds

## Test Plan

- [x] Added 11 unit tests for countdown removal logic
- [x] Added packaging service integration test
- [x] Added orchestrator countdown pipeline tests
- [x] All 103 emulator tests pass
- [x] Verified manually with playable CDG+MP3 output

## Files Changed

- `karaoke_gen/lyrics_transcriber/output/cdg.py` - Core fix
- `backend/workers/video_worker_orchestrator.py` - Pipeline changes
- `backend/services/packaging_service.py` - Param passing
- `backend/tests/test_cdg_countdown.py` - New test file
- `backend/tests/test_video_worker_orchestrator.py` - Pipeline tests
- `docs/LESSONS-LEARNED.md` - Documented the pattern
- `docs/README.md` - Added to known issues

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

@coderabbitai ignore

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 12:18:23 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 b7c2ae185f Allow admin restart from in_review state (#381)
## Summary

Fixes 400 error when attempting to restart a job that's in `in_review`
state.

## Problem

Job 1eab1172 failed to restart with error:
```
400: Cannot restart job in 'in_review' state. 
Allowed states: awaiting_review, ...
```

The job was in `in_review` state (user had opened the review UI), but
the restart endpoint only allowed `awaiting_review` (waiting for review
to start).

## Root Cause

`RESTART_ALLOWED_STATES` in `backend/api/routes/admin.py:2036` didn't
include `in_review` state.

**State distinction:**
- `awaiting_review` - Job waiting for user to open review UI
- `in_review` - User has opened review UI and is actively reviewing

## Solution

Add `in_review` to `RESTART_ALLOWED_STATES`:

```python
RESTART_ALLOWED_STATES = {
    "pending",
    "complete",
    "failed",
    "awaiting_review",
    "in_review",  # ← Added
    "awaiting_audio_selection",
    "awaiting_instrumental_selection",
    "instrumental_selected",
    "prep_complete",
}
```

## Changes

- `backend/api/routes/admin.py` - Add `in_review` to allowed states
- `pyproject.toml` - Bump to 0.115.5

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 12:17:32 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 af4b3a7c29 Enable AudioShake language auto-detection for international lyrics (#379)
## Summary

Fixes lyrics translation issue where non-English songs (like Russian)
were being translated to English instead of transcribed in their
original language.

## Problem

Job 1eab1172 with Russian lyrics "я куплю тебе дом" was transcribed by
AudioShake, but the output was in English instead of Russian. The
transcription API was translating the lyrics rather than preserving the
original language.

## Root Cause

`karaoke_gen/lyrics_transcriber/transcribers/audioshake.py:152` was
hardcoding `"language": "en"` in the AudioShake API request, which
forced English output regardless of the audio's actual language.

## Solution

Remove the hardcoded language parameter to enable AudioShake's built-in
language auto-detection:

```python
# Before (forced English):
"targets": [{
    "model": "alignment",
    "formats": ["json"],
    "language": "en"  # ← Forced English
}]

# After (auto-detect):
"targets": [{
    "model": "alignment",
    "formats": ["json"]
    # Language omitted - AudioShake auto-detects from 100+ languages
}]
```

## Testing

Verified with direct AudioShake API test:
- Created task WITHOUT language parameter → HTTP 200 ✓
- Response shows `language: null` → Auto-detection enabled ✓
- AudioShake supports 100+ languages including Russian, Chinese,
Spanish, Arabic, etc.

## Impact

- ✅ Russian, Chinese, and other non-English songs will now be
transcribed correctly
- ✅ No configuration needed - works globally with all languages
- ✅ Existing English songs continue to work (auto-detected as English)

## Changes

- `karaoke_gen/lyrics_transcriber/transcribers/audioshake.py` - Remove
hardcoded language parameter
- `docs/LESSONS-LEARNED.md` - Document pattern for API integration
- `pyproject.toml` - Bump to 0.115.4

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 11:45:23 -08:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 67cd6494b4 Fix admin restart state transition for preserve_audio_stems mode (#377)
## Summary

Fixes a state machine validation error when using the admin "Full
Restart" button with preserve_audio_stems mode.

## Problem

When restarting job 1eab1172 after the Unicode fix (PR #375), the
restart failed with:
```
Invalid state transition for job 1eab1172: transcribing -> JobStatus.GENERATING_SCREENS. 
Valid transitions: ['correcting', 'failed']
```

## Root Cause

The restart endpoint was setting the job status to `transcribing`, then
triggering the screens worker. The screens worker immediately tries to
transition to `GENERATING_SCREENS`, but `transcribing ->
generating_screens` is not a valid transition according to the state
machine in `backend/models/job.py`.

## Fix

Changed the restart endpoint to set status to `downloading` instead of
`transcribing`. According to STATE_TRANSITIONS:
- ✅ `downloading` → `generating_screens` (valid)
- ✅ `audio_complete` → `generating_screens` (valid)
- ✅ `lyrics_complete` → `generating_screens` (valid)
- ❌ `transcribing` → `generating_screens` (invalid)

Using `downloading` is the safest choice as it's the most flexible entry
state.

## Changes

- `backend/api/routes/admin.py:2161` - Change status from "transcribing"
to "downloading"
- `docs/LESSONS-LEARNED.md` - Added lesson about validating state
transitions in admin operations
- `pyproject.toml` - Bump version to 0.115.3

## Testing

- Existing tests pass (8 pre-existing emulator test failures unrelated
to this change)
- Will verify in production by restarting job 1eab1172 after deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-05 18:32:46 +00:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 d9bb8b3181 fix: sanitize Unicode in temp file paths (FFmpeg exit 183) (#375)
## Summary
Fixes job 1eab1172 which failed with FFmpeg exit code 183 due to
Cyrillic characters in temp subtitle file paths. Python's
`str.isalnum()` returns True for ALL Unicode alphanumeric characters
(Cyrillic, Arabic, Chinese, emoji, etc.), not just ASCII.

## Root Cause
- Job `1eab1172` with Russian title "я куплю тебе дом" failed
- Temp subtitle path contained Cyrillic:
`temp_subtitles_zivert___я_куплю_тебе_дом_1770150285852.ass`
- FFmpeg cannot handle Unicode in filter argument paths → exit code 183

## Changes
- **karaoke_gen/lyrics_transcriber/output/video.py** (lines 222, 282):
- Changed `c.isalnum()` to `c.isascii() and c.isalnum()` for temp
subtitle paths
  - Affects both `generate_video()` and `generate_preview_video()`

- **karaoke_gen/utils/remote_cli.py** (line 1970):
- Changed `c.isalnum()` to `c.isascii() and c.isalnum()` for output
folder names
- Added fallback to `job_{job_id}` if folder name is empty or contains
no alphanumeric chars

- **Tests added** (21 comprehensive tests):
- `test_video_generator_unicode.py` (10 tests): Cyrillic, Chinese,
Arabic, emoji, accented chars
- `test_remote_cli_unicode_folders.py` (11 tests): Folder sanitization,
edge cases, real failure case

- **Documentation**:
- Updated `docs/LESSONS-LEARNED.md` with "Unicode in File Paths (Feb
2026)" section
  - Updated `docs/README.md` with recent changes
  - Bumped version 0.115.1 → 0.115.2

## Testing
- [x] All 1088 unit tests pass (68.91% coverage, target 69%)
- [x] 21 new comprehensive Unicode tests covering:
  - Cyrillic, Chinese, Arabic, emoji, accented characters
  - Empty folder name edge case
  - Separator-only folder name edge case
  - Exact real-world failure case from job 1eab1172
- [x] Verified temp file paths are ASCII-only
- [x] Verified folder names safe for all filesystems

## Review
- [x] CodeRabbit CLI review completed locally (2 cycles)
- [x] All identified issues addressed:
  - Empty folder name fallback added
  - Separator-only folder name guard enhanced
- [x] Test coverage maintained at target

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-05 09:13:33 -08:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 2b1e4ead00 fix(admin): perform audio search in override-audio-source endpoint (#374)
## Summary

Fixes the override-audio-source endpoint which was transitioning jobs to
`awaiting_audio_selection` without actually performing the search,
leaving jobs stuck with no results to select from.

## Changes

- Now performs actual audio search after clearing state
- Stores search results in state_data before transitioning
- Updated response model to include `search_results_count` and `error`
fields
- Updated API docs to reflect new behavior

## Test plan

- [x] Ran test_admin_job_reset.py - all 39 tests pass
- [x] Manually tested on job 284ed6a6 (populated search results)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-04 22:22:41 -08:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 b7b5d5c814 feat(admin): add MFY restart and override admin tools (#373)
## Summary

Add three new admin endpoints for better Made-For-You order management,
plus clean up the admin toolbar to remove confusing low-level buttons.

### New Endpoints

1. **`POST /api/admin/jobs/{job_id}/regenerate-screens`**
   - Regenerates title/end screens with current artist/title metadata
- Use after editing artist/title to update screens without full
reprocessing

2. **`POST /api/admin/jobs/{job_id}/restart`**
- Full restart that actually triggers workers (fixes the "stuck in
pending" issue)
   - Option to preserve audio stems for faster restart
   - Handles YouTube, audio search, and file upload jobs appropriately

3. **`POST /api/admin/jobs/{job_id}/override-audio-source`**
   - Switch from YouTube URL to audio search mode
   - Allows finding higher quality audio for MFY orders

### Frontend Changes

- Add new toolbar buttons: **Regen Screens**, **Full Restart**, **Audio
Search**
- Remove confusing low-level buttons that could leave jobs in broken
states:
  - "Reset to Start" (replaced by "Full Restart")
  - "Clear Workers" (low-level escape hatch)
  - "Trigger" (low-level escape hatch)
- Add confirmation dialogs with clear explanations

### Use Case

When a customer submits a Made-For-You order with:
- Wrong artist/title → Edit fields, click "Regen Screens"
- Low-quality YouTube URL → Click "Audio Search" to switch to audio
search mode
- Job stuck after reset → Click "Full Restart" to actually trigger
workers

## Test plan

- [ ] Verify new endpoints return proper responses
- [ ] Verify "Regen Screens" regenerates screens with updated metadata
- [ ] Verify "Full Restart" with preserve_audio_stems=true triggers
screens worker
- [ ] Verify "Full Restart" with preserve_audio_stems=false re-downloads
YouTube audio
- [ ] Verify "Audio Search" transitions job to awaiting_audio_selection
- [ ] Verify old confusing buttons are removed from toolbar

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-04 18:57:17 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 747f7cf079 test: fix mock return values to match real API contract (#372)
## Summary
- Fixed 22 test mocks that incorrectly returned `True` instead of `None`
for `update_job()`
- Added emulator integration tests proving the real API contract
- Added test using real production job 984da08b as fixture to validate
the fix
- Documented lesson learned and added mock guidance to TESTING.md

## Context
PR #371 fixed an admin reset endpoint that always returned 500. The root
cause was that tests mocked `update_job()` to return `True`, but the
real method returns `None` on success. Code checking `if not success:`
was always `True` with `None`.

This PR ensures our tests would have caught the bug before it shipped
by:
1. Fixing all mock return values to match the real API contract
2. Adding integration tests that prove `update_job()` returns `None`
3. Documenting this pattern to prevent future occurrences

## Test plan
- [x] All 38 admin reset unit tests pass
- [x] New emulator integration tests verify API contract
- [x] Real production job fixture test validates the fix

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 16:37:10 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 2f46b3754d fix: admin reset endpoint always returned 500 (#371)
## Summary

The admin job reset endpoint (`POST /api/admin/jobs/{job_id}/reset`)
always returned a 500 error due to a bug introduced in PR #370.

**Root cause:** The code checked `if not success:` but `update_job()`
returns `None` (not a boolean). Since `not None` is `True`, this always
raised the HTTPException.

**Fix:** Removed the erroneous check. `update_job()` raises exceptions
on failure, so checking the return value is unnecessary.

## Test plan

- [x] All 38 admin reset tests pass locally
- [x] All 100 emulator integration tests pass locally
- [ ] Verify reset button works in production admin dashboard

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- @coderabbitai ignore -->

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 15:42:26 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 5901137724 feat: make state machine transitions robust with explicit failure handling (#370)
## Summary

Jobs 06cfea29 and 984da08b got stuck at `pending` status because state
transitions failed silently. This comprehensive fix makes the state
machine more robust through several improvements:

- **Explicit failure handling**: `transition_to_state()` now raises
`InvalidStateTransitionError` by default instead of returning False
silently
- **Centralized processing pattern**: New `start_job_processing()`
helper ensures atomic transition + worker trigger across all handlers
- **Worker validation**: All workers now validate job status before
processing via `validate_worker_can_run()`
- **Runtime consistency checks**: New `job_health_service.py` detects
stuck jobs and state inconsistencies
- **Health endpoint**: `/health/job-consistency` endpoint for admin
dashboard integration
- **Admin button fixes**: Reset to pending now clears parallel
processing flags; removed deprecated states

## Changes

### Core Services
- `services/job_manager.py`: Added `InvalidStateTransitionError`,
`start_job_processing()`, and updated `transition_to_state()` to raise
by default
- `services/job_health_service.py`: New service for consistency checks
and worker validation
- `exceptions.py`: Added `InvalidStateTransitionError` exception class

### API Routes  
- `routes/health.py`: Added `/health/job-consistency` endpoint
- `routes/admin.py`: Updated STATE_DATA_CLEAR_KEYS and
ALLOWED_RESET_STATES
- `routes/audio_search.py`: Fixed to properly transition from
DOWNLOADING_AUDIO to DOWNLOADING
- `routes/users.py`: Added error handling for state transition failures
- `routes/file_upload.py`: Uses centralized `start_job_processing()`

### Workers
- All workers (audio, lyrics, screens, video, render_video): Added
status validation via `validate_worker_can_run()`

### Frontend
- `admin/jobs/page.tsx`: Replaced "Lyrics" + "Inst." buttons with
combined "Review" button

### Tests
- 26 tests for job_health_service
- 11 integration tests for state machine flow
- 4 tests for job-consistency endpoint
- Updated existing tests for new exception behavior

### Documentation
- `docs/LESSONS-LEARNED.md`: Added "Make State Transitions Loud, Not
Silent" entry
- `docs/ARCHITECTURE.md`: Added state machine robustness note
- `docs/archive/2026-02-02-state-machine-robustness-plan.md`: Complete
implementation plan

## Test plan

- [x] All 1850+ unit tests pass
- [x] CodeRabbit review issues addressed
- [ ] Manual verification of admin reset buttons in staging
- [ ] Verify stuck jobs are properly detected by
`/health/job-consistency`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- @coderabbitai ignore -->

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-02 22:48:57 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 b0cbd28c8d fix(infrastructure): Import existing Firestore index into Pulumi state (#369)
## Summary
- Adds `import_` directive to the `firestore-index-jobs-customer-mfy`
resource to import the existing index in GCP into Pulumi state
- The index was created before Pulumi managed it, causing a 409 conflict
error on deploy

## Context
PR #367 added this index definition, but the index already existed in
GCP (created manually). This caused the infrastructure deployment to
fail with:
```
Error creating Index: googleapi: Error 409: index already exists
```

The `import_` directive tells Pulumi to adopt the existing resource
rather than trying to create a new one.

## Test plan
- [ ] CI passes
- [ ] Pulumi successfully imports the existing index on deploy

> [!NOTE]
> After this PR deploys successfully, the `import_` line can be removed
in a follow-up PR since the resource will already be in Pulumi state.

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 22:33:26 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 c8d3fa46c4 fix: update E2E tests for combined lyrics/instrumental review UI (#368)
## Summary
- Updates production E2E tests to work with the combined
lyrics/instrumental review UI
- The "Complete Review" button was changed to "Proceed to Instrumental
Review" in a previous update
- Instrumental selection now happens on the same page via hash
navigation instead of a separate page

## Changes
- **happy-path-real-user.spec.ts**: 
- Step 7 now clicks "Proceed to Instrumental Review" instead of
"Complete Review"
- Step 8 handles instrumental selection on the same page (via hash
navigation)
- Removed the old flow that waited for job status change and opened a
new page
- **lyrics-review-only.spec.ts**: Updated to use new button name
- **test-existing-job.spec.ts**: Updated to use new button name

## Test plan
- [x] Verified test passes locally (6.2 minutes)
- [x] Full flow tested: signup → job creation → audio selection → lyrics
review → instrumental selection → completion → cleanup

## Notes
This fixes the daily scheduled E2E test that has been failing since the
UI was combined.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 22:21:06 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 6ebf076576 fix: E2E test issues - add Firestore index and fix job endpoint URL (#367)
## Summary
- Add missing Firestore composite index for `customer_email +
made_for_you + created_at` queries (required for test-webhook
made-for-you order processing)
- Fix `getJobDetails` helper to use `/api/jobs/{job_id}` instead of
`/api/admin/jobs/{job_id}` (the admin route doesn't have a GET method)

## Test plan
- [x] All production E2E tests pass:
  - `stripe-webhook-security.spec.ts` (9 passed)
  - `stripe-credit-purchase.spec.ts` (3 passed)
  - `stripe-made-for-you-search.spec.ts` (2 passed, 1 skipped)
  - `stripe-made-for-you-youtube.spec.ts` (3 passed)

Note: The Firestore index was already created manually via gcloud. This
PR adds it to Pulumi IaC so it's tracked in infrastructure as code.

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 21:54:35 -08:00
574 changed files with 17341 additions and 1917 deletions

No files matched your search

+407
View File
@@ -0,0 +1,407 @@
# Plan: Add Retry Logic for Flacfetch Download Failures
**Created:** 2026-02-10
**Branch:** feat/sess-20260210-1158-investigate-job-749141f8
**Status:** Draft
## Overview
Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"
Investigation revealed **multiple related issues** when flacfetch network calls fail:
1. **No retry logic** - A single network hiccup causes immediate job failure
2. **Missing source info** - When download fails, source_id/source_name aren't saved to job
3. **Broken retry button** - Can't retry because no source info was persisted
4. **Silent UI failures** - Frontend doesn't show retry error messages to user
### Root Cause
Audio search flow saves selection info to `state_data`, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.
### Solution
This plan fixes all four issues:
- Add retry logic to flacfetch client (network resilience)
- Persist download info before attempting download (enables retry)
- Update retry endpoint to handle audio search jobs
- Show error messages in frontend when retry fails
## Requirements
### Functional Requirements - Flacfetch Retry Logic
- [x] Retry `download_by_id()` on `httpx.RequestError` (network failures)
- [x] Use exponential backoff spread over ~5 minutes
- [x] Configurable max retries via environment variable
- [x] Log each retry attempt with context
- [x] Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
- [x] Do retry on HTTP 5xx errors (server errors - transient)
### Functional Requirements - Persist Download Info
- [x] Save source_id, source_name, target_file, download_url to job document
- [x] Save BEFORE attempting download (not after)
- [x] Store in top-level fields for easy retry access
- [x] Preserve existing state_data fields (selected_audio_index, etc.)
### Functional Requirements - Retry Endpoint
- [x] Add case for audio search jobs in `/jobs/{id}/retry` endpoint
- [x] Check for saved download params (source_id, source_name)
- [x] Re-download using saved params
- [x] Clear error state before retrying
- [x] Trigger appropriate workers after successful download
### Functional Requirements - Frontend Error Handling
- [x] Show toast notification when retry fails
- [x] Display actual error message from API
- [x] Don't just console.error - make it visible to user
### Non-Functional Requirements
- [x] Minimal code changes (use existing patterns)
- [x] Clear error messages that include retry context
- [x] Backwards compatible (no breaking changes)
- [x] Should not significantly increase total timeout duration
## Technical Approach
### Architecture Decisions
**Option 1: Tenacity library (Recommended)**
- ✅ Industry standard, well-tested
- ✅ Declarative retry configuration
- ✅ Built-in exponential backoff
- ✅ Already used in project (check `poetry.lock`)
- ❌ New dependency if not already present
**Option 2: Manual retry loop**
- ✅ No new dependencies
- ✅ Full control over logic
- ❌ More code to maintain
- ❌ Easy to get wrong (edge cases)
**Decision:** Use **tenacity** if already in dependencies, otherwise implement manual retry loop.
### Retry Strategy
```python
# Retry on network errors only (RequestError)
# Do NOT retry on:
# - HTTP 4xx (client errors like 400, 401, 404)
# - Already completed downloads
# Do retry on:
# - httpx.RequestError (connection timeout, refused, DNS failure)
# - HTTP 5xx (server errors - service temporarily down)
Max retries: 8 (total 9 attempts)
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
- Attempt 1: immediate
- Attempt 2: wait 10s
- Attempt 3: wait 20s
- Attempt 4: wait 40s
- Attempt 5: wait 60s (capped)
- Attempt 6: wait 60s (capped)
- Attempt 7: wait 60s (capped)
- Attempt 8: wait 60s (capped)
- Attempt 9: wait 60s (capped)
Total max time: ~370s (6+ minutes) of backoff + request timeouts
This gives the flacfetch service time to recover from restarts,
network issues, or temporary VM problems.
```
### Configuration
Add to `backend/config.py`:
```python
# Flacfetch client retry settings
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
```
### Error Message Improvements
Change from:
```
Download by ID request failed:
```
To:
```
Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out
```
## Implementation Steps
### Phase 1: Check Dependencies
1. [ ] Check if `tenacity` is in `pyproject.toml` dependencies
2. [ ] If not present, add `tenacity = "^8.2.0"` to dependencies
3. [ ] Run `poetry lock` if dependencies changed
### Phase 2: Add Configuration
4. [ ] Add retry configuration to `backend/config.py`
- `flacfetch_retry_max_attempts = 9`
- `flacfetch_retry_min_wait = 10.0`
- `flacfetch_retry_max_wait = 60.0`
### Phase 3: Implement Flacfetch Retry Logic
5. [ ] Import tenacity in `backend/services/flacfetch_client.py`
6. [ ] Create retry decorator with:
- Retry on `httpx.RequestError`
- Retry on `httpx.HTTPStatusError` with 5xx status
- Stop on `httpx.HTTPStatusError` with 4xx status
- Exponential backoff with cap at 60s
- Before-sleep logging callback
7. [ ] Apply decorator to `download_by_id()` method
8. [ ] Improve error messages to include:
- Number of attempts made
- Original exception type and message
### Phase 4: Persist Download Info Before Download
9. [ ] Update `audio_search.py:_download_and_start_processing()`
10. [ ] After getting source info from selected result (line 307-311)
11. [ ] **Save to job document BEFORE download attempt:**
```python
job_manager.update_job(job_id, {
'audio_source_type': 'audio_search',
'source_name': source_name,
'source_id': source_id,
'target_file': target_file,
'download_url': download_url,
})
```
12. [ ] Then proceed with download (existing code)
13. [ ] Add to Job model if needed (check if fields exist)
### Phase 5: Update Retry Endpoint for Audio Search
14. [ ] Update `backend/api/routes/jobs.py:retry_job()`
15. [ ] Add new case before "no input audio" error (line ~1088)
16. [ ] Check if job has saved download params:
```python
elif job.audio_source_type == 'audio_search' and job.source_id:
# Has audio search download info - retry download
```
17. [ ] Re-call audio search download logic with saved params
18. [ ] Or refactor download into reusable function
### Phase 6: Frontend Error Handling
19. [ ] Update `frontend/components/job/JobActions.tsx`
20. [ ] Add toast notification library (check if exists)
21. [ ] In `handleRetry()` catch block, show toast:
```typescript
catch (error) {
const message = error.response?.data?.detail || "Failed to retry job"
toast.error(message)
console.error("Failed to retry job:", error)
}
```
### Phase 7: Testing - Flacfetch Retry
22. [ ] Add unit tests to `backend/tests/test_flacfetch_client.py`
23. [ ] Test retry on RequestError (mock fails N times then succeeds)
24. [ ] Test no retry on 4xx errors
25. [ ] Test retry on 5xx errors
26. [ ] Test max retries exhausted
27. [ ] Verify logging output includes attempt numbers
### Phase 8: Testing - End-to-End
28. [ ] Test audio search job with flacfetch down
29. [ ] Verify source info is saved even when download fails
30. [ ] Verify retry button works after source info saved
31. [ ] Test retry succeeds when flacfetch comes back up
32. [ ] Verify error toast appears when retry fails
### Phase 9: Documentation
33. [ ] Update `backend/services/flacfetch_client.py` docstring
34. [ ] Add to `docs/LESSONS-LEARNED.md`:
- "Always add retries for external service calls"
- "Save operation parameters before attempting operations"
- "Persist enough state to enable retry from any failure point"
35. [ ] Update `docs/ARCHITECTURE.md` if needed
36. [ ] Document the new job fields in API.md
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/config.py` | Modify | Add retry configuration variables |
| `backend/services/flacfetch_client.py` | Modify | Add retry logic to `download_by_id()` |
| `backend/api/routes/audio_search.py` | Modify | Save download info before attempting download |
| `backend/api/routes/jobs.py` | Modify | Add audio search case to retry endpoint |
| `backend/models/job.py` | Modify (maybe) | Add fields: source_name, source_id, target_file, download_url |
| `frontend/components/job/JobActions.tsx` | Modify | Show error toast when retry fails |
| `backend/tests/test_flacfetch_client.py` | Modify | Add retry tests |
| `backend/tests/test_audio_search.py` | Modify | Test download info persistence |
| `backend/tests/test_jobs.py` | Modify | Test retry with audio search jobs |
| `pyproject.toml` | Modify (maybe) | Add tenacity if not present |
| `docs/LESSONS-LEARNED.md` | Modify | Document patterns |
| `docs/API.md` | Modify | Document new job fields |
## Testing Strategy
### Unit Tests (`backend/tests/test_flacfetch_client.py`)
1. **Test retry on RequestError**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_network_error():
"""Should retry 3 times on connection errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
# Fail 2 times, then succeed
mock_post.side_effect = [
httpx.RequestError("Connection timeout"),
httpx.RequestError("Connection refused"),
Mock(json=lambda: {"download_id": "test123"}, status_code=200),
]
result = await client.download_by_id("YouTube", "abc123")
assert result == "test123"
assert mock_post.call_count == 3
```
2. **Test no retry on 4xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_no_retry_on_client_error():
"""Should NOT retry on 400/401/404 errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
response = Mock(status_code=404, text="Not found")
response.raise_for_status.side_effect = httpx.HTTPStatusError(
"Not found", request=Mock(), response=response
)
mock_post.return_value = response
with pytest.raises(FlacfetchServiceError, match="404"):
await client.download_by_id("YouTube", "abc123")
assert mock_post.call_count == 1 # No retries
```
3. **Test retry on 5xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_server_error():
"""Should retry on 500/503 errors."""
# Similar to test 1 but with HTTP 503 errors
```
4. **Test max retries exhausted**
```python
@pytest.mark.asyncio
async def test_download_by_id_fails_after_max_retries():
"""Should fail with clear message after max retries."""
# All attempts fail, verify error message includes attempt count
```
### Integration Tests
5. **Test with actual flacfetch service** (if available in test environment)
- Verify retry doesn't break normal operation
- Test with deliberate network issues
### Manual Testing
6. **Simulate network failure**
```bash
# Temporarily stop flacfetch VM
gcloud compute instances stop flacfetch-service --zone=us-central1-a
# Submit a test job
# Observe logs showing retry attempts
# Restart VM during retries
gcloud compute instances start flacfetch-service --zone=us-central1-a
# Job should succeed
```
7. **Test with intentional delays**
- Add sleep in flacfetch API handler
- Verify retries don't cause issues
## Open Questions
- [x] ~~Is tenacity already in dependencies?~~ (Will check during implementation)
- [x] ~~What's the appropriate total timeout budget?~~ - 9 attempts over ~6 minutes
- [x] ~~Should we fix the Retry button UX in this PR?~~ - YES, it's the same root cause
- [x] ~~Why did job 749141f8 fail before saving audio source info?~~ - Download info saved AFTER download, not before
- [ ] Should we also add retries to `wait_for_download()`?
- Probably NO - that already has its own polling/timeout logic
- [ ] Should we add circuit breaker pattern?
- Probably NO for now - keep it simple
- Could add in future if we see cascading failures
- [ ] Should retry logic be applied to other flacfetch methods?
- `start_download()` - probably YES (same pattern)
- `search()` - maybe YES (search can also have network issues)
- `get_download_status()` - probably NO (polling already handles this)
- **Decision**: Apply to `start_download()` too, skip `search()` for now
- [ ] Which toast library should frontend use?
- Check if already exists (react-hot-toast, sonner, etc.)
- If not, add lightweight library
- [ ] Should we save download params to state_data or top-level fields?
- Top-level fields more visible and easier to query
- **Decision**: Top-level fields (source_id, source_name, etc.)
## Rollback Plan
If this causes issues:
1. **Quick rollback**: Set env var `FLACFETCH_RETRY_MAX_ATTEMPTS=1` in Cloud Run
- Disables retries without code change
- Can deploy in ~2 minutes
2. **Full rollback**: Revert the PR
- Git revert the merge commit
- Deploy previous version
3. **Monitoring**: Watch for:
- Increased request duration to flacfetch
- Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
- Duplicate download attempts causing issues
## Success Criteria
- [ ] Jobs with transient flacfetch failures succeed instead of failing
- [ ] Error messages are more informative (show attempts and root cause)
- [ ] No increase in false positives (jobs that should fail, now hang)
- [ ] All tests pass
- [ ] No impact on successful jobs (retry overhead is minimal)
## Related Issues
### Job 749141f8 - The Incident
**What happened:**
- User searched for audio via audio search
- Selected "Elizabeth Moen - Songbird"
- Backend tried to download via flacfetch
- Flacfetch call failed with network error (connection timeout/refused)
- Job immediately failed
- No source info saved → Retry button broken
- User sees "Retry" button but clicking does nothing (silent 400 error)
**How this fix helps:**
1. ✅ Flacfetch retry logic → Download would have succeeded after retry
2. ✅ Persist download info → Even if all retries fail, user can retry later
3. ✅ Retry endpoint fix → Retry button will actually work
4. ✅ Frontend toast → User sees helpful error if retry still fails
**Note:** Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.
### Future Work (Not in This PR)
Similar pattern should be applied to other external service calls:
- AudioShake API (lyrics transcription)
- Modal API (audio separation)
- YouTube direct API calls (not via flacfetch)
- Email sending (SendGrid)
## References
- [Tenacity documentation](https://tenacity.readthedocs.io/)
- [LESSONS-LEARNED.md](docs/LESSONS-LEARNED.md) - External service response format mismatches
- [flacfetch README](../../flacfetch/README.md) - Understanding the service
+279
View File
@@ -0,0 +1,279 @@
# Plan: Private (Non-Published) Tracks
**Created:** 2026-02-24
**Branch:** feat/sess-20260224-1149-private-tracks
**Status:** Implemented
## Overview
Add a `is_private` flag to jobs that changes distribution behavior:
- **Dropbox**: Uses `Tracks-NonPublished` folder (not `Tracks-Organized`) with brand code `NOMADNP` (not `NOMAD`)
- **YouTube**: Skipped entirely
- **Google Drive**: Skipped entirely
- **Email**: Completion email includes Dropbox link but no YouTube section
- **Discord**: Still notified (for internal tracking)
Available to all users at job creation and toggleable by admin on existing jobs.
## Requirements
- [ ] All users can mark a job as "Private (Non-Published)" when creating it
- [ ] Admin can toggle `is_private` on any existing job via admin dashboard
- [ ] Private jobs upload to Dropbox at `Tracks-NonPublished` with `NOMADNP-XXXX` brand codes
- [ ] Private jobs skip YouTube and Google Drive uploads entirely
- [ ] Completion emails omit YouTube section when no YouTube URL (private jobs + any other edge case)
- [ ] When admin toggles a completed job to private, existing outputs are auto-deleted
- [ ] Discord notifications still fire for private jobs
- [ ] Existing (non-private) jobs are completely unaffected (backward compatible)
## Technical Approach
### Core Design: Single Boolean Flag with Runtime Overrides
Store `is_private: bool = False` on the job model. At distribution time, a helper function returns effective distribution parameters based on this flag. This means:
- **Toggling is_private automatically changes all distribution behavior** - no need to manually update dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id
- **Job fields still store the "normal" defaults** - the override happens at the config-to-orchestrator boundary
- **Admin can flip one field** and everything adapts
### Key Helper Function
```python
# In job_defaults_service.py
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
"""Return distribution settings, applying private overrides if is_private=True."""
if getattr(job, 'is_private', False):
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=settings.default_private_dropbox_path,
brand_prefix=settings.default_private_brand_prefix,
enable_youtube_upload=False,
gdrive_folder_id=None,
# Keep other settings from job (discord, youtube_description, etc.)
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
# Non-private: return job's own settings
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=getattr(job, 'youtube_description_template', None),
)
```
This helper is used in both distribution paths (orchestrator and native fallback).
### Distribution Paths That Need Changes
Two code paths handle distribution (see LESSONS-LEARNED: "Fix Both Sides of Dual Code Paths"):
1. **Orchestrator path** (`video_worker_orchestrator.py`): Primary path. `create_orchestrator_config_from_job()` uses the helper to build config.
2. **Native distribution path** (`video_worker.py` → `_handle_native_distribution()`): Fallback path. Also uses the helper instead of reading job fields directly.
### Email Template Change
Add conditional YouTube section removal in `render_template()`, matching the existing pattern for `feedback_url`:
```python
# Remove YouTube section if no URL provided
if not variables.get("youtube_url"):
result = re.sub(
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
'\n',
result,
flags=re.DOTALL
)
variables["youtube_url"] = ""
```
And change `render_job_completion()` to pass `None` instead of `"[YouTube URL not available]"` when youtube_url is empty.
### Admin Toggle Auto-Delete
When admin PATCH updates `is_private` from False to True on a job that has existing outputs, reuse the existing delete-outputs logic to remove YouTube + GDrive + Dropbox outputs. Admin can then "Reprocess" to re-distribute to the private Dropbox path.
## Implementation Steps
### Phase 1: Backend Model & Config (foundation)
1. [ ] **Add `is_private` to Job model** (`backend/models/job.py`)
- Add `is_private: bool = False` field after `made_for_you` field
- Add to `JobCreate` model as well
2. [ ] **Add private distribution config** (`backend/config.py`)
- Add `default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")`
- Add `default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")`
3. [ ] **Add helper function** (`backend/services/job_defaults_service.py`)
- Add `get_effective_distribution_for_job(job)` function that returns distribution settings with private overrides applied
### Phase 2: Backend API (job creation + admin)
4. [ ] **Add `is_private` to request models** (`backend/models/requests.py`)
- Add `is_private: Optional[bool] = None` to `URLSubmissionRequest`
- Add `is_private: Optional[bool] = None` to `AudioSearchRequest`
5. [ ] **Update URL creation endpoint** (`backend/api/routes/jobs.py`)
- Accept and pass through `is_private` field to `JobCreate`
6. [ ] **Update file upload endpoint** (`backend/api/routes/file_upload.py`)
- Accept `is_private` Form parameter
- Pass through to `JobCreate`
7. [ ] **Update audio search endpoint** (`backend/api/routes/audio_search.py`)
- Accept `is_private` in `AudioSearchRequest`
- Pass through to `JobCreate`
8. [ ] **Update admin endpoints** (`backend/api/routes/admin.py`)
- Add `is_private: Optional[bool] = None` to `JobUpdateRequest`
- Add `"is_private"` to `EDITABLE_JOB_FIELDS`
- In the PATCH handler: detect when `is_private` changes from False→True on a completed job, auto-delete existing outputs (call existing delete-outputs service logic)
### Phase 3: Distribution Pipeline
9. [ ] **Update orchestrator config creation** (`backend/workers/video_worker_orchestrator.py`)
- In `create_orchestrator_config_from_job()`: Use `get_effective_distribution_for_job()` helper to determine dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id instead of reading directly from job
10. [ ] **Update native distribution fallback** (`backend/workers/video_worker.py`)
- In `_handle_native_distribution()`: Use `get_effective_distribution_for_job()` helper instead of reading directly from job fields
### Phase 4: Email Template
11. [ ] **Add YouTube section conditional removal** (`backend/services/template_service.py`)
- In `render_template()`: Add regex to remove YouTube paragraph when `youtube_url` is empty (matching existing `feedback_url` pattern)
- In `render_job_completion()`: Pass `None` for youtube_url when empty (instead of `"[YouTube URL not available]"`)
### Phase 5: Frontend
12. [ ] **Add private checkbox to job creation form** (`frontend/components/job/JobSubmission.tsx`)
- Add "Private (Non-Published)" checkbox below the existing "Skip lyrics review" checkbox
- Show for all users (not just admin)
- Include brief description text: "Private tracks go to Dropbox only, not YouTube or Google Drive"
13. [ ] **Update API interfaces** (`frontend/lib/api.ts`)
- Add `is_private?: boolean` to all job creation option interfaces
- Add `is_private` to `JobUpdateRequest` interface
- Pass through in `uploadJob()`, `createJobFromUrl()`, `searchAudio()` methods
14. [ ] **Add private toggle to admin job detail** (`frontend/app/admin/jobs/page.tsx`)
- Add `is_private` as an inline-editable field in the job overview section (use Switch component)
- Show current state with visual indicator (e.g., lock icon for private jobs)
- When toggling to private on a completed job, show confirmation dialog about auto-deleting outputs
### Phase 6: Environment Configuration
15. [ ] **Update CI/deployment config** (`.github/workflows/ci.yml` or Cloud Run env)
- Add `DEFAULT_PRIVATE_DROPBOX_PATH` environment variable (e.g., `/Karaoke/Tracks-NonPublished`)
- Add `DEFAULT_PRIVATE_BRAND_PREFIX` environment variable (e.g., `NOMADNP`)
- These should be configured in Cloud Run service and CI test environment
### Phase 7: Testing
16. [ ] **Unit tests for private distribution logic** (`backend/tests/test_private_tracks.py`)
- Test `get_effective_distribution_for_job()` with private and non-private jobs
- Test `create_orchestrator_config_from_job()` applies private overrides correctly
- Test `_run_distribution()` skips YouTube and GDrive when private config applied
- Test `_upload_to_dropbox()` uses private path and brand prefix
- Test brand code generation with NOMADNP prefix in Tracks-NonPublished
- Test `_handle_native_distribution()` respects private overrides
- Test non-private jobs are completely unaffected (regression tests)
17. [ ] **Unit tests for email changes** (add to `backend/tests/test_email_service.py` or `test_template_service.py`)
- Test YouTube section removed from email when youtube_url is None
- Test YouTube section present when youtube_url is provided
- Test Dropbox section still present for private jobs
- Test full email rendering for private job scenario
18. [ ] **Unit tests for admin toggle** (add to `backend/tests/test_admin.py` or new file)
- Test admin can update is_private field
- Test toggling is_private to True on completed job triggers output deletion
- Test toggling is_private to True on non-completed job does NOT delete outputs
- Test toggling is_private to False does nothing special
19. [ ] **Integration test** (`backend/tests/emulator/` or `test_distribution_services.py`)
- Test full distribution pipeline with is_private=True: verify Dropbox path, brand prefix, YouTube skip, GDrive skip
20. [ ] **Frontend E2E regression test** (`frontend/e2e/regression/`)
- Test private checkbox appears in job creation form
- Test private toggle in admin dashboard
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `is_private: bool = False` to Job and JobCreate |
| `backend/config.py` | Modify | Add `default_private_dropbox_path` and `default_private_brand_prefix` |
| `backend/services/job_defaults_service.py` | Modify | Add `get_effective_distribution_for_job()` helper |
| `backend/models/requests.py` | Modify | Add `is_private` to URL and AudioSearch request models |
| `backend/api/routes/jobs.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/file_upload.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/audio_search.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/admin.py` | Modify | Add `is_private` to editable fields + auto-delete logic |
| `backend/workers/video_worker_orchestrator.py` | Modify | Use helper in `create_orchestrator_config_from_job()` |
| `backend/workers/video_worker.py` | Modify | Use helper in `_handle_native_distribution()` |
| `backend/services/template_service.py` | Modify | Conditional YouTube section removal |
| `frontend/components/job/JobSubmission.tsx` | Modify | Add private checkbox |
| `frontend/lib/api.ts` | Modify | Add `is_private` to interfaces and API calls |
| `frontend/app/admin/jobs/page.tsx` | Modify | Add private toggle to admin detail |
| `backend/tests/test_private_tracks.py` | Create | Comprehensive unit tests for private track logic |
## Testing Strategy
### Unit Tests (~25-30 tests in `test_private_tracks.py`)
- `get_effective_distribution_for_job()` with private=True and False
- `create_orchestrator_config_from_job()` private overrides (YouTube=False, GDrive=None, private Dropbox path, NOMADNP brand)
- Distribution stage skips: YouTube not called, GDrive not called, Dropbox IS called with correct path
- Brand code generation uses NOMADNP prefix and Tracks-NonPublished path
- Native distribution fallback applies same overrides
- **Regression tests**: Non-private jobs have unchanged behavior (same paths, same brand code, YouTube+GDrive+Dropbox all fire)
### Email Template Tests (~5 tests)
- YouTube section stripped when youtube_url is None
- YouTube section preserved when youtube_url has value
- Dropbox section always present
- Full private job email rendering (Dropbox + no YouTube)
- Full non-private job email rendering (both present) - regression
### Admin Toggle Tests (~5 tests)
- is_private editable via PATCH
- Toggle to private on completed job deletes outputs
- Toggle to private on pending/in-progress job does NOT delete
- Toggle to non-private does nothing special
### Frontend Tests
- E2E: Private checkbox visible and functional in job creation
- E2E: Admin toggle visible and includes confirmation dialog
### Manual Verification
- Create a private job end-to-end, verify Dropbox folder in Tracks-NonPublished with NOMADNP code
- Verify completion email has Dropbox link but no YouTube section
- Toggle existing completed job to private, verify outputs deleted
## Regression Risk Analysis
| Risk | Mitigation |
|------|-----------|
| Non-private jobs affected | `is_private` defaults to `False`; helper returns unchanged values for non-private. Explicit regression tests. |
| Dual code paths (orchestrator vs native) | Both paths use same helper function. Tests cover both paths. (Lesson: "Fix Both Sides of Dual Code Paths") |
| Email template regex breaks other content | Regex targets specific YouTube paragraph text. Tests verify non-YouTube content untouched. |
| Brand code namespace collision | NOMADNP codes are in a completely separate Dropbox folder (Tracks-NonPublished), no overlap with NOMAD codes in Tracks-Organized |
| Admin auto-delete on wrong job | Only triggers when `is_private` changes from False→True AND job has terminal status (complete). Confirmation dialog in frontend. |
| Made-for-you orders | MFY webhook doesn't set is_private (defaults False). Admin can toggle after. No change to MFY flow. |
| Existing Firestore documents | Missing `is_private` field defaults to `False` via model default. No migration needed. |
## Open Questions
- [x] Who sees private toggle? → **Everyone** (all users at creation, admin on existing)
- [x] Auto-delete on toggle? → **Yes**, auto-delete all outputs when toggling to private on completed job
- [x] Email format? → **Remove YouTube section** when no YouTube URL (improves all edge cases)
- [x] Discord? → **Still notify** (useful for internal tracking)
## Rollback Plan
1. **Feature flag**: The `is_private` field defaults to `False`, so setting `DEFAULT_PRIVATE_DROPBOX_PATH` to empty/None effectively disables private distribution
2. **No schema migration needed**: Missing `is_private` field on existing Firestore documents defaults to `False`
3. **Email change is backward-compatible**: The YouTube section removal only happens when youtube_url is None, which is already an edge case showing "[YouTube URL not available]"
4. **Frontend**: Private checkbox is additive UI, removing it just means users can't create new private jobs
+1 -1
View File
@@ -1242,7 +1242,7 @@ jobs:
--min-instances 4 \
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
--vpc-egress private-ranges-only \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true" \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest"
- name: Verify deployment and version
+25 -8
View File
@@ -1,7 +1,12 @@
# Makefile for karaoke-gen project
# Run `make help` to see available commands
.PHONY: help install install-backend install-frontend build-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
.PHONY: help install install-backend install-frontend build-frontend dev-install test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
# Per-stage timeout in seconds (default: 10 minutes)
# Prevents hangs from stale dev servers, broken emulators, etc.
# Override with: make test TEST_TIMEOUT=300
TEST_TIMEOUT ?= 600
# Default target
help:
@@ -12,7 +17,8 @@ help:
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
@echo " make test-e2e - Run emulator tests with auto-start/stop"
@echo " make install - Install all dependencies (backend + frontend)"
@echo " make build-frontend - Build frontend and copy to Python package (for local testing)"
@echo " make build-frontend - Build frontend and copy to Python package"
@echo " make dev-install - Build frontend + pip install -e . (for local CLI testing)"
@echo " make lint - Run linter checks"
@echo " make emulators-start - Start GCP emulators for local development"
@echo " make emulators-stop - Stop GCP emulators"
@@ -34,23 +40,34 @@ install-frontend:
install: install-backend install-frontend
# Build frontend and copy to Python package (for local CLI testing)
# Build frontend and copy to Python package
build-frontend: install-frontend
@echo "=== Building frontend ==="
cd frontend && npm run build
@echo "=== Copying build to Python package ==="
rm -rf karaoke_gen/nextjs_frontend/out
mkdir -p karaoke_gen/nextjs_frontend/out
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
@echo "✅ Frontend built and ready for local testing"
@echo "✅ Frontend built and copied to karaoke_gen/nextjs_frontend/out/"
# Build frontend + install package in editable mode for local CLI testing
# After running this, `karaoke-gen` CLI will use the freshly built frontend.
dev-install: build-frontend
@echo "=== Installing package in editable mode ==="
pip install -e .
@echo ""
@echo "✅ Ready! You can now run karaoke-gen with the updated frontend."
@echo " Example: karaoke-gen input.flac \"Artist\" \"Title\""
# Run unit tests for karaoke_gen package
test-unit: install-backend
@echo "=== Running karaoke_gen unit tests ==="
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
timeout $(TEST_TIMEOUT) poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
# Run backend unit tests (excludes emulator tests)
test-backend-unit: install-backend
@echo "=== Running backend unit tests ==="
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
# Run E2E integration tests with emulators
test-e2e: install-backend
@@ -62,7 +79,7 @@ test-e2e: install-backend
export GCS_BUCKET_NAME=test-bucket && \
export ADMIN_TOKENS=test-admin-token && \
export ENVIRONMENT=test && \
poetry run pytest backend/tests/emulator/ -v; \
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/emulator/ -v; \
TEST_RESULT=$$?; \
./scripts/stop-emulators.sh; \
exit $$TEST_RESULT
@@ -75,7 +92,7 @@ test-backend: test-unit test-backend-unit test-e2e
# Run frontend tests (unit + E2E)
test-frontend: install-frontend
@echo "=== Running frontend tests ==="
cd frontend && npm run test:all
cd frontend && timeout $(TEST_TIMEOUT) npm run test:all
# Run ALL tests (backend + frontend) - use this before committing!
test: test-backend test-frontend
+820 -21
View File
@@ -20,6 +20,7 @@ from backend.services.user_service import get_user_service, UserService, USERS_C
from backend.services.job_manager import JobManager
from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchServiceError
from backend.services.storage_service import StorageService
from backend.services.audio_search_service import get_audio_search_service, NoResultsError, AudioSearchError
from backend.models.job import JobStatus
from backend.utils.test_data import is_test_email
from karaoke_gen.utils import sanitize_filename
@@ -95,6 +96,7 @@ class JobUpdateRequest(BaseModel):
enable_youtube_upload: Optional[bool] = None
non_interactive: Optional[bool] = None
prep_only: Optional[bool] = None
is_private: Optional[bool] = None
class JobUpdateResponse(BaseModel):
@@ -122,6 +124,7 @@ EDITABLE_JOB_FIELDS = {
"enable_youtube_upload",
"non_interactive",
"prep_only",
"is_private",
}
@@ -911,6 +914,7 @@ async def update_job(
- customer_email, customer_notes: Made-for-you order info
- brand_prefix: Brand code prefix
- non_interactive, prep_only: Workflow options
- is_private: Private (non-published) track mode
- discord_webhook_url: Notification URL
- youtube_description, youtube_description_template: YouTube settings
@@ -949,26 +953,63 @@ async def update_job(
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Perform the update
success = job_manager.update_job(job_id, updates)
# Check if we're toggling is_private from False to True on a completed job
# If so, auto-delete existing outputs (YouTube, Dropbox, GDrive)
toggling_to_private = (
"is_private" in updates
and updates["is_private"] is True
and not getattr(job, 'is_private', False)
)
if not success:
# Perform the update (update_job returns None; exceptions indicate failure)
try:
job_manager.update_job(job_id, updates)
except Exception as e:
raise HTTPException(
status_code=500,
detail="Failed to update job. Please try again."
) from e
# Auto-delete outputs when toggling to private on a job with existing outputs
auto_deleted = False
if toggling_to_private and job.status in TERMINAL_STATES and not job.outputs_deleted_at:
state_data = job.state_data or {}
has_outputs = (
state_data.get('youtube_url')
or state_data.get('dropbox_link')
or state_data.get('gdrive_files')
)
if has_outputs:
try:
logger.info(
f"Auto-deleting outputs for job {job_id} (toggled to private by {admin_email})"
)
# Call the delete-outputs endpoint directly (same module)
await delete_job_outputs(job_id, auth_data)
auto_deleted = True
except HTTPException:
# delete_job_outputs raises HTTPException on validation failures,
# but we've already validated terminal state + not-already-deleted
logger.warning(f"Auto-delete outputs skipped for job {job_id} (validation)")
except Exception as e:
logger.warning(f"Auto-delete outputs failed for job {job_id}: {e}")
# Log the admin action
logger.info(
f"Admin {admin_email} updated job {job_id}. "
f"Updated fields: {list(updates.keys())}"
f"{' (outputs auto-deleted)' if auto_deleted else ''}"
)
message = f"Successfully updated {len(updates)} field(s)"
if auto_deleted:
message += ". Existing outputs were auto-deleted (job set to private)."
return JobUpdateResponse(
status="success",
job_id=job_id,
updated_fields=list(updates.keys()),
message=f"Successfully updated {len(updates)} field(s)",
message=message,
)
@@ -1012,8 +1053,8 @@ class TriggerWorkerResponse(BaseModel):
ALLOWED_RESET_STATES = {
"pending",
"awaiting_audio_selection",
"awaiting_review",
"awaiting_instrumental_selection",
"awaiting_review", # Combined review (lyrics + instrumental selection)
# Note: awaiting_instrumental_selection removed - deprecated by combined review flow (Jan 2026)
"instrumental_selected", # Re-run video generation with same settings
}
@@ -1034,6 +1075,9 @@ STATE_DATA_CLEAR_KEYS = {
"render_progress",
"screens_progress",
"encoding_progress",
# Parallel processing flags - must clear to allow workers to run again
"audio_complete",
"lyrics_complete",
],
"awaiting_audio_selection": [
"audio_selection",
@@ -1046,6 +1090,7 @@ STATE_DATA_CLEAR_KEYS = {
"encoding_progress",
],
"awaiting_review": [
# Combined review - user reviews lyrics AND selects instrumental in one step
"review_complete",
"corrected_lyrics",
"instrumental_selection",
@@ -1054,13 +1099,7 @@ STATE_DATA_CLEAR_KEYS = {
"screens_progress",
"encoding_progress",
],
"awaiting_instrumental_selection": [
"instrumental_selection",
"video_progress",
"render_progress",
"screens_progress",
"encoding_progress",
],
# Note: awaiting_instrumental_selection removed - deprecated by combined review flow (Jan 2026)
"instrumental_selected": [
# Clear video/encoding/distribution state to allow re-processing
# Preserves: instrumental_selection, lyrics_metadata, review data
@@ -1148,14 +1187,8 @@ async def reset_job(
}
# Perform the update with state_data clearing
# We need to set the cleared keys to DELETE_FIELD
success = job_manager.update_job(job_id, updates)
if not success:
raise HTTPException(
status_code=500,
detail="Failed to reset job. Please try again."
)
# Note: update_job() raises exceptions on failure, returns None on success
job_manager.update_job(job_id, updates)
# Clear the state data keys and error state using direct Firestore update
from google.cloud.firestore_v1 import DELETE_FIELD, ArrayUnion
@@ -1334,6 +1367,69 @@ async def trigger_worker(
)
# =============================================================================
# Prepare Review Audio Endpoint
# =============================================================================
class PrepareReviewAudioResponse(BaseModel):
"""Response from prepare-review-audio endpoint."""
status: str
job_id: str
transcoded_files: List[str]
message: str
@router.post("/jobs/{job_id}/prepare-review-audio", response_model=PrepareReviewAudioResponse)
async def prepare_review_audio(
job_id: str,
auth_data: AuthResult = Depends(require_admin),
):
"""
Transcode review audio to OGG Opus for a job (admin only).
Use this to backfill transcoded audio for existing jobs that were
created before eager transcoding was deployed. Transcodes the main
input audio and all stem files to OGG Opus 128kbps.
Idempotent: skips files that are already transcoded.
"""
admin_email = auth_data.user_email or "unknown"
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
try:
import asyncio
from backend.services.audio_transcoding_service import AudioTranscodingService
transcoding = AudioTranscodingService()
transcoded = await asyncio.to_thread(transcoding.prepare_review_audio_for_job, job)
logger.info(
f"Admin {admin_email} prepared review audio for job {job_id}: "
f"{len(transcoded)} files transcoded"
)
return PrepareReviewAudioResponse(
status="success",
job_id=job_id,
transcoded_files=transcoded,
message=f"Transcoded {len(transcoded)} files to OGG Opus",
)
except Exception as e:
logger.error(
f"Admin {admin_email} failed to prepare review audio for job {job_id}: {e}",
exc_info=True,
)
raise HTTPException(
status_code=500,
detail=f"Failed to prepare review audio: {str(e)}",
)
# =============================================================================
# Clear Worker State Endpoint
# =============================================================================
@@ -1698,6 +1794,7 @@ async def get_job_completion_message(
title=job.title,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
is_private=getattr(job, 'is_private', False),
)
# Build subject: "NOMAD-1178: Artist - Title (Your karaoke video is ready!)"
@@ -1762,6 +1859,7 @@ async def send_job_completion_email(
title=job.title,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
is_private=getattr(job, 'is_private', False),
)
# Send the email
@@ -1863,3 +1961,704 @@ async def impersonate_user(
user_email=target_email,
message=f"Now impersonating {target_email}",
)
# =============================================================================
# Regenerate Screens Endpoint
# =============================================================================
class RegenerateScreensResponse(BaseModel):
"""Response from regenerate screens endpoint."""
status: str
job_id: str
message: str
previous_screens_deleted: bool
worker_triggered: bool
error: Optional[str] = None
# States where regenerate screens is allowed
REGENERATE_SCREENS_ALLOWED_STATES = {
"complete",
"failed",
"awaiting_review",
"awaiting_instrumental_selection",
"instrumental_selected",
"prep_complete",
}
@router.post("/jobs/{job_id}/regenerate-screens", response_model=RegenerateScreensResponse)
async def regenerate_screens(
job_id: str,
auth_data: AuthResult = Depends(require_admin),
):
"""
Regenerate title and end screens with current artist/title metadata (admin only).
Use this when you've edited the artist or title fields and need the
title/end screens to reflect the new metadata.
This endpoint:
1. Validates the job is in an appropriate state
2. Deletes existing screen files from GCS
3. Triggers the screens worker to regenerate with current metadata
4. Returns immediately (does not wait for completion)
Monitor progress via the job detail page or logs.
Allowed states: complete, failed, awaiting_review, awaiting_instrumental_selection,
instrumental_selected, prep_complete
"""
admin_email = auth_data.user_email or "unknown"
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Validate job state
if job.status not in REGENERATE_SCREENS_ALLOWED_STATES:
raise HTTPException(
status_code=400,
detail=f"Cannot regenerate screens for job in '{job.status}' state. "
f"Allowed states: {', '.join(sorted(REGENERATE_SCREENS_ALLOWED_STATES))}"
)
# Validate job has artist and title
if not job.artist or not job.title:
raise HTTPException(
status_code=400,
detail="Job must have artist and title set to regenerate screens"
)
# Check that audio and lyrics processing completed at some point
state_data = job.state_data or {}
audio_progress = state_data.get('audio_progress', {})
lyrics_progress = state_data.get('lyrics_progress', {})
if audio_progress.get('stage') != 'audio_complete':
raise HTTPException(
status_code=400,
detail="Audio processing must be complete before regenerating screens"
)
if lyrics_progress.get('stage') != 'lyrics_complete':
raise HTTPException(
status_code=400,
detail="Lyrics processing must be complete before regenerating screens"
)
# Delete existing screen files from GCS
storage = StorageService()
screens_deleted = False
try:
# Delete screen video and image files
screen_paths = [
f"jobs/{job_id}/screens/title.mov",
f"jobs/{job_id}/screens/title.jpg",
f"jobs/{job_id}/screens/title.png",
f"jobs/{job_id}/screens/end.mov",
f"jobs/{job_id}/screens/end.jpg",
f"jobs/{job_id}/screens/end.png",
]
for path in screen_paths:
try:
storage.delete_file(path)
except Exception:
pass # File may not exist
screens_deleted = True
logger.info(f"Admin {admin_email} deleted existing screens for job {job_id}")
except Exception as e:
logger.warning(f"Error deleting screens for job {job_id}: {e}")
# Clear screens_progress to allow worker to run
from google.cloud.firestore_v1 import DELETE_FIELD, ArrayUnion
db = job_manager.firestore.db
job_ref = db.collection("jobs").document(job_id)
# Update state to allow screens worker to run
job_ref.update({
"state_data.screens_progress": DELETE_FIELD,
"state_data.audio_complete": True,
"state_data.lyrics_complete": True,
"timeline": ArrayUnion([{
"status": job.status,
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Admin {admin_email} triggered screen regeneration"
}])
})
# Trigger screens worker
worker_triggered = False
error_msg = None
try:
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
worker_triggered = await worker_service.trigger_screens_worker(job_id)
if worker_triggered:
logger.info(f"Admin {admin_email} triggered screens regeneration for job {job_id}")
else:
error_msg = "Worker service returned False - check service logs"
except Exception as e:
error_msg = str(e)
logger.error(f"Error triggering screens worker for job {job_id}: {e}", exc_info=True)
return RegenerateScreensResponse(
status="success" if worker_triggered else "error",
job_id=job_id,
message="Screens regeneration started" if worker_triggered else "Failed to trigger screens worker",
previous_screens_deleted=screens_deleted,
worker_triggered=worker_triggered,
error=error_msg,
)
# =============================================================================
# Restart Job Endpoint
# =============================================================================
class RestartJobRequest(BaseModel):
"""Request model for restarting a job."""
preserve_audio_stems: bool = False # If true, keep existing stems (faster)
delete_outputs: bool = True # Delete existing output files from GCS
class RestartJobResponse(BaseModel):
"""Response from restart job endpoint."""
status: str
job_id: str
message: str
previous_status: str
new_status: str
cleared_data: List[str]
deleted_gcs_paths: List[str]
workers_triggered: List[str]
error: Optional[str] = None
# States where restart is allowed
RESTART_ALLOWED_STATES = {
"pending",
"complete",
"failed",
"awaiting_review",
"in_review", # User opened review UI but hasn't completed
"awaiting_audio_selection",
"awaiting_instrumental_selection",
"instrumental_selected",
"prep_complete",
}
@router.post("/jobs/{job_id}/restart", response_model=RestartJobResponse)
async def restart_job(
job_id: str,
request: RestartJobRequest,
auth_data: AuthResult = Depends(require_admin),
):
"""
Fully restart a job from the beginning (admin only).
Unlike reset (which just changes state), restart actually triggers
the appropriate workers to begin processing again.
Options:
- preserve_audio_stems: If True, keeps existing audio separation (faster restart).
Useful when you just need to regenerate screens with updated metadata.
- delete_outputs: If True (default), deletes existing output files from GCS.
When preserve_audio_stems=True:
- Keeps stems, lyrics, and transcription data
- Clears screens, video, encoding state
- Triggers screens worker immediately
When preserve_audio_stems=False:
- Clears all processing state
- For YouTube URL jobs: triggers download then audio/lyrics workers
- For audio search jobs: transitions to awaiting_audio_selection
- For file upload jobs: triggers audio/lyrics workers
Allowed states: pending, complete, failed, awaiting_review, awaiting_audio_selection,
awaiting_instrumental_selection, instrumental_selected, prep_complete
"""
admin_email = auth_data.user_email or "unknown"
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Validate job state
if job.status not in RESTART_ALLOWED_STATES:
raise HTTPException(
status_code=400,
detail=f"Cannot restart job in '{job.status}' state. "
f"Allowed states: {', '.join(sorted(RESTART_ALLOWED_STATES))}"
)
previous_status = job.status
state_data = job.state_data or {}
cleared_keys = []
deleted_paths = []
workers_triggered = []
error_msg = None
from google.cloud.firestore_v1 import DELETE_FIELD, ArrayUnion
db = job_manager.firestore.db
job_ref = db.collection("jobs").document(job_id)
# Delete output files if requested
if request.delete_outputs:
storage = StorageService()
# Delete screens
for screen_file in ["title.mov", "title.jpg", "title.png", "end.mov", "end.jpg", "end.png"]:
try:
path = f"jobs/{job_id}/screens/{screen_file}"
storage.delete_file(path)
deleted_paths.append(path)
except Exception:
pass
# Delete final outputs
for output_file in ["final_4k.mp4", "final_720p.mp4", "with_vocals.mkv", "preview.mp4"]:
try:
path = f"jobs/{job_id}/output/{output_file}"
storage.delete_file(path)
deleted_paths.append(path)
except Exception:
pass
if request.preserve_audio_stems:
# Quick restart: keep stems/lyrics, just regenerate screens and video
# Validate that audio and lyrics completed
audio_progress = state_data.get('audio_progress', {})
lyrics_progress = state_data.get('lyrics_progress', {})
if audio_progress.get('stage') != 'audio_complete':
raise HTTPException(
status_code=400,
detail="Cannot preserve audio stems - audio processing was not complete"
)
if lyrics_progress.get('stage') != 'lyrics_complete':
raise HTTPException(
status_code=400,
detail="Cannot preserve audio stems - lyrics processing was not complete"
)
# Clear only screens/video/encoding state
keys_to_clear = [
"screens_progress",
"video_progress",
"render_progress",
"encoding_progress",
"review_complete",
"instrumental_selection",
]
update_payload = {}
for key in keys_to_clear:
if key in state_data:
update_payload[f"state_data.{key}"] = DELETE_FIELD
cleared_keys.append(key)
# Set coordination flags to trigger screens worker
update_payload["state_data.audio_complete"] = True
update_payload["state_data.lyrics_complete"] = True
update_payload["status"] = "downloading" # Allows transition to generating_screens
update_payload["message"] = "Restarting with preserved audio stems"
update_payload["error_message"] = DELETE_FIELD
update_payload["error_details"] = DELETE_FIELD
update_payload["timeline"] = ArrayUnion([{
"status": "downloading",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Admin {admin_email} restarted job (preserving audio stems)"
}])
job_ref.update(update_payload)
new_status = "downloading"
# Trigger screens worker
try:
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
if await worker_service.trigger_screens_worker(job_id):
workers_triggered.append("screens")
logger.info(f"Admin {admin_email} triggered screens worker for restart of job {job_id}")
else:
error_msg = "Failed to trigger screens worker"
except Exception as e:
error_msg = f"Error triggering screens worker: {e}"
logger.error(f"Error triggering screens worker for job {job_id}: {e}", exc_info=True)
else:
# Full restart: clear everything and start from beginning
keys_to_clear = [
"audio_complete",
"lyrics_complete",
"audio_progress",
"lyrics_progress",
"screens_progress",
"video_progress",
"render_progress",
"encoding_progress",
"review_complete",
"corrected_lyrics",
"instrumental_selection",
"instrumental_options",
"lyrics_metadata",
]
update_payload = {}
for key in keys_to_clear:
if key in state_data:
update_payload[f"state_data.{key}"] = DELETE_FIELD
cleared_keys.append(key)
update_payload["error_message"] = DELETE_FIELD
update_payload["error_details"] = DELETE_FIELD
# Determine what workers to trigger based on job input type
has_youtube_url = bool(job.url)
has_uploaded_file = bool(job.input_media_gcs_path)
has_audio_search = bool(state_data.get('audio_search_results') or state_data.get('audio_selection'))
if has_youtube_url and not has_audio_search:
# YouTube URL job - need to re-download and process
# Clear the downloaded file to force re-download
update_payload["input_media_gcs_path"] = DELETE_FIELD
cleared_keys.append("input_media_gcs_path")
update_payload["status"] = "downloading"
update_payload["message"] = "Re-downloading audio from YouTube"
update_payload["timeline"] = ArrayUnion([{
"status": "downloading",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Admin {admin_email} restarted job - downloading from YouTube"
}])
job_ref.update(update_payload)
new_status = "downloading"
# Download YouTube audio using YouTubeDownloadService, then trigger processing
try:
from backend.services.youtube_download_service import (
get_youtube_download_service,
YouTubeDownloadError,
)
from backend.services.worker_service import get_worker_service
youtube_service = get_youtube_download_service()
audio_gcs_path = await youtube_service.download(
url=job.url,
job_id=job_id,
artist=job.artist,
title=job.title,
)
# Update job with downloaded audio path
job_manager.update_job(job_id, {'input_media_gcs_path': audio_gcs_path})
workers_triggered.append("download")
logger.info(f"YouTube audio re-downloaded to GCS: {audio_gcs_path}")
# Now trigger audio and lyrics workers
worker_service = get_worker_service()
if await worker_service.trigger_audio_worker(job_id):
workers_triggered.append("audio")
if await worker_service.trigger_lyrics_worker(job_id):
workers_triggered.append("lyrics")
if "audio" not in workers_triggered and "lyrics" not in workers_triggered:
error_msg = "Audio downloaded but failed to trigger processing workers"
except YouTubeDownloadError as e:
error_msg = f"YouTube download failed: {e}"
logger.error(f"YouTube download failed for job {job_id}: {e}")
# Update job to failed state
job_manager.mark_job_failed(job_id, f"YouTube download failed: {e}")
new_status = "failed"
except Exception as e:
error_msg = f"Error during YouTube restart: {e}"
logger.error(f"Error restarting YouTube job {job_id}: {e}", exc_info=True)
elif has_audio_search:
# Audio search job - go back to audio selection
# Clear audio search results to allow re-search
for key in ["audio_search_results", "audio_search_count", "remote_search_id", "audio_selection"]:
if key in state_data:
update_payload[f"state_data.{key}"] = DELETE_FIELD
cleared_keys.append(key)
update_payload["input_media_gcs_path"] = DELETE_FIELD
update_payload["status"] = "awaiting_audio_selection"
update_payload["message"] = "Ready for audio source selection"
update_payload["timeline"] = ArrayUnion([{
"status": "awaiting_audio_selection",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Admin {admin_email} restarted job - select audio source"
}])
job_ref.update(update_payload)
new_status = "awaiting_audio_selection"
# No workers to trigger - waiting for admin to select audio
elif has_uploaded_file:
# File upload job - just trigger audio/lyrics workers
update_payload["status"] = "downloading"
update_payload["message"] = "Restarting audio and lyrics processing"
update_payload["timeline"] = ArrayUnion([{
"status": "downloading",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Admin {admin_email} restarted job from beginning"
}])
job_ref.update(update_payload)
new_status = "downloading"
try:
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
if await worker_service.trigger_audio_worker(job_id):
workers_triggered.append("audio")
if await worker_service.trigger_lyrics_worker(job_id):
workers_triggered.append("lyrics")
if not workers_triggered:
error_msg = "Failed to trigger audio/lyrics workers"
except Exception as e:
error_msg = f"Error triggering workers: {e}"
logger.error(f"Error triggering workers for job {job_id}: {e}", exc_info=True)
else:
# Unknown job type - just set to pending
update_payload["status"] = "pending"
update_payload["message"] = "Job reset - manual intervention may be needed"
update_payload["timeline"] = ArrayUnion([{
"status": "pending",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Admin {admin_email} restarted job (no automatic worker trigger)"
}])
job_ref.update(update_payload)
new_status = "pending"
error_msg = "Job type unclear - may need manual worker trigger"
logger.info(
f"Admin {admin_email} restarted job {job_id}: "
f"{previous_status} -> {new_status}, cleared: {cleared_keys}, workers: {workers_triggered}"
)
return RestartJobResponse(
status="success" if workers_triggered or new_status == "awaiting_audio_selection" else "partial",
job_id=job_id,
message=f"Job restarted from {previous_status} to {new_status}",
previous_status=previous_status,
new_status=new_status,
cleared_data=cleared_keys,
deleted_gcs_paths=deleted_paths,
workers_triggered=workers_triggered,
error=error_msg,
)
# =============================================================================
# Override Audio Source Endpoint
# =============================================================================
class OverrideAudioSourceRequest(BaseModel):
"""Request model for overriding audio source."""
source_type: str # "audio_search" - switch to audio search mode
class OverrideAudioSourceResponse(BaseModel):
"""Response from override audio source endpoint."""
status: str
job_id: str
message: str
previous_source: str # "youtube", "audio_search", "file_upload"
new_source: str
cleared_data: List[str]
new_status: str
search_results_count: Optional[int] = None
error: Optional[str] = None
# States where audio source override is allowed
OVERRIDE_AUDIO_ALLOWED_STATES = {
"pending",
"complete",
"failed",
"awaiting_review",
"awaiting_audio_selection",
"instrumental_selected",
"prep_complete",
}
@router.post("/jobs/{job_id}/override-audio-source", response_model=OverrideAudioSourceResponse)
async def override_audio_source(
job_id: str,
request: OverrideAudioSourceRequest,
auth_data: AuthResult = Depends(require_admin),
):
"""
Override the audio source for a job (admin only).
Use this when a Made-For-You order was submitted with a YouTube URL
but you want to use audio search to find higher quality audio instead.
Currently supports switching to audio_search mode, which:
1. Clears existing audio-related state
2. Performs an audio search using the job's artist/title
3. Stores search results in state_data
4. Transitions job to awaiting_audio_selection with results ready
5. Admin can then select an audio source from the search results
Allowed states: pending, complete, failed, awaiting_review, awaiting_audio_selection,
instrumental_selected, prep_complete
"""
admin_email = auth_data.user_email or "unknown"
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Validate job state
if job.status not in OVERRIDE_AUDIO_ALLOWED_STATES:
raise HTTPException(
status_code=400,
detail=f"Cannot override audio source for job in '{job.status}' state. "
f"Allowed states: {', '.join(sorted(OVERRIDE_AUDIO_ALLOWED_STATES))}"
)
# Validate source type
if request.source_type != "audio_search":
raise HTTPException(
status_code=400,
detail="Only 'audio_search' source type is currently supported"
)
# Determine current source type
state_data = job.state_data or {}
if state_data.get('audio_selection') or state_data.get('audio_search_results'):
previous_source = "audio_search"
elif job.url:
previous_source = "youtube"
elif job.input_media_gcs_path:
previous_source = "file_upload"
else:
previous_source = "unknown"
previous_status = job.status
cleared_keys = []
from google.cloud.firestore_v1 import DELETE_FIELD, ArrayUnion
db = job_manager.firestore.db
job_ref = db.collection("jobs").document(job_id)
# Clear all audio-related state to start fresh with audio search
keys_to_clear = [
"audio_complete",
"lyrics_complete",
"audio_progress",
"lyrics_progress",
"screens_progress",
"video_progress",
"render_progress",
"encoding_progress",
"review_complete",
"corrected_lyrics",
"instrumental_selection",
"instrumental_options",
"lyrics_metadata",
"audio_search_results",
"audio_search_count",
"remote_search_id",
"audio_selection",
]
update_payload = {}
for key in keys_to_clear:
if key in state_data:
update_payload[f"state_data.{key}"] = DELETE_FIELD
cleared_keys.append(key)
# Clear job-level audio fields
update_payload["url"] = DELETE_FIELD # Clear YouTube URL
update_payload["input_media_gcs_path"] = DELETE_FIELD # Clear downloaded file
update_payload["error_message"] = DELETE_FIELD
update_payload["error_details"] = DELETE_FIELD
# Set audio search fields based on current artist/title
search_artist = job.artist
search_title = job.title
update_payload["audio_search_artist"] = search_artist
update_payload["audio_search_title"] = search_title
# First transition to searching_audio
update_payload["status"] = "searching_audio"
update_payload["message"] = f"Searching for: {search_artist} - {search_title}"
update_payload["progress"] = 5
update_payload["timeline"] = ArrayUnion([{
"status": "searching_audio",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Admin {admin_email} switched audio source from {previous_source} to audio_search"
}])
job_ref.update(update_payload)
# Perform the actual audio search
search_results_count = None
error_msg = None
try:
audio_search_service = get_audio_search_service()
search_results = audio_search_service.search(search_artist, search_title)
search_results_count = len(search_results)
# Store results in job state_data
results_dicts = [r.to_dict() for r in search_results]
state_data_update = {
'audio_search_results': results_dicts,
'audio_search_count': search_results_count,
}
if audio_search_service.last_remote_search_id:
state_data_update['remote_search_id'] = audio_search_service.last_remote_search_id
# Transition to awaiting_audio_selection with results
new_status = "awaiting_audio_selection"
job_ref.update({
"state_data": state_data_update,
"status": new_status,
"message": f"Found {search_results_count} audio sources - select one",
"progress": 10,
})
logger.info(
f"Admin {admin_email} overrode audio source for job {job_id}: "
f"{previous_source} -> audio_search, found {search_results_count} results"
)
except NoResultsError as e:
new_status = "failed"
error_msg = f"No audio sources found for: {search_artist} - {search_title}"
job_ref.update({
"status": new_status,
"message": error_msg,
"error_message": error_msg,
})
logger.warning(f"Audio search failed for job {job_id}: {error_msg}")
except AudioSearchError as e:
new_status = "failed"
error_msg = f"Audio search failed: {e}"
job_ref.update({
"status": new_status,
"message": error_msg,
"error_message": error_msg,
})
logger.error(f"Audio search error for job {job_id}: {e}")
return OverrideAudioSourceResponse(
status="success" if not error_msg else "error",
job_id=job_id,
message=f"Found {search_results_count} audio sources - select one in the admin panel." if not error_msg
else error_msg,
previous_source=previous_source,
new_source="audio_search",
cleared_data=cleared_keys,
new_status=new_status,
search_results_count=search_results_count,
error=error_msg,
)
+113 -61
View File
@@ -46,6 +46,7 @@ from backend.version import VERSION
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType, AuthResult
from backend.middleware.tenant import get_tenant_config_from_request
from backend.exceptions import InsufficientCreditsError, RateLimitExceededError
from pathlib import Path
logger = logging.getLogger(__name__)
@@ -134,6 +135,9 @@ class AudioSearchRequest(BaseModel):
# Non-interactive mode
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
# Private (non-published) track mode
is_private: bool = Field(False, description="Private track: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive")
@validator('artist', 'title', 'lyrics_artist', 'lyrics_title', 'display_artist', 'display_title')
def normalize_text_fields(cls, v):
"""Normalize text fields to standardize Unicode characters."""
@@ -236,14 +240,6 @@ def extract_request_metadata(request: Request, created_from: str = "audio_search
}
async def _trigger_workers_parallel(job_id: str) -> None:
"""Trigger both audio and lyrics workers in parallel."""
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
def _extract_gcs_path(filepath: str) -> str:
"""
Extract the path portion from a GCS path or gs:// URL.
@@ -262,27 +258,20 @@ def _extract_gcs_path(filepath: str) -> str:
return filepath
async def _download_and_start_processing(
def _validate_and_prepare_selection(
job_id: str,
selection_index: int,
audio_search_service,
background_tasks: BackgroundTasks,
) -> Dict[str, Any]:
"""
Download selected audio and start job processing.
This is called either:
- Immediately after search if auto_download=True
- When user calls the select endpoint
For remote flacfetch downloads (torrent sources), the file is downloaded
on the flacfetch VM and uploaded directly to GCS. For local downloads
(YouTube), the file is downloaded locally and then uploaded to GCS.
Validate selection and transition job to DOWNLOADING_AUDIO state.
Returns selection info dict with the selected result details.
Raises HTTPException on validation errors.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Get search results from state_data
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
@@ -296,9 +285,6 @@ async def _download_and_start_processing(
selected = search_results[selection_index]
# Get remote_search_id from state_data (stored during initial search)
remote_search_id = job.state_data.get('remote_search_id')
# Transition to downloading state
job_manager.transition_to_state(
job_id=job_id,
@@ -310,8 +296,59 @@ async def _download_and_start_processing(
'selected_audio_provider': selected['provider'],
}
)
# Save download params to job BEFORE attempting download
# This enables retry even if download fails
source_name = selected.get('provider')
source_id = selected.get('source_id')
target_file = selected.get('target_file')
download_url = selected.get('url')
job_manager.update_job(job_id, {
'audio_source_type': 'audio_search',
'source_name': source_name,
'source_id': source_id,
'target_file': target_file,
'download_url': download_url,
})
logger.info(
f"Job {job_id}: Saved download params before attempt: "
f"source_name={source_name}, source_id={source_id}"
)
return {
'selected_index': selection_index,
'selected_title': selected['title'],
'selected_artist': selected['artist'],
'selected_provider': selected['provider'],
}
async def _download_audio_and_trigger_workers(
job_id: str,
selection_index: int,
audio_search_service,
) -> None:
"""
Download audio and trigger processing workers.
This runs as a background task after the select endpoint returns.
Also called synchronously from the auto_download path during search.
For remote flacfetch downloads (torrent sources), the file is downloaded
on the flacfetch VM and uploaded directly to GCS. For local downloads
(YouTube), the file is downloaded locally and then uploaded to GCS.
"""
try:
job = job_manager.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found during background download")
return
search_results = job.state_data.get('audio_search_results', [])
selected = search_results[selection_index]
remote_search_id = job.state_data.get('remote_search_id')
# Get source info from selected result
source_name = selected.get('provider')
source_id = selected.get('source_id')
@@ -433,36 +470,36 @@ async def _download_and_start_processing(
shutil.rmtree(temp_dir)
except Exception as e:
logger.warning(f"Failed to clean up temp files: {e}")
# Update job with GCS path and transition to DOWNLOADING
# Update job with GCS path
job_manager.update_job(job_id, {
'input_media_gcs_path': audio_gcs_path,
'filename': filename,
})
# Transition from DOWNLOADING_AUDIO to DOWNLOADING and trigger workers
# Note: Can't use start_job_processing() here because that expects PENDING status,
# but we're in DOWNLOADING_AUDIO state from the audio search flow.
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=15,
message="Audio downloaded, starting processing"
)
# Trigger workers
background_tasks.add_task(_trigger_workers_parallel, job_id)
return {
'selected_index': selection_index,
'selected_title': selected['title'],
'selected_artist': selected['artist'],
'selected_provider': selected['provider'],
}
# Trigger workers directly
worker_service = get_worker_service()
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
except DownloadError as e:
logger.error(f"Download failed for job {job_id}: {e}")
job_manager.fail_job(job_id, f"Audio download failed: {e}")
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
except Exception as e:
logger.error(f"Download failed for job {job_id}: {e}", exc_info=True)
job_manager.fail_job(job_id, f"Audio download failed: {e}")
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
@router.post("/audio-search/search", response_model=AudioSearchResponse)
@@ -588,6 +625,7 @@ async def search_audio(
auto_download=body.auto_download,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
is_private=body.is_private,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
)
@@ -727,16 +765,22 @@ async def search_audio(
# If auto_download, select best and start processing
if body.auto_download:
best_index = audio_search_service.select_best(search_results)
logger.info(f"Auto-download enabled, selecting result {best_index}")
selection_info = await _download_and_start_processing(
selection_info = _validate_and_prepare_selection(
job_id=job_id,
selection_index=best_index,
audio_search_service=audio_search_service,
background_tasks=background_tasks,
)
# For auto_download, run download in background so this endpoint returns quickly
background_tasks.add_task(
_download_audio_and_trigger_workers,
job_id,
best_index,
audio_search_service,
)
return AudioSearchResponse(
status="success",
job_id=job_id,
@@ -823,6 +867,8 @@ async def search_audio(
except HTTPException:
raise
except (InsufficientCreditsError, RateLimitExceededError):
raise
except Exception as e:
logger.error(f"Error in audio search: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@@ -870,24 +916,23 @@ async def select_audio_source(
):
"""
Select an audio source and start job processing.
This endpoint:
1. Validates the job is awaiting selection
2. Downloads the selected audio
3. Uploads to GCS
4. Triggers audio and lyrics workers
This endpoint returns immediately after validating the selection and
transitioning the job to DOWNLOADING_AUDIO state. The actual download
runs as a background task so the UI can close the selection dialog
without waiting for the download to complete.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Verify job is awaiting selection
if job.status != JobStatus.AWAITING_AUDIO_SELECTION:
raise HTTPException(
status_code=400,
detail=f"Job is not awaiting audio selection (status: {job.status})"
)
# Get search service instance
# Note: With download_by_id, we no longer need to re-search to populate the cache.
# The source_id stored in job.state_data['audio_search_results'] is sufficient.
@@ -897,18 +942,25 @@ async def select_audio_source(
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
raise HTTPException(status_code=400, detail="No search results cached for this job")
selection_info = await _download_and_start_processing(
# Validate selection and transition to DOWNLOADING_AUDIO (returns immediately)
selection_info = _validate_and_prepare_selection(
job_id=job_id,
selection_index=body.selection_index,
audio_search_service=audio_search_service,
background_tasks=background_tasks,
)
# Schedule the actual download as a background task
background_tasks.add_task(
_download_audio_and_trigger_workers,
job_id,
body.selection_index,
audio_search_service,
)
return AudioSelectResponse(
status="success",
job_id=job_id,
message="Audio selected and download started",
message="Audio selected, download starting",
selected_index=selection_info['selected_index'],
selected_title=selection_info['selected_title'],
selected_artist=selection_info['selected_artist'],
+70 -56
View File
@@ -39,6 +39,7 @@ from backend.services.metrics import metrics
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType, AuthResult
from backend.middleware.tenant import get_tenant_config_from_request
from backend.exceptions import InsufficientCreditsError, RateLimitExceededError
from backend.services.youtube_download_service import (
get_youtube_download_service,
YouTubeDownloadError,
@@ -111,6 +112,9 @@ class CreateJobFromUrlRequest(BaseModel):
# Non-interactive mode
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
# Private (non-published) track mode
is_private: bool = Field(False, description="Private track: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive")
class CreateJobFromUrlResponse(BaseModel):
"""Response from creating a job from URL."""
@@ -172,6 +176,9 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
# Non-interactive mode
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
# Private (non-published) track mode
is_private: bool = Field(False, description="Private track: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive")
class SignedUploadUrl(BaseModel):
"""Signed URL for uploading a file."""
@@ -256,19 +263,6 @@ ALLOWED_IMAGE_EXTENSIONS = {'.png', '.jpg', '.jpeg', '.gif', '.webp'}
ALLOWED_FONT_EXTENSIONS = {'.ttf', '.otf', '.woff', '.woff2'}
async def _trigger_workers_parallel(job_id: str) -> None:
"""
Trigger both audio and lyrics workers in parallel.
FastAPI's BackgroundTasks runs async tasks sequentially, so we use
asyncio.gather to ensure both workers start at the same time.
"""
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
async def _trigger_audio_worker_only(job_id: str) -> None:
"""
Trigger only the audio worker.
@@ -374,6 +368,8 @@ async def upload_and_create_job(
other_stems_models: Optional[str] = Form(None, description="Comma-separated list of models for other stems (bass, drums, guitar, etc.)"),
# Non-interactive mode
non_interactive: bool = Form(False, description="Skip interactive steps (lyrics review, instrumental selection)"),
# Private (non-published) track mode
is_private: bool = Form(False, description="Private track: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive"),
):
"""
Upload an audio file and create a karaoke generation job with full style configuration.
@@ -474,10 +470,11 @@ async def upload_and_create_job(
# Validate credentials for requested distribution services (including defaults)
# This prevents accepting jobs that will fail later due to missing credentials
# Skip YouTube/GDrive checks for private jobs (they won't use those services)
invalid_services = []
credential_manager = get_credential_manager()
if effective_enable_youtube_upload:
if effective_enable_youtube_upload and not is_private:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -487,7 +484,7 @@ async def upload_and_create_job(
if result.status != CredentialStatus.VALID:
invalid_services.append(f"dropbox ({result.message})")
if dist.gdrive_folder_id:
if dist.gdrive_folder_id and not is_private:
result = credential_manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"gdrive ({result.message})")
@@ -587,6 +584,8 @@ async def upload_and_create_job(
request_metadata=request_metadata,
# Non-interactive mode
non_interactive=non_interactive,
# Private (non-published) track mode
is_private=is_private,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
)
@@ -755,18 +754,17 @@ async def upload_and_create_job(
if style_assets:
logger.info(f"Style assets: {list(style_assets.keys())}")
# Transition job to DOWNLOADING state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=5,
message="Files uploaded, preparing to process"
# Use centralized start_job_processing which handles:
# 1. Validates job exists and has input_media_gcs_path
# 2. Transitions to DOWNLOADING (raises InvalidStateTransitionError if invalid)
# 3. Triggers audio + lyrics workers in parallel
background_tasks.add_task(
job_manager.start_job_processing,
job_id,
5, # progress
"Files uploaded, preparing to process"
)
# Trigger workers in parallel using asyncio.gather
# (FastAPI's BackgroundTasks runs async tasks sequentially)
background_tasks.add_task(_trigger_workers_parallel, job_id)
# Build distribution services info for response
distribution_services: Dict[str, Any] = {}
@@ -814,6 +812,8 @@ async def upload_and_create_job(
except HTTPException:
raise
except (InsufficientCreditsError, RateLimitExceededError):
raise
except Exception as e:
logger.error(f"Error uploading files: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e)) from e
@@ -890,6 +890,8 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
dropbox_path: Optional[str] = Field(None, description="Dropbox folder path for organized output")
gdrive_folder_id: Optional[str] = Field(None, description="Google Drive folder ID for public share uploads")
# Private (non-published) track mode
is_private: bool = Field(False, description="Private track: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive")
@@ -1036,10 +1038,11 @@ async def create_job_with_upload_urls(
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
# Skip YouTube/GDrive checks for private jobs (they won't use those services)
invalid_services = []
credential_manager = get_credential_manager()
if effective_enable_youtube_upload:
if effective_enable_youtube_upload and not body.is_private:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1049,7 +1052,7 @@ async def create_job_with_upload_urls(
if result.status != CredentialStatus.VALID:
invalid_services.append(f"dropbox ({result.message})")
if dist.gdrive_folder_id:
if dist.gdrive_folder_id and not body.is_private:
result = credential_manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"gdrive ({result.message})")
@@ -1118,6 +1121,7 @@ async def create_job_with_upload_urls(
other_stems_models=body.other_stems_models,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
is_private=body.is_private,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
)
@@ -1175,6 +1179,8 @@ async def create_job_with_upload_urls(
except HTTPException:
raise
except (InsufficientCreditsError, RateLimitExceededError):
raise
except Exception as e:
logger.error(f"Error creating job with upload URLs: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e)) from e
@@ -1305,17 +1311,17 @@ async def mark_uploads_complete(
logger.info(f"Validated uploads for job {job_id}: {body.uploaded_files}")
# Transition job to DOWNLOADING state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=5,
message="Files uploaded, preparing to process"
# Use centralized start_job_processing which handles:
# 1. Validates job exists and has input_media_gcs_path
# 2. Transitions to DOWNLOADING (raises InvalidStateTransitionError if invalid)
# 3. Triggers audio + lyrics workers in parallel
background_tasks.add_task(
job_manager.start_job_processing,
job_id,
5, # progress
"Files uploaded, preparing to process"
)
# Trigger workers in parallel
background_tasks.add_task(_trigger_workers_parallel, job_id)
# Get distribution services info for response
credential_manager = get_credential_manager()
distribution_services: Dict[str, Any] = {}
@@ -1478,10 +1484,11 @@ async def create_job_from_url(
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
# Skip YouTube/GDrive checks for private jobs (they won't use those services)
invalid_services = []
credential_manager = get_credential_manager()
if effective_enable_youtube_upload:
if effective_enable_youtube_upload and not body.is_private:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1491,7 +1498,7 @@ async def create_job_from_url(
if result.status != CredentialStatus.VALID:
invalid_services.append(f"dropbox ({result.message})")
if dist.gdrive_folder_id:
if dist.gdrive_folder_id and not body.is_private:
result = credential_manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"gdrive ({result.message})")
@@ -1558,6 +1565,7 @@ async def create_job_from_url(
other_stems_models=body.other_stems_models,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
is_private=body.is_private,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
)
@@ -1588,14 +1596,6 @@ async def create_job_from_url(
if title:
logger.info(f" Title: {title}")
# Transition job to DOWNLOADING state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=5,
message="Starting audio download from URL"
)
# For YouTube URLs, download audio NOW using YouTubeDownloadService
# This uses remote flacfetch (if configured) to avoid bot detection on Cloud Run
if _is_youtube_url(body.url):
@@ -1617,14 +1617,16 @@ async def create_job_from_url(
logger.info(f"YouTube audio downloaded to GCS: {audio_gcs_path}")
# Now trigger both workers in parallel (audio already downloaded)
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=10,
message="Audio downloaded, starting processing"
# Use centralized start_job_processing which handles:
# 1. Validates job exists and has input_media_gcs_path
# 2. Transitions PENDING → DOWNLOADING (raises if invalid)
# 3. Triggers audio + lyrics workers in parallel
background_tasks.add_task(
job_manager.start_job_processing,
job_id,
10, # progress
"Audio downloaded, starting processing"
)
background_tasks.add_task(_trigger_workers_parallel, job_id)
except YouTubeDownloadError as e:
logger.error(f"YouTube download failed: {e}")
@@ -1634,8 +1636,14 @@ async def create_job_from_url(
detail=f"YouTube download failed: {e}"
)
else:
# For non-YouTube URLs, trigger ONLY audio worker first
# For non-YouTube URLs, transition to DOWNLOADING and trigger audio worker
# The audio worker will download the URL and trigger lyrics worker after
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=5,
message="Starting audio download from URL"
)
background_tasks.add_task(_trigger_audio_worker_only, job_id)
return CreateJobFromUrlResponse(
@@ -1649,6 +1657,8 @@ async def create_job_from_url(
except HTTPException:
raise
except (InsufficientCreditsError, RateLimitExceededError):
raise
except Exception as e:
logger.error(f"Error creating job from URL: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e)) from e
@@ -1785,10 +1795,11 @@ async def create_finalise_only_job(
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate distribution credentials if services are requested
# Skip YouTube/GDrive checks for private jobs (they won't use those services)
invalid_services = []
credential_manager = get_credential_manager()
if effective_enable_youtube_upload:
if effective_enable_youtube_upload and not body.is_private:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1798,7 +1809,7 @@ async def create_finalise_only_job(
if result.status != CredentialStatus.VALID:
invalid_services.append(f"dropbox ({result.message})")
if dist.gdrive_folder_id:
if dist.gdrive_folder_id and not body.is_private:
result = credential_manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"gdrive ({result.message})")
@@ -1856,6 +1867,7 @@ async def create_finalise_only_job(
finalise_only=True,
keep_brand_code=body.keep_brand_code,
request_metadata=request_metadata,
is_private=body.is_private,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
)
@@ -1913,6 +1925,8 @@ async def create_finalise_only_job(
except HTTPException:
raise
except (InsufficientCreditsError, RateLimitExceededError):
raise
except Exception as e:
logger.error(f"Error creating finalise-only job: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e)) from e
+66
View File
@@ -16,6 +16,9 @@ from backend.services.encoding_service import get_encoding_service
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
from backend.services.job_health_service import check_job_consistency_detailed
from backend.services.job_manager import JobManager
from backend.models.job import JobStatus
router = APIRouter()
logger = logging.getLogger(__name__)
@@ -430,6 +433,69 @@ async def preload_status() -> Dict[str, Any]:
}
@router.get("/health/job-consistency")
async def job_consistency_check(
status: Optional[str] = None,
limit: int = 50
) -> Dict[str, Any]:
"""
Check job consistency across the system.
This endpoint identifies jobs with state inconsistencies that may indicate
bugs in the state machine or trigger logic. Use this to find stuck jobs
or jobs in invalid states.
Args:
status: Optional filter by job status (e.g., "pending", "downloading")
limit: Maximum number of jobs to check (default 50)
Returns:
Dictionary with:
- total_checked: Number of jobs checked
- inconsistent_count: Number of jobs with issues
- inconsistent_jobs: List of job details with their issues
- summary: Summary of issue types found
"""
job_manager = JobManager()
# Query jobs to check
query_params = {}
if status:
try:
job_status = JobStatus(status)
query_params['status'] = job_status
except ValueError:
return {
"error": f"Invalid status: {status}",
"valid_statuses": [s.value for s in JobStatus]
}
# Get jobs from Firestore
jobs = job_manager.list_jobs(limit=limit, **query_params)
# Check each job for consistency
inconsistent_jobs = []
issue_summary = {}
for job in jobs:
result = check_job_consistency_detailed(job)
if not result['is_healthy']:
inconsistent_jobs.append(result)
# Count issue types
for issue in result['issues']:
issue_type = issue.split(':')[0] if ':' in issue else issue
issue_summary[issue_type] = issue_summary.get(issue_type, 0) + 1
return {
"total_checked": len(jobs),
"inconsistent_count": len(inconsistent_jobs),
"inconsistent_jobs": inconsistent_jobs,
"summary": issue_summary,
"status_filter": status,
"limit": limit,
}
@router.get("/readiness")
async def readiness_check() -> Dict[str, str]:
"""Readiness check endpoint for Cloud Run."""
+454 -7
View File
@@ -8,9 +8,14 @@ Handles job lifecycle endpoints including:
- Job deletion and cancellation
"""
import asyncio
import json
import logging
import os
import subprocess
import tempfile
from pathlib import Path
from typing import List, Optional, Dict, Any
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request, UploadFile, File
from backend.models.job import Job, JobCreate, JobResponse, JobStatus
from backend.models.requests import (
@@ -20,6 +25,7 @@ from backend.models.requests import (
CancelJobRequest,
InstrumentalSelection,
CompleteReviewRequest,
CreateCustomInstrumentalRequest,
)
from backend.services.job_manager import JobManager
from backend.services.worker_service import get_worker_service
@@ -31,6 +37,7 @@ from backend.services.auth_service import AuthResult
from backend.services.metrics import metrics
from backend.middleware.tenant import get_tenant_from_request
from backend.utils.test_data import is_test_email
from backend.exceptions import InsufficientCreditsError, RateLimitExceededError
logger = logging.getLogger(__name__)
@@ -115,7 +122,8 @@ async def create_job(
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=request.youtube_description,
webhook_url=request.webhook_url,
user_email=user_email
user_email=user_email,
is_private=request.is_private or False,
)
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
@@ -133,6 +141,8 @@ async def create_job(
job_id=job.job_id,
message="Job created successfully. Processing started."
)
except (InsufficientCreditsError, RateLimitExceededError):
raise
except Exception as e:
logger.error(f"Error creating job: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@@ -188,7 +198,33 @@ def _check_job_ownership(job: Job, auth_result: AuthResult) -> bool:
return True
@router.get("", response_model=List[Job])
_SUMMARY_STATE_DATA_KEYS = {
'brand_code', 'youtube_url', 'dropbox_link',
'audio_progress', 'lyrics_progress',
'audio_complete', 'lyrics_complete',
'backing_vocals_analysis',
}
_SUMMARY_FILE_URLS_KEYS = {'finals', 'videos', 'packages'}
_HIDE_COMPLETED_STATUSES = ['complete', 'prep_complete', 'failed', 'cancelled']
def _prune_state_data(data: Dict[str, Any]) -> Dict[str, Any]:
"""Strip state_data down to dashboard-required keys."""
sd = data.get('state_data')
if isinstance(sd, dict):
data['state_data'] = {k: v for k, v in sd.items() if k in _SUMMARY_STATE_DATA_KEYS}
return data
def _prune_file_urls(data: Dict[str, Any]) -> Dict[str, Any]:
"""Strip file_urls down to dashboard-required keys."""
fu = data.get('file_urls')
if isinstance(fu, dict):
data['file_urls'] = {k: v for k, v in fu.items() if k in _SUMMARY_FILE_URLS_KEYS}
return data
@router.get("", response_model=None)
async def list_jobs(
request: Request,
status: Optional[JobStatus] = None,
@@ -198,8 +234,10 @@ async def list_jobs(
created_before: Optional[str] = None,
exclude_test: bool = True,
limit: int = 100,
fields: Optional[str] = None,
hide_completed: bool = False,
auth_result: AuthResult = Depends(require_auth)
) -> List[Job]:
):
"""
List jobs with optional filters.
@@ -214,9 +252,13 @@ async def list_jobs(
created_before: Filter jobs created before this ISO datetime
exclude_test: If True (default), exclude jobs from test users (admin only)
limit: Maximum number of jobs to return (default 100)
fields: Set to "summary" for reduced payload with only dashboard-required fields
hide_completed: If True, exclude terminal-status jobs (complete, prep_complete, failed, cancelled)
Returns:
List of jobs matching filters, ordered by created_at descending
List of jobs matching filters, ordered by created_at descending.
When fields=summary, returns List[dict] (pruned).
Otherwise returns List[Job] (full model).
"""
from datetime import datetime
@@ -253,6 +295,32 @@ async def list_jobs(
# Tenant users only see jobs from their tenant
tenant_id = get_tenant_from_request(request)
# --- Summary mode: field-projected query returning dicts ---
if fields == "summary":
exclude_statuses = _HIDE_COMPLETED_STATUSES if hide_completed else None
jobs_dicts = job_manager.list_jobs_summary(
status=status,
exclude_statuses=exclude_statuses,
environment=environment,
client_id=client_id,
created_after=created_after_dt,
created_before=created_before_dt,
user_email=user_email_filter,
tenant_id=tenant_id,
limit=limit,
)
# Exclude test user jobs (Python-side, same as full mode)
if exclude_test and auth_result.is_admin:
jobs_dicts = [j for j in jobs_dicts if not is_test_email(j.get('user_email') or "")]
# Safety-net pruning (in case Firestore returned extra nested keys)
jobs_dicts = [_prune_file_urls(_prune_state_data(j)) for j in jobs_dicts]
logger.debug(f"Listed {len(jobs_dicts)} summary jobs for user={auth_result.user_email}")
return jobs_dicts
# --- Full mode: existing behaviour (List[Job]) ---
jobs = job_manager.list_jobs(
status=status,
environment=environment,
@@ -536,8 +604,14 @@ async def submit_corrections(
)
try:
# Store corrected lyrics in state_data
job_manager.update_state_data(job_id, 'corrected_lyrics', submission.corrections)
# Strip reference_lyrics from Firestore save to avoid exceeding 1MB document
# limit. reference_lyrics can be very large (e.g. Genius fetching non-lyrics
# content like screenplays). The full data is preserved in the GCS upload below.
corrections_for_firestore = {
k: v for k, v in submission.corrections.items()
if k != 'reference_lyrics'
}
job_manager.update_state_data(job_id, 'corrected_lyrics', corrections_for_firestore)
if submission.user_notes:
job_manager.update_state_data(job_id, 'review_notes', submission.user_notes)
@@ -570,6 +644,229 @@ async def submit_corrections(
raise HTTPException(status_code=500, detail=str(e))
@router.post("/{job_id}/create-custom-instrumental")
async def create_custom_instrumental(
job_id: str,
request: CreateCustomInstrumentalRequest,
auth_result: AuthResult = Depends(require_auth)
) -> dict:
"""
Create a custom instrumental by muting regions of backing vocals.
Downloads the clean instrumental and backing vocals stems, applies mute
regions, combines them, uploads to GCS, and returns a signed URL for playback.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if not _check_job_ownership(job, auth_result):
raise HTTPException(status_code=403, detail="You don't have permission to modify this job")
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
# Get stem paths from job
stems = job.file_urls.get('stems', {})
clean_path = stems.get('instrumental_clean')
backing_path = stems.get('backing_vocals')
if not clean_path or not backing_path:
raise HTTPException(
status_code=400,
detail="Job missing required stems (instrumental_clean and/or backing_vocals)"
)
try:
from backend.services.audio_editing_service import AudioEditingService
from karaoke_gen.instrumental_review import MuteRegion
storage = StorageService()
editing_service = AudioEditingService(storage_service=storage)
# Convert request mute regions to domain model
mute_regions = [
MuteRegion(start_seconds=r.start_seconds, end_seconds=r.end_seconds)
for r in request.mute_regions
]
# Create custom instrumental and upload to GCS (offload blocking IO to thread)
output_path = f"jobs/{job_id}/stems/custom_instrumental.flac"
result = await asyncio.get_event_loop().run_in_executor(
None,
lambda: editing_service.create_custom_instrumental(
gcs_clean_instrumental_path=clean_path,
gcs_backing_vocals_path=backing_path,
mute_regions=mute_regions,
gcs_output_path=output_path,
job_id=job_id,
)
)
# Store the custom instrumental path in job file_urls
job_manager.update_file_url(job_id, 'stems', 'custom_instrumental', output_path)
# Generate signed URL for playback
from backend.services.audio_transcoding_service import AudioTranscodingService
transcoding = AudioTranscodingService(storage_service=storage)
audio_url = await transcoding.get_review_audio_url_async(output_path, expiration_minutes=120)
logger.info(f"Job {job_id}: Custom instrumental created with {len(mute_regions)} mute regions")
return {
"status": "success",
"message": f"Custom instrumental created with {len(mute_regions)} mute regions",
"audio_url": audio_url,
"muted_duration_seconds": result.total_muted_duration_seconds,
}
except Exception as e:
logger.error(f"Error creating custom instrumental for job {job_id}: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.post("/{job_id}/upload-instrumental")
async def upload_custom_instrumental(
job_id: str,
file: UploadFile = File(...),
auth_result: AuthResult = Depends(require_auth)
) -> dict:
"""
Upload a custom instrumental audio file for use during review.
Accepts any audio format supported by pydub/ffmpeg (mp3, wav, flac, ogg, etc.).
The file is stored to GCS and its path recorded in the job's stems metadata.
The user can then select 'custom' as their instrumental_selection when completing review.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if not _check_job_ownership(job, auth_result):
raise HTTPException(status_code=403, detail="You don't have permission to modify this job")
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
# Determine extension from filename or content type
original_filename = file.filename or "instrumental"
suffix = Path(original_filename).suffix.lower()
if not suffix:
content_type = file.content_type or ""
suffix_map = {
"audio/mpeg": ".mp3",
"audio/mp3": ".mp3",
"audio/wav": ".wav",
"audio/x-wav": ".wav",
"audio/flac": ".flac",
"audio/x-flac": ".flac",
"audio/ogg": ".ogg",
"audio/aac": ".aac",
"audio/mp4": ".m4a",
}
suffix = suffix_map.get(content_type, ".audio")
storage = StorageService()
tmp_path = None
flac_path = None
try:
with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
tmp_path = tmp.name
content = await file.read()
tmp.write(content)
# Get duration of uploaded file using pydub
from pydub import AudioSegment
audio_segment = AudioSegment.from_file(tmp_path)
upload_duration = len(audio_segment) / 1000.0
# Get duration of original job audio to validate match
original_duration = await _get_audio_duration_ffprobe_signed(job_id, job, storage)
if original_duration is not None:
diff = abs(upload_duration - original_duration)
if diff > 0.5:
raise HTTPException(
status_code=400,
detail=(
f"Duration mismatch: uploaded file is {upload_duration:.1f}s "
f"but original audio is {original_duration:.1f}s. "
f"The instrumental must be exactly {original_duration:.1f}s (±0.5s)."
),
)
# Convert to FLAC for consistency with the rest of the pipeline
# (GCE encoding worker searches for *.flac patterns)
flac_path = tmp_path.rsplit('.', 1)[0] + '.flac' if '.' in tmp_path else tmp_path + '.flac'
if suffix != '.flac':
audio_segment.export(flac_path, format='flac')
else:
flac_path = tmp_path # Already FLAC, no conversion needed
# Upload to GCS (always as .flac)
output_path = f"jobs/{job_id}/stems/custom_instrumental.flac"
storage.upload_file(flac_path, output_path)
except HTTPException:
raise
except Exception as e:
logger.error(f"Job {job_id}: Error processing uploaded instrumental: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Failed to process audio file: {str(e)}")
finally:
if tmp_path and os.path.exists(tmp_path):
os.unlink(tmp_path)
if flac_path and flac_path != tmp_path and os.path.exists(flac_path):
os.unlink(flac_path)
# Record in job file_urls
job_manager.update_file_url(job_id, 'stems', 'custom_instrumental', output_path)
logger.info(f"Job {job_id}: Custom instrumental uploaded ({upload_duration:.1f}s) to {output_path}")
return {
"status": "success",
"duration_seconds": upload_duration,
"message": f"Custom instrumental uploaded ({upload_duration:.1f}s)",
}
async def _get_audio_duration_ffprobe_signed(job_id: str, job, storage: StorageService) -> Optional[float]:
"""
Get the duration of the job's original audio using ffprobe on a signed GCS URL.
Uses ffprobe which reads only the container header - fast even for large files.
Returns None if the duration cannot be determined (non-fatal, upload proceeds).
"""
gcs_path = job.input_media_gcs_path
if not gcs_path:
return None
try:
signed_url = storage.generate_signed_url(gcs_path, expiration_minutes=5)
def _run_ffprobe() -> float:
result = subprocess.run(
['ffprobe', '-v', 'quiet', '-print_format', 'json', '-show_format', signed_url],
capture_output=True, text=True, timeout=30,
)
if result.returncode != 0:
raise ValueError(f"ffprobe error: {result.stderr[:200]}")
data = json.loads(result.stdout)
return float(data['format']['duration'])
return await asyncio.get_event_loop().run_in_executor(None, _run_ffprobe)
except Exception as e:
logger.warning(f"Job {job_id}: Could not determine original audio duration: {e}")
return None
@router.post("/{job_id}/complete-review")
async def complete_review(
job_id: str,
@@ -1085,6 +1382,144 @@ async def retry_job(
"retry_stage": "from_beginning"
}
# If we have audio search download params, retry the download
elif (job.audio_source_type == 'audio_search' and
job.source_name and job.source_id):
logger.info(
f"Job {job_id}: Has audio search params, retrying download: "
f"source_name={job.source_name}, source_id={job.source_id}"
)
# Import needed services
from backend.services.audio_search_service import get_audio_search_service
from backend.services.youtube_download_service import (
get_youtube_download_service,
YouTubeDownloadError
)
from backend.services.audio_search_service import DownloadError
import tempfile
import os
# Clear error state
job_manager.update_job(job_id, {
'error_message': None,
'error_details': None,
})
# Reset to DOWNLOADING_AUDIO state
if not job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING_AUDIO,
progress=12,
message=f"Retrying audio download from {job.source_name}"
):
raise HTTPException(
status_code=500,
detail="Failed to transition job status for retry"
)
# Retry download using saved params
# This is similar to the logic in audio_search.py:_download_and_start_processing
audio_search_service = get_audio_search_service()
try:
if job.source_name == 'YouTube':
# Use YouTubeDownloadService
youtube_service = get_youtube_download_service()
video_id = job.source_id
try:
audio_gcs_path = await youtube_service.download_by_id(
video_id=video_id,
job_id=job_id,
artist=job.artist,
title=job.title,
)
filename = os.path.basename(audio_gcs_path)
except YouTubeDownloadError as e:
job_manager.fail_job(job_id, f"Audio download retry failed: YouTube download failed: {e}")
raise HTTPException(status_code=500, detail=f"YouTube download failed: {e}")
elif job.source_name in ['RED', 'OPS', 'Spotify']:
# Use audio search service for torrent/Spotify sources
if not audio_search_service.is_remote_enabled():
job_manager.fail_job(
job_id,
f"Cannot retry: flacfetch service not configured for {job.source_name}"
)
raise HTTPException(
status_code=400,
detail=f"Cannot download from {job.source_name} without remote flacfetch service."
)
gcs_destination = f"uploads/{job_id}/audio/"
try:
result = audio_search_service.download_by_id(
source_name=job.source_name,
source_id=job.source_id,
output_dir="",
target_file=job.target_file,
download_url=job.download_url,
gcs_path=gcs_destination,
)
audio_gcs_path = result.filepath
if audio_gcs_path.startswith('gs://'):
audio_gcs_path = audio_gcs_path.replace('gs://karaoke-gen-storage/', '', 1)
filename = os.path.basename(result.filepath)
except DownloadError as e:
job_manager.fail_job(job_id, f"Audio download retry failed: {e}")
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
else:
job_manager.fail_job(
job_id,
f"Cannot retry: unknown source type {job.source_name}"
)
raise HTTPException(
status_code=400,
detail=f"Unknown source type: {job.source_name}"
)
# Update job with GCS path
job_manager.update_job(job_id, {
'input_media_gcs_path': audio_gcs_path,
'filename': filename,
})
# Transition to DOWNLOADING and trigger workers
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=15,
message="Audio downloaded, starting processing"
)
# Trigger workers in background
async def trigger_workers():
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
background_tasks.add_task(trigger_workers)
return {
"status": "success",
"job_id": job_id,
"job_status": "downloading",
"message": f"Job retry started: re-downloading from {job.source_name}",
"retry_stage": "audio_download"
}
except HTTPException:
raise
except Exception as e:
logger.error(f"Error retrying audio download for job {job_id}: {e}", exc_info=True)
job_manager.fail_job(job_id, f"Audio download retry failed: {e}")
raise HTTPException(status_code=500, detail=f"Download retry failed: {e}")
else:
# No input audio available - job needs to be resubmitted
raise HTTPException(
@@ -1233,6 +1668,18 @@ async def cleanup_distribution(
"status": "success" if success else "failed",
"path": full_path
}
# Recycle the brand code number so it can be reused
if success:
try:
from backend.services.brand_code_service import (
BrandCodeService, get_brand_code_service
)
prefix, number = BrandCodeService.parse_brand_code(brand_code)
get_brand_code_service().recycle_brand_code(prefix, number)
results["dropbox"]["recycled_brand_code"] = True
except (ValueError, Exception) as e:
logger.warning(f"Failed to recycle brand code {brand_code}: {e}")
results["dropbox"]["recycled_brand_code"] = False
else:
results["dropbox"] = {"status": "failed", "reason": "Dropbox credentials not configured"}
except Exception as e:
+54 -51
View File
@@ -22,7 +22,7 @@ from pathlib import Path
from typing import Dict, Any, Set, Tuple
from fastapi import APIRouter, HTTPException, Request, Depends
from fastapi.responses import FileResponse, StreamingResponse
from fastapi.responses import FileResponse, RedirectResponse, StreamingResponse
from starlette.background import BackgroundTask
from backend.models.job import JobStatus
@@ -138,27 +138,40 @@ async def get_correction_data(
}
# === Add instrumental data for combined review ===
from backend.services.audio_transcoding_service import AudioTranscodingService
transcoding = AudioTranscodingService(storage_service=storage)
# Get instrumental stem URLs
stems = job.file_urls.get('stems', {})
clean_url = stems.get('instrumental_clean')
backing_url = stems.get('instrumental_with_backing')
# Build instrumental options with signed URLs
# Build instrumental options with transcoded signed URLs (OGG Opus)
instrumental_options = []
url_tasks = {}
if clean_url:
url_tasks['clean'] = transcoding.get_review_audio_url_async(clean_url, expiration_minutes=120)
if backing_url:
url_tasks['with_backing'] = transcoding.get_review_audio_url_async(backing_url, expiration_minutes=120)
signed_urls = {}
if url_tasks:
results = await asyncio.gather(*url_tasks.values())
signed_urls = dict(zip(url_tasks.keys(), results))
if clean_url:
instrumental_options.append({
"id": "clean",
"label": "Clean Instrumental",
"description": "No backing vocals - just the music",
"audio_url": storage.generate_signed_url(clean_url, expiration_minutes=120),
"audio_url": signed_urls['clean'],
})
if backing_url:
instrumental_options.append({
"id": "with_backing",
"label": "Instrumental with Backing Vocals",
"description": "Includes harmonies and background vocals",
"audio_url": storage.generate_signed_url(backing_url, expiration_minutes=120),
"audio_url": signed_urls['with_backing'],
})
corrections_data['instrumental_options'] = instrumental_options
@@ -213,49 +226,32 @@ async def get_audio_no_hash(
async def _stream_audio(job_id: str):
"""
Stream the audio file for playback in the review interface.
Redirect to a signed GCS URL for audio playback in the review interface.
Serves transcoded OGG Opus (~3 MB) instead of raw FLAC (~35 MB).
Falls back to FLAC if transcoded version is not available.
"""
from backend.services.audio_transcoding_service import AudioTranscodingService
job_manager = JobManager()
storage = StorageService()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
audio_gcs_path = job.input_media_gcs_path
if not audio_gcs_path:
raise HTTPException(status_code=404, detail="Audio file not found")
# Download to temp file and stream
try:
# Create temp file
with tempfile.NamedTemporaryFile(delete=False, suffix=".flac") as tmp:
tmp_path = tmp.name
storage.download_file(audio_gcs_path, tmp_path)
# Determine content type
if audio_gcs_path.endswith('.flac'):
media_type = "audio/flac"
elif audio_gcs_path.endswith('.wav'):
media_type = "audio/wav"
elif audio_gcs_path.endswith('.mp3'):
media_type = "audio/mpeg"
else:
media_type = "audio/mpeg"
logger.info(f"Job {job_id}: Streaming audio for review")
return FileResponse(
tmp_path,
media_type=media_type,
filename=os.path.basename(audio_gcs_path),
background=BackgroundTask(os.unlink, tmp_path),
)
transcoding = AudioTranscodingService()
signed_url = await transcoding.get_review_audio_url_async(audio_gcs_path, expiration_minutes=120)
logger.info(f"Job {job_id}: Redirecting to signed URL for audio review")
return RedirectResponse(url=signed_url, status_code=302)
except Exception as e:
logger.error(f"Job {job_id}: Error streaming audio: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Error streaming audio: {str(e)}")
logger.error(f"Job {job_id}: Error generating audio signed URL: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=f"Error serving audio: {str(e)}")
@router.post("/{job_id}/complete")
@@ -296,8 +292,9 @@ async def complete_review(
)
valid_selections = ["clean", "with_backing"]
# Also allow "custom" for user-provided instrumentals
if job.existing_instrumental_gcs_path:
# Allow "custom" for mute-region instrumentals or user-uploaded instrumentals
stems = job.file_urls.get("stems", {}) if job.file_urls else {}
if job.existing_instrumental_gcs_path or stems.get("custom_instrumental"):
valid_selections.append("custom")
if instrumental_selection not in valid_selections:
@@ -785,23 +782,29 @@ async def get_instrumental_analysis(
backing_analysis = job.state_data.get('backing_vocals_analysis', {})
# Get stem URLs
from backend.services.audio_transcoding_service import AudioTranscodingService
transcoding = AudioTranscodingService(storage_service=storage)
stems = job.file_urls.get('stems', {})
clean_url = stems.get('instrumental_clean')
backing_url = stems.get('instrumental_with_backing')
# Build audio URLs with signed access
audio_urls = {}
if clean_url:
audio_urls['clean'] = storage.generate_signed_url(clean_url, expiration_minutes=120)
if backing_url:
audio_urls['with_backing'] = storage.generate_signed_url(backing_url, expiration_minutes=120)
if job.input_media_gcs_path:
audio_urls['original'] = storage.generate_signed_url(job.input_media_gcs_path, expiration_minutes=120)
# Check for backing vocals stem for playback
backing_vocals_path = stems.get('backing_vocals')
# Build audio URLs with transcoded signed access (OGG Opus, parallel)
url_tasks = {}
if clean_url:
url_tasks['clean'] = transcoding.get_review_audio_url_async(clean_url, expiration_minutes=120)
if backing_url:
url_tasks['with_backing'] = transcoding.get_review_audio_url_async(backing_url, expiration_minutes=120)
if job.input_media_gcs_path:
url_tasks['original'] = transcoding.get_review_audio_url_async(job.input_media_gcs_path, expiration_minutes=120)
if backing_vocals_path:
audio_urls['backing_vocals'] = storage.generate_signed_url(backing_vocals_path, expiration_minutes=120)
url_tasks['backing_vocals'] = transcoding.get_review_audio_url_async(backing_vocals_path, expiration_minutes=120)
audio_urls = {}
if url_tasks:
results = await asyncio.gather(*url_tasks.values())
audio_urls = dict(zip(url_tasks.keys(), results))
# Format response to match InstrumentalAnalysis type
return {
+16 -9
View File
@@ -65,6 +65,7 @@ from backend.services.youtube_download_service import (
get_youtube_download_service,
YouTubeDownloadError,
)
from backend.exceptions import InvalidStateTransitionError
logger = logging.getLogger(__name__)
@@ -476,14 +477,12 @@ async def _handle_made_for_you_order(
"""
from backend.models.job import JobCreate, JobStatus
from backend.services.job_manager import JobManager
from backend.services.worker_service import get_worker_service
from backend.services.audio_search_service import (
get_audio_search_service,
NoResultsError,
AudioSearchError,
)
from backend.services.storage_service import StorageService
import asyncio
import tempfile
import os
@@ -501,7 +500,6 @@ async def _handle_made_for_you_order(
try:
job_manager = JobManager()
worker_service = get_worker_service()
storage_service = StorageService()
# Apply default theme (Nomad) - same as audio_search endpoint
@@ -602,13 +600,22 @@ async def _handle_made_for_you_order(
'filename': os.path.basename(audio_gcs_path),
})
logger.info(f"Job {job_id}: YouTube audio downloaded to {audio_gcs_path}, triggering workers")
logger.info(f"Job {job_id}: YouTube audio downloaded to {audio_gcs_path}")
# Now trigger both workers in parallel (audio already downloaded)
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
# Use centralized helper that handles transition + worker triggers
# This ensures consistent state machine flow and raises on invalid transitions
try:
await job_manager.start_job_processing(job_id)
except InvalidStateTransitionError as e:
logger.error(f"Job {job_id}: Failed to start processing - invalid state transition: {e}")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.FAILED,
progress=0,
message=f"Failed to start processing: {str(e)}",
raise_on_invalid=False # Don't raise if already failed
)
raise
except YouTubeDownloadError as e:
logger.error(f"Job {job_id}: YouTube download failed: {e}")
+11 -1
View File
@@ -97,7 +97,13 @@ class Settings(BaseSettings):
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
# Flacfetch client retry settings
# Retries help handle transient network failures when calling the flacfetch service
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
# Default distribution settings (can be overridden per-request)
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
@@ -110,6 +116,10 @@ class Settings(BaseSettings):
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
# Private (non-published) track distribution settings
default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")
default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")
# Rate Limiting Configuration
# Enable/disable rate limiting system-wide (useful for development)
enable_rate_limiting: bool = os.getenv("ENABLE_RATE_LIMITING", "true").lower() in ("true", "1", "yes")
+58
View File
@@ -64,3 +64,61 @@ class IPBlockedError(Exception):
self.message = message
self.ip_address = ip_address
super().__init__(message)
class InsufficientCreditsError(Exception):
"""
Raised when a user attempts to create a job without sufficient credits.
Attributes:
message: Human-readable error message
credits_available: Number of credits the user currently has
credits_required: Number of credits required (always 1 for now)
"""
def __init__(
self,
message: str,
credits_available: int = 0,
credits_required: int = 1
):
self.message = message
self.credits_available = credits_available
self.credits_required = credits_required
super().__init__(message)
class InvalidStateTransitionError(Exception):
"""
Raised when an invalid job state transition is attempted.
This exception is raised by JobManager.transition_to_state() when the
requested transition is not allowed by the state machine defined in
STATE_TRANSITIONS.
Example: Trying to transition from PENDING to GENERATING_SCREENS is invalid
because PENDING can only transition to DOWNLOADING, SEARCHING_AUDIO, FAILED,
or CANCELLED.
Attributes:
message: Human-readable error message
job_id: The job ID that failed to transition
from_status: Current status of the job
to_status: Attempted target status
valid_transitions: List of valid transitions from current status
"""
def __init__(
self,
message: str,
job_id: str = "",
from_status: str = "",
to_status: str = "",
valid_transitions: list = None
):
self.message = message
self.job_id = job_id
self.from_status = from_status
self.to_status = to_status
self.valid_transitions = valid_transitions or []
super().__init__(message)
+15 -1
View File
@@ -167,7 +167,7 @@ app.include_router(tenant.router) # Tenant/white-label configuration (no /api p
# Exception handler for rate limiting
from fastapi import Request
from fastapi.responses import JSONResponse
from backend.exceptions import RateLimitExceededError
from backend.exceptions import RateLimitExceededError, InsufficientCreditsError
@app.exception_handler(RateLimitExceededError)
@@ -187,6 +187,20 @@ async def rate_limit_exception_handler(request: Request, exc: RateLimitExceededE
)
@app.exception_handler(InsufficientCreditsError)
async def insufficient_credits_exception_handler(request: Request, exc: InsufficientCreditsError):
"""Handle insufficient credits errors with 402 Payment Required status."""
return JSONResponse(
status_code=402,
content={
"detail": exc.message,
"credits_available": exc.credits_available,
"credits_required": exc.credits_required,
"buy_url": "/#pricing",
},
)
@app.get("/")
async def root():
"""Root endpoint."""
+14 -1
View File
@@ -205,7 +205,14 @@ class Job(BaseModel):
title: Optional[str] = None
filename: Optional[str] = None # Original uploaded filename
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
# Audio source tracking (for retry capability)
audio_source_type: Optional[str] = None # Source type: 'audio_search', 'youtube_url', 'file_upload'
source_name: Optional[str] = None # Provider name (RED, OPS, YouTube, Spotify) - for audio_search
source_id: Optional[str] = None # Source-specific ID (torrent ID, video ID, track ID) - for audio_search
target_file: Optional[str] = None # For torrents, specific file to extract - for audio_search
download_url: Optional[str] = None # Direct download URL if available - for audio_search
# User preferences
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
@@ -292,6 +299,9 @@ class Job(BaseModel):
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Output deletion tracking (for admin cleanup without deleting job)
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
@@ -499,6 +509,9 @@ class JobCreate(BaseModel):
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Request metadata (set by API endpoint from request headers)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
+2 -1
View File
@@ -10,7 +10,7 @@ class URLSubmissionRequest(BaseModel):
url: HttpUrl
artist: Optional[str] = None # Auto-detected if not provided
title: Optional[str] = None # Auto-detected if not provided
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
@@ -18,6 +18,7 @@ class URLSubmissionRequest(BaseModel):
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
is_private: Optional[bool] = None # Private (non-published) track mode
class UploadSubmissionRequest(BaseModel):
+265
View File
@@ -0,0 +1,265 @@
#!/usr/bin/env python3
"""
Generate a Mermaid state diagram from the job state machine.
This script reads the STATE_TRANSITIONS dictionary from models/job.py
and generates a Mermaid diagram that can be embedded in documentation.
Usage:
python scripts/generate_state_diagram.py > docs/state-machine.mmd
# Or to update ARCHITECTURE.md directly:
python scripts/generate_state_diagram.py --update-docs
Created as part of state machine robustness improvements (2026-02-02).
"""
import sys
import argparse
from pathlib import Path
# Add backend to path for imports
backend_dir = Path(__file__).parent.parent
sys.path.insert(0, str(backend_dir))
# Import the state machine data
# Note: Must be run from an environment where karaoke_gen is installed
try:
from models.job import STATE_TRANSITIONS, JobStatus
except ImportError:
# Fallback: define the state machine inline for documentation purposes
from enum import Enum
class JobStatus(str, Enum):
PENDING = "pending"
SEARCHING_AUDIO = "searching_audio"
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection"
DOWNLOADING_AUDIO = "downloading_audio"
DOWNLOADING = "downloading"
SEPARATING_STAGE1 = "separating_stage1"
SEPARATING_STAGE2 = "separating_stage2"
AUDIO_COMPLETE = "audio_complete"
TRANSCRIBING = "transcribing"
CORRECTING = "correcting"
LYRICS_COMPLETE = "lyrics_complete"
GENERATING_SCREENS = "generating_screens"
APPLYING_PADDING = "applying_padding"
AWAITING_REVIEW = "awaiting_review"
IN_REVIEW = "in_review"
REVIEW_COMPLETE = "review_complete"
RENDERING_VIDEO = "rendering_video"
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection"
INSTRUMENTAL_SELECTED = "instrumental_selected"
GENERATING_VIDEO = "generating_video"
ENCODING = "encoding"
PACKAGING = "packaging"
UPLOADING = "uploading"
NOTIFYING = "notifying"
COMPLETE = "complete"
PREP_COMPLETE = "prep_complete"
FAILED = "failed"
CANCELLED = "cancelled"
QUEUED = "queued"
PROCESSING = "processing"
ERROR = "error"
READY_FOR_FINALIZATION = "ready_for_finalization"
FINALIZING = "finalizing"
STATE_TRANSITIONS = {
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.FAILED],
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED],
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
}
def generate_mermaid() -> str:
"""
Generate a Mermaid state diagram from STATE_TRANSITIONS.
Returns:
Mermaid diagram as a string
"""
lines = ["stateDiagram-v2"]
lines.append(" direction TB")
lines.append("")
# Group states by category for better visualization
categories = {
"Initial": [JobStatus.PENDING],
"Audio Search": [
JobStatus.SEARCHING_AUDIO,
JobStatus.AWAITING_AUDIO_SELECTION,
JobStatus.DOWNLOADING_AUDIO,
],
"Processing": [
JobStatus.DOWNLOADING,
JobStatus.SEPARATING_STAGE1,
JobStatus.SEPARATING_STAGE2,
JobStatus.AUDIO_COMPLETE,
JobStatus.TRANSCRIBING,
JobStatus.CORRECTING,
JobStatus.LYRICS_COMPLETE,
],
"Screens": [
JobStatus.GENERATING_SCREENS,
JobStatus.APPLYING_PADDING,
],
"Review": [
JobStatus.AWAITING_REVIEW,
JobStatus.IN_REVIEW,
JobStatus.REVIEW_COMPLETE,
],
"Video": [
JobStatus.RENDERING_VIDEO,
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.GENERATING_VIDEO,
JobStatus.ENCODING,
JobStatus.PACKAGING,
],
"Distribution": [
JobStatus.UPLOADING,
JobStatus.NOTIFYING,
],
"Terminal": [
JobStatus.COMPLETE,
JobStatus.PREP_COMPLETE,
JobStatus.FAILED,
JobStatus.CANCELLED,
],
}
# Add state definitions with notes for key states
lines.append(" %% State definitions")
key_states = {
JobStatus.PENDING: "Job created",
JobStatus.AWAITING_AUDIO_SELECTION: "⚠️ USER ACTION",
JobStatus.AWAITING_REVIEW: "⚠️ USER ACTION",
JobStatus.COMPLETE: "✅ Success",
JobStatus.FAILED: "❌ Error",
}
for status, note in key_states.items():
lines.append(f" {status.value}: {status.value}")
lines.append(f" note right of {status.value}: {note}")
lines.append("")
lines.append(" %% Transitions")
# Generate transitions
for from_status, to_statuses in STATE_TRANSITIONS.items():
for to_status in to_statuses:
# Skip deprecated statuses
if to_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
continue
if from_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
continue
lines.append(f" {from_status.value} --> {to_status.value}")
# Add start and end markers
lines.append("")
lines.append(" %% Start and end")
lines.append(" [*] --> pending")
lines.append(" complete --> [*]")
lines.append(" prep_complete --> [*]")
lines.append(" failed --> [*]")
lines.append(" cancelled --> [*]")
return "\n".join(lines)
def generate_markdown_section() -> str:
"""
Generate a markdown section with the state diagram.
Returns:
Markdown string with the diagram
"""
diagram = generate_mermaid()
return f"""## Job State Machine
The following diagram shows all valid state transitions for a karaoke generation job.
Key states:
- **PENDING**: Initial state when job is created
- **AWAITING_AUDIO_SELECTION**: Human selects audio source (for artist+title search)
- **AWAITING_REVIEW**: Human reviews lyrics and selects instrumental
- **COMPLETE**: All processing finished successfully
```mermaid
{diagram}
```
### Parallel Processing
Jobs use parallel processing for audio separation and lyrics transcription:
1. **DOWNLOADING** triggers both:
- Audio worker → SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE
- Lyrics worker → TRANSCRIBING → CORRECTING → LYRICS_COMPLETE
2. When BOTH complete, the screens worker is triggered automatically.
### Human Review Points
Jobs wait for human input at these points:
- **AWAITING_AUDIO_SELECTION**: User selects audio source from search results
- **AWAITING_REVIEW**: User reviews/corrects lyrics AND selects instrumental
### Error Handling
- Any state can transition to **FAILED** on unrecoverable errors
- **CANCELLED** is set when user cancels the job
"""
def main():
parser = argparse.ArgumentParser(description="Generate state machine diagram")
parser.add_argument(
"--update-docs",
action="store_true",
help="Update ARCHITECTURE.md with the diagram"
)
parser.add_argument(
"--markdown",
action="store_true",
help="Output markdown section instead of raw Mermaid"
)
args = parser.parse_args()
if args.update_docs:
# TODO: Implement auto-update of ARCHITECTURE.md
print("Auto-update of ARCHITECTURE.md not yet implemented")
print("Please copy the output below to the appropriate section:")
print()
print(generate_markdown_section())
elif args.markdown:
print(generate_markdown_section())
else:
print(generate_mermaid())
if __name__ == "__main__":
main()
@@ -0,0 +1,181 @@
"""
Audio transcoding service for review UI optimization.
Transcodes FLAC audio files to OGG Opus for efficient browser playback.
Original FLAC files are never modified — transcoded versions are cached
in GCS at jobs/{job_id}/review-audio/{filename}.ogg.
Eager transcoding: called from screens_worker before AWAITING_REVIEW
so all files are cached before the user opens the review UI.
Falls back to original FLAC signed URLs if transcoded version is missing.
"""
import asyncio
import hashlib
import logging
import os
import subprocess
import tempfile
from pathlib import Path
from typing import Optional
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
class AudioTranscodingService:
"""
Transcodes audio files to OGG Opus and caches them in GCS.
Used by the review UI to serve ~3 MB OGG files instead of 35+ MB FLACs.
"""
def __init__(self, storage_service: Optional[StorageService] = None):
self.storage = storage_service or StorageService()
def _get_cache_path(self, source_gcs_path: str) -> str:
"""
Derive the cache path for a transcoded file.
Examples:
jobs/abc123/input/song.flac -> jobs/abc123/review-audio/song.ogg
jobs/abc123/stems/instrumental_clean.flac -> jobs/abc123/review-audio/instrumental_clean.ogg
"""
parts = Path(source_gcs_path).parts
# Find "jobs/{job_id}" prefix
try:
jobs_idx = list(parts).index("jobs")
job_id = parts[jobs_idx + 1]
except (ValueError, IndexError):
# Fallback: use a hash of the full path to avoid collisions
job_id = "_" + hashlib.sha256(source_gcs_path.encode()).hexdigest()[:12]
filename = Path(source_gcs_path).stem + ".ogg"
return f"jobs/{job_id}/review-audio/{filename}"
def _transcode_and_upload(self, source_gcs_path: str, cache_path: str) -> str:
"""
Download FLAC from GCS, transcode to OGG Opus, upload to cache path.
Returns the cache GCS path on success.
Raises on failure.
"""
with tempfile.TemporaryDirectory() as temp_dir:
local_input = os.path.join(temp_dir, "input.flac")
local_output = os.path.join(temp_dir, "output.ogg")
# Download source
self.storage.download_file(source_gcs_path, local_input)
# Transcode: FLAC -> OGG Opus 128kbps
cmd = [
"ffmpeg", "-hide_banner", "-loglevel", "error",
"-y",
"-i", local_input,
"-c:a", "libopus",
"-b:a", "128k",
"-vn",
local_output,
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
if result.returncode != 0:
raise RuntimeError(
f"ffmpeg transcoding failed (exit {result.returncode}): {result.stderr}"
)
# Upload transcoded file
self.storage.upload_file(local_output, cache_path)
return cache_path
def transcode_if_needed(self, source_gcs_path: str) -> str:
"""
Return the GCS path of the transcoded OGG file, transcoding if not cached.
Returns the cache path (always an OGG file in review-audio/).
Raises if transcoding fails.
"""
cache_path = self._get_cache_path(source_gcs_path)
if self.storage.file_exists(cache_path):
logger.debug(f"Cache hit: {cache_path}")
return cache_path
logger.info(f"Cache miss, transcoding: {source_gcs_path} -> {cache_path}")
return self._transcode_and_upload(source_gcs_path, cache_path)
def get_review_audio_url(
self, source_gcs_path: str, expiration_minutes: int = 120
) -> str:
"""
Get a signed URL for the transcoded audio, falling back to original FLAC.
"""
try:
cache_path = self.transcode_if_needed(source_gcs_path)
return self.storage.generate_signed_url(cache_path, expiration_minutes)
except Exception as e:
# Source may be deleted but cache from earlier transcode persists
try:
cache_path = self._get_cache_path(source_gcs_path)
if self.storage.file_exists(cache_path):
logger.info(f"Source gone but cache exists, serving: {cache_path}")
return self.storage.generate_signed_url(cache_path, expiration_minutes)
except Exception:
pass # Fall through to FLAC fallback below
logger.warning(
f"Transcoding failed for {source_gcs_path}, falling back to FLAC: {e}"
)
return self.storage.generate_signed_url(source_gcs_path, expiration_minutes)
async def get_review_audio_url_async(
self, source_gcs_path: str, expiration_minutes: int = 120
) -> str:
"""Async wrapper around get_review_audio_url via asyncio.to_thread."""
return await asyncio.to_thread(
self.get_review_audio_url, source_gcs_path, expiration_minutes
)
def prepare_review_audio_for_job(self, job) -> list[str]:
"""
Transcode all review audio files for a job (eager transcoding).
Called from screens_worker before AWAITING_REVIEW transition
and from the admin backfill endpoint.
Returns list of cache paths that were transcoded.
"""
job_id = job.job_id
transcoded = []
# Collect all audio paths to transcode
paths_to_transcode = []
# Main input audio
if job.input_media_gcs_path:
paths_to_transcode.append(("input", job.input_media_gcs_path))
# Stems
stems = job.file_urls.get("stems", {})
for stem_key in ["instrumental_clean", "instrumental_with_backing", "backing_vocals"]:
path = stems.get(stem_key)
if path:
paths_to_transcode.append((stem_key, path))
for label, source_path in paths_to_transcode:
try:
cache_path = self.transcode_if_needed(source_path)
transcoded.append(cache_path)
logger.info(f"[{job_id}] Transcoded {label}: {cache_path}")
except Exception as e:
# Non-fatal: review UI will fall back to FLAC
logger.warning(
f"[{job_id}] Failed to transcode {label} ({source_path}): {e}"
)
logger.info(
f"[{job_id}] Review audio preparation complete: "
f"{len(transcoded)}/{len(paths_to_transcode)} files transcoded"
)
return transcoded
+7 -27
View File
@@ -437,10 +437,11 @@ class AuthService:
def increment_token_usage(self, token: str, job_id: str) -> bool:
"""
Increment usage count for a token and track the job.
Increment usage count for an auth_token and track the job.
For session tokens (magic link auth), this deducts a credit from the user.
For auth_tokens, this increments the usage count.
Note: Credit deduction for session-based users is handled in
JobManager.create_job(), not here. This method only tracks usage
for legacy auth_tokens stored in Firestore.
Args:
token: The access token
@@ -465,32 +466,11 @@ class AuthService:
if token in self.admin_tokens:
return True
# Check if this is a session token (STRIPE type from validate_token means it's a session)
# Session tokens are not in auth_tokens collection
# Only increment usage for auth_tokens in Firestore
# Session tokens (magic link auth) have credits managed by JobManager.create_job()
token_data = self.firestore.get_token(token)
if not token_data:
# This is a session token - deduct credit from user
from backend.services.user_service import get_user_service
user_service = get_user_service()
# Get the user email from the session
is_valid, user, _ = user_service.validate_session(token)
if not is_valid or not user:
logger.error(f"Session token validation failed during usage increment")
return False
# Deduct one credit
success, new_balance, deduct_message = user_service.deduct_credit(
user.email, job_id, reason="job_creation"
)
if success:
logger.info(f"Deducted credit for user {user.email} (remaining: {new_balance})")
return True
else:
logger.error(f"Failed to deduct credit for user {user.email}: {deduct_message}")
return False
return True
# Regular auth_token - increment usage in Firestore
try:
+225
View File
@@ -0,0 +1,225 @@
"""
Brand code allocation service using Firestore atomic counters.
Replaces the previous Dropbox folder-scanning approach which was susceptible to
TOCTOU race conditions when concurrent jobs read the same folder state.
Uses Firestore transactions to atomically allocate unique brand codes.
Supports recycling numbers when E2E cleanup deletes test brand codes.
"""
import logging
import re
from datetime import datetime, timezone
from typing import Optional
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
BRAND_CODE_COUNTERS_COLLECTION = "brand_code_counters"
# Singleton instance
_brand_code_service: Optional["BrandCodeService"] = None
class BrandCodeService:
"""Atomic brand code allocation using Firestore counters."""
def __init__(self, db: Optional[firestore.Client] = None):
if db is None:
self.db = firestore.Client(project=settings.google_cloud_project)
else:
self.db = db
def allocate_brand_code(self, prefix: str, dropbox_path: str) -> str:
"""
Atomically allocate the next brand code for a given prefix.
Uses a Firestore transaction to ensure no two concurrent callers
receive the same number. If recycled numbers are available (from
E2E cleanup), the smallest recycled number is reused first.
On first use for a prefix, initializes the counter by scanning
Dropbox to find the current max number.
Args:
prefix: Brand prefix (e.g., "NOMAD" or "NOMADNP")
dropbox_path: Dropbox path for one-time initialization scan
Returns:
Brand code string (e.g., "NOMAD-1264")
"""
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
@firestore.transactional
def allocate_in_transaction(transaction):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
# First use — initialize from Dropbox scan
next_number = self._get_initial_next_number(prefix, dropbox_path)
allocated = next_number
transaction.set(doc_ref, {
"prefix": prefix,
"next_number": next_number + 1,
"recycled": [],
"initialized_at": datetime.now(timezone.utc),
"last_updated": datetime.now(timezone.utc),
})
return allocated
data = doc.to_dict()
recycled = data.get("recycled", [])
if recycled:
# Reuse smallest recycled number
recycled.sort()
allocated = recycled.pop(0)
transaction.update(doc_ref, {
"recycled": recycled,
"last_updated": datetime.now(timezone.utc),
})
else:
# Allocate next sequential number
allocated = data["next_number"]
transaction.update(doc_ref, {
"next_number": allocated + 1,
"last_updated": datetime.now(timezone.utc),
})
return allocated
transaction = self.db.transaction()
number = allocate_in_transaction(transaction)
brand_code = f"{prefix}-{number:04d}"
logger.info(f"Allocated brand code: {brand_code}")
return brand_code
def recycle_brand_code(self, prefix: str, number: int) -> None:
"""
Return a brand code number to the recycled pool.
Called when E2E cleanup deletes a brand code's distribution.
The number will be reused on the next allocation for this prefix.
Args:
prefix: Brand prefix (e.g., "NOMAD")
number: The numeric portion to recycle (e.g., 1234)
"""
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
@firestore.transactional
def recycle_in_transaction(transaction):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
# Counter doesn't exist yet — nothing to recycle into.
# The number will naturally be below next_number when
# the counter is initialized, so it would create a gap.
# Just log and skip.
logger.warning(
f"Cannot recycle {prefix}-{number:04d}: counter doc doesn't exist"
)
return
data = doc.to_dict()
next_number = data.get("next_number", 0)
if number <= 0 or number >= next_number:
logger.warning(
f"Cannot recycle {prefix}-{number:04d}: "
f"invalid (next_number={next_number})"
)
return
recycled = data.get("recycled", [])
if number not in recycled:
recycled.append(number)
transaction.update(doc_ref, {
"recycled": recycled,
"last_updated": datetime.now(timezone.utc),
})
logger.info(f"Recycled brand code number: {prefix}-{number:04d}")
else:
logger.warning(f"Number {number} already in recycled pool for {prefix}")
transaction = self.db.transaction()
recycle_in_transaction(transaction)
def _get_initial_next_number(self, prefix: str, dropbox_path: str) -> int:
"""
Scan Dropbox once to determine the starting next_number.
This is only called when the counter document doesn't exist yet
(one-time migration from the old Dropbox-scanning approach).
Args:
prefix: Brand prefix to scan for
dropbox_path: Dropbox path containing organized folders
Returns:
The next number to allocate (max existing + 1, minimum 1)
"""
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if not dropbox.is_configured:
logger.warning(
f"Dropbox not configured for initialization of {prefix} counter, starting at 1"
)
return 1
folders = dropbox.list_folders(dropbox_path)
pattern = re.compile(rf"^{re.escape(prefix)}-(\d{{4}})")
existing_nums: set[int] = set()
for folder in folders:
match = pattern.match(folder)
if match:
existing_nums.add(int(match.group(1)))
if existing_nums:
next_num = max(existing_nums) + 1
logger.info(
f"Initialized {prefix} counter from Dropbox: "
f"max existing={max(existing_nums)}, next_number={next_num}"
)
return next_num
logger.info(f"No existing {prefix} codes found in Dropbox, starting at 1")
return 1
except Exception as e:
logger.error(f"Failed to initialize {prefix} counter from Dropbox: {e}")
raise
@staticmethod
def parse_brand_code(brand_code: str) -> tuple[str, int]:
"""
Parse a brand code string into prefix and number.
Args:
brand_code: e.g., "NOMAD-1234" or "NOMADNP-0012"
Returns:
Tuple of (prefix, number), e.g., ("NOMAD", 1234)
Raises:
ValueError: If brand_code doesn't match expected format
"""
match = re.match(r"^([A-Z]+)-(\d+)$", brand_code)
if not match:
raise ValueError(f"Invalid brand code format: {brand_code}")
return match.group(1), int(match.group(2))
def get_brand_code_service() -> BrandCodeService:
"""Get or create the singleton BrandCodeService instance."""
global _brand_code_service
if _brand_code_service is None:
_brand_code_service = BrandCodeService()
return _brand_code_service
+4
View File
@@ -111,6 +111,10 @@ class DropboxService:
"""
Calculate next brand code from existing folders, filling gaps in the sequence.
NOTE: This method is only used for one-time initialization of the Firestore
counter (see brand_code_service.py). All runtime brand code allocation now
goes through BrandCodeService.allocate_brand_code() for atomic safety.
Scans the folder for existing brand codes like "NOMAD-0001" and
returns the first available code, filling any gaps left by deleted outputs.
+76
View File
@@ -173,6 +173,82 @@ class FirestoreService:
logger.error(f"Error listing jobs: {e}")
raise
# Fields needed by the dashboard summary view
SUMMARY_FIELD_PATHS = [
'job_id', 'status', 'progress', 'created_at', 'artist', 'title',
'error_message', 'non_interactive', 'outputs_deleted_at', 'user_email',
'state_data.brand_code', 'state_data.youtube_url', 'state_data.dropbox_link',
'state_data.audio_progress', 'state_data.lyrics_progress',
'state_data.audio_complete', 'state_data.lyrics_complete',
'state_data.backing_vocals_analysis',
'file_urls.finals', 'file_urls.videos', 'file_urls.packages',
]
def list_jobs_summary(
self,
status: Optional[JobStatus] = None,
exclude_statuses: Optional[List[str]] = None,
environment: Optional[str] = None,
client_id: Optional[str] = None,
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
user_email: Optional[str] = None,
tenant_id: Optional[str] = None,
limit: int = 100,
) -> List[Dict[str, Any]]:
"""
List jobs with Firestore field projection for reduced payload.
Uses ``query.select()`` so Firestore only reads the requested fields,
drastically cutting bandwidth and deserialization cost.
Returns raw dicts (not Job models) because projected documents would
fail Pydantic validation for missing required fields.
"""
try:
query = self.db.collection(self.collection)
if status:
query = query.where(filter=FieldFilter('status', '==', status.value))
# NOTE: exclude_statuses is filtered in Python after the query
# because Firestore's not-in filter combined with order_by(created_at)
# requires a composite index. Since documents are tiny after select(),
# Python filtering is fast and avoids the index requirement.
if environment:
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
if client_id:
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
if user_email:
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
if tenant_id:
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
if created_after:
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
if created_before:
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
query = query.select(self.SUMMARY_FIELD_PATHS)
docs = query.stream()
results = [doc.to_dict() for doc in docs]
# Apply exclude_statuses filter in Python (see note above)
if exclude_statuses:
results = [r for r in results if r.get('status') not in exclude_statuses]
return results
except Exception as e:
logger.error(f"Error listing jobs summary: {e}")
raise
def delete_jobs_by_filter(
self,
environment: Optional[str] = None,
+157 -55
View File
@@ -34,6 +34,8 @@ from typing import Any, Dict, List, Optional
import httpx
from backend.config import get_settings
logger = logging.getLogger(__name__)
@@ -42,6 +44,77 @@ class FlacfetchServiceError(Exception):
pass
def should_retry_flacfetch_error(exception: Exception) -> bool:
"""
Determine if a flacfetch exception should be retried.
Retry on:
- httpx.RequestError (network failures: timeout, connection refused, etc.)
- httpx.HTTPStatusError with 5xx status (server errors)
Don't retry on:
- httpx.HTTPStatusError with 4xx status (client errors: bad request, auth, not found)
"""
if isinstance(exception, httpx.RequestError):
# Network-level failures - always retry
return True
if isinstance(exception, httpx.HTTPStatusError):
# HTTP errors - only retry on 5xx (server errors)
return exception.response.status_code >= 500
# Don't retry on other exceptions (e.g., FlacfetchServiceError after wrapping)
return False
async def with_retry(func, *args, **kwargs):
"""
Execute an async function with retry logic using exponential backoff.
This implementation uses manual retry logic instead of tenacity decorators
for better testability and runtime configuration.
"""
settings = get_settings()
max_attempts = settings.flacfetch_retry_max_attempts
min_wait = settings.flacfetch_retry_min_wait
max_wait = settings.flacfetch_retry_max_wait
last_exception = None
for attempt in range(1, max_attempts + 1):
try:
return await func(*args, **kwargs)
except Exception as e:
last_exception = e
# Check if we should retry this exception
if not should_retry_flacfetch_error(e):
# Don't retry - raise immediately
raise
# Check if we have more attempts left
if attempt >= max_attempts:
# No more attempts - raise the last exception
logger.warning(
f"Flacfetch request failed after {max_attempts} attempts: {e}"
)
raise
# Calculate wait time with exponential backoff
wait_time = min(min_wait * (2 ** (attempt - 1)), max_wait)
logger.warning(
f"Flacfetch request failed (attempt {attempt}/{max_attempts}): {e}. "
f"Retrying in {wait_time:.1f}s..."
)
# Wait before next attempt
await asyncio.sleep(wait_time)
# Should never reach here, but just in case
raise last_exception
class FlacfetchClient:
"""
Client for remote flacfetch HTTP API.
@@ -148,7 +221,7 @@ class FlacfetchClient:
raise FlacfetchServiceError(f"Search request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
async def download(
self,
search_id: str,
@@ -158,47 +231,59 @@ class FlacfetchClient:
) -> str:
"""
Start downloading an audio file.
Retries automatically on network errors and server errors (5xx) using
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
Args:
search_id: Search ID from previous search
result_index: Index of result to download
output_filename: Optional custom filename (without extension)
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
Returns:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure
FlacfetchServiceError: On download start failure (after all retries exhausted)
"""
try:
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
return await with_retry(self._download_impl, search_id, result_index, output_filename, gcs_path)
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
async def _download_impl(
self,
search_id: str,
result_index: int,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""Implementation of download without error wrapping - lets httpx exceptions bubble up."""
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
async def download_by_id(
self,
source_name: str,
@@ -214,6 +299,9 @@ class FlacfetchClient:
This is useful when you have stored the source_id from a previous search
and want to download later without re-searching.
Retries automatically on network errors and server errors (5xx) using
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
Args:
source_name: Provider name (RED, OPS, YouTube, Spotify)
source_id: Source-specific ID (torrent ID, video ID, track ID)
@@ -226,40 +314,54 @@ class FlacfetchClient:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure
FlacfetchServiceError: On download start failure (after all retries exhausted)
"""
try:
async with httpx.AsyncClient() as client:
payload = {
"source_name": source_name,
"source_id": source_id,
}
if output_filename:
payload["output_filename"] = output_filename
if target_file:
payload["target_file"] = target_file
if download_url:
payload["download_url"] = download_url
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download-by-id",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
return await with_retry(
self._download_by_id_impl,
source_name, source_id, output_filename, target_file, download_url, gcs_path
)
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
async def _download_by_id_impl(
self,
source_name: str,
source_id: str,
output_filename: Optional[str] = None,
target_file: Optional[str] = None,
download_url: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""Implementation of download_by_id without error wrapping - lets httpx exceptions bubble up."""
async with httpx.AsyncClient() as client:
payload = {
"source_name": source_name,
"source_id": source_id,
}
if output_filename:
payload["output_filename"] = output_filename
if target_file:
payload["target_file"] = target_file
if download_url:
payload["download_url"] = download_url
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download-by-id",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
"""
Get the current status of a download.
+23 -3
View File
@@ -3,6 +3,7 @@ import glob as glob_module
import json
import logging
import os
import re
import subprocess
import sys
import tempfile
@@ -13,6 +14,7 @@ from typing import Optional
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
from google.cloud import storage
from packaging.version import Version
from pydantic import BaseModel
logging.basicConfig(level=logging.INFO)
@@ -254,12 +256,23 @@ def ensure_latest_wheel():
# Find the downloaded wheel (get the latest by version sorting)
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
# Filter out karaoke_gen-current.whl (not a valid PEP 427 wheel name)
wheels = [w for w in wheels if '-current' not in w]
if not wheels:
logger.warning("No wheel found in GCS, using fallback encoding logic")
return False
# Sort to get latest version
wheel_path = sorted(wheels)[-1]
# Sort to get latest version (by semantic version, not alphabetically)
def extract_version(wheel_path):
"""Extract version from wheel filename like karaoke_gen-0.116.1-py3-none-any.whl"""
match = re.search(r'karaoke_gen-([0-9.]+)-', wheel_path)
if match:
return Version(match.group(1))
return Version("0.0.0") # Fallback for unparseable filenames
wheels.sort(key=extract_version, reverse=True)
wheel_path = wheels[0]
logger.info(f"Installing wheel: {wheel_path}")
# Install (or upgrade) the wheel
@@ -359,7 +372,14 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
instrumental_selection = config.get("instrumental_selection", "clean")
logger.info(f"Instrumental selection from config: {instrumental_selection}")
if instrumental_selection == "with_backing":
if instrumental_selection == "custom":
# User uploaded a custom instrumental or created one via mute-region editing
instrumental = find_file(
work_dir,
"*custom_instrumental*.flac", "*Instrumental Custom*.flac",
"*custom_instrumental*.mp3",
)
elif instrumental_selection == "with_backing":
# User selected instrumental with backing vocals
instrumental = find_file(
work_dir,
+39
View File
@@ -61,6 +61,45 @@ def get_effective_distribution_settings(
)
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
"""
Get effective distribution settings for a job, applying private overrides if is_private=True.
For private (non-published) jobs, this overrides:
- dropbox_path → private Dropbox path (Tracks-NonPublished)
- brand_prefix → private brand prefix (NOMADNP)
- enable_youtube_upload → False (no YouTube for private)
- gdrive_folder_id → None (no Google Drive for private)
For non-private jobs, returns the job's own distribution settings unchanged.
Args:
job: Job object (or any object with distribution fields)
Returns:
EffectiveDistributionSettings with private overrides applied if needed
"""
if getattr(job, 'is_private', False):
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=settings.default_private_dropbox_path,
brand_prefix=settings.default_private_brand_prefix,
enable_youtube_upload=False,
gdrive_folder_id=None,
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
# Non-private: return job's own settings
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=getattr(job, 'youtube_description_template', None),
)
def resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool] = None,
+221
View File
@@ -0,0 +1,221 @@
"""
Job health and consistency checking service.
Provides functions to detect state inconsistencies in jobs that might indicate
bugs in the state machine or trigger logic.
Created as part of state machine robustness improvements (2026-02-02).
"""
import logging
from typing import List, Dict, Any, Optional
from backend.models.job import Job, JobStatus
logger = logging.getLogger(__name__)
# Jobs in these statuses should NOT have audio_complete flag set
# (unless they're actively processing or beyond)
STATUSES_BEFORE_AUDIO_START = {
JobStatus.PENDING,
JobStatus.SEARCHING_AUDIO,
JobStatus.AWAITING_AUDIO_SELECTION,
}
# Jobs in these statuses should NOT have lyrics_complete flag set
STATUSES_BEFORE_LYRICS_START = {
JobStatus.PENDING,
JobStatus.SEARCHING_AUDIO,
JobStatus.AWAITING_AUDIO_SELECTION,
}
# Statuses where screens_progress.stage='complete' is valid
STATUSES_AFTER_SCREENS_COMPLETE = {
JobStatus.AWAITING_REVIEW,
JobStatus.IN_REVIEW,
JobStatus.REVIEW_COMPLETE,
JobStatus.RENDERING_VIDEO,
JobStatus.AWAITING_INSTRUMENTAL_SELECTION, # Legacy
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.GENERATING_VIDEO,
JobStatus.ENCODING,
JobStatus.PACKAGING,
JobStatus.UPLOADING,
JobStatus.NOTIFYING,
JobStatus.COMPLETE,
JobStatus.PREP_COMPLETE,
}
def check_job_consistency(job: Job) -> List[str]:
"""
Check for state inconsistencies in a job.
This function detects situations where state_data flags indicate
progress that doesn't match the job status - which usually indicates
a bug in the state machine or trigger logic.
Args:
job: Job object to check
Returns:
List of issue descriptions found. Empty list means no issues.
"""
issues = []
state_data = job.state_data or {}
# Normalize status to JobStatus enum if it's a string
status = job.status
if isinstance(status, str):
try:
status = JobStatus(status)
except ValueError:
issues.append(f"invalid_status: '{job.status}' is not a valid JobStatus")
return issues
# Check: audio_complete=True but status still in pre-audio states
if state_data.get('audio_complete') and status in STATUSES_BEFORE_AUDIO_START:
issues.append(
f"audio_complete_status_mismatch: audio_complete=True but status={status.value}"
)
# Check: lyrics_complete=True but status still in pre-lyrics states
if state_data.get('lyrics_complete') and status in STATUSES_BEFORE_LYRICS_START:
issues.append(
f"lyrics_complete_status_mismatch: lyrics_complete=True but status={status.value}"
)
# Check: screens_progress.stage='complete' but status not in post-screens states
screens_progress = state_data.get('screens_progress', {})
if isinstance(screens_progress, dict) and screens_progress.get('stage') == 'complete':
if status not in STATUSES_AFTER_SCREENS_COMPLETE:
issues.append(
f"screens_complete_status_mismatch: screens_progress.stage=complete but status={status.value}"
)
# Check: job has input_media_gcs_path but is stuck at pending
# This was the exact bug that got jobs 06cfea29 and 984da08b stuck
if job.input_media_gcs_path and status == JobStatus.PENDING:
# This could be legitimate if job was just created and workers haven't started
# But if it's been more than a few minutes, it's likely stuck
# We flag it as a warning, not an error
issues.append(
f"pending_with_input_media: has input_media_gcs_path but status=pending (may be stuck)"
)
return issues
def check_job_consistency_detailed(job: Job) -> Dict[str, Any]:
"""
Check job consistency with detailed information for debugging.
Returns a dictionary with:
- job_id: The job ID
- status: Current job status
- issues: List of issue descriptions
- state_data_summary: Summary of relevant state_data fields
- is_healthy: Boolean indicating if no issues were found
Args:
job: Job object to check
Returns:
Dictionary with detailed consistency information
"""
issues = check_job_consistency(job)
state_data = job.state_data or {}
return {
"job_id": job.job_id,
"status": job.status.value if hasattr(job.status, 'value') else str(job.status),
"issues": issues,
"state_data_summary": {
"audio_complete": state_data.get('audio_complete', False),
"lyrics_complete": state_data.get('lyrics_complete', False),
"screens_progress_stage": state_data.get('screens_progress', {}).get('stage'),
"has_input_media_gcs_path": bool(job.input_media_gcs_path),
},
"is_healthy": len(issues) == 0,
}
def get_worker_valid_statuses() -> Dict[str, List[JobStatus]]:
"""
Get mapping of worker names to their valid job statuses.
This is used by workers to validate they should run for a given job.
Returns:
Dictionary mapping worker name to list of valid JobStatus values
"""
return {
"audio_worker": [
JobStatus.DOWNLOADING,
JobStatus.DOWNLOADING_AUDIO,
JobStatus.SEPARATING_STAGE1,
JobStatus.SEPARATING_STAGE2,
],
"lyrics_worker": [
JobStatus.DOWNLOADING,
JobStatus.TRANSCRIBING,
JobStatus.CORRECTING,
],
"screens_worker": [
# Screens worker can be triggered when both audio+lyrics complete
# which can happen in any of these states
JobStatus.DOWNLOADING,
JobStatus.AUDIO_COMPLETE,
JobStatus.LYRICS_COMPLETE,
# Also valid if re-running after a failure
JobStatus.GENERATING_SCREENS,
],
"video_worker": [
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.REVIEW_COMPLETE,
JobStatus.RENDERING_VIDEO,
JobStatus.GENERATING_VIDEO,
],
"render_video_worker": [
JobStatus.REVIEW_COMPLETE,
JobStatus.RENDERING_VIDEO,
],
}
def validate_worker_can_run(worker_name: str, job: Job) -> Optional[str]:
"""
Validate that a worker should run for the given job.
Args:
worker_name: Name of the worker (e.g., "screens_worker")
job: Job object to validate
Returns:
None if valid, or an error message string if invalid
"""
valid_statuses_map = get_worker_valid_statuses()
if worker_name not in valid_statuses_map:
# Unknown worker, can't validate
return None
valid_statuses = valid_statuses_map[worker_name]
# Normalize status
status = job.status
if isinstance(status, str):
try:
status = JobStatus(status)
except ValueError:
return f"Invalid job status: {job.status}"
if status not in valid_statuses:
valid_names = [s.value for s in valid_statuses]
return (
f"{worker_name} called but job status is {status.value}, "
f"expected one of {valid_names}. This may indicate a bug in the trigger logic."
)
return None
+195 -18
View File
@@ -14,7 +14,7 @@ from datetime import datetime
from typing import Optional, Dict, Any, List
from backend.config import settings
from backend.exceptions import RateLimitExceededError
from backend.exceptions import RateLimitExceededError, InvalidStateTransitionError, InsufficientCreditsError
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
from backend.models.worker_log import WorkerLogEntry
from backend.services.firestore_service import FirestoreService
@@ -80,6 +80,18 @@ class JobManager:
limit_value=settings.rate_limit_jobs_per_day
)
# Check credits (skip for admins)
if job_create.user_email and not is_admin:
from backend.services.user_service import get_user_service
user_service = get_user_service()
if not user_service.has_credits(job_create.user_email):
credits_available = user_service.check_credits(job_create.user_email)
raise InsufficientCreditsError(
message="You're out of credits. Buy more to continue creating karaoke videos.",
credits_available=credits_available,
credits_required=1,
)
# Enforce theme requirement - all jobs must have a theme
# This prevents unstyled videos from ever being generated
if not job_create.theme_id:
@@ -130,11 +142,33 @@ class JobManager:
made_for_you=job_create.made_for_you,
customer_email=job_create.customer_email,
customer_notes=job_create.customer_notes,
# Private (non-published) track mode
is_private=job_create.is_private,
)
self.firestore.create_job(job)
logger.info(f"Created new job {job_id} with status PENDING")
# Deduct credit atomically (after job is persisted so we have job_id for transaction record)
if job_create.user_email and not is_admin:
from backend.services.user_service import get_user_service
user_service = get_user_service()
success, _remaining, deduct_msg = user_service.deduct_credit(
job_create.user_email, job_id, reason="job_creation"
)
if not success:
# Deduction failed (e.g., race condition) - delete the job and raise error
logger.error(f"Credit deduction failed for job {job_id}: {deduct_msg}")
try:
self.firestore.delete_job(job_id)
except Exception:
logger.exception(f"Failed to delete job {job_id} after credit deduction failure")
raise InsufficientCreditsError(
message="Failed to deduct credit. Please try again.",
credits_available=0,
credits_required=1,
)
# Record job creation for rate limiting (after successful persistence)
if job_create.user_email:
try:
@@ -210,6 +244,31 @@ class JobManager:
limit=limit
)
def list_jobs_summary(
self,
status: Optional[JobStatus] = None,
exclude_statuses: Optional[List[str]] = None,
environment: Optional[str] = None,
client_id: Optional[str] = None,
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
user_email: Optional[str] = None,
tenant_id: Optional[str] = None,
limit: int = 100,
) -> List[Dict[str, Any]]:
"""List jobs with field projection (summary mode)."""
return self.firestore.list_jobs_summary(
status=status,
exclude_statuses=exclude_statuses,
environment=environment,
client_id=client_id,
created_after=created_after,
created_before=created_before,
user_email=user_email,
tenant_id=tenant_id,
limit=limit,
)
def delete_jobs_by_filter(
self,
environment: Optional[str] = None,
@@ -316,53 +375,87 @@ class JobManager:
self.firestore.delete_job(job_id)
logger.info(f"Deleted job {job_id}")
def validate_state_transition(self, job_id: str, new_status: JobStatus) -> bool:
def validate_state_transition(
self,
job_id: str,
new_status: JobStatus,
raise_on_invalid: bool = True
) -> bool:
"""
Validate that a state transition is legal.
Args:
job_id: Job ID
new_status: Target state
raise_on_invalid: If True, raise InvalidStateTransitionError on invalid
transitions. If False, return False (legacy behavior).
Returns:
True if transition is valid, False otherwise
True if transition is valid
Raises:
InvalidStateTransitionError: If transition is invalid and raise_on_invalid=True
ValueError: If job not found and raise_on_invalid=True
"""
job = self.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
error_msg = f"Job {job_id} not found"
logger.error(error_msg)
if raise_on_invalid:
raise ValueError(error_msg)
return False
current_status = job.status
valid_transitions = STATE_TRANSITIONS.get(current_status, [])
if new_status not in valid_transitions:
logger.error(
error_msg = (
f"Invalid state transition for job {job_id}: "
f"{current_status} -> {new_status}. "
f"Valid transitions: {valid_transitions}"
f"Valid transitions: {[s.value for s in valid_transitions]}"
)
logger.error(error_msg)
if raise_on_invalid:
raise InvalidStateTransitionError(
message=error_msg,
job_id=job_id,
from_status=current_status,
to_status=new_status.value if hasattr(new_status, 'value') else str(new_status),
valid_transitions=[s.value for s in valid_transitions]
)
return False
return True
def transition_to_state(
self,
job_id: str,
new_status: JobStatus,
progress: Optional[int] = None,
message: Optional[str] = None,
state_data_updates: Optional[Dict[str, Any]] = None
state_data_updates: Optional[Dict[str, Any]] = None,
raise_on_invalid: bool = True
) -> bool:
"""
Transition job to new state with validation.
Args:
job_id: Job ID
new_status: Target state
progress: Progress percentage (0-100)
message: Timeline message
state_data_updates: Updates to state_data field
raise_on_invalid: If True (default), raise InvalidStateTransitionError
on invalid transitions. If False, return False silently.
Returns:
True if transition succeeded, False otherwise
True if transition succeeded, False if failed (only when raise_on_invalid=False)
Raises:
InvalidStateTransitionError: If transition is invalid and raise_on_invalid=True
"""
if not self.validate_state_transition(job_id, new_status):
if not self.validate_state_transition(job_id, new_status, raise_on_invalid=raise_on_invalid):
return False
updates = {
@@ -498,6 +591,7 @@ class JobManager:
youtube_url=youtube_url,
dropbox_url=dropbox_url,
brand_code=brand_code,
is_private=getattr(job, 'is_private', False),
)
# Try to get existing event loop, create new one if none exists
@@ -721,11 +815,32 @@ class JobManager:
)
logger.error(f"Job {job_id} failed: {error_message}")
# Auto-refund credit on failure (skip for admin-owned jobs)
try:
job = self.get_job(job_id)
if job and job.user_email:
from backend.services.auth_service import is_admin_email
from backend.services.user_service import get_user_service
if not is_admin_email(job.user_email):
user_service = get_user_service()
refund_ok, new_balance, refund_msg = user_service.refund_credit(
job.user_email, job_id, reason="job_failed"
)
if refund_ok:
logger.info(f"Refunded credit for failed job {job_id} to {_mask_email(job.user_email)} (balance: {new_balance})")
else:
logger.warning(f"Credit refund failed for job {job_id}: {refund_msg}")
except Exception as refund_err:
# Don't fail the fail_job call if refund fails
logger.warning(f"Error refunding credit for job {job_id}: {refund_err}")
return True
except Exception as e:
logger.error(f"Error marking job {job_id} as failed: {e}")
return False
def update_file_url(self, job_id: str, category: str, file_type: str, url: str) -> None:
"""
Update a file URL in the job's file_urls structure.
@@ -768,7 +883,69 @@ class JobManager:
lyrics_complete = job.state_data.get('lyrics_complete', False)
return audio_complete and lyrics_complete
async def start_job_processing(
self,
job_id: str,
progress: int = 15,
message: str = "Starting audio and lyrics processing"
) -> None:
"""
Transition job to DOWNLOADING and trigger audio + lyrics workers.
This is the single entry point for starting job processing after audio
is available. All handlers (file_upload, webhook, audio_search) should
use this method to ensure consistent state transitions.
The method:
1. Validates job exists and has input_media_gcs_path
2. Transitions to DOWNLOADING status (raises if invalid)
3. Triggers audio and lyrics workers in parallel
Args:
job_id: Job ID to start processing
progress: Progress percentage to set (default 15)
message: Timeline message (default "Starting audio and lyrics processing")
Raises:
InvalidStateTransitionError: If job can't transition to DOWNLOADING
ValueError: If job not found or missing input_media_gcs_path
Example:
# After downloading audio to GCS:
job_manager.update_job(job_id, {'input_media_gcs_path': audio_path})
await job_manager.start_job_processing(job_id)
"""
import asyncio
from backend.services.worker_service import get_worker_service
job = self.get_job(job_id)
if not job:
raise ValueError(f"Job {job_id} not found")
if not job.input_media_gcs_path:
raise ValueError(
f"Job {job_id} missing input_media_gcs_path. "
"Audio must be downloaded to GCS before starting processing."
)
# Transition to DOWNLOADING - this will raise if invalid
self.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=progress,
message=message
)
# Trigger both workers in parallel
worker_service = get_worker_service()
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
logger.info(f"Job {job_id}: Started processing (audio + lyrics workers triggered)")
def mark_audio_complete(self, job_id: str) -> None:
"""
Mark audio processing as complete and check if can proceed.
@@ -71,6 +71,7 @@ class JobNotificationService:
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
brand_code: Optional[str] = None,
is_private: bool = False,
) -> bool:
"""
Send job completion email to user.
@@ -84,6 +85,7 @@ class JobNotificationService:
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
brand_code: Release ID (e.g., "NOMAD-1178")
is_private: If True, omit YouTube section (private/non-published tracks)
Returns:
True if email was sent successfully
@@ -106,6 +108,7 @@ class JobNotificationService:
title=title,
job_id=job_id,
feedback_url=FEEDBACK_FORM_URL,
is_private=is_private,
)
# Send the email with CC to admin
@@ -218,6 +221,7 @@ class JobNotificationService:
title: Optional[str] = None,
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
is_private: bool = False,
) -> str:
"""
Get the rendered completion message for a job (for admin copy functionality).
@@ -229,6 +233,7 @@ class JobNotificationService:
title: Song title
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
is_private: If True, omit YouTube section (private/non-published tracks)
Returns:
Rendered message content as plain text
@@ -241,6 +246,7 @@ class JobNotificationService:
title=title,
job_id=job_id,
feedback_url=FEEDBACK_FORM_URL,
is_private=is_private,
)
+7
View File
@@ -57,6 +57,8 @@ class PackagingService:
title: str,
output_mp3_path: Optional[str] = None,
output_cdg_path: Optional[str] = None,
lrc_has_countdown_padding: bool = False,
countdown_padding_seconds: float = 3.0,
) -> Tuple[str, Optional[str], Optional[str]]:
"""
Create a CDG package (ZIP containing CDG and MP3 files).
@@ -69,6 +71,9 @@ class PackagingService:
title: Song title for metadata
output_mp3_path: Optional path for the extracted MP3 file
output_cdg_path: Optional path for the extracted CDG file
lrc_has_countdown_padding: If True, LRC contains countdown-shifted timestamps
that need adjustment since CDG uses instrumental audio without padding
countdown_padding_seconds: Duration of countdown padding to compensate for
Returns:
Tuple of (zip_path, mp3_path, cdg_path)
@@ -131,6 +136,8 @@ class PackagingService:
title=title,
artist=artist,
cdg_styles=self.cdg_styles,
lrc_has_countdown_padding=lrc_has_countdown_padding,
countdown_padding_seconds=countdown_padding_seconds,
)
# Rename ZIP to expected output path if different
+11
View File
@@ -212,6 +212,7 @@ class TemplateService:
title: Optional[str] = None,
job_id: Optional[str] = None,
feedback_url: Optional[str] = None,
is_private: bool = False,
) -> str:
"""
Render the job completion email template.
@@ -224,11 +225,21 @@ class TemplateService:
title: Song title
job_id: Job ID
feedback_url: Feedback form URL (optional)
is_private: If True, remove YouTube section entirely (private/non-published tracks)
Returns:
Rendered email content
"""
template = self.get_job_completion_template()
# For private tracks, remove the YouTube section entirely
if is_private:
template = re.sub(
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
'\n',
template,
)
variables = {
"name": name or "there",
"youtube_url": youtube_url or "[YouTube URL not available]",
+1 -1
View File
@@ -48,7 +48,7 @@ class UserService:
"""Service for user management and authentication."""
# Number of free credits granted to new users
NEW_USER_FREE_CREDITS = 1
NEW_USER_FREE_CREDITS = 2
def __init__(self):
"""Initialize user service with Firestore client."""
@@ -0,0 +1,122 @@
"""
Integration tests for admin job reset endpoint.
These tests use the real Firestore emulator and real JobManager,
catching bugs that unit tests with mocks might miss (e.g., incorrect
assumptions about return values).
Run with: scripts/run-emulator-tests.sh
"""
import pytest
from datetime import datetime, UTC
from backend.tests.emulator.conftest import emulators_running
# Skip all tests if emulators aren't running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
# Only import if emulators are running
if emulators_running():
from backend.models.job import Job, JobStatus
from backend.services.job_manager import JobManager
from backend.services.firestore_service import FirestoreService
class TestUpdateJobReturnValue:
"""
Critical tests documenting that update_job() returns None, not boolean.
This test class exists because a production bug (PR #371) was caused by
assuming update_job() returns a boolean success indicator. The real API
returns None on success and raises on failure. Mocks that return True
hide this behavior.
"""
@pytest.fixture
def job_manager(self):
"""Create a real JobManager using emulator."""
return JobManager()
@pytest.fixture
def firestore_service(self):
"""Create a real FirestoreService using emulator."""
return FirestoreService()
@pytest.fixture
def test_job(self, firestore_service):
"""Create a test job for verification."""
job_id = f"update-return-test-{datetime.now(UTC).timestamp()}"
job = Job(
job_id=job_id,
status=JobStatus.COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Test Song",
)
firestore_service.create_job(job)
yield job
# Cleanup
try:
firestore_service.delete_job(job_id)
except Exception:
pass
def test_update_job_returns_none_not_boolean(self, job_manager, test_job):
"""
Verify update_job() returns None, not a boolean.
This test exists because a production bug was introduced when code
assumed update_job() returns a boolean. The method actually returns
None on success and raises an exception on failure.
Bug pattern that this test prevents:
```python
success = job_manager.update_job(job_id, updates)
if not success: # BUG: None is falsy, so this is always True!
raise HTTPException(status_code=500, ...)
```
Correct pattern:
```python
# update_job() raises on failure, returns None on success
job_manager.update_job(job_id, updates)
```
"""
# update_job() should return None, not True/False
result = job_manager.update_job(
test_job.job_id,
{"status": JobStatus.PENDING.value}
)
# The actual return value is None, not True
assert result is None, (
"update_job() should return None, not a boolean. "
"Code checking 'if not update_job(...)' will always be True!"
)
def test_update_job_on_nonexistent_job(self, job_manager):
"""
Document update_job() behavior for non-existent jobs.
In production Firestore, update() on non-existent doc raises NotFound.
The key insight for preventing the admin reset bug is understanding
that update_job() returns None (not boolean) on success.
"""
try:
result = job_manager.update_job(
"nonexistent-job-id-12345",
{"status": JobStatus.PENDING.value}
)
# If no exception raised (emulator quirk), verify return is still None
assert result is None, "update_job should return None even on non-existent doc"
except Exception:
# Exception is the expected behavior in production Firestore
pass
@@ -0,0 +1,69 @@
"""
Integration tests for admin job reset using REAL job fixture from production.
This test validates that update_job() returns None, not a boolean
(the root cause of PR #371 bug).
Uses actual data from production job 984da08b which needed admin reset.
Run with: scripts/run-emulator-tests.sh
"""
import pytest
from datetime import datetime, UTC
from backend.tests.emulator.conftest import emulators_running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
if emulators_running():
from backend.models.job import Job, JobStatus
from backend.services.job_manager import JobManager
class TestAdminResetBugFix:
"""
Tests that prove the PR #371 bug fix works using real emulator.
The bug: admin reset endpoint returned 500 because:
success = job_manager.update_job(job_id, updates)
if not success: # BUG: update_job returns None, not True!
raise HTTPException(500)
"""
def test_update_job_returns_none_not_boolean(self):
"""
Prove that update_job() returns None, not a boolean.
This is the root cause of the bug. Any code that does
'if not update_job(...)' will always be True.
"""
jm = JobManager()
job_id = f"test-{datetime.now(UTC).timestamp()}"
job = Job(
job_id=job_id,
status=JobStatus.COMPLETE,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Mark Mallman",
title="Minneapolis",
)
try:
jm.firestore.create_job(job)
# THE KEY TEST: update_job returns None
result = jm.update_job(job_id, {"status": "pending"})
assert result is None, (
"update_job() must return None. "
"Code checking 'if not success:' would always fail!"
)
finally:
try:
jm.firestore.delete_job(job_id)
except Exception:
pass
@@ -218,16 +218,16 @@ class TestCountdownStateTransitions:
"""
Countdown state should survive job state transitions.
When job moves from RENDERING_VIDEO to ENCODING,
When job moves from GENERATING_VIDEO to ENCODING,
the lyrics_metadata should remain intact.
"""
job_id = f"transition-test-{datetime.now(UTC).timestamp()}"
try:
# Create job in RENDERING_VIDEO state
# Create job in GENERATING_VIDEO state (valid predecessor to ENCODING)
job = Job(
job_id=job_id,
status=JobStatus.RENDERING_VIDEO,
status=JobStatus.GENERATING_VIDEO,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test",
@@ -241,7 +241,7 @@ class TestCountdownStateTransitions:
)
firestore_service.create_job(job)
# Transition to ENCODING
# Transition to ENCODING (valid transition from GENERATING_VIDEO)
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.ENCODING,
@@ -229,6 +229,12 @@ class TestMadeForYouYouTubeOrderFlow:
mock_job.job_id = "test-job-workers"
mock_job_manager.create_job.return_value = mock_job
# start_job_processing is async - mock it to call workers directly
async def mock_start_job_processing(job_id, progress=15, message=""):
await mock_worker_service_local.trigger_audio_worker(job_id)
await mock_worker_service_local.trigger_lyrics_worker(job_id)
mock_job_manager.start_job_processing = AsyncMock(side_effect=mock_start_job_processing)
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
@@ -245,7 +251,7 @@ class TestMadeForYouYouTubeOrderFlow:
email_service=mock_email_service,
)
# Verify workers were triggered
# Verify workers were triggered via start_job_processing
mock_worker_service_local.trigger_audio_worker.assert_called_once_with(mock_job.job_id)
mock_worker_service_local.trigger_lyrics_worker.assert_called_once_with(mock_job.job_id)
@@ -269,6 +275,9 @@ class TestMadeForYouYouTubeOrderFlow:
mock_job.job_id = "test-job-notifications"
mock_job_manager.create_job.return_value = mock_job
# start_job_processing is async - mock it as simple AsyncMock
mock_job_manager.start_job_processing = AsyncMock()
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
@@ -389,6 +398,12 @@ class TestMadeForYouYouTubeOrderFlow:
mock_job.job_id = "test-job-no-search"
mock_job_manager.create_job.return_value = mock_job
# start_job_processing is async - mock it to call workers directly
async def mock_start_job_processing(job_id, progress=15, message=""):
await mock_worker_service_local.trigger_audio_worker(job_id)
await mock_worker_service_local.trigger_lyrics_worker(job_id)
mock_job_manager.start_job_processing = AsyncMock(side_effect=mock_start_job_processing)
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.audio_search_service.get_audio_search_service') as mock_get_audio, \
@@ -412,7 +427,7 @@ class TestMadeForYouYouTubeOrderFlow:
# Audio search should NOT be called for YouTube URL orders
mock_audio_service.search.assert_not_called()
# Workers should be triggered directly
# Workers should be triggered directly via start_job_processing
mock_worker_service_local.trigger_audio_worker.assert_called_once()
@@ -548,6 +563,12 @@ class TestMadeForYouWebhookIntegration:
mock_job.job_id = "test-job-webhook-flow"
mock_job_manager.create_job.return_value = mock_job
# start_job_processing is async - mock it to call workers directly
async def mock_start_job_processing(job_id, progress=15, message=""):
await mock_worker_service_local.trigger_audio_worker(job_id)
await mock_worker_service_local.trigger_lyrics_worker(job_id)
mock_job_manager.start_job_processing = AsyncMock(side_effect=mock_start_job_processing)
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service_local), \
patch('backend.services.storage_service.StorageService'), \
@@ -573,7 +594,7 @@ class TestMadeForYouWebhookIntegration:
# 2. Verify YouTube download was called
mock_youtube_download_service.download.assert_called_once()
# 3. Verify workers were triggered
# 3. Verify workers were triggered via start_job_processing
mock_worker_service_local.trigger_audio_worker.assert_called_once()
mock_worker_service_local.trigger_lyrics_worker.assert_called_once()
@@ -0,0 +1,348 @@
"""
Integration tests for the job state machine flow.
These tests verify complete state transitions without mocking the state machine,
ensuring the flow works correctly end-to-end (with mocked external services).
Created as part of state machine robustness improvements (2026-02-02).
"""
import pytest
from datetime import datetime, UTC
from unittest.mock import Mock, AsyncMock, patch, MagicMock
# Mock Firestore before importing JobManager
import sys
sys.modules['google.cloud.firestore'] = MagicMock()
from backend.models.job import Job, JobStatus
from backend.services.job_manager import JobManager, InvalidStateTransitionError
@pytest.fixture
def mock_firestore_service():
"""Mock FirestoreService."""
with patch('backend.services.job_manager.FirestoreService') as mock:
service = Mock()
mock.return_value = service
yield service
@pytest.fixture
def job_manager(mock_firestore_service):
"""Create JobManager instance with mocked Firestore."""
return JobManager()
class TestMadeForYouYouTubeFlow:
"""
Integration tests for Made-For-You YouTube order flow.
This was the flow that caused jobs 06cfea29 and 984da08b to get stuck.
The fix ensures proper state transitions from PENDING -> DOWNLOADING.
"""
@pytest.mark.asyncio
async def test_complete_flow_pending_to_downloading(self, job_manager, mock_firestore_service):
"""
Test the flow from PENDING to DOWNLOADING for YouTube orders.
This verifies the fix for the bug where jobs got stuck at PENDING
because start_job_processing wasn't called properly.
"""
# Create initial job in PENDING state
job = Job(
job_id="test-mfy-001",
artist="Test Artist",
title="Test Song",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
input_media_gcs_path="gs://bucket/test.mp3",
state_data={},
)
# Access the firestore through the job_manager's instance
job_manager.firestore.get_job.return_value = job
job_manager.firestore.update_job_status.return_value = True
# Mock worker service
with patch('backend.services.worker_service.get_worker_service') as mock_get_worker:
mock_worker = Mock()
mock_worker.trigger_audio_worker = AsyncMock()
mock_worker.trigger_lyrics_worker = AsyncMock()
mock_get_worker.return_value = mock_worker
# Call start_job_processing - this is what _handle_made_for_you_order now uses
await job_manager.start_job_processing("test-mfy-001")
# Verify transition to DOWNLOADING was called
job_manager.firestore.update_job_status.assert_called()
call_kwargs = job_manager.firestore.update_job_status.call_args.kwargs
assert call_kwargs['status'] == JobStatus.DOWNLOADING
# Verify both workers were triggered
mock_worker.trigger_audio_worker.assert_called_once_with("test-mfy-001")
mock_worker.trigger_lyrics_worker.assert_called_once_with("test-mfy-001")
@pytest.mark.asyncio
async def test_fails_if_job_stuck_at_wrong_state(self, job_manager, mock_firestore_service):
"""
Test that starting processing fails if job is in wrong state.
Jobs that are already past PENDING (e.g., DOWNLOADING) shouldn't
be able to start processing again - this would indicate a bug.
"""
# Create job already in DOWNLOADING state
job = Job(
job_id="test-mfy-002",
artist="Test Artist",
title="Test Song",
status=JobStatus.DOWNLOADING, # Already past PENDING
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
input_media_gcs_path="gs://bucket/test.mp3",
state_data={},
)
job_manager.firestore.get_job.return_value = job
# Attempting to start processing should raise InvalidStateTransitionError
# because DOWNLOADING -> DOWNLOADING is not a valid transition
with pytest.raises(InvalidStateTransitionError):
await job_manager.start_job_processing("test-mfy-002")
class TestParallelProcessingCoordination:
"""
Integration tests for parallel audio + lyrics processing coordination.
Both workers run in parallel and set flags when complete.
Screens worker is triggered when both are done.
"""
def test_audio_complete_updates_state_data(self, job_manager, mock_firestore_service):
"""Test that mark_audio_complete updates state_data correctly."""
job = Job(
job_id="test-parallel-001",
artist="Test Artist",
title="Test Song",
status=JobStatus.DOWNLOADING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={},
)
job_manager.firestore.get_job.return_value = job
job_manager.firestore.update_job.return_value = True
# Call mark_audio_complete
job_manager.mark_audio_complete("test-parallel-001")
# Verify update_job was called to set the audio_complete flag
job_manager.firestore.update_job.assert_called()
def test_lyrics_complete_updates_state_data(self, job_manager, mock_firestore_service):
"""Test that mark_lyrics_complete updates state_data correctly."""
job = Job(
job_id="test-parallel-002",
artist="Test Artist",
title="Test Song",
status=JobStatus.DOWNLOADING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={},
)
job_manager.firestore.get_job.return_value = job
job_manager.firestore.update_job.return_value = True
# Call mark_lyrics_complete
job_manager.mark_lyrics_complete("test-parallel-002")
# Verify update_job was called to set the lyrics_complete flag
job_manager.firestore.update_job.assert_called()
def test_both_complete_triggers_screens(self, job_manager, mock_firestore_service):
"""Test that screens worker is triggered when both complete."""
# First call: Job with audio already complete (for update_state_data)
# Second call: Job with both complete (for check_parallel_processing_complete)
job_audio_only = Job(
job_id="test-parallel-003",
artist="Test Artist",
title="Test Song",
status=JobStatus.DOWNLOADING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={'audio_complete': True},
)
job_both_complete = Job(
job_id="test-parallel-003",
artist="Test Artist",
title="Test Song",
status=JobStatus.DOWNLOADING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={'audio_complete': True, 'lyrics_complete': True},
)
# Return different jobs on subsequent calls
job_manager.firestore.get_job.side_effect = [job_audio_only, job_both_complete]
job_manager.firestore.update_job.return_value = True
# Mock the screens worker trigger
with patch.object(job_manager, '_trigger_screens_worker') as mock_trigger:
# Now lyrics completes
job_manager.mark_lyrics_complete("test-parallel-003")
# Verify screens worker was triggered (both complete)
mock_trigger.assert_called_once_with("test-parallel-003")
class TestStateTransitionValidation:
"""
Integration tests for state transition validation.
Ensures invalid transitions are rejected with clear errors.
"""
def test_valid_transition_succeeds(self, job_manager, mock_firestore_service):
"""Test that valid transitions succeed."""
job = Job(
job_id="test-transition-001",
artist="Test Artist",
title="Test Song",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={},
)
job_manager.firestore.get_job.return_value = job
job_manager.firestore.update_job_status.return_value = True
# PENDING -> DOWNLOADING is valid
result = job_manager.transition_to_state(
job_id="test-transition-001",
new_status=JobStatus.DOWNLOADING,
progress=10,
message="Starting processing"
)
assert result is True
def test_invalid_transition_raises_by_default(self, job_manager, mock_firestore_service):
"""Test that invalid transitions raise exception by default."""
job = Job(
job_id="test-transition-002",
artist="Test Artist",
title="Test Song",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={},
)
job_manager.firestore.get_job.return_value = job
# PENDING -> COMPLETE is invalid (skips many steps)
with pytest.raises(InvalidStateTransitionError) as exc_info:
job_manager.transition_to_state(
job_id="test-transition-002",
new_status=JobStatus.COMPLETE,
)
assert "pending" in str(exc_info.value).lower()
assert "complete" in str(exc_info.value).lower()
def test_invalid_transition_returns_false_when_not_raising(self, job_manager, mock_firestore_service):
"""Test that invalid transitions return False when raise_on_invalid=False."""
job = Job(
job_id="test-transition-003",
artist="Test Artist",
title="Test Song",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={},
)
job_manager.firestore.get_job.return_value = job
# PENDING -> COMPLETE is invalid, but with raise_on_invalid=False
result = job_manager.transition_to_state(
job_id="test-transition-003",
new_status=JobStatus.COMPLETE,
raise_on_invalid=False
)
assert result is False
class TestAdminResetFlow:
"""
Integration tests for admin reset functionality.
Admin resets bypass normal state machine validation but must
properly clear state_data flags to avoid inconsistent state.
"""
def test_reset_to_pending_clears_parallel_flags(self, job_manager, mock_firestore_service):
"""
Test that resetting to pending clears audio_complete and lyrics_complete.
This was a bug where the Start button would reset to pending but
leave these flags set, causing inconsistent state.
"""
# This is tested via the admin endpoint, not directly in JobManager
# The test in test_admin_job_reset.py covers this
pass
class TestConsistencyWithHealthService:
"""
Integration tests that verify the job health service correctly
identifies inconsistencies in real job flows.
"""
def test_detects_stuck_job_scenario(self, job_manager, mock_firestore_service):
"""
Test that the health service would detect the exact bug that
caused jobs 06cfea29 and 984da08b to get stuck.
"""
from backend.services.job_health_service import check_job_consistency
# Simulate the stuck job scenario:
# - Job has input_media_gcs_path (YouTube download completed)
# - But status is still PENDING (transition never happened)
stuck_job = Job(
job_id="test-stuck-001",
artist="Test Artist",
title="Test Song",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
input_media_gcs_path="gs://bucket/downloaded.mp3",
state_data={},
)
issues = check_job_consistency(stuck_job)
# Should detect the pending_with_input_media issue
assert len(issues) > 0
assert any("pending_with_input_media" in issue for issue in issues)
def test_detects_flag_status_mismatch(self, job_manager, mock_firestore_service):
"""
Test that the health service detects when flags don't match status.
"""
from backend.services.job_health_service import check_job_consistency
# Simulate: audio_complete=True but status still PENDING
# (would happen if mark_audio_complete ran but transition didn't)
inconsistent_job = Job(
job_id="test-inconsistent-001",
artist="Test Artist",
title="Test Song",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={'audio_complete': True},
)
issues = check_job_consistency(inconsistent_job)
# Should detect the audio_complete_status_mismatch issue
assert len(issues) > 0
assert any("audio_complete_status_mismatch" in issue for issue in issues)
+12 -12
View File
@@ -75,7 +75,7 @@ class TestAudioSearchYouTubeFlow:
}
# Import the route function
from backend.api.routes.audio_search import _download_and_start_processing
from backend.api.routes.audio_search import _download_audio_and_trigger_workers
# Patch dependencies
with patch(
@@ -87,20 +87,19 @@ class TestAudioSearchYouTubeFlow:
), patch(
'backend.api.routes.audio_search.storage_service',
mock_services['storage_service']
), patch(
'backend.api.routes.audio_search.worker_service',
mock_services['worker_service']
):
# Create a mock audio_search_service
mock_audio_search_service = MagicMock()
mock_audio_search_service.is_remote_enabled = MagicMock(return_value=True)
# Create a mock background_tasks
mock_background_tasks = MagicMock()
# Call the download function
result = await _download_and_start_processing(
# Call the download function (now runs as background task, returns None)
await _download_audio_and_trigger_workers(
job_id="test_job_123",
selection_index=0,
audio_search_service=mock_audio_search_service,
background_tasks=mock_background_tasks,
)
# Verify YouTubeDownloadService was used
@@ -131,7 +130,7 @@ class TestAudioSearchYouTubeFlow:
'audio_search_count': 1,
}
from backend.api.routes.audio_search import _download_and_start_processing
from backend.api.routes.audio_search import _download_audio_and_trigger_workers
# Mock audio_search_service for torrent download
mock_audio_search_service = MagicMock()
@@ -149,14 +148,15 @@ class TestAudioSearchYouTubeFlow:
), patch(
'backend.api.routes.audio_search.storage_service',
mock_services['storage_service']
), patch(
'backend.api.routes.audio_search.worker_service',
mock_services['worker_service']
):
mock_background_tasks = MagicMock()
result = await _download_and_start_processing(
# Call the download function (now runs as background task, returns None)
await _download_audio_and_trigger_workers(
job_id="test_job_123",
selection_index=0,
audio_search_service=mock_audio_search_service,
background_tasks=mock_background_tasks,
)
# YouTubeDownloadService should NOT be called for torrent
+126 -43
View File
@@ -84,7 +84,7 @@ class TestResetJobToPending:
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -104,7 +104,7 @@ class TestResetJobToPending:
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -128,7 +128,7 @@ class TestResetJobToAwaitingAudioSelection:
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -149,7 +149,7 @@ class TestResetJobToAwaitingReview:
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -162,27 +162,6 @@ class TestResetJobToAwaitingReview:
assert data["new_status"] == "awaiting_review"
class TestResetJobToAwaitingInstrumentalSelection:
"""Tests for resetting job to AWAITING_INSTRUMENTAL_SELECTION state."""
def test_reset_to_awaiting_instrumental_selection(self, client, mock_job):
"""Test resetting a job to AWAITING_INSTRUMENTAL_SELECTION state."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "awaiting_instrumental_selection"},
)
assert response.status_code == 200
data = response.json()
assert data["new_status"] == "awaiting_instrumental_selection"
class TestResetJobValidation:
"""Tests for reset endpoint validation."""
@@ -268,7 +247,7 @@ class TestResetJobLogging:
patch('backend.api.routes.admin.logger') as mock_logger:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -291,7 +270,7 @@ class TestResetJobTimeline:
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -340,7 +319,7 @@ class TestResetJobClearsStateData:
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -353,12 +332,57 @@ class TestResetJobClearsStateData:
data = response.json()
assert "cleared_data" in data or "state_data" in str(data)
def test_reset_to_pending_clears_parallel_processing_flags(self, client):
"""Test that resetting to PENDING clears audio_complete and lyrics_complete.
This is critical for proper re-processing - without clearing these flags,
workers may think audio/lyrics processing is already done.
"""
from backend.models.job import Job, JobStatus
from datetime import datetime, UTC
# Create a job with audio_complete and lyrics_complete set
mock_job = Job(
job_id="test-job-123",
status=JobStatus.COMPLETE,
artist="Test Artist",
title="Test Title",
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={
"audio_complete": True,
"lyrics_complete": True,
"audio_search_results": [{"source": "test"}],
}
)
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "pending"},
)
assert response.status_code == 200
data = response.json()
# Verify the cleared_data includes the parallel processing flags
cleared_keys = data.get("cleared_data", [])
assert "audio_complete" in cleared_keys, \
f"audio_complete should be cleared. Cleared: {cleared_keys}"
assert "lyrics_complete" in cleared_keys, \
f"lyrics_complete should be cleared. Cleared: {cleared_keys}"
def test_reset_to_awaiting_review_preserves_audio_data(self, client, mock_job):
"""Test that resetting to AWAITING_REVIEW preserves audio stems."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -368,12 +392,15 @@ class TestResetJobClearsStateData:
assert response.status_code == 200
def test_reset_to_instrumental_preserves_review_data(self, client, mock_job):
"""Test that resetting to AWAITING_INSTRUMENTAL_SELECTION preserves review data."""
def test_rejects_deprecated_awaiting_instrumental_selection(self, client, mock_job):
"""Test that awaiting_instrumental_selection is no longer a valid reset target.
This state was deprecated in Jan 2026 when lyrics review and instrumental
selection were combined into a single review step.
"""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -381,7 +408,9 @@ class TestResetJobClearsStateData:
json={"target_state": "awaiting_instrumental_selection"},
)
assert response.status_code == 200
# Should be rejected as invalid
assert response.status_code == 400
assert "invalid" in response.json()["detail"].lower()
class TestResetJobToInstrumentalSelected:
@@ -396,7 +425,7 @@ class TestResetJobToInstrumentalSelected:
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
# Mock worker service with async method
@@ -421,7 +450,7 @@ class TestResetJobToInstrumentalSelected:
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
# Mock worker service
@@ -449,7 +478,7 @@ class TestResetJobToInstrumentalSelected:
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
# Mock worker service failure
@@ -486,7 +515,7 @@ class TestResetJobToInstrumentalSelected:
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
@@ -521,7 +550,7 @@ class TestResetJobClearsErrorState:
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -625,7 +654,7 @@ class TestResetIncludesEncodingProgress:
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -650,7 +679,7 @@ class TestResetWorkerTriggerResponseFields:
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
@@ -675,7 +704,7 @@ class TestResetWorkerTriggerResponseFields:
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
@@ -701,7 +730,7 @@ class TestResetWorkerTriggerResponseFields:
patch('backend.services.worker_service.get_worker_service') as mock_ws:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
mock_worker_service = Mock()
@@ -725,7 +754,7 @@ class TestResetWorkerTriggerResponseFields:
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm.update_job.return_value = None # Matches real API: returns None, raises on error
mock_jm_class.return_value = mock_jm
response = client.post(
@@ -933,3 +962,57 @@ class TestTriggerWorkerEndpoint:
app.dependency_overrides[require_admin] = original_override
else:
app.dependency_overrides[require_admin] = get_mock_admin
class TestResetEndpointWithRealJobData:
"""
Tests using real production job data (job 984da08b) as fixture.
This validates the admin reset endpoint works correctly with realistic
state_data from a production job that needed manual admin reset.
"""
# Real state_data from production job 984da08b
REAL_JOB_STATE_DATA = {
"backing_vocals_analysis": {"has_audible_content": True},
"video_progress": {"stage": "running"}, # Stale progress
"lyrics_complete": True,
"audio_complete": True
}
def test_reset_endpoint_returns_200_with_none_return_value(self, client):
"""
Test that reset endpoint returns 200 (not 500) when update_job
returns None, matching the real API behavior.
Before fix: endpoint checked 'if not success:' which was always True
After fix: endpoint doesn't check return value
Uses real production job 984da08b data as fixture.
"""
# Mock job with real production data
mock_job = Mock(spec=Job)
mock_job.job_id = "984da08b"
mock_job.status = JobStatus.COMPLETE
mock_job.artist = "Mark Mallman"
mock_job.title = "Minneapolis"
mock_job.theme_id = "nomad"
mock_job.user_email = "madeforyou@nomadkaraoke.com"
mock_job.file_urls = {"stems": {}, "lyrics": {}}
mock_job.state_data = self.REAL_JOB_STATE_DATA.copy()
mock_job.timeline = []
with patch('backend.api.routes.admin.JobManager') as mock_jm:
mock_jm.return_value.get_job.return_value = mock_job
# KEY: return None (matching real API contract - not boolean!)
mock_jm.return_value.update_job.return_value = None
response = client.post(
"/api/admin/jobs/984da08b/reset",
json={"target_state": "pending"},
)
# This is the bug fix - before it was 500, now 200
assert response.status_code == 200
assert response.json()["status"] == "success"
assert response.json()["new_status"] == "pending"
+81 -3
View File
@@ -7,7 +7,7 @@ route logic without hitting real cloud services.
import pytest
import json
from datetime import datetime, UTC
from unittest.mock import MagicMock, AsyncMock, patch
from unittest.mock import MagicMock, AsyncMock, patch, Mock
from fastapi.testclient import TestClient
from io import BytesIO
@@ -131,13 +131,91 @@ class TestHealthRoutes:
assert data["queue_length"] == 5
class TestJobConsistencyEndpoint:
"""Tests for the /health/job-consistency endpoint."""
@pytest.fixture
def client(self):
"""Create test client."""
with patch('backend.api.routes.health.JobManager') as mock_jm_class:
from backend.main import app
return TestClient(app)
def test_job_consistency_returns_empty_when_no_jobs(self, client):
"""Test endpoint returns empty results when no jobs exist."""
with patch('backend.api.routes.health.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.list_jobs.return_value = []
mock_jm_class.return_value = mock_jm
response = client.get("/api/health/job-consistency")
assert response.status_code == 200
data = response.json()
assert data["total_checked"] == 0
assert data["inconsistent_count"] == 0
assert data["inconsistent_jobs"] == []
def test_job_consistency_detects_issues(self, client):
"""Test endpoint detects inconsistent jobs."""
from backend.models.job import Job, JobStatus
from datetime import datetime, UTC
stuck_job = Job(
job_id="stuck-123",
artist="Test",
title="Song",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data={'audio_complete': True},
input_media_gcs_path="gs://bucket/audio.mp3",
)
with patch('backend.api.routes.health.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.list_jobs.return_value = [stuck_job]
mock_jm_class.return_value = mock_jm
response = client.get("/api/health/job-consistency")
assert response.status_code == 200
data = response.json()
assert data["total_checked"] == 1
assert data["inconsistent_count"] == 1
assert len(data["inconsistent_jobs"]) == 1
assert data["inconsistent_jobs"][0]["job_id"] == "stuck-123"
def test_job_consistency_filters_by_status(self, client):
"""Test endpoint filters by status parameter."""
with patch('backend.api.routes.health.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.list_jobs.return_value = []
mock_jm_class.return_value = mock_jm
response = client.get("/api/health/job-consistency?status=pending")
assert response.status_code == 200
data = response.json()
assert data["status_filter"] == "pending"
def test_job_consistency_rejects_invalid_status(self, client):
"""Test endpoint rejects invalid status parameter."""
with patch('backend.api.routes.health.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm_class.return_value = mock_jm
response = client.get("/api/health/job-consistency?status=invalid_status")
assert response.status_code == 200
data = response.json()
assert "error" in data
assert "Invalid status" in data["error"]
class TestJobRoutes:
"""Tests for jobs.py routes.
Note: These tests verify the route module structure.
Full integration tests are in test_api_integration.py.
"""
def test_jobs_router_exists(self):
"""Test jobs router can be imported."""
from backend.api.routes import jobs
+265 -77
View File
@@ -1140,37 +1140,48 @@ class TestFlacfetchIntegration:
class TestAudioSearchApiRouteDownload:
"""
Tests for _download_and_start_processing function in audio_search routes.
Tests for audio search download functions in audio_search routes.
This tests the API layer's handling of the download flow, including:
- Correct routing between local and remote downloads
- GCS path handling for remote downloads
- Error handling
- Background task scheduling (select endpoint returns immediately)
"""
@pytest.fixture
def mock_job_manager(self):
"""Mock the job manager singleton."""
with patch('backend.api.routes.audio_search.job_manager') as mock:
yield mock
@pytest.fixture
def mock_storage_service(self):
"""Mock the storage service singleton."""
with patch('backend.api.routes.audio_search.storage_service') as mock:
yield mock
def test_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
@pytest.fixture
def mock_worker_service(self):
"""Mock the worker service."""
with patch('backend.api.routes.audio_search.get_worker_service') as mock:
worker = Mock()
worker.trigger_audio_worker = AsyncMock()
worker.trigger_lyrics_worker = AsyncMock()
mock.return_value = worker
yield worker
def test_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service, mock_worker_service):
"""
Test that remote torrent downloads include GCS path for direct upload.
This was the bug: we were calling download() without gcs_path for
torrent sources, causing flacfetch VM to return a local path that
Cloud Run couldn't access.
"""
from backend.api.routes.audio_search import _download_and_start_processing
from backend.api.routes.audio_search import _download_audio_and_trigger_workers
import asyncio
# Setup mock job with RED search result
mock_job = Mock()
mock_job.state_data = {
@@ -1184,49 +1195,45 @@ class TestAudioSearchApiRouteDownload:
mock_job.audio_search_artist = 'Avril Lavigne'
mock_job.audio_search_title = 'Unwanted'
mock_job_manager.get_job.return_value = mock_job
# Create mock audio search service with remote client
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = True
# Mock download to return GCS path
mock_download_result = Mock()
mock_download_result.filepath = "gs://bucket/uploads/job123/audio/Avril Lavigne - Unwanted.flac"
mock_audio_service.download.return_value = mock_download_result
# Create mock background tasks
mock_bg_tasks = Mock()
# Run the async function
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
loop.run_until_complete(
_download_audio_and_trigger_workers(
job_id="job123",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=mock_bg_tasks,
)
)
finally:
loop.close()
# CRITICAL: Verify download was called with gcs_path for remote torrent source
mock_audio_service.download.assert_called_once()
call_kwargs = mock_audio_service.download.call_args.kwargs
# The gcs_path should be set for remote torrent downloads
assert 'gcs_path' in call_kwargs
assert call_kwargs['gcs_path'] == "uploads/job123/audio/"
def test_youtube_remote_download_uses_youtube_service(self, mock_job_manager, mock_storage_service):
def test_youtube_remote_download_uses_youtube_service(self, mock_job_manager, mock_storage_service, mock_worker_service):
"""
Test that YouTube downloads use YouTubeDownloadService.
After the consolidation refactor, all YouTube downloads go through
YouTubeDownloadService instead of audio_search_service.download_by_id().
"""
from backend.api.routes.audio_search import _download_and_start_processing
from backend.api.routes.audio_search import _download_audio_and_trigger_workers
import asyncio
# Setup mock job with YouTube search result that has source_id
@@ -1237,7 +1244,7 @@ class TestAudioSearchApiRouteDownload:
'artist': 'Kenny Chesney',
'provider': 'YouTube',
'quality': 'Opus 128kbps',
'source_id': 'qmTjyGH9ro4', # YouTube video ID (11 chars)
'source_id': 'qmTjyGH9ro4',
'url': 'https://www.youtube.com/watch?v=qmTjyGH9ro4',
}]
}
@@ -1256,20 +1263,15 @@ class TestAudioSearchApiRouteDownload:
)
mock_youtube_service._extract_video_id = Mock(return_value="qmTjyGH9ro4")
# Create mock background tasks
mock_bg_tasks = Mock()
# Patch get_youtube_download_service to return our mock
with patch('backend.api.routes.audio_search.get_youtube_download_service', return_value=mock_youtube_service):
# Run the async function
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
loop.run_until_complete(
_download_audio_and_trigger_workers(
job_id="job456",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=mock_bg_tasks,
)
)
finally:
@@ -1288,12 +1290,12 @@ class TestAudioSearchApiRouteDownload:
# audio_search_service.download_by_id should NOT be called for YouTube
mock_audio_service.download_by_id.assert_not_called()
def test_youtube_download_always_uses_youtube_service(self, mock_job_manager, mock_storage_service):
def test_youtube_download_always_uses_youtube_service(self, mock_job_manager, mock_storage_service, mock_worker_service):
"""
Test that YouTube downloads always use YouTubeDownloadService,
regardless of remote enabled status. The service handles fallback internally.
"""
from backend.api.routes.audio_search import _download_and_start_processing
from backend.api.routes.audio_search import _download_audio_and_trigger_workers
import asyncio
# Setup mock job with YouTube search result (use valid 11-char video ID)
@@ -1304,7 +1306,7 @@ class TestAudioSearchApiRouteDownload:
'artist': 'Avril Lavigne',
'provider': 'YouTube',
'quality': 'Opus 128kbps',
'source_id': 'abc123xyz45', # Valid 11-char ID
'source_id': 'abc123xyz45',
'url': 'https://www.youtube.com/watch?v=abc123xyz45',
}]
}
@@ -1323,20 +1325,15 @@ class TestAudioSearchApiRouteDownload:
)
mock_youtube_service._extract_video_id = Mock(return_value="abc123xyz45")
# Create mock background tasks
mock_bg_tasks = Mock()
# Patch get_youtube_download_service to return our mock
with patch('backend.api.routes.audio_search.get_youtube_download_service', return_value=mock_youtube_service):
# Run the async function
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
loop.run_until_complete(
_download_audio_and_trigger_workers(
job_id="job789",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=mock_bg_tasks,
)
)
finally:
@@ -1344,12 +1341,12 @@ class TestAudioSearchApiRouteDownload:
# YouTubeDownloadService.download_by_id should always be called for YouTube
mock_youtube_service.download_by_id.assert_called_once()
def test_handles_gcs_path_response_correctly(self, mock_job_manager, mock_storage_service):
def test_handles_gcs_path_response_correctly(self, mock_job_manager, mock_storage_service, mock_worker_service):
"""Test that GCS path responses are parsed correctly."""
from backend.api.routes.audio_search import _download_and_start_processing
from backend.api.routes.audio_search import _download_audio_and_trigger_workers
import asyncio
# Setup mock job with RED search result
mock_job = Mock()
mock_job.state_data = {
@@ -1361,33 +1358,32 @@ class TestAudioSearchApiRouteDownload:
}]
}
mock_job_manager.get_job.return_value = mock_job
# Create mock audio search service
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = True
# Mock download to return full GCS path
mock_download_result = Mock()
mock_download_result.filepath = "gs://karaoke-gen-bucket/uploads/job789/audio/test.flac"
mock_audio_service.download.return_value = mock_download_result
# Run the async function
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
loop.run_until_complete(
_download_audio_and_trigger_workers(
job_id="job789",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=Mock(),
)
)
finally:
loop.close()
# Verify job was updated with correct GCS path (without gs://bucket/ prefix)
update_calls = mock_job_manager.update_job.call_args_list
# Find the call that sets input_media_gcs_path
gcs_path_set = False
for call in update_calls:
@@ -1397,24 +1393,17 @@ class TestAudioSearchApiRouteDownload:
assert gcs_path == "uploads/job789/audio/test.flac"
gcs_path_set = True
break
assert gcs_path_set, "input_media_gcs_path was not set in job update"
def test_torrent_download_requires_remote(self, mock_job_manager, mock_storage_service):
"""
Test that torrent sources (RED/OPS) require remote flacfetch.
Torrent downloads can only happen on the flacfetch VM which has:
- BitTorrent peer connectivity
- Private tracker credentials
- Long-running seeding capability
Cloud Run cannot download torrents, so remote must be enabled.
When running as a background task, the error is caught and the job is failed.
"""
from backend.api.routes.audio_search import _download_and_start_processing
from backend.services.audio_search_service import DownloadError
from backend.api.routes.audio_search import _download_audio_and_trigger_workers
import asyncio
import pytest
# Setup mock job with RED search result
mock_job = Mock()
@@ -1422,7 +1411,7 @@ class TestAudioSearchApiRouteDownload:
'audio_search_results': [{
'title': 'Test',
'artist': 'Test',
'provider': 'RED', # Torrent source requires remote
'provider': 'RED',
'quality': 'FLAC',
}]
}
@@ -1430,21 +1419,220 @@ class TestAudioSearchApiRouteDownload:
# Create mock audio search service WITHOUT remote client
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = False # REMOTE DISABLED
mock_audio_service.is_remote_enabled.return_value = False
loop = asyncio.new_event_loop()
try:
# Should raise an error because RED requires remote
with pytest.raises(Exception) as exc_info:
loop.run_until_complete(
_download_and_start_processing(
job_id="job_no_remote",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=Mock(),
)
# The background task catches exceptions and fails the job
loop.run_until_complete(
_download_audio_and_trigger_workers(
job_id="job_no_remote",
selection_index=0,
audio_search_service=mock_audio_service,
)
# Verify the error message mentions remote flacfetch
assert "remote flacfetch" in str(exc_info.value).lower() or "flacfetch" in str(exc_info.value).lower()
)
finally:
loop.close()
# Verify the job was failed with an appropriate error message
mock_job_manager.fail_job.assert_called_once()
fail_args = mock_job_manager.fail_job.call_args
assert fail_args.args[0] == "job_no_remote"
assert "remote flacfetch" in fail_args.args[1].lower() or "flacfetch" in fail_args.args[1].lower()
def test_validate_and_prepare_selection(self, mock_job_manager, mock_storage_service):
"""
Test that _validate_and_prepare_selection transitions job state and returns
selection info without performing the download.
"""
from backend.api.routes.audio_search import _validate_and_prepare_selection
# Setup mock job
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Waterloo',
'artist': 'ABBA',
'provider': 'RED',
'quality': 'FLAC',
'source_id': 'abc123',
'target_file': 'Waterloo.flac',
'url': '',
}]
}
mock_job_manager.get_job.return_value = mock_job
result = _validate_and_prepare_selection(
job_id="job_test",
selection_index=0,
)
# Should return selection info
assert result['selected_index'] == 0
assert result['selected_title'] == 'Waterloo'
assert result['selected_artist'] == 'ABBA'
assert result['selected_provider'] == 'RED'
# Should transition to DOWNLOADING_AUDIO
mock_job_manager.transition_to_state.assert_called_once()
call_kwargs = mock_job_manager.transition_to_state.call_args.kwargs
assert call_kwargs['new_status'] == JobStatus.DOWNLOADING_AUDIO
# Should save download params
mock_job_manager.update_job.assert_called_once()
update_data = mock_job_manager.update_job.call_args.args[1]
assert update_data['audio_source_type'] == 'audio_search'
assert update_data['source_name'] == 'RED'
def test_validate_and_prepare_selection_invalid_index(self, mock_job_manager, mock_storage_service):
"""Test _validate_and_prepare_selection raises HTTPException for invalid index."""
from backend.api.routes.audio_search import _validate_and_prepare_selection
from fastapi import HTTPException
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{'title': 'Song', 'artist': 'Artist', 'provider': 'RED'}]
}
mock_job_manager.get_job.return_value = mock_job
with pytest.raises(HTTPException) as exc_info:
_validate_and_prepare_selection(job_id="job_test", selection_index=99)
assert exc_info.value.status_code == 400
assert "Invalid selection index" in exc_info.value.detail
def test_validate_and_prepare_selection_job_not_found(self, mock_job_manager, mock_storage_service):
"""Test _validate_and_prepare_selection raises HTTPException when job not found."""
from backend.api.routes.audio_search import _validate_and_prepare_selection
from fastapi import HTTPException
mock_job_manager.get_job.return_value = None
with pytest.raises(HTTPException) as exc_info:
_validate_and_prepare_selection(job_id="nonexistent", selection_index=0)
assert exc_info.value.status_code == 404
def test_validate_and_prepare_selection_no_results(self, mock_job_manager, mock_storage_service):
"""Test _validate_and_prepare_selection raises HTTPException when no search results."""
from backend.api.routes.audio_search import _validate_and_prepare_selection
from fastapi import HTTPException
mock_job = Mock()
mock_job.state_data = {'audio_search_results': []}
mock_job_manager.get_job.return_value = mock_job
with pytest.raises(HTTPException) as exc_info:
_validate_and_prepare_selection(job_id="job_test", selection_index=0)
assert exc_info.value.status_code == 400
assert "No search results" in exc_info.value.detail
def test_download_background_job_not_found(self, mock_job_manager, mock_storage_service):
"""Test _download_audio_and_trigger_workers handles missing job gracefully."""
from backend.api.routes.audio_search import _download_audio_and_trigger_workers
import asyncio
mock_job_manager.get_job.return_value = None
mock_audio_service = Mock()
loop = asyncio.new_event_loop()
try:
# Should not raise - background tasks must handle errors gracefully
loop.run_until_complete(
_download_audio_and_trigger_workers(
job_id="gone",
selection_index=0,
audio_search_service=mock_audio_service,
)
)
finally:
loop.close()
# Should not try to fail a non-existent job
mock_job_manager.fail_job.assert_not_called()
def test_download_triggers_workers_on_success(self, mock_job_manager, mock_storage_service, mock_worker_service):
"""Test that _download_audio_and_trigger_workers triggers both workers after download."""
from backend.api.routes.audio_search import _download_audio_and_trigger_workers
import asyncio
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Test',
'artist': 'Artist',
'provider': 'RED',
'quality': 'FLAC',
}]
}
mock_job_manager.get_job.return_value = mock_job
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = True
mock_download_result = Mock()
mock_download_result.filepath = "gs://bucket/uploads/job_w/audio/test.flac"
mock_audio_service.download.return_value = mock_download_result
loop = asyncio.new_event_loop()
try:
loop.run_until_complete(
_download_audio_and_trigger_workers(
job_id="job_w",
selection_index=0,
audio_search_service=mock_audio_service,
)
)
finally:
loop.close()
# Both workers should be triggered
mock_worker_service.trigger_audio_worker.assert_called_once_with("job_w")
mock_worker_service.trigger_lyrics_worker.assert_called_once_with("job_w")
# Job should transition to DOWNLOADING
transition_calls = mock_job_manager.transition_to_state.call_args_list
assert any(
c.kwargs.get('new_status') == JobStatus.DOWNLOADING
for c in transition_calls
), "Job should transition to DOWNLOADING status"
def test_download_generic_exception_fails_job(self, mock_job_manager, mock_storage_service):
"""Test that generic exceptions in background download fail the job gracefully."""
from backend.api.routes.audio_search import _download_audio_and_trigger_workers
import asyncio
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Test',
'artist': 'Artist',
'provider': 'RED',
'quality': 'FLAC',
}]
}
mock_job_manager.get_job.return_value = mock_job
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = True
mock_audio_service.download.side_effect = RuntimeError("Unexpected connection error")
loop = asyncio.new_event_loop()
try:
# Should not raise - background tasks catch all exceptions
loop.run_until_complete(
_download_audio_and_trigger_workers(
job_id="job_err",
selection_index=0,
audio_search_service=mock_audio_service,
)
)
finally:
loop.close()
# Job should be failed with error message
mock_job_manager.fail_job.assert_called_once()
fail_args = mock_job_manager.fail_job.call_args
assert fail_args.args[0] == "job_err"
assert "Unexpected connection error" in fail_args.args[1]
@@ -0,0 +1,396 @@
"""
Tests for audio_transcoding_service.py - Review audio transcoding to OGG Opus.
These tests mock StorageService and subprocess to verify:
- Cache path derivation for various GCS paths
- Cache hit path (no ffmpeg call)
- Cache miss with successful transcode
- Fallback to FLAC on transcoding failure
- Bulk prepare_review_audio_for_job transcodes all expected files
"""
import pytest
from unittest.mock import Mock, patch, call, ANY
class TestGetCachePath:
"""Test _get_cache_path() derivation logic."""
def _make_service(self):
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
return AudioTranscodingService(storage_service=mock_storage)
def test_input_audio_path(self):
service = self._make_service()
result = service._get_cache_path("jobs/abc123/input/song.flac")
assert result == "jobs/abc123/review-audio/song.ogg"
def test_stem_path(self):
service = self._make_service()
result = service._get_cache_path("jobs/abc123/stems/instrumental_clean.flac")
assert result == "jobs/abc123/review-audio/instrumental_clean.ogg"
def test_backing_vocals_path(self):
service = self._make_service()
result = service._get_cache_path("jobs/abc123/stems/backing_vocals.flac")
assert result == "jobs/abc123/review-audio/backing_vocals.ogg"
def test_nested_job_path(self):
service = self._make_service()
result = service._get_cache_path("jobs/4bed6f80/stems/instrumental_with_backing.flac")
assert result == "jobs/4bed6f80/review-audio/instrumental_with_backing.ogg"
def test_wav_extension(self):
service = self._make_service()
result = service._get_cache_path("jobs/abc123/input/song.wav")
assert result == "jobs/abc123/review-audio/song.ogg"
def test_no_jobs_prefix_fallback_uses_hash(self):
service = self._make_service()
result = service._get_cache_path("other/path/audio.flac")
# Should use a hash-based fallback, not a literal "_unknown"
assert result.startswith("jobs/_")
assert result.endswith("/review-audio/audio.ogg")
assert len(result.split("/")[1]) == 13 # "_" + 12-char hash
def test_no_jobs_prefix_different_paths_different_hashes(self):
service = self._make_service()
result1 = service._get_cache_path("other/path/audio.flac")
result2 = service._get_cache_path("another/path/audio.flac")
# Different source paths should produce different cache paths
assert result1 != result2
class TestTranscodeIfNeeded:
"""Test transcode_if_needed() cache hit/miss logic."""
@patch("backend.services.audio_transcoding_service.subprocess")
def test_cache_hit_skips_transcode(self, mock_subprocess):
"""When cached file exists, return it without transcoding."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = True
service = AudioTranscodingService(storage_service=mock_storage)
result = service.transcode_if_needed("jobs/abc/input/song.flac")
assert result == "jobs/abc/review-audio/song.ogg"
mock_storage.file_exists.assert_called_once_with("jobs/abc/review-audio/song.ogg")
mock_subprocess.run.assert_not_called()
mock_storage.download_file.assert_not_called()
@patch("backend.services.audio_transcoding_service.subprocess")
def test_cache_miss_transcodes_and_uploads(self, mock_subprocess):
"""When cached file is missing, download + transcode + upload."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = False
mock_subprocess.run.return_value = Mock(returncode=0, stderr="")
service = AudioTranscodingService(storage_service=mock_storage)
result = service.transcode_if_needed("jobs/abc/input/song.flac")
assert result == "jobs/abc/review-audio/song.ogg"
# Verify download from correct source path
download_args = mock_storage.download_file.call_args[0]
assert download_args[0] == "jobs/abc/input/song.flac"
# Verify ffmpeg command
mock_subprocess.run.assert_called_once()
ffmpeg_call = mock_subprocess.run.call_args
ffmpeg_cmd = ffmpeg_call[0][0]
assert "ffmpeg" in ffmpeg_cmd[0]
assert "libopus" in ffmpeg_cmd
assert "128k" in ffmpeg_cmd
assert "-vn" in ffmpeg_cmd
# Verify timeout is set
assert ffmpeg_call[1]["timeout"] == 120
# Verify upload to correct cache path
upload_args = mock_storage.upload_file.call_args[0]
assert upload_args[1] == "jobs/abc/review-audio/song.ogg"
class TestGetReviewAudioUrl:
"""Test get_review_audio_url() with fallback behavior."""
@patch("backend.services.audio_transcoding_service.subprocess")
def test_returns_transcoded_url_on_success(self, mock_subprocess):
"""On success, return signed URL for transcoded file."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = True
mock_storage.generate_signed_url.return_value = "https://signed-url/song.ogg"
service = AudioTranscodingService(storage_service=mock_storage)
result = service.get_review_audio_url("jobs/abc/input/song.flac")
assert result == "https://signed-url/song.ogg"
mock_storage.generate_signed_url.assert_called_once_with(
"jobs/abc/review-audio/song.ogg", 120
)
def test_falls_back_to_flac_on_error(self):
"""On transcoding error with no cache, fall back to original FLAC signed URL."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = False
mock_storage.download_file.side_effect = Exception("GCS download failed")
mock_storage.generate_signed_url.return_value = "https://signed-url/song.flac"
service = AudioTranscodingService(storage_service=mock_storage)
result = service.get_review_audio_url("jobs/abc/input/song.flac")
assert result == "https://signed-url/song.flac"
mock_storage.generate_signed_url.assert_called_once_with(
"jobs/abc/input/song.flac", 120
)
# file_exists called twice: once in transcode_if_needed (cache miss),
# once in fallback (cache still missing → fall through to FLAC)
assert mock_storage.file_exists.call_count == 2
def test_serves_cache_when_source_deleted(self):
"""When source is deleted but OGG cache exists, serve the cache."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
# First file_exists (in transcode_if_needed): cache miss
# download_file fails: source deleted
# Second file_exists (in fallback): cache found
mock_storage.file_exists.side_effect = [False, True]
mock_storage.download_file.side_effect = Exception("blob not found")
mock_storage.generate_signed_url.return_value = "https://signed-url/song.ogg"
service = AudioTranscodingService(storage_service=mock_storage)
result = service.get_review_audio_url("jobs/abc/input/song.flac")
assert result == "https://signed-url/song.ogg"
mock_storage.generate_signed_url.assert_called_once_with(
"jobs/abc/review-audio/song.ogg", 120
)
def test_falls_back_to_flac_when_cache_check_also_fails(self):
"""When source is deleted and file_exists() also throws, fall back to FLAC."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
# First file_exists (in transcode_if_needed): cache miss
# download_file fails: source deleted
# Second file_exists (in fallback): also throws
mock_storage.file_exists.side_effect = [False, Exception("GCS unavailable")]
mock_storage.download_file.side_effect = Exception("blob not found")
mock_storage.generate_signed_url.return_value = "https://signed-url/song.flac"
service = AudioTranscodingService(storage_service=mock_storage)
result = service.get_review_audio_url("jobs/abc/input/song.flac")
assert result == "https://signed-url/song.flac"
mock_storage.generate_signed_url.assert_called_once_with(
"jobs/abc/input/song.flac", 120
)
@patch("backend.services.audio_transcoding_service.subprocess")
def test_custom_expiration_minutes(self, mock_subprocess):
"""Custom expiration_minutes is passed through to generate_signed_url."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = True
mock_storage.generate_signed_url.return_value = "https://signed-url/song.ogg"
service = AudioTranscodingService(storage_service=mock_storage)
service.get_review_audio_url("jobs/abc/input/song.flac", expiration_minutes=60)
mock_storage.generate_signed_url.assert_called_once_with(
"jobs/abc/review-audio/song.ogg", 60
)
@patch("backend.services.audio_transcoding_service.subprocess")
def test_falls_back_on_ffmpeg_failure(self, mock_subprocess):
"""On ffmpeg failure, fall back to original FLAC signed URL."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = False
mock_subprocess.run.return_value = Mock(returncode=1, stderr="codec error")
service = AudioTranscodingService(storage_service=mock_storage)
result = service.get_review_audio_url("jobs/abc/input/song.flac")
# Should fall back to FLAC URL
mock_storage.generate_signed_url.assert_called_once_with(
"jobs/abc/input/song.flac", 120
)
class TestPrepareReviewAudioForJob:
"""Test prepare_review_audio_for_job() bulk transcoding."""
@patch("backend.services.audio_transcoding_service.subprocess")
def test_transcodes_all_expected_files(self, mock_subprocess):
"""Should transcode input + all stems and return correct cache paths."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = True # All cache hits
service = AudioTranscodingService(storage_service=mock_storage)
mock_job = Mock()
mock_job.job_id = "abc123"
mock_job.input_media_gcs_path = "jobs/abc123/input/song.flac"
mock_job.file_urls = {
"stems": {
"instrumental_clean": "jobs/abc123/stems/instrumental_clean.flac",
"instrumental_with_backing": "jobs/abc123/stems/instrumental_with_backing.flac",
"backing_vocals": "jobs/abc123/stems/backing_vocals.flac",
}
}
result = service.prepare_review_audio_for_job(mock_job)
assert len(result) == 4
assert "jobs/abc123/review-audio/song.ogg" in result
assert "jobs/abc123/review-audio/instrumental_clean.ogg" in result
assert "jobs/abc123/review-audio/instrumental_with_backing.ogg" in result
assert "jobs/abc123/review-audio/backing_vocals.ogg" in result
assert mock_storage.file_exists.call_count == 4
# No ffmpeg calls since all are cache hits
mock_subprocess.run.assert_not_called()
@patch("backend.services.audio_transcoding_service.subprocess")
def test_continues_on_individual_failure(self, mock_subprocess):
"""If one file fails to transcode, others should still succeed."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
# First call (input): cache miss + download fails
# Subsequent calls: cache hits
mock_storage.file_exists.side_effect = [False, True, True, True]
mock_storage.download_file.side_effect = Exception("download failed")
service = AudioTranscodingService(storage_service=mock_storage)
mock_job = Mock()
mock_job.job_id = "abc123"
mock_job.input_media_gcs_path = "jobs/abc123/input/song.flac"
mock_job.file_urls = {
"stems": {
"instrumental_clean": "jobs/abc123/stems/instrumental_clean.flac",
"instrumental_with_backing": "jobs/abc123/stems/instrumental_with_backing.flac",
"backing_vocals": "jobs/abc123/stems/backing_vocals.flac",
}
}
result = service.prepare_review_audio_for_job(mock_job)
# 3 out of 4 succeeded (the 3 cache hits)
assert len(result) == 3
@patch("backend.services.audio_transcoding_service.subprocess")
def test_handles_missing_stems(self, mock_subprocess):
"""Should handle jobs with missing stem paths gracefully."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = True
service = AudioTranscodingService(storage_service=mock_storage)
mock_job = Mock()
mock_job.job_id = "abc123"
mock_job.input_media_gcs_path = "jobs/abc123/input/song.flac"
mock_job.file_urls = {"stems": {}} # No stems
result = service.prepare_review_audio_for_job(mock_job)
# Only the input audio
assert len(result) == 1
@patch("backend.services.audio_transcoding_service.subprocess")
def test_handles_no_input_audio(self, mock_subprocess):
"""Should handle jobs with no input_media_gcs_path."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = True
service = AudioTranscodingService(storage_service=mock_storage)
mock_job = Mock()
mock_job.job_id = "abc123"
mock_job.input_media_gcs_path = None
mock_job.file_urls = {
"stems": {
"instrumental_clean": "jobs/abc123/stems/instrumental_clean.flac",
}
}
result = service.prepare_review_audio_for_job(mock_job)
assert len(result) == 1
@patch("backend.services.audio_transcoding_service.subprocess")
def test_handles_no_stems_key(self, mock_subprocess):
"""Should handle jobs where file_urls has no stems key at all."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = True
service = AudioTranscodingService(storage_service=mock_storage)
mock_job = Mock()
mock_job.job_id = "abc123"
mock_job.input_media_gcs_path = "jobs/abc123/input/song.flac"
mock_job.file_urls = {} # No stems key
result = service.prepare_review_audio_for_job(mock_job)
assert len(result) == 1
assert result[0] == "jobs/abc123/review-audio/song.ogg"
class TestInit:
"""Test AudioTranscodingService initialization."""
@patch("backend.services.audio_transcoding_service.StorageService")
def test_creates_default_storage_service(self, mock_storage_class):
"""Creates a StorageService when none is provided."""
from backend.services.audio_transcoding_service import AudioTranscodingService
service = AudioTranscodingService()
mock_storage_class.assert_called_once()
assert service.storage is mock_storage_class.return_value
def test_uses_provided_storage_service(self):
"""Uses the provided storage service instead of creating one."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
service = AudioTranscodingService(storage_service=mock_storage)
assert service.storage is mock_storage
class TestGetReviewAudioUrlAsync:
"""Test async wrapper."""
@pytest.mark.asyncio
@patch("backend.services.audio_transcoding_service.subprocess")
async def test_async_wrapper_delegates(self, mock_subprocess):
"""Async wrapper should call sync method via to_thread."""
from backend.services.audio_transcoding_service import AudioTranscodingService
mock_storage = Mock()
mock_storage.file_exists.return_value = True
mock_storage.generate_signed_url.return_value = "https://signed/url"
service = AudioTranscodingService(storage_service=mock_storage)
result = await service.get_review_audio_url_async("jobs/abc/input/song.flac")
assert result == "https://signed/url"
+148
View File
@@ -0,0 +1,148 @@
"""
Tests for download_audio() upload persistence in audio_worker.py.
Verifies that uploaded files (uploads/ prefix) are copied to jobs/ prefix
so they survive the 7-day GCS lifecycle rule on uploads/.
"""
import os
import pytest
from unittest.mock import Mock, patch, AsyncMock
@pytest.mark.asyncio
@patch("backend.workers.audio_worker.JobManager")
async def test_upload_persisted_to_jobs_prefix(MockJobManager):
"""Uploaded audio (uploads/ prefix) should be copied to jobs/{id}/input/."""
from backend.workers.audio_worker import download_audio
mock_storage = Mock()
mock_jm = Mock()
mock_job = Mock()
mock_job.input_media_gcs_path = "uploads/abc123/audio/My Song.flac"
mock_job.filename = "My Song.flac"
mock_job.url = None
mock_job.file_urls = {}
result = await download_audio(
job_id="abc123",
temp_dir="/tmp/test",
storage=mock_storage,
job=mock_job,
job_manager_instance=mock_jm,
)
# Should download from original upload path
mock_storage.download_file.assert_called_once_with(
"uploads/abc123/audio/My Song.flac", os.path.join("/tmp/test", "My Song.flac")
)
# Should re-upload to persistent jobs/ path
mock_storage.upload_file.assert_called_once_with(
os.path.join("/tmp/test", "My Song.flac"),
"jobs/abc123/input/My Song.flac",
)
# Should update job metadata
mock_jm.update_job.assert_called_once_with(
"abc123", {"input_media_gcs_path": "jobs/abc123/input/My Song.flac"}
)
mock_jm.update_file_url.assert_called_once_with(
"abc123", "input", "audio", "jobs/abc123/input/My Song.flac"
)
assert result == os.path.join("/tmp/test", "My Song.flac")
@pytest.mark.asyncio
@patch("backend.workers.audio_worker.JobManager")
async def test_jobs_prefix_not_re_persisted(MockJobManager):
"""Files already in jobs/ prefix should NOT be re-uploaded."""
from backend.workers.audio_worker import download_audio
mock_storage = Mock()
mock_jm = Mock()
mock_job = Mock()
mock_job.input_media_gcs_path = "jobs/abc123/input/song.flac"
mock_job.filename = "song.flac"
mock_job.url = None
mock_job.file_urls = {}
result = await download_audio(
job_id="abc123",
temp_dir="/tmp/test",
storage=mock_storage,
job=mock_job,
job_manager_instance=mock_jm,
)
# Should download but NOT re-upload
mock_storage.download_file.assert_called_once()
mock_storage.upload_file.assert_not_called()
mock_jm.update_job.assert_not_called()
@pytest.mark.asyncio
@patch("backend.workers.audio_worker.JobManager")
async def test_upload_persistence_creates_job_manager_if_none(MockJobManager):
"""If no job_manager_instance provided, creates one for persistence."""
from backend.workers.audio_worker import download_audio
mock_storage = Mock()
mock_jm_instance = Mock()
MockJobManager.return_value = mock_jm_instance
mock_job = Mock()
mock_job.input_media_gcs_path = "uploads/def456/audio/track.flac"
mock_job.filename = "track.flac"
mock_job.url = None
mock_job.file_urls = {}
await download_audio(
job_id="def456",
temp_dir="/tmp/test",
storage=mock_storage,
job=mock_job,
job_manager_instance=None,
)
# Should have created a JobManager
MockJobManager.assert_called_once()
# Should use the created instance
mock_jm_instance.update_job.assert_called_once()
mock_jm_instance.update_file_url.assert_called_once()
@pytest.mark.asyncio
@patch("backend.workers.audio_worker.JobManager")
async def test_persistence_failure_does_not_block_download(MockJobManager):
"""If upload_file fails during persistence, download still succeeds."""
from backend.workers.audio_worker import download_audio
mock_storage = Mock()
mock_storage.upload_file.side_effect = Exception("GCS upload failed")
mock_jm = Mock()
mock_job = Mock()
mock_job.input_media_gcs_path = "uploads/abc123/audio/song.flac"
mock_job.filename = "song.flac"
mock_job.url = None
mock_job.file_urls = {}
result = await download_audio(
job_id="abc123",
temp_dir="/tmp/test",
storage=mock_storage,
job=mock_job,
job_manager_instance=mock_jm,
)
# Download succeeded despite persistence failure
assert result == os.path.join("/tmp/test", "song.flac")
mock_storage.download_file.assert_called_once()
# upload_file was attempted but failed
mock_storage.upload_file.assert_called_once()
# Metadata update should NOT have been called (upload failed first)
mock_jm.update_job.assert_not_called()
+29
View File
@@ -32,6 +32,35 @@ class TestAuthServiceBasics:
assert hasattr(AuthService, 'validate_token') or hasattr(AuthService, 'verify_token')
class TestIncrementTokenUsage:
"""Tests for increment_token_usage after credit deduction cleanup."""
def test_session_token_does_not_deduct_credits(self):
"""
Verify that increment_token_usage does NOT deduct credits for session tokens.
Credit deduction is handled by JobManager.create_job(), not here.
This test ensures the dead code removal didn't break the method.
"""
mock_firestore = MagicMock()
# Session token: not in admin_tokens, not in Firestore auth_tokens
mock_firestore.get_token.return_value = None
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
patch.dict('os.environ', {'ADMIN_TOKENS': 'admin-token'}):
from backend.services.auth_service import AuthService
service = AuthService()
# Mock validate_token to return valid session user
with patch.object(service, 'validate_token', return_value=(True, 'stripe', 5, 'OK')):
result = service.increment_token_usage("session-token-123", "job-abc")
# Should return True (no-op for session tokens)
assert result is True
# Should NOT try to deduct credits via user_service
# (If it did, it would fail since user_service is not mocked)
class TestFirestoreServiceBasics:
"""Basic tests for FirestoreService."""
+590
View File
@@ -0,0 +1,590 @@
"""
Tests for BrandCodeService - atomic brand code allocation via Firestore counters.
Tests cover:
- Sequential allocation returns unique codes
- Recycled numbers are reused (smallest first)
- One-time initialization from Dropbox scan
- Both NOMAD and NOMADNP prefixes work independently
- Recycle then allocate round-trip
- parse_brand_code utility
"""
import pytest
from unittest.mock import Mock, MagicMock, patch
from datetime import datetime, timezone
import sys
sys.modules.setdefault('google.cloud.firestore', MagicMock())
sys.modules.setdefault('google.cloud.storage', MagicMock())
def _make_doc_snapshot(exists, data=None):
"""Create a mock Firestore document snapshot."""
doc = Mock()
doc.exists = exists
if data:
doc.to_dict.return_value = data
return doc
def _setup_service_and_mocks(mock_db, doc_snapshot):
"""Common setup: create service with fake transactional decorator."""
mock_doc_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_doc_ref
mock_doc_ref.get.return_value = doc_snapshot
mock_transaction = Mock()
mock_db.transaction.return_value = mock_transaction
return mock_doc_ref, mock_transaction
class TestBrandCodeServiceAllocate:
"""Test allocate_brand_code method."""
@pytest.fixture
def mock_db(self):
return MagicMock()
def test_allocate_sequential_number(self, mock_db):
"""Test allocation returns next sequential number when no recycled numbers."""
doc_snapshot = _make_doc_snapshot(True, {
"prefix": "NOMAD",
"next_number": 1264,
"recycled": [],
})
mock_doc_ref, mock_transaction = _setup_service_and_mocks(mock_db, doc_snapshot)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
result = svc.allocate_brand_code("NOMAD", "/Karaoke/Tracks-Organized")
assert result == "NOMAD-1264"
# Transaction.update should have been called to increment next_number
mock_transaction.update.assert_called_once()
update_args = mock_transaction.update.call_args[0]
assert update_args[0] is mock_doc_ref # first arg is doc_ref
assert update_args[1]["next_number"] == 1265
def test_allocate_uses_recycled_number(self, mock_db):
"""Test allocation pops smallest recycled number first."""
doc_snapshot = _make_doc_snapshot(True, {
"prefix": "NOMAD",
"next_number": 1264,
"recycled": [1250, 1230, 1245],
})
mock_doc_ref, mock_transaction = _setup_service_and_mocks(mock_db, doc_snapshot)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
result = svc.allocate_brand_code("NOMAD", "/path")
assert result == "NOMAD-1230"
# Verify recycled list was updated (1230 removed, sorted remainder)
update_args = mock_transaction.update.call_args[0][1]
assert update_args["recycled"] == [1245, 1250]
def test_allocate_initializes_from_dropbox_when_no_counter(self, mock_db):
"""Test first allocation scans Dropbox and creates counter doc."""
doc_snapshot = _make_doc_snapshot(False)
mock_doc_ref, mock_transaction = _setup_service_and_mocks(mock_db, doc_snapshot)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
with patch.object(svc, '_get_initial_next_number', return_value=1264):
result = svc.allocate_brand_code("NOMAD", "/Karaoke/Tracks-Organized")
assert result == "NOMAD-1264"
# Verify counter doc was created via transaction.set
mock_transaction.set.assert_called_once()
set_args = mock_transaction.set.call_args[0]
assert set_args[0] is mock_doc_ref
assert set_args[1]["prefix"] == "NOMAD"
assert set_args[1]["next_number"] == 1265
assert set_args[1]["recycled"] == []
def test_allocate_nomadnp_prefix(self, mock_db):
"""Test NOMADNP prefix works independently."""
doc_snapshot = _make_doc_snapshot(True, {
"prefix": "NOMADNP",
"next_number": 13,
"recycled": [],
})
_setup_service_and_mocks(mock_db, doc_snapshot)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
result = svc.allocate_brand_code("NOMADNP", "/Karaoke/NP")
assert result == "NOMADNP-0013"
mock_db.collection.assert_called_with("brand_code_counters")
mock_db.collection.return_value.document.assert_called_with("NOMADNP")
def test_allocate_formats_with_leading_zeros(self, mock_db):
"""Test brand codes are zero-padded to 4 digits."""
doc_snapshot = _make_doc_snapshot(True, {
"prefix": "NOMADNP",
"next_number": 5,
"recycled": [],
})
_setup_service_and_mocks(mock_db, doc_snapshot)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
result = svc.allocate_brand_code("NOMADNP", "/path")
assert result == "NOMADNP-0005"
class TestBrandCodeServiceRecycle:
"""Test recycle_brand_code method."""
@pytest.fixture
def mock_db(self):
return MagicMock()
def test_recycle_adds_number_to_pool(self, mock_db):
"""Test recycling adds number to the recycled list."""
doc_snapshot = _make_doc_snapshot(True, {
"prefix": "NOMAD",
"next_number": 1264,
"recycled": [1250],
})
mock_doc_ref, mock_transaction = _setup_service_and_mocks(mock_db, doc_snapshot)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
svc.recycle_brand_code("NOMAD", 1230)
update_args = mock_transaction.update.call_args[0][1]
assert 1230 in update_args["recycled"]
assert 1250 in update_args["recycled"]
def test_recycle_rejects_number_at_or_above_next(self, mock_db):
"""Test recycling a number >= next_number is rejected."""
doc_snapshot = _make_doc_snapshot(True, {
"prefix": "NOMAD",
"next_number": 1264,
"recycled": [],
})
mock_doc_ref, mock_transaction = _setup_service_and_mocks(mock_db, doc_snapshot)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
# Number equal to next_number — hasn't been allocated yet
svc.recycle_brand_code("NOMAD", 1264)
mock_transaction.update.assert_not_called()
def test_recycle_rejects_zero_or_negative(self, mock_db):
"""Test recycling zero or negative numbers is rejected."""
doc_snapshot = _make_doc_snapshot(True, {
"prefix": "NOMAD",
"next_number": 1264,
"recycled": [],
})
mock_doc_ref, mock_transaction = _setup_service_and_mocks(mock_db, doc_snapshot)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
svc.recycle_brand_code("NOMAD", 0)
mock_transaction.update.assert_not_called()
def test_recycle_skips_duplicate(self, mock_db):
"""Test recycling a number already in the pool doesn't duplicate it."""
doc_snapshot = _make_doc_snapshot(True, {
"prefix": "NOMAD",
"next_number": 1264,
"recycled": [1250],
})
mock_doc_ref, mock_transaction = _setup_service_and_mocks(mock_db, doc_snapshot)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
svc.recycle_brand_code("NOMAD", 1250)
mock_transaction.update.assert_not_called()
def test_recycle_no_counter_doc_logs_warning(self, mock_db):
"""Test recycling when counter doc doesn't exist logs warning and returns."""
doc_snapshot = _make_doc_snapshot(False)
mock_doc_ref, mock_transaction = _setup_service_and_mocks(mock_db, doc_snapshot)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
svc.recycle_brand_code("NOMAD", 1230)
mock_transaction.update.assert_not_called()
mock_transaction.set.assert_not_called()
class TestBrandCodeServiceRecycleRoundTrip:
"""Test recycle-then-allocate round trip."""
@pytest.fixture
def mock_db(self):
return MagicMock()
def test_recycle_then_allocate_reuses_number(self, mock_db):
"""Test that a recycled number is returned on next allocation."""
mock_doc_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_doc_ref
mock_transaction = Mock()
mock_db.transaction.return_value = mock_transaction
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
# Step 1: Recycle number 1230
mock_doc_ref.get.return_value = _make_doc_snapshot(True, {
"prefix": "NOMAD",
"next_number": 1264,
"recycled": [],
})
svc.recycle_brand_code("NOMAD", 1230)
# Step 2: Allocate — should get recycled 1230
mock_doc_ref.get.return_value = _make_doc_snapshot(True, {
"prefix": "NOMAD",
"next_number": 1264,
"recycled": [1230],
})
result = svc.allocate_brand_code("NOMAD", "/path")
assert result == "NOMAD-1230"
class TestInitializeFromDropbox:
"""Test _get_initial_next_number."""
@pytest.fixture
def mock_db(self):
return MagicMock()
def test_init_from_dropbox_finds_max(self, mock_db):
"""Test initialization finds the max existing number and adds 1."""
mock_dropbox = Mock()
mock_dropbox.is_configured = True
mock_dropbox.list_folders.return_value = [
"NOMAD-1260 - Artist A - Song A",
"NOMAD-1261 - Artist B - Song B",
"NOMAD-1263 - Artist C - Song C",
"Some Other Folder",
]
with patch('backend.services.brand_code_service.settings') as mock_settings, \
patch('backend.services.dropbox_service.get_dropbox_service', return_value=mock_dropbox):
mock_settings.google_cloud_project = 'test-project'
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
result = svc._get_initial_next_number("NOMAD", "/Karaoke/Tracks-Organized")
assert result == 1264 # max(1260, 1261, 1263) + 1
def test_init_from_dropbox_no_existing(self, mock_db):
"""Test initialization returns 1 when no existing codes found."""
mock_dropbox = Mock()
mock_dropbox.is_configured = True
mock_dropbox.list_folders.return_value = ["Some Random Folder"]
with patch('backend.services.brand_code_service.settings') as mock_settings, \
patch('backend.services.dropbox_service.get_dropbox_service', return_value=mock_dropbox):
mock_settings.google_cloud_project = 'test-project'
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
result = svc._get_initial_next_number("NOMAD", "/path")
assert result == 1
def test_init_from_dropbox_not_configured(self, mock_db):
"""Test initialization returns 1 when Dropbox not configured."""
mock_dropbox = Mock()
mock_dropbox.is_configured = False
with patch('backend.services.brand_code_service.settings') as mock_settings, \
patch('backend.services.dropbox_service.get_dropbox_service', return_value=mock_dropbox):
mock_settings.google_cloud_project = 'test-project'
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
result = svc._get_initial_next_number("NOMAD", "/path")
assert result == 1
def test_init_nomadnp_prefix_independent(self, mock_db):
"""Test NOMADNP initialization only finds NOMADNP codes, not NOMAD."""
mock_dropbox = Mock()
mock_dropbox.is_configured = True
mock_dropbox.list_folders.return_value = [
"NOMADNP-0010 - Artist A",
"NOMADNP-0012 - Artist B",
"NOMAD-1263 - Artist C", # Should NOT match NOMADNP prefix
]
with patch('backend.services.brand_code_service.settings') as mock_settings, \
patch('backend.services.dropbox_service.get_dropbox_service', return_value=mock_dropbox):
mock_settings.google_cloud_project = 'test-project'
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
result = svc._get_initial_next_number("NOMADNP", "/Karaoke/NP")
assert result == 13 # max(10, 12) + 1
class TestParseBrandCode:
"""Test parse_brand_code static method."""
def test_parse_nomad_code(self):
from backend.services.brand_code_service import BrandCodeService
prefix, number = BrandCodeService.parse_brand_code("NOMAD-1234")
assert prefix == "NOMAD"
assert number == 1234
def test_parse_nomadnp_code(self):
from backend.services.brand_code_service import BrandCodeService
prefix, number = BrandCodeService.parse_brand_code("NOMADNP-0012")
assert prefix == "NOMADNP"
assert number == 12
def test_parse_leading_zeros(self):
from backend.services.brand_code_service import BrandCodeService
prefix, number = BrandCodeService.parse_brand_code("NOMAD-0001")
assert prefix == "NOMAD"
assert number == 1
def test_parse_invalid_format_raises(self):
from backend.services.brand_code_service import BrandCodeService
with pytest.raises(ValueError, match="Invalid brand code format"):
BrandCodeService.parse_brand_code("invalid")
def test_parse_no_dash_raises(self):
from backend.services.brand_code_service import BrandCodeService
with pytest.raises(ValueError, match="Invalid brand code format"):
BrandCodeService.parse_brand_code("NOMAD1234")
def test_parse_lowercase_raises(self):
from backend.services.brand_code_service import BrandCodeService
with pytest.raises(ValueError, match="Invalid brand code format"):
BrandCodeService.parse_brand_code("nomad-1234")
class TestCleanupDistributionRecycling:
"""Test that cleanup_distribution endpoint recycles brand codes correctly."""
def test_parse_and_recycle_nomad_brand_code(self):
"""Test that a NOMAD brand code is correctly parsed and recycled."""
from backend.services.brand_code_service import BrandCodeService
prefix, number = BrandCodeService.parse_brand_code("NOMAD-1234")
assert prefix == "NOMAD"
assert number == 1234
def test_parse_and_recycle_nomadnp_brand_code(self):
"""Test that a NOMADNP brand code is correctly parsed and recycled."""
from backend.services.brand_code_service import BrandCodeService
prefix, number = BrandCodeService.parse_brand_code("NOMADNP-0012")
assert prefix == "NOMADNP"
assert number == 12
def test_recycle_called_after_successful_dropbox_delete(self):
"""Test the integration pattern: parse brand code, then recycle the number."""
mock_db = MagicMock()
mock_doc_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_doc_ref
mock_transaction = Mock()
mock_db.transaction.return_value = mock_transaction
mock_doc_ref.get.return_value = _make_doc_snapshot(True, {
"prefix": "NOMAD",
"next_number": 1264,
"recycled": [],
})
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
# Simulate what cleanup_distribution does:
# 1. Parse the brand code
brand_code = "NOMAD-1234"
prefix, number = BrandCodeService.parse_brand_code(brand_code)
# 2. Recycle it
svc.recycle_brand_code(prefix, number)
# Verify transaction.update was called to add 1234 to recycled pool
mock_transaction.update.assert_called_once()
update_args = mock_transaction.update.call_args[0][1]
assert 1234 in update_args["recycled"]
def test_recycle_failure_does_not_raise(self):
"""Test that recycling a brand code with no counter doc doesn't raise."""
mock_db = MagicMock()
mock_doc_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_doc_ref
mock_transaction = Mock()
mock_db.transaction.return_value = mock_transaction
mock_doc_ref.get.return_value = _make_doc_snapshot(False)
with patch('backend.services.brand_code_service.firestore') as mock_fs, \
patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
def fake_transactional(func):
def wrapper(transaction, *args, **kwargs):
return func(transaction, *args, **kwargs)
return wrapper
mock_fs.transactional = fake_transactional
from backend.services.brand_code_service import BrandCodeService
svc = BrandCodeService(db=mock_db)
# Should not raise — cleanup should be resilient
svc.recycle_brand_code("NOMAD", 1234)
mock_transaction.update.assert_not_called()
class TestGetBrandCodeService:
"""Test singleton getter."""
def test_returns_same_instance(self):
"""Test get_brand_code_service returns singleton."""
with patch('backend.services.brand_code_service.settings') as mock_settings:
mock_settings.google_cloud_project = 'test-project'
import backend.services.brand_code_service as module
# Reset singleton
module._brand_code_service = None
svc1 = module.get_brand_code_service()
svc2 = module.get_brand_code_service()
assert svc1 is svc2
# Clean up
module._brand_code_service = None
+249
View File
@@ -0,0 +1,249 @@
"""
Tests for CDG countdown handling.
Tests cover:
- _remove_countdown_from_lyrics() helper method
- CDG generation with countdown-padded LRC files
- Timestamp adjustment for instrumental audio sync
"""
import pytest
from karaoke_gen.lyrics_transcriber.output.cdg import CDGGenerator
class TestCDGCountdownRemoval:
"""Test the _remove_countdown_from_lyrics helper method."""
@pytest.fixture
def generator(self, tmp_path):
"""Create a CDGGenerator instance for testing."""
return CDGGenerator(output_dir=str(tmp_path))
def test_removes_countdown_segment(self, generator):
"""Test that countdown segment is removed from lyrics data."""
# Simulated LRC data with countdown at 0.1s and first word at 3.6s
lyrics_data = [
{"timestamp": 10, "text": "3... 2... 1..."}, # 0.1s = 10 centiseconds
{"timestamp": 360, "text": "IF"}, # 3.6s = 360 centiseconds
{"timestamp": 380, "text": "YOU"},
{"timestamp": 624, "text": "COULD"},
]
# Remove 3 second countdown padding
result = generator._remove_countdown_from_lyrics(lyrics_data, 3.0)
# Countdown segment should be removed (timestamp < 300cs)
assert len(result) == 3
assert result[0]["text"] == "IF"
# Timestamps should be shifted back by 300cs (3 seconds)
assert result[0]["timestamp"] == 60 # 360 - 300
assert result[1]["timestamp"] == 80 # 380 - 300
assert result[2]["timestamp"] == 324 # 624 - 300
def test_handles_multiple_countdown_segments(self, generator):
"""Test with multiple early segments (e.g., multi-part countdown)."""
lyrics_data = [
{"timestamp": 10, "text": "3"},
{"timestamp": 100, "text": "2"},
{"timestamp": 200, "text": "1"},
{"timestamp": 360, "text": "HELLO"},
{"timestamp": 500, "text": "WORLD"},
]
result = generator._remove_countdown_from_lyrics(lyrics_data, 3.0)
# All segments before 300cs should be removed
assert len(result) == 2
assert result[0]["text"] == "HELLO"
assert result[0]["timestamp"] == 60 # 360 - 300
assert result[1]["text"] == "WORLD"
assert result[1]["timestamp"] == 200 # 500 - 300
def test_handles_no_countdown_segment(self, generator):
"""Test when LRC has no countdown segment but flag is set."""
# All timestamps are after the countdown duration
lyrics_data = [
{"timestamp": 400, "text": "FIRST"},
{"timestamp": 500, "text": "SECOND"},
]
result = generator._remove_countdown_from_lyrics(lyrics_data, 3.0)
# Should still shift timestamps back
assert len(result) == 2
assert result[0]["timestamp"] == 100 # 400 - 300
assert result[1]["timestamp"] == 200 # 500 - 300
def test_handles_different_padding_durations(self, generator):
"""Test with different countdown padding durations."""
lyrics_data = [
{"timestamp": 50, "text": "COUNTDOWN"}, # 0.5s
{"timestamp": 560, "text": "WORD"}, # 5.6s
]
# With 5s padding (500cs)
result = generator._remove_countdown_from_lyrics(lyrics_data, 5.0)
assert len(result) == 1
assert result[0]["text"] == "WORD"
assert result[0]["timestamp"] == 60 # 560 - 500
def test_preserves_text_unchanged(self, generator):
"""Test that text content is preserved exactly."""
lyrics_data = [
{"timestamp": 10, "text": "3... 2... 1..."},
{"timestamp": 360, "text": "HELLO, WORLD!"},
{"timestamp": 500, "text": "IT'S ME"},
]
result = generator._remove_countdown_from_lyrics(lyrics_data, 3.0)
assert result[0]["text"] == "HELLO, WORLD!"
assert result[1]["text"] == "IT'S ME"
def test_empty_lyrics_list(self, generator):
"""Test with empty lyrics list."""
result = generator._remove_countdown_from_lyrics([], 3.0)
assert result == []
def test_all_segments_in_countdown_period(self, generator):
"""Test when all segments are within countdown period."""
lyrics_data = [
{"timestamp": 10, "text": "A"},
{"timestamp": 100, "text": "B"},
{"timestamp": 200, "text": "C"},
]
result = generator._remove_countdown_from_lyrics(lyrics_data, 3.0)
# All segments removed - should return empty after shifting
assert len(result) == 0
def test_segment_exactly_at_boundary(self, generator):
"""Test with segment timestamp exactly at countdown boundary."""
lyrics_data = [
{"timestamp": 299, "text": "ALMOST"}, # Just before 3s
{"timestamp": 300, "text": "EXACTLY"}, # Exactly at 3s
{"timestamp": 301, "text": "AFTER"}, # Just after 3s
]
result = generator._remove_countdown_from_lyrics(lyrics_data, 3.0)
# timestamp < countdown_cs means 299 is removed, 300 and 301 stay
assert len(result) == 2
assert result[0]["text"] == "EXACTLY"
assert result[0]["timestamp"] == 0 # 300 - 300
assert result[1]["text"] == "AFTER"
assert result[1]["timestamp"] == 1 # 301 - 300
def test_fractional_countdown_duration(self, generator):
"""Test with fractional countdown duration (e.g., 3.5 seconds)."""
lyrics_data = [
{"timestamp": 100, "text": "EARLY"}, # 1s
{"timestamp": 340, "text": "MIDDLE"}, # 3.4s - still in countdown
{"timestamp": 360, "text": "AFTER"}, # 3.6s - after 3.5s
]
result = generator._remove_countdown_from_lyrics(lyrics_data, 3.5)
# 350cs boundary
assert len(result) == 1
assert result[0]["text"] == "AFTER"
assert result[0]["timestamp"] == 10 # 360 - 350
class TestCDGGenerateFromLRCWithCountdown:
"""Test generate_cdg_from_lrc with countdown parameters."""
@pytest.fixture
def generator(self, tmp_path):
"""Create a CDGGenerator instance for testing."""
return CDGGenerator(output_dir=str(tmp_path))
def test_passes_countdown_params_to_parse(self, generator, tmp_path, mocker):
"""Test that countdown params are used when processing LRC."""
# Create a test LRC file with countdown
lrc_file = tmp_path / "test.lrc"
lrc_file.write_text("""[re:MidiCo]
[00:00.100]1:/3... 2... 1...
[00:03.600]1:/If
[00:03.800]1:you
""")
audio_file = tmp_path / "test.flac"
audio_file.write_text("fake audio")
# Mock the methods that would actually generate the CDG
mock_remove = mocker.patch.object(
generator, "_remove_countdown_from_lyrics", wraps=generator._remove_countdown_from_lyrics
)
mocker.patch.object(generator, "_validate_and_setup_font")
mocker.patch.object(generator, "_create_toml_file", return_value=str(tmp_path / "test.toml"))
mocker.patch.object(generator, "_compose_cdg")
mocker.patch.object(generator, "_find_cdg_zip", return_value=str(tmp_path / "test.zip"))
mocker.patch.object(generator, "_extract_cdg_files")
mocker.patch.object(generator, "_get_cdg_path", return_value=str(tmp_path / "test.cdg"))
mocker.patch.object(generator, "_get_mp3_path", return_value=str(tmp_path / "test.mp3"))
mocker.patch.object(generator, "_verify_output_files")
# Create mock files
(tmp_path / "test.toml").write_text("")
(tmp_path / "test.zip").write_text("")
(tmp_path / "test.cdg").write_text("")
(tmp_path / "test.mp3").write_text("")
# Call with countdown padding enabled
generator.generate_cdg_from_lrc(
lrc_file=str(lrc_file),
audio_file=str(audio_file),
title="Test",
artist="Artist",
cdg_styles={"font_path": None}, # Minimal styles to avoid validation
lrc_has_countdown_padding=True,
countdown_padding_seconds=3.0,
)
# Verify _remove_countdown_from_lyrics was called
mock_remove.assert_called_once()
# The method is called with (lyrics_data, countdown_padding_seconds)
call_args = mock_remove.call_args
# Check positional args - second arg should be countdown_padding_seconds
assert call_args[0][1] == 3.0
def test_skips_countdown_processing_when_disabled(self, generator, tmp_path, mocker):
"""Test that countdown processing is skipped when flag is False."""
lrc_file = tmp_path / "test.lrc"
lrc_file.write_text("[00:03.600]1:/Test\n")
audio_file = tmp_path / "test.flac"
audio_file.write_text("fake audio")
mock_remove = mocker.patch.object(generator, "_remove_countdown_from_lyrics")
mocker.patch.object(generator, "_validate_and_setup_font")
mocker.patch.object(generator, "_create_toml_file", return_value=str(tmp_path / "test.toml"))
mocker.patch.object(generator, "_compose_cdg")
mocker.patch.object(generator, "_find_cdg_zip", return_value=str(tmp_path / "test.zip"))
mocker.patch.object(generator, "_extract_cdg_files")
mocker.patch.object(generator, "_get_cdg_path", return_value=str(tmp_path / "test.cdg"))
mocker.patch.object(generator, "_get_mp3_path", return_value=str(tmp_path / "test.mp3"))
mocker.patch.object(generator, "_verify_output_files")
# Create mock files
(tmp_path / "test.toml").write_text("")
(tmp_path / "test.zip").write_text("")
(tmp_path / "test.cdg").write_text("")
(tmp_path / "test.mp3").write_text("")
# Call with countdown padding DISABLED (default)
generator.generate_cdg_from_lrc(
lrc_file=str(lrc_file),
audio_file=str(audio_file),
title="Test",
artist="Artist",
cdg_styles={"font_path": None},
lrc_has_countdown_padding=False,
)
# Verify _remove_countdown_from_lyrics was NOT called
mock_remove.assert_not_called()
+262
View File
@@ -0,0 +1,262 @@
"""
Tests for credit enforcement in job creation and failure flows.
Verifies that:
- Job creation requires credits (unless admin)
- Credits are deducted on job creation
- Credits are refunded on job failure
- New users get 2 welcome credits
"""
import pytest
from unittest.mock import Mock, MagicMock, patch
from datetime import datetime
# Mock Firestore before importing modules that use it
import sys
sys.modules.setdefault('google.cloud.firestore', MagicMock())
from backend.services.job_manager import JobManager
from backend.models.job import Job, JobCreate, JobStatus
from backend.exceptions import InsufficientCreditsError
@pytest.fixture
def mock_firestore_service():
"""Mock FirestoreService."""
with patch('backend.services.job_manager.FirestoreService') as mock:
service = Mock()
mock.return_value = service
yield service
@pytest.fixture
def mock_storage_service():
"""Mock StorageService."""
with patch('backend.services.job_manager.StorageService') as mock:
service = Mock()
mock.return_value = service
yield service
@pytest.fixture
def job_manager(mock_firestore_service, mock_storage_service):
"""Create JobManager with mocked dependencies."""
return JobManager()
@pytest.fixture
def mock_user_service():
"""Mock UserService via get_user_service (patched at source module)."""
service = Mock()
with patch('backend.services.user_service.get_user_service', return_value=service):
yield service
@pytest.fixture
def mock_rate_limit_service():
"""Mock rate limit service to always allow."""
service = Mock()
service.check_user_job_limit.return_value = (True, 5, "OK")
service.record_job_creation.return_value = None
with patch('backend.services.rate_limit_service.get_rate_limit_service', return_value=service):
yield service
class TestCreditCheckOnJobCreation:
"""Test credit checks during job creation."""
def test_job_creation_succeeds_with_credits(
self, job_manager, mock_firestore_service, mock_user_service, mock_rate_limit_service
):
"""Job creation succeeds when user has credits, and credit is deducted."""
mock_user_service.has_credits.return_value = True
mock_user_service.deduct_credit.return_value = (True, 4, "Credit deducted. 4 remaining")
job = job_manager.create_job(
JobCreate(artist="Test", title="Song", theme_id="nomad", user_email="user@test.com"),
is_admin=False,
)
assert job.status == JobStatus.PENDING
mock_user_service.has_credits.assert_called_once_with("user@test.com")
mock_user_service.deduct_credit.assert_called_once()
# Verify deduct_credit was called with user email, job_id, and reason
call_args = mock_user_service.deduct_credit.call_args
assert call_args[0][0] == "user@test.com"
def test_job_creation_fails_without_credits(
self, job_manager, mock_firestore_service, mock_user_service, mock_rate_limit_service
):
"""Job creation raises InsufficientCreditsError when user has no credits."""
mock_user_service.has_credits.return_value = False
mock_user_service.check_credits.return_value = 0
with pytest.raises(InsufficientCreditsError) as exc_info:
job_manager.create_job(
JobCreate(artist="Test", title="Song", theme_id="nomad", user_email="broke@test.com"),
is_admin=False,
)
assert exc_info.value.credits_available == 0
assert exc_info.value.credits_required == 1
# Verify Firestore create_job was NOT called (fail-fast before job creation)
mock_firestore_service.create_job.assert_not_called()
def test_admin_bypasses_credit_check(
self, job_manager, mock_firestore_service, mock_user_service, mock_rate_limit_service
):
"""Admin users bypass credit check entirely."""
job = job_manager.create_job(
JobCreate(artist="Test", title="Song", theme_id="nomad", user_email="admin@nomadkaraoke.com"),
is_admin=True,
)
assert job.status == JobStatus.PENDING
mock_user_service.has_credits.assert_not_called()
mock_user_service.deduct_credit.assert_not_called()
def test_job_without_user_email_skips_credit_check(
self, job_manager, mock_firestore_service, mock_user_service, mock_rate_limit_service
):
"""Jobs without user_email (e.g., API token auth) skip credit check."""
job = job_manager.create_job(
JobCreate(artist="Test", title="Song", theme_id="nomad"),
is_admin=False,
)
assert job.status == JobStatus.PENDING
mock_user_service.has_credits.assert_not_called()
def test_deduction_failure_deletes_job_and_raises(
self, job_manager, mock_firestore_service, mock_user_service, mock_rate_limit_service
):
"""If credit deduction fails after job creation, the job is deleted."""
mock_user_service.has_credits.return_value = True
mock_user_service.deduct_credit.return_value = (False, 0, "Insufficient credits")
with pytest.raises(InsufficientCreditsError):
job_manager.create_job(
JobCreate(artist="Test", title="Song", theme_id="nomad", user_email="user@test.com"),
is_admin=False,
)
# Job was created then deleted
mock_firestore_service.create_job.assert_called_once()
mock_firestore_service.delete_job.assert_called_once()
class TestCreditRefundOnJobFailure:
"""Test credit refund when jobs fail."""
def test_credit_refunded_on_job_failure(
self, job_manager, mock_firestore_service, mock_user_service
):
"""Credit is refunded when a non-admin job fails."""
mock_job = Mock(spec=Job)
mock_job.user_email = "user@test.com"
mock_firestore_service.get_job.return_value = mock_job
mock_user_service.refund_credit.return_value = (True, 1, "Refunded")
with patch('backend.services.auth_service.is_admin_email', return_value=False):
result = job_manager.fail_job("job123", "Processing error")
assert result is True
mock_user_service.refund_credit.assert_called_once_with(
"user@test.com", "job123", reason="job_failed"
)
def test_credit_refund_skipped_for_admin_jobs(
self, job_manager, mock_firestore_service, mock_user_service
):
"""Credit refund is skipped for admin-owned jobs."""
mock_job = Mock(spec=Job)
mock_job.user_email = "admin@nomadkaraoke.com"
mock_firestore_service.get_job.return_value = mock_job
with patch('backend.services.auth_service.is_admin_email', return_value=True):
result = job_manager.fail_job("job123", "Processing error")
assert result is True
mock_user_service.refund_credit.assert_not_called()
def test_refund_failure_does_not_fail_job_marking(
self, job_manager, mock_firestore_service, mock_user_service
):
"""If refund fails, the job is still marked as failed."""
mock_job = Mock(spec=Job)
mock_job.user_email = "user@test.com"
mock_firestore_service.get_job.return_value = mock_job
mock_user_service.refund_credit.side_effect = Exception("Firestore error")
with patch('backend.services.auth_service.is_admin_email', return_value=False):
result = job_manager.fail_job("job123", "Processing error")
assert result is True # Job still marked as failed
def test_no_refund_when_job_has_no_user_email(
self, job_manager, mock_firestore_service, mock_user_service
):
"""No refund attempted when job has no user_email."""
mock_job = Mock(spec=Job)
mock_job.user_email = None
mock_firestore_service.get_job.return_value = mock_job
result = job_manager.fail_job("job123", "Processing error")
assert result is True
mock_user_service.refund_credit.assert_not_called()
class TestCreditAndRateLimitOrdering:
"""Test interaction between rate limit and credit checks."""
def test_rate_limited_user_not_charged_credit(
self, job_manager, mock_firestore_service, mock_user_service
):
"""User who is rate limited should NOT have a credit deducted."""
mock_rate_limit_service = Mock()
mock_rate_limit_service.check_user_job_limit.return_value = (False, 0, "Rate limit exceeded")
mock_rate_limit_service.get_user_job_count_today.return_value = 5
with patch('backend.services.rate_limit_service.get_rate_limit_service', return_value=mock_rate_limit_service):
from backend.exceptions import RateLimitExceededError
with pytest.raises(RateLimitExceededError):
job_manager.create_job(
JobCreate(artist="Test", title="Song", theme_id="nomad", user_email="user@test.com"),
is_admin=False,
)
# Credit check should never be called because rate limit fails first
mock_user_service.has_credits.assert_not_called()
mock_user_service.deduct_credit.assert_not_called()
class TestWelcomeCredits:
"""Test new user welcome credits."""
def test_new_user_gets_2_welcome_credits(self):
"""Verify NEW_USER_FREE_CREDITS is set to 2."""
from backend.services.user_service import UserService
assert UserService.NEW_USER_FREE_CREDITS == 2
class TestInsufficientCreditsError:
"""Test the InsufficientCreditsError exception."""
def test_exception_attributes(self):
"""Verify exception stores all attributes."""
err = InsufficientCreditsError(
message="No credits",
credits_available=0,
credits_required=1,
)
assert err.message == "No credits"
assert err.credits_available == 0
assert err.credits_required == 1
assert str(err) == "No credits"
def test_exception_defaults(self):
"""Verify default values."""
err = InsufficientCreditsError(message="Out of credits")
assert err.credits_available == 0
assert err.credits_required == 1
File diff suppressed because it is too large. Load diff
+17 -14
View File
@@ -777,21 +777,22 @@ class TestVideoWorkerDistribution:
async def test_handle_native_distribution_dropbox_not_configured(self):
"""Test Dropbox upload skipped when service not configured."""
from backend.workers.video_worker import _handle_native_distribution
mock_job = MagicMock()
mock_job.dropbox_path = "/test/path"
mock_job.brand_prefix = "TEST"
mock_job.gdrive_folder_id = None
mock_job.is_private = False
mock_job.artist = "Test"
mock_job.title = "Song"
mock_job.state_data = {}
mock_job_log = MagicMock()
mock_job_manager = MagicMock()
mock_dropbox = MagicMock()
mock_dropbox.is_configured = False
with patch('backend.services.dropbox_service.get_dropbox_service', return_value=mock_dropbox):
await _handle_native_distribution(
job_id="test-123",
@@ -801,7 +802,7 @@ class TestVideoWorkerDistribution:
temp_dir="/tmp/test",
result={},
)
# Should log warning about not configured
mock_job_log.warning.assert_called()
@@ -809,21 +810,22 @@ class TestVideoWorkerDistribution:
async def test_handle_native_distribution_gdrive_not_configured(self):
"""Test Google Drive upload skipped when service not configured."""
from backend.workers.video_worker import _handle_native_distribution
mock_job = MagicMock()
mock_job.dropbox_path = None
mock_job.brand_prefix = None
mock_job.gdrive_folder_id = "test_folder_id"
mock_job.is_private = False
mock_job.artist = "Test"
mock_job.title = "Song"
mock_job.state_data = {}
mock_job_log = MagicMock()
mock_job_manager = MagicMock()
mock_gdrive = MagicMock()
mock_gdrive.is_configured = False
with patch('backend.services.gdrive_service.get_gdrive_service', return_value=mock_gdrive):
await _handle_native_distribution(
job_id="test-123",
@@ -833,7 +835,7 @@ class TestVideoWorkerDistribution:
temp_dir="/tmp/test",
result={},
)
# Should log warning about not configured
mock_job_log.warning.assert_called()
@@ -841,18 +843,19 @@ class TestVideoWorkerDistribution:
async def test_handle_native_distribution_handles_import_error(self):
"""Test that import errors for services are handled gracefully."""
from backend.workers.video_worker import _handle_native_distribution
mock_job = MagicMock()
mock_job.dropbox_path = "/test/path"
mock_job.brand_prefix = "TEST"
mock_job.gdrive_folder_id = None
mock_job.is_private = False
mock_job.artist = "Test"
mock_job.title = "Song"
mock_job.state_data = {}
mock_job_log = MagicMock()
mock_job_manager = MagicMock()
# Simulate import error
with patch('backend.services.dropbox_service.get_dropbox_service', side_effect=ImportError("No module")):
await _handle_native_distribution(
@@ -863,6 +866,6 @@ class TestVideoWorkerDistribution:
temp_dir="/tmp/test",
result={},
)
# Should log warning about import error
mock_job_log.warning.assert_called()
+197
View File
@@ -630,3 +630,200 @@ class TestGetFlacfetchClient:
assert client1 is not client2
assert client2.base_url == "http://other:8080"
class TestFlacfetchRetryLogic:
"""Test retry logic for flacfetch download methods."""
@pytest.mark.asyncio
async def test_download_by_id_retries_on_request_error(self):
"""Test download_by_id retries on network errors (RequestError)."""
# Mock config with retry settings
mock_settings = Mock()
mock_settings.flacfetch_retry_max_attempts = 3
mock_settings.flacfetch_retry_min_wait = 0.01 # Fast retries for testing
mock_settings.flacfetch_retry_max_wait = 0.02
with patch("backend.services.flacfetch_client.get_settings", return_value=mock_settings):
client = FlacfetchClient(
base_url="http://test:8080",
api_key="test-key",
)
with patch("httpx.AsyncClient") as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
# Fail twice with network error, then succeed
mock_response = Mock()
mock_response.json.return_value = {"download_id": "test123"}
mock_response.raise_for_status = Mock()
mock_client.post.side_effect = [
httpx.RequestError("Connection timeout"),
httpx.RequestError("Connection refused"),
mock_response, # Success on third attempt
]
result = await client.download_by_id("YouTube", "abc123")
assert result == "test123"
assert mock_client.post.call_count == 3
@pytest.mark.asyncio
async def test_download_by_id_no_retry_on_4xx(self):
"""Test download_by_id does NOT retry on 4xx client errors."""
mock_settings = Mock()
mock_settings.flacfetch_retry_max_attempts = 3
mock_settings.flacfetch_retry_min_wait = 0.01
mock_settings.flacfetch_retry_max_wait = 0.02
with patch("backend.services.flacfetch_client.get_settings", return_value=mock_settings):
client = FlacfetchClient(
base_url="http://test:8080",
api_key="test-key",
)
with patch("httpx.AsyncClient") as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
# 404 error
mock_response = Mock()
mock_response.status_code = 404
mock_response.text = "Not found"
def raise_status_error():
raise httpx.HTTPStatusError(
"Not found",
request=Mock(),
response=mock_response
)
mock_response.raise_for_status.side_effect = raise_status_error
mock_client.post.return_value = mock_response
with pytest.raises(FlacfetchServiceError, match="404"):
await client.download_by_id("YouTube", "abc123")
# Should only attempt once (no retries for 4xx)
assert mock_client.post.call_count == 1
@pytest.mark.asyncio
async def test_download_by_id_retries_on_5xx(self):
"""Test download_by_id retries on 5xx server errors."""
mock_settings = Mock()
mock_settings.flacfetch_retry_max_attempts = 3
mock_settings.flacfetch_retry_min_wait = 0.01
mock_settings.flacfetch_retry_max_wait = 0.02
with patch("backend.services.flacfetch_client.get_settings", return_value=mock_settings):
client = FlacfetchClient(
base_url="http://test:8080",
api_key="test-key",
)
with patch("httpx.AsyncClient") as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
# 503 error twice, then success
mock_503_response = Mock()
mock_503_response.status_code = 503
mock_503_response.text = "Service Unavailable"
def raise_503():
raise httpx.HTTPStatusError(
"Service Unavailable",
request=Mock(),
response=mock_503_response
)
mock_success_response = Mock()
mock_success_response.json.return_value = {"download_id": "test456"}
mock_success_response.raise_for_status = Mock()
mock_503_response_1 = Mock()
mock_503_response_1.status_code = 503
mock_503_response_1.text = "Service Unavailable"
mock_503_response_1.raise_for_status.side_effect = raise_503
mock_503_response_2 = Mock()
mock_503_response_2.status_code = 503
mock_503_response_2.text = "Service Unavailable"
mock_503_response_2.raise_for_status.side_effect = raise_503
mock_client.post.side_effect = [
mock_503_response_1,
mock_503_response_2,
mock_success_response,
]
result = await client.download_by_id("YouTube", "abc123")
assert result == "test456"
assert mock_client.post.call_count == 3
@pytest.mark.asyncio
async def test_download_by_id_fails_after_max_retries(self):
"""Test download_by_id fails with clear message after all retries exhausted."""
mock_settings = Mock()
mock_settings.flacfetch_retry_max_attempts = 3
mock_settings.flacfetch_retry_min_wait = 0.01
mock_settings.flacfetch_retry_max_wait = 0.02
with patch("backend.services.flacfetch_client.get_settings", return_value=mock_settings):
client = FlacfetchClient(
base_url="http://test:8080",
api_key="test-key",
)
with patch("httpx.AsyncClient") as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
# All attempts fail
mock_client.post.side_effect = httpx.RequestError("Connection timeout")
with pytest.raises(FlacfetchServiceError, match="request failed"):
await client.download_by_id("YouTube", "abc123")
# Should have attempted max times
assert mock_client.post.call_count == 3
@pytest.mark.asyncio
async def test_download_retries_same_as_download_by_id(self):
"""Test download() method has same retry behavior as download_by_id()."""
mock_settings = Mock()
mock_settings.flacfetch_retry_max_attempts = 3
mock_settings.flacfetch_retry_min_wait = 0.01
mock_settings.flacfetch_retry_max_wait = 0.02
with patch("backend.services.flacfetch_client.get_settings", return_value=mock_settings):
client = FlacfetchClient(
base_url="http://test:8080",
api_key="test-key",
)
with patch("httpx.AsyncClient") as mock_client_cls:
mock_client = AsyncMock()
mock_client_cls.return_value.__aenter__.return_value = mock_client
# Fail twice, then succeed
mock_response = Mock()
mock_response.json.return_value = {"download_id": "test789"}
mock_response.raise_for_status = Mock()
mock_client.post.side_effect = [
httpx.RequestError("Network error"),
httpx.RequestError("Network error"),
mock_response,
]
result = await client.download(
search_id="search123",
result_index=0,
)
assert result == "test789"
assert mock_client.post.call_count == 3
+229
View File
@@ -9,7 +9,9 @@ to avoid dependency issues in CI/CD environments.
"""
import pytest
import re
from pathlib import Path
from packaging.version import Version
# Copy the find_file function here to test it in isolation
@@ -227,3 +229,230 @@ class TestInstrumentalSelectionRegression:
assert result is not None
name = result.name.lower()
assert "clean" in name, f"Expected clean instrumental but found: {result.name}"
class TestWheelFilenameFiltering:
"""Test wheel filename filtering to prevent installation of invalid wheels.
Bug: The ensure_latest_wheel() function downloads all wheels including
karaoke_gen-current.whl, which is not a valid PEP 427 wheel filename
(missing version and platform tags). When pip tries to install it,
it fails with: "Invalid wheel filename (wrong number of parts)"
Fix: Filter out wheels containing '-current' before selecting which to install.
"""
def filter_wheels(self, wheels):
"""Replicate the filtering logic from ensure_latest_wheel()."""
return [w for w in wheels if '-current' not in w]
def test_filters_out_current_wheel(self):
"""Test that karaoke_gen-current.whl is filtered out."""
wheels = [
"/tmp/karaoke_gen-0.116.0-py3-none-any.whl",
"/tmp/karaoke_gen-current.whl",
"/tmp/karaoke_gen-0.115.0-py3-none-any.whl",
]
filtered = self.filter_wheels(wheels)
assert len(filtered) == 2
assert "/tmp/karaoke_gen-current.whl" not in filtered
assert "/tmp/karaoke_gen-0.116.0-py3-none-any.whl" in filtered
assert "/tmp/karaoke_gen-0.115.0-py3-none-any.whl" in filtered
def test_returns_empty_when_only_current_wheel(self):
"""Test that filtering returns empty list when only current wheel exists."""
wheels = ["/tmp/karaoke_gen-current.whl"]
filtered = self.filter_wheels(wheels)
assert filtered == []
def test_selects_latest_version_after_filtering(self):
"""Test that the latest version is selected after filtering out current.
Note: This test verifies filtering but uses simple sorting.
See TestWheelVersionSorting for comprehensive semantic version sorting tests.
"""
wheels = [
"/tmp/karaoke_gen-0.114.0-py3-none-any.whl",
"/tmp/karaoke_gen-current.whl",
"/tmp/karaoke_gen-0.116.0-py3-none-any.whl",
"/tmp/karaoke_gen-0.115.0-py3-none-any.whl",
]
filtered = self.filter_wheels(wheels)
# Verify current wheel was filtered out
assert len(filtered) == 3
assert all("-current" not in w for w in filtered)
def test_no_filtering_when_no_current_wheel(self):
"""Test that valid wheels pass through when no current wheel exists."""
wheels = [
"/tmp/karaoke_gen-0.116.0-py3-none-any.whl",
"/tmp/karaoke_gen-0.115.0-py3-none-any.whl",
]
filtered = self.filter_wheels(wheels)
assert len(filtered) == 2
assert filtered == wheels
def test_filters_current_in_different_paths(self):
"""Test that current wheel is filtered regardless of path."""
wheels = [
"/different/path/karaoke_gen-current.whl",
"/tmp/karaoke_gen-0.116.0-py3-none-any.whl",
]
filtered = self.filter_wheels(wheels)
assert len(filtered) == 1
assert "/tmp/karaoke_gen-0.116.0-py3-none-any.whl" in filtered
class TestWheelVersionSorting:
"""Test semantic version sorting for wheel selection.
Bug: The ensure_latest_wheel() function uses sorted(wheels)[-1] which does
alphabetical sorting, not semantic version sorting. This causes version 0.99.9
to be selected over 0.116.1 because string comparison puts '9' > '1'.
Fix: Extract version from filename using regex and sort by packaging.version.Version
"""
def extract_version(self, wheel_path):
"""Replicate the version extraction logic from ensure_latest_wheel()."""
match = re.search(r'karaoke_gen-([0-9.]+)-', wheel_path)
if match:
return Version(match.group(1))
return Version("0.0.0") # Fallback for unparseable filenames
def select_latest_wheel(self, wheels):
"""Replicate the full selection logic from ensure_latest_wheel()."""
# Filter out -current wheels
filtered = [w for w in wheels if '-current' not in w]
if not filtered:
return None
# Sort by semantic version
filtered.sort(key=self.extract_version, reverse=True)
return filtered[0]
def test_semantic_version_sorting_regression(self):
"""Test the exact bug: 0.99.9 should NOT be selected over 0.116.1"""
wheels = [
"/tmp/karaoke_gen-0.99.9-py3-none-any.whl",
"/tmp/karaoke_gen-0.116.1-py3-none-any.whl",
]
latest = self.select_latest_wheel(wheels)
# Semantic version: 0.116.1 > 0.99.9
assert "0.116.1" in latest
assert "0.99.9" not in latest
def test_alphabetical_would_fail(self):
"""Demonstrate that alphabetical sorting gives wrong result."""
wheels = [
"/tmp/karaoke_gen-0.99.9-py3-none-any.whl",
"/tmp/karaoke_gen-0.116.1-py3-none-any.whl",
]
# This is the OLD buggy behavior (alphabetical)
alphabetical_latest = sorted(wheels)[-1]
# Alphabetical puts 0.99.9 last (WRONG!)
assert "0.99.9" in alphabetical_latest
def test_version_extraction(self):
"""Test that version is correctly extracted from wheel filename."""
test_cases = [
("/tmp/karaoke_gen-0.116.1-py3-none-any.whl", "0.116.1"),
("/tmp/karaoke_gen-0.99.9-py3-none-any.whl", "0.99.9"),
("/tmp/karaoke_gen-1.0.0-py3-none-any.whl", "1.0.0"),
("/tmp/karaoke_gen-0.1.2-py3-none-any.whl", "0.1.2"),
]
for wheel_path, expected_version in test_cases:
version = self.extract_version(wheel_path)
assert str(version) == expected_version
def test_version_extraction_fallback(self):
"""Test that unparseable filenames get fallback version."""
invalid_wheels = [
"/tmp/karaoke_gen-current.whl",
"/tmp/karaoke_gen-latest.whl",
"/tmp/invalid-filename.whl",
]
for wheel in invalid_wheels:
version = self.extract_version(wheel)
assert version == Version("0.0.0")
def test_complex_version_sorting(self):
"""Test sorting with multiple versions including patch, minor, major."""
wheels = [
"/tmp/karaoke_gen-0.99.9-py3-none-any.whl",
"/tmp/karaoke_gen-0.116.0-py3-none-any.whl",
"/tmp/karaoke_gen-0.116.1-py3-none-any.whl",
"/tmp/karaoke_gen-1.0.0-py3-none-any.whl",
"/tmp/karaoke_gen-0.115.0-py3-none-any.whl",
"/tmp/karaoke_gen-0.1.0-py3-none-any.whl",
]
latest = self.select_latest_wheel(wheels)
# 1.0.0 is the highest version
assert "1.0.0" in latest
def test_version_sorting_with_current_wheel(self):
"""Test that current wheel is filtered before version sorting."""
wheels = [
"/tmp/karaoke_gen-0.99.9-py3-none-any.whl",
"/tmp/karaoke_gen-current.whl",
"/tmp/karaoke_gen-0.116.1-py3-none-any.whl",
]
latest = self.select_latest_wheel(wheels)
# Current is filtered out, 0.116.1 is selected
assert "0.116.1" in latest
assert "current" not in latest
def test_patch_version_comparison(self):
"""Test that patch versions are compared correctly."""
wheels = [
"/tmp/karaoke_gen-0.116.0-py3-none-any.whl",
"/tmp/karaoke_gen-0.116.1-py3-none-any.whl",
"/tmp/karaoke_gen-0.116.2-py3-none-any.whl",
]
latest = self.select_latest_wheel(wheels)
assert "0.116.2" in latest
def test_minor_version_comparison(self):
"""Test that minor versions are compared correctly."""
wheels = [
"/tmp/karaoke_gen-0.115.0-py3-none-any.whl",
"/tmp/karaoke_gen-0.116.0-py3-none-any.whl",
"/tmp/karaoke_gen-0.200.0-py3-none-any.whl",
]
latest = self.select_latest_wheel(wheels)
assert "0.200.0" in latest
def test_empty_wheels_list(self):
"""Test that empty list returns None."""
wheels = []
latest = self.select_latest_wheel(wheels)
assert latest is None
def test_only_current_wheel_returns_none(self):
"""Test that list with only current wheel returns None."""
wheels = ["/tmp/karaoke_gen-current.whl"]
latest = self.select_latest_wheel(wheels)
assert latest is None
@@ -269,10 +269,12 @@ class TestUrlJobWorkerSequencing:
async def test_parallel_worker_triggers_both_workers(self):
"""
_trigger_workers_parallel should trigger both workers (for uploaded files).
Note: This function now lives in jobs.py (direct job creation endpoint).
Most code paths use job_manager.start_job_processing() instead.
"""
from backend.api.routes.file_upload import _trigger_workers_parallel
from backend.api.routes.jobs import _trigger_workers_parallel
with patch('backend.api.routes.file_upload.worker_service') as mock_ws:
with patch('backend.api.routes.jobs.worker_service') as mock_ws:
mock_ws.trigger_audio_worker = AsyncMock()
mock_ws.trigger_lyrics_worker = AsyncMock()
@@ -379,7 +381,7 @@ class TestAudioSearchCacheIndependence:
def test_download_code_uses_youtube_download_service(self):
"""
Verify that _download_and_start_processing uses YouTubeDownloadService.
Verify that _download_audio_and_trigger_workers uses YouTubeDownloadService.
After the consolidation refactor (2025-01), all YouTube downloads go through
YouTubeDownloadService instead of using download_from_url directly. This
@@ -388,14 +390,14 @@ class TestAudioSearchCacheIndependence:
from backend.api.routes import audio_search
import inspect
# Get the source of _download_and_start_processing
source = inspect.getsource(audio_search._download_and_start_processing)
# Get the source of _download_audio_and_trigger_workers
source = inspect.getsource(audio_search._download_audio_and_trigger_workers)
# Verify YouTubeDownloadService is used for YouTube
assert "source_name == 'YouTube'" in source, \
"_download_and_start_processing must check for YouTube source"
"_download_audio_and_trigger_workers must check for YouTube source"
assert 'youtube_service' in source or 'get_youtube_download_service' in source, \
"_download_and_start_processing must use YouTubeDownloadService for YouTube"
"_download_audio_and_trigger_workers must use YouTubeDownloadService for YouTube"
def test_youtube_download_service_import_exists(self):
"""
+262
View File
@@ -0,0 +1,262 @@
"""
Tests for the job health service.
Tests consistency checks and worker validation functions.
"""
import pytest
from datetime import datetime, UTC
from backend.models.job import Job, JobStatus
from backend.services.job_health_service import (
check_job_consistency,
check_job_consistency_detailed,
validate_worker_can_run,
get_worker_valid_statuses,
)
def create_test_job(
job_id: str = "test-job-123",
status: JobStatus = JobStatus.PENDING,
state_data: dict = None,
input_media_gcs_path: str = None,
) -> Job:
"""Helper to create test Job objects."""
return Job(
job_id=job_id,
artist="Test Artist",
title="Test Song",
status=status,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
state_data=state_data or {},
input_media_gcs_path=input_media_gcs_path,
)
class TestCheckJobConsistency:
"""Tests for check_job_consistency function."""
def test_healthy_job_has_no_issues(self):
"""Test that a properly configured job has no issues."""
job = create_test_job(
status=JobStatus.DOWNLOADING,
state_data={},
)
issues = check_job_consistency(job)
assert issues == []
def test_detects_audio_complete_with_pending_status(self):
"""Test detection of audio_complete flag with pending status."""
job = create_test_job(
status=JobStatus.PENDING,
state_data={'audio_complete': True},
)
issues = check_job_consistency(job)
assert len(issues) == 1
assert "audio_complete_status_mismatch" in issues[0]
assert "audio_complete=True but status=pending" in issues[0]
def test_detects_lyrics_complete_with_pending_status(self):
"""Test detection of lyrics_complete flag with pending status."""
job = create_test_job(
status=JobStatus.PENDING,
state_data={'lyrics_complete': True},
)
issues = check_job_consistency(job)
assert len(issues) == 1
assert "lyrics_complete_status_mismatch" in issues[0]
def test_detects_screens_complete_with_wrong_status(self):
"""Test detection of screens complete but status not in post-screens states."""
job = create_test_job(
status=JobStatus.DOWNLOADING,
state_data={'screens_progress': {'stage': 'complete'}},
)
issues = check_job_consistency(job)
assert len(issues) == 1
assert "screens_complete_status_mismatch" in issues[0]
def test_screens_complete_valid_with_awaiting_review(self):
"""Test that screens complete is valid with AWAITING_REVIEW status."""
job = create_test_job(
status=JobStatus.AWAITING_REVIEW,
state_data={'screens_progress': {'stage': 'complete'}},
)
issues = check_job_consistency(job)
assert issues == []
def test_screens_complete_valid_with_complete_status(self):
"""Test that screens complete is valid with COMPLETE status."""
job = create_test_job(
status=JobStatus.COMPLETE,
state_data={'screens_progress': {'stage': 'complete'}},
)
issues = check_job_consistency(job)
assert issues == []
def test_detects_pending_with_input_media_path(self):
"""Test detection of job stuck at pending with input_media_gcs_path."""
job = create_test_job(
status=JobStatus.PENDING,
input_media_gcs_path="gs://bucket/audio.mp3",
)
issues = check_job_consistency(job)
assert len(issues) == 1
assert "pending_with_input_media" in issues[0]
def test_multiple_issues_detected(self):
"""Test that multiple issues are detected simultaneously."""
job = create_test_job(
status=JobStatus.PENDING,
state_data={
'audio_complete': True,
'lyrics_complete': True,
},
input_media_gcs_path="gs://bucket/audio.mp3",
)
issues = check_job_consistency(job)
assert len(issues) == 3
def test_handles_invalid_status_string(self):
"""Test handling of invalid status string."""
job = create_test_job(status=JobStatus.PENDING)
job.status = "invalid_status" # type: ignore
issues = check_job_consistency(job)
assert len(issues) == 1
assert "invalid_status" in issues[0]
def test_handles_empty_state_data(self):
"""Test handling of empty or None state_data."""
job = create_test_job(
status=JobStatus.DOWNLOADING,
state_data=None,
)
issues = check_job_consistency(job)
assert issues == []
class TestCheckJobConsistencyDetailed:
"""Tests for check_job_consistency_detailed function."""
def test_returns_detailed_info(self):
"""Test that detailed info is returned."""
job = create_test_job(
status=JobStatus.PENDING,
state_data={'audio_complete': True},
input_media_gcs_path="gs://bucket/audio.mp3",
)
result = check_job_consistency_detailed(job)
assert result['job_id'] == "test-job-123"
assert result['status'] == "pending"
assert len(result['issues']) == 2
assert result['state_data_summary']['audio_complete'] is True
assert result['state_data_summary']['lyrics_complete'] is False
assert result['state_data_summary']['has_input_media_gcs_path'] is True
assert result['is_healthy'] is False
def test_healthy_job_returns_is_healthy_true(self):
"""Test that healthy job returns is_healthy=True."""
job = create_test_job(status=JobStatus.DOWNLOADING)
result = check_job_consistency_detailed(job)
assert result['is_healthy'] is True
assert result['issues'] == []
class TestValidateWorkerCanRun:
"""Tests for validate_worker_can_run function."""
def test_audio_worker_valid_with_downloading_status(self):
"""Test audio worker is valid with DOWNLOADING status."""
job = create_test_job(status=JobStatus.DOWNLOADING)
error = validate_worker_can_run("audio_worker", job)
assert error is None
def test_audio_worker_invalid_with_pending_status(self):
"""Test audio worker is invalid with PENDING status."""
job = create_test_job(status=JobStatus.PENDING)
error = validate_worker_can_run("audio_worker", job)
assert error is not None
assert "audio_worker called but job status is pending" in error
def test_lyrics_worker_valid_with_downloading_status(self):
"""Test lyrics worker is valid with DOWNLOADING status."""
job = create_test_job(status=JobStatus.DOWNLOADING)
error = validate_worker_can_run("lyrics_worker", job)
assert error is None
def test_screens_worker_valid_with_audio_complete_status(self):
"""Test screens worker is valid with AUDIO_COMPLETE status."""
job = create_test_job(status=JobStatus.AUDIO_COMPLETE)
error = validate_worker_can_run("screens_worker", job)
assert error is None
def test_screens_worker_valid_with_lyrics_complete_status(self):
"""Test screens worker is valid with LYRICS_COMPLETE status."""
job = create_test_job(status=JobStatus.LYRICS_COMPLETE)
error = validate_worker_can_run("screens_worker", job)
assert error is None
def test_screens_worker_invalid_with_awaiting_review_status(self):
"""Test screens worker is invalid with AWAITING_REVIEW status (too late)."""
job = create_test_job(status=JobStatus.AWAITING_REVIEW)
error = validate_worker_can_run("screens_worker", job)
assert error is not None
assert "screens_worker called but job status is awaiting_review" in error
def test_video_worker_valid_with_instrumental_selected_status(self):
"""Test video worker is valid with INSTRUMENTAL_SELECTED status."""
job = create_test_job(status=JobStatus.INSTRUMENTAL_SELECTED)
error = validate_worker_can_run("video_worker", job)
assert error is None
def test_video_worker_invalid_with_pending_status(self):
"""Test video worker is invalid with PENDING status."""
job = create_test_job(status=JobStatus.PENDING)
error = validate_worker_can_run("video_worker", job)
assert error is not None
assert "video_worker called but job status is pending" in error
def test_render_video_worker_valid_with_review_complete_status(self):
"""Test render video worker is valid with REVIEW_COMPLETE status."""
job = create_test_job(status=JobStatus.REVIEW_COMPLETE)
error = validate_worker_can_run("render_video_worker", job)
assert error is None
def test_unknown_worker_returns_none(self):
"""Test that unknown worker names return None (no validation)."""
job = create_test_job(status=JobStatus.PENDING)
error = validate_worker_can_run("unknown_worker", job)
assert error is None
def test_handles_invalid_status_string(self):
"""Test handling of invalid status string."""
job = create_test_job(status=JobStatus.PENDING)
job.status = "invalid_status" # type: ignore
error = validate_worker_can_run("audio_worker", job)
assert error is not None
assert "Invalid job status" in error
class TestGetWorkerValidStatuses:
"""Tests for get_worker_valid_statuses function."""
def test_returns_dict_with_all_workers(self):
"""Test that all known workers are included."""
statuses = get_worker_valid_statuses()
assert "audio_worker" in statuses
assert "lyrics_worker" in statuses
assert "screens_worker" in statuses
assert "video_worker" in statuses
assert "render_video_worker" in statuses
def test_audio_worker_includes_downloading(self):
"""Test that audio worker includes DOWNLOADING status."""
statuses = get_worker_valid_statuses()
assert JobStatus.DOWNLOADING in statuses["audio_worker"]
def test_lyrics_worker_includes_downloading(self):
"""Test that lyrics worker includes DOWNLOADING status."""
statuses = get_worker_valid_statuses()
assert JobStatus.DOWNLOADING in statuses["lyrics_worker"]
+285 -2
View File
@@ -5,7 +5,7 @@ Tests the job lifecycle management without requiring actual Firestore connection
Uses mocking to isolate the business logic.
"""
import pytest
from unittest.mock import Mock, MagicMock, patch
from unittest.mock import Mock, MagicMock, patch, AsyncMock
from datetime import datetime, UTC
# Mock Firestore before importing JobManager
@@ -14,6 +14,7 @@ sys.modules['google.cloud.firestore'] = MagicMock()
from backend.services.job_manager import JobManager
from backend.models.job import Job, JobCreate, JobStatus
from backend.exceptions import InvalidStateTransitionError
@pytest.fixture
@@ -427,7 +428,7 @@ class TestMadeForYouFieldMapping:
brand_prefix="NOMAD",
)
job = job_manager.create_job(job_create)
job = job_manager.create_job(job_create, is_admin=True)
# Verify all made-for-you fields
assert job.made_for_you is True
@@ -568,6 +569,288 @@ class TestDeleteStateDataKeys:
assert result == []
# =============================================================================
# InvalidStateTransitionError Tests
# =============================================================================
class TestInvalidStateTransitionError:
"""Tests for InvalidStateTransitionError exception."""
def test_exception_stores_all_attributes(self):
"""Test exception stores job_id, from_status, to_status, and valid_transitions."""
error = InvalidStateTransitionError(
message="Test error",
job_id="test-123",
from_status="pending",
to_status="generating_screens",
valid_transitions=["downloading", "searching_audio", "failed"]
)
assert error.message == "Test error"
assert error.job_id == "test-123"
assert error.from_status == "pending"
assert error.to_status == "generating_screens"
assert error.valid_transitions == ["downloading", "searching_audio", "failed"]
assert str(error) == "Test error"
def test_exception_defaults(self):
"""Test exception has sensible defaults."""
error = InvalidStateTransitionError(message="Minimal error")
assert error.message == "Minimal error"
assert error.job_id == ""
assert error.from_status == ""
assert error.to_status == ""
assert error.valid_transitions == []
# =============================================================================
# State Transition Validation Tests (raise_on_invalid behavior)
# =============================================================================
class TestValidateStateTransitionRaisesBehavior:
"""Tests for validate_state_transition with raise_on_invalid parameter."""
def test_invalid_transition_raises_by_default(self, job_manager, mock_firestore_service):
"""Test that invalid transition raises InvalidStateTransitionError by default."""
mock_firestore_service.get_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
# PENDING cannot transition directly to GENERATING_SCREENS
with pytest.raises(InvalidStateTransitionError) as exc_info:
job_manager.validate_state_transition("test123", JobStatus.GENERATING_SCREENS)
error = exc_info.value
assert error.job_id == "test123"
assert error.from_status == JobStatus.PENDING
assert error.to_status == "generating_screens"
assert "downloading" in error.valid_transitions
def test_invalid_transition_returns_false_when_not_raising(self, job_manager, mock_firestore_service):
"""Test that invalid transition returns False when raise_on_invalid=False."""
mock_firestore_service.get_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
# PENDING cannot transition directly to GENERATING_SCREENS
result = job_manager.validate_state_transition(
"test123",
JobStatus.GENERATING_SCREENS,
raise_on_invalid=False
)
assert result is False
def test_valid_transition_returns_true(self, job_manager, mock_firestore_service):
"""Test that valid transition returns True."""
mock_firestore_service.get_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
# PENDING can transition to DOWNLOADING
result = job_manager.validate_state_transition("test123", JobStatus.DOWNLOADING)
assert result is True
def test_job_not_found_raises_value_error(self, job_manager, mock_firestore_service):
"""Test that missing job raises ValueError when raise_on_invalid=True."""
mock_firestore_service.get_job.return_value = None
with pytest.raises(ValueError, match="Job nonexistent not found"):
job_manager.validate_state_transition("nonexistent", JobStatus.DOWNLOADING)
def test_job_not_found_returns_false_when_not_raising(self, job_manager, mock_firestore_service):
"""Test that missing job returns False when raise_on_invalid=False."""
mock_firestore_service.get_job.return_value = None
result = job_manager.validate_state_transition(
"nonexistent",
JobStatus.DOWNLOADING,
raise_on_invalid=False
)
assert result is False
class TestTransitionToStateRaisesBehavior:
"""Tests for transition_to_state with raise_on_invalid parameter."""
def test_invalid_transition_raises_by_default(self, job_manager, mock_firestore_service):
"""Test that invalid transition raises InvalidStateTransitionError by default."""
mock_firestore_service.get_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
# PENDING cannot transition directly to AWAITING_REVIEW
with pytest.raises(InvalidStateTransitionError) as exc_info:
job_manager.transition_to_state("test123", JobStatus.AWAITING_REVIEW)
error = exc_info.value
assert error.job_id == "test123"
def test_invalid_transition_returns_false_when_not_raising(self, job_manager, mock_firestore_service):
"""Test that invalid transition returns False when raise_on_invalid=False."""
mock_firestore_service.get_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
result = job_manager.transition_to_state(
"test123",
JobStatus.AWAITING_REVIEW,
raise_on_invalid=False
)
assert result is False
# Verify update was NOT called since transition was invalid
mock_firestore_service.update_job.assert_not_called()
def test_valid_transition_updates_job(self, job_manager, mock_firestore_service):
"""Test that valid transition updates the job."""
mock_firestore_service.get_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC)
)
result = job_manager.transition_to_state(
"test123",
JobStatus.DOWNLOADING,
progress=15,
message="Starting processing"
)
assert result is True
# transition_to_state uses update_job_status internally
mock_firestore_service.update_job_status.assert_called()
call_kwargs = mock_firestore_service.update_job_status.call_args.kwargs
assert call_kwargs['job_id'] == "test123"
assert call_kwargs['status'] == JobStatus.DOWNLOADING
# =============================================================================
# start_job_processing Tests
# =============================================================================
class TestStartJobProcessing:
"""Tests for start_job_processing method."""
@pytest.mark.asyncio
async def test_raises_if_job_not_found(self, job_manager, mock_firestore_service):
"""Test raises ValueError if job doesn't exist."""
mock_firestore_service.get_job.return_value = None
with pytest.raises(ValueError, match="Job missing-job not found"):
await job_manager.start_job_processing("missing-job")
@pytest.mark.asyncio
async def test_raises_if_missing_input_media_path(self, job_manager, mock_firestore_service):
"""Test raises ValueError if job has no input_media_gcs_path."""
mock_firestore_service.get_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
input_media_gcs_path=None # Missing
)
with pytest.raises(ValueError, match="missing input_media_gcs_path"):
await job_manager.start_job_processing("test123")
@pytest.mark.asyncio
async def test_raises_if_invalid_state_transition(self, job_manager, mock_firestore_service):
"""Test raises InvalidStateTransitionError if job can't transition to DOWNLOADING."""
mock_firestore_service.get_job.return_value = Job(
job_id="test123",
status=JobStatus.AWAITING_REVIEW, # Can't go back to DOWNLOADING
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
input_media_gcs_path="gs://bucket/audio.mp3"
)
with pytest.raises(InvalidStateTransitionError):
await job_manager.start_job_processing("test123")
@pytest.mark.asyncio
async def test_transitions_and_triggers_workers(self, job_manager, mock_firestore_service):
"""Test successful flow: transitions to DOWNLOADING and triggers workers."""
mock_firestore_service.get_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
input_media_gcs_path="gs://bucket/audio.mp3"
)
with patch('backend.services.worker_service.get_worker_service') as mock_get_worker:
mock_worker = Mock()
mock_worker.trigger_audio_worker = AsyncMock()
mock_worker.trigger_lyrics_worker = AsyncMock()
mock_get_worker.return_value = mock_worker
await job_manager.start_job_processing("test123")
# Verify transition was called (uses update_job_status internally)
mock_firestore_service.update_job_status.assert_called()
call_kwargs = mock_firestore_service.update_job_status.call_args.kwargs
assert call_kwargs['status'] == JobStatus.DOWNLOADING
# Verify both workers were triggered
mock_worker.trigger_audio_worker.assert_called_once_with("test123")
mock_worker.trigger_lyrics_worker.assert_called_once_with("test123")
@pytest.mark.asyncio
async def test_workers_triggered_in_parallel(self, job_manager, mock_firestore_service):
"""Test that audio and lyrics workers are triggered concurrently."""
mock_firestore_service.get_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
input_media_gcs_path="gs://bucket/audio.mp3"
)
call_order = []
async def track_audio(*args):
call_order.append(('audio_start', datetime.now()))
# Simulate some work
call_order.append(('audio_end', datetime.now()))
async def track_lyrics(*args):
call_order.append(('lyrics_start', datetime.now()))
# Simulate some work
call_order.append(('lyrics_end', datetime.now()))
with patch('backend.services.worker_service.get_worker_service') as mock_get_worker:
mock_worker = Mock()
mock_worker.trigger_audio_worker = AsyncMock(side_effect=track_audio)
mock_worker.trigger_lyrics_worker = AsyncMock(side_effect=track_lyrics)
mock_get_worker.return_value = mock_worker
await job_manager.start_job_processing("test123")
# Both workers should have been called
assert any('audio_start' in str(c) for c in call_order)
assert any('lyrics_start' in str(c) for c in call_order)
if __name__ == "__main__":
pytest.main([__file__, "-v"])
+72 -9
View File
@@ -1332,10 +1332,13 @@ class TestHandleMadeForYouOrder:
mock_theme_service
):
"""
Orders with YouTube URL should download audio then trigger workers (no search).
Orders with YouTube URL should download audio then start processing (no search).
Note: The download step was added in Jan 2026 to fix job 811ec4e5 failure.
See docs/LESSONS-LEARNED.md "YouTube URL Downloads Must Happen Before Workers".
The handler now uses job_manager.start_job_processing() which handles
transitioning to DOWNLOADING and triggering workers.
"""
from backend.api.routes.users import _handle_made_for_you_order
@@ -1348,18 +1351,16 @@ class TestHandleMadeForYouOrder:
"youtube_url": "https://youtube.com/watch?v=abc123",
}
mock_worker_service = MagicMock()
mock_worker_service.trigger_audio_worker = AsyncMock()
mock_worker_service.trigger_lyrics_worker = AsyncMock()
# Mock YouTube download service to return a GCS path
mock_youtube_service = MagicMock()
mock_youtube_service.download = AsyncMock(
return_value="uploads/test-job-123/audio/test.webm"
)
# Mock start_job_processing as an async method
mock_job_manager.start_job_processing = AsyncMock()
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.worker_service.get_worker_service', return_value=mock_worker_service), \
patch('backend.services.audio_search_service.get_audio_search_service') as mock_get_audio, \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
@@ -1380,9 +1381,8 @@ class TestHandleMadeForYouOrder:
# Audio should be downloaded first
mock_youtube_service.download.assert_called_once()
# Workers should be triggered after download
mock_worker_service.trigger_audio_worker.assert_called_once()
mock_worker_service.trigger_lyrics_worker.assert_called_once()
# start_job_processing should be called (handles transition + workers)
mock_job_manager.start_job_processing.assert_called_once_with("test-job-123")
@pytest.mark.asyncio
async def test_stripe_session_marked_processed(
@@ -1588,6 +1588,69 @@ class TestMadeForYouYouTubeDownloadContract:
f"input_media_gcs_path must be set BEFORE workers are triggered. " \
f"Call order was: {call_order}"
@pytest.mark.asyncio
async def test_youtube_url_calls_start_job_processing_after_download(
self, mock_job_manager, mock_email_service, mock_user_service,
mock_theme_service, youtube_order_metadata
):
"""
CRITICAL: start_job_processing must be called AFTER download completes.
The handler uses job_manager.start_job_processing() which:
1. Transitions job to DOWNLOADING
2. Triggers audio and lyrics workers
Without start_job_processing, workers would run but all status updates
would fail silently because transitions from PENDING to GENERATING_SCREENS
or AWAITING_REVIEW are invalid.
This bug caused made-for-you orders to get stuck at PENDING status even though
all processing completed successfully. See job IDs 06cfea29 and 984da08b.
"""
from backend.api.routes.users import _handle_made_for_you_order
# Track call order to verify sequence
call_order = []
# Mock YouTubeDownloadService to return a GCS path
mock_youtube_service = MagicMock()
mock_youtube_service.download = AsyncMock(
return_value="uploads/test-job-123/audio/test.webm",
side_effect=lambda **kwargs: (
call_order.append('youtube_download'),
"uploads/test-job-123/audio/test.webm"
)[1]
)
# Track when start_job_processing is called
mock_job_manager.start_job_processing = AsyncMock(
side_effect=lambda job_id: call_order.append('start_job_processing')
)
with patch('backend.services.job_manager.JobManager', return_value=mock_job_manager), \
patch('backend.services.storage_service.StorageService'), \
patch('backend.api.routes.users.get_theme_service', return_value=mock_theme_service), \
patch('backend.api.routes.users.get_youtube_download_service', return_value=mock_youtube_service):
await _handle_made_for_you_order(
session_id="sess_123",
metadata=youtube_order_metadata,
user_service=mock_user_service,
email_service=mock_email_service,
)
# CRITICAL: start_job_processing must be called
assert 'start_job_processing' in call_order, \
f"start_job_processing must be called. Call order was: {call_order}"
# CRITICAL: start_job_processing must happen AFTER download
download_index = call_order.index('youtube_download')
processing_index = call_order.index('start_job_processing')
assert download_index < processing_index, \
f"YouTube download must complete BEFORE start_job_processing. " \
f"Call order was: {call_order}"
@pytest.mark.asyncio
async def test_youtube_download_called_with_correct_params(
self, mock_job_manager, mock_email_service, mock_user_service,
+26
View File
@@ -75,6 +75,32 @@ class TestMain:
assert 'karaoke' in app.title.lower() or app.title
class TestExceptionHandlers:
"""Tests for global exception handlers in main.py."""
@pytest.mark.asyncio
async def test_insufficient_credits_handler_returns_402_response(self):
"""Test that the insufficient credits handler returns HTTP 402 with correct body."""
from backend.main import insufficient_credits_exception_handler
from backend.exceptions import InsufficientCreditsError
exc = InsufficientCreditsError(
message="No credits remaining",
credits_available=0,
credits_required=1,
)
response = await insufficient_credits_exception_handler(request=None, exc=exc)
assert response.status_code == 402
import json
body = json.loads(response.body)
assert body["detail"] == "No credits remaining"
assert body["credits_available"] == 0
assert body["credits_required"] == 1
assert body["buy_url"] == "/#pricing"
class TestDependencies:
"""Tests for dependencies.py."""
+53
View File
@@ -243,6 +243,59 @@ class TestPackagingServiceCDGPackage:
title="Test Title",
artist="Test Artist",
cdg_styles=styles,
lrc_has_countdown_padding=False,
countdown_padding_seconds=3.0,
)
assert os.path.isfile(zip_path)
@patch("karaoke_gen.lyrics_transcriber.output.cdg.CDGGenerator")
def test_create_cdg_package_with_countdown_padding(self, mock_cdg_generator_class):
"""Test CDG package creation with countdown padding parameters."""
with tempfile.TemporaryDirectory() as tmpdir:
lrc_file = os.path.join(tmpdir, "test.lrc")
audio_file = os.path.join(tmpdir, "test.flac")
zip_path = os.path.join(tmpdir, "output.zip")
generated_zip = os.path.join(tmpdir, "generated.zip")
with open(lrc_file, "w") as f:
f.write("[00:00.00]Test lyrics")
with open(audio_file, "w") as f:
f.write("fake audio")
# Create the zip file that would be generated
with zipfile.ZipFile(generated_zip, "w") as zf:
zf.writestr("test.mp3", "fake mp3")
zf.writestr("test.cdg", "fake cdg")
mock_generator = MagicMock()
mock_generator.generate_cdg_from_lrc.return_value = (
"test.cdg", "test.mp3", generated_zip
)
mock_cdg_generator_class.return_value = mock_generator
styles = {"font_size": 24}
service = PackagingService(cdg_styles=styles)
# Call with countdown padding enabled
result = service.create_cdg_package(
lrc_file=lrc_file,
audio_file=audio_file,
output_zip_path=zip_path,
artist="Test Artist",
title="Test Title",
lrc_has_countdown_padding=True,
countdown_padding_seconds=3.5,
)
# Verify countdown params are passed to generator
mock_generator.generate_cdg_from_lrc.assert_called_once_with(
lrc_file=lrc_file,
audio_file=audio_file,
title="Test Title",
artist="Test Artist",
cdg_styles=styles,
lrc_has_countdown_padding=True,
countdown_padding_seconds=3.5,
)
assert os.path.isfile(zip_path)
+419
View File
@@ -0,0 +1,419 @@
"""
Tests for private (non-published) tracks feature.
Tests cover:
- Job model is_private field
- Distribution settings overrides for private jobs
- Email template YouTube section removal
- Admin toggle with auto-delete behavior
"""
import pytest
from datetime import datetime
from unittest.mock import Mock, patch, MagicMock
from types import SimpleNamespace
from backend.models.job import Job, JobCreate
from backend.services.job_defaults_service import (
get_effective_distribution_for_job,
EffectiveDistributionSettings,
)
from backend.services.template_service import TemplateService
class TestJobModelIsPrivate:
"""Tests for is_private field on Job and JobCreate models."""
def test_job_defaults_to_not_private(self):
"""Job.is_private defaults to False."""
now = datetime.utcnow().isoformat()
job = Job(job_id="test-123", status="pending", created_at=now, updated_at=now)
assert job.is_private is False
def test_job_can_be_created_as_private(self):
"""Job can be explicitly set as private."""
now = datetime.utcnow().isoformat()
job = Job(job_id="test-123", status="pending", is_private=True, created_at=now, updated_at=now)
assert job.is_private is True
def test_job_create_defaults_to_not_private(self):
"""JobCreate.is_private defaults to False."""
job_create = JobCreate(artist="Test", title="Song")
assert job_create.is_private is False
def test_job_create_can_be_private(self):
"""JobCreate can be explicitly set as private."""
job_create = JobCreate(artist="Test", title="Song", is_private=True)
assert job_create.is_private is True
def test_create_job_persists_is_private(self):
"""JobManager.create_job should persist is_private from JobCreate to Job."""
from backend.services.job_manager import JobManager
manager = JobManager()
manager.firestore = Mock()
manager.firestore.create_job = Mock()
job_create = JobCreate(
artist="Test",
title="Song",
is_private=True,
theme_id="nomad",
)
job = manager.create_job(job_create, is_admin=True)
assert job.is_private is True
manager.firestore.create_job.assert_called_once()
saved_job = manager.firestore.create_job.call_args[0][0]
assert saved_job.is_private is True
def test_create_job_defaults_is_private_false(self):
"""JobManager.create_job should default is_private to False."""
from backend.services.job_manager import JobManager
manager = JobManager()
manager.firestore = Mock()
manager.firestore.create_job = Mock()
job_create = JobCreate(
artist="Test",
title="Song",
theme_id="nomad",
)
job = manager.create_job(job_create, is_admin=True)
assert job.is_private is False
class TestGetEffectiveDistributionForJob:
"""Tests for get_effective_distribution_for_job() helper."""
@patch('backend.services.job_defaults_service.get_settings')
def test_private_job_uses_private_dropbox_path(self, mock_settings):
"""Private job should use the private Dropbox path."""
mock_settings.return_value = SimpleNamespace(
default_private_dropbox_path="/Tracks-NonPublished",
default_private_brand_prefix="NOMADNP",
)
job = SimpleNamespace(
is_private=True,
dropbox_path="/Tracks-Organized",
brand_prefix="NOMAD",
enable_youtube_upload=True,
gdrive_folder_id="some-folder-id",
discord_webhook_url="https://discord.com/hook",
youtube_description_template="template",
)
dist = get_effective_distribution_for_job(job)
assert dist.dropbox_path == "/Tracks-NonPublished"
assert dist.brand_prefix == "NOMADNP"
@patch('backend.services.job_defaults_service.get_settings')
def test_private_job_disables_youtube(self, mock_settings):
"""Private job should have YouTube upload disabled."""
mock_settings.return_value = SimpleNamespace(
default_private_dropbox_path="/Tracks-NonPublished",
default_private_brand_prefix="NOMADNP",
)
job = SimpleNamespace(
is_private=True,
enable_youtube_upload=True,
discord_webhook_url=None,
)
dist = get_effective_distribution_for_job(job)
assert dist.enable_youtube_upload is False
assert dist.youtube_description is None
@patch('backend.services.job_defaults_service.get_settings')
def test_private_job_disables_gdrive(self, mock_settings):
"""Private job should have Google Drive folder ID set to None."""
mock_settings.return_value = SimpleNamespace(
default_private_dropbox_path="/Tracks-NonPublished",
default_private_brand_prefix="NOMADNP",
)
job = SimpleNamespace(
is_private=True,
gdrive_folder_id="some-folder-id",
discord_webhook_url=None,
)
dist = get_effective_distribution_for_job(job)
assert dist.gdrive_folder_id is None
@patch('backend.services.job_defaults_service.get_settings')
def test_private_job_preserves_discord(self, mock_settings):
"""Private job should still have Discord webhook preserved."""
mock_settings.return_value = SimpleNamespace(
default_private_dropbox_path="/Tracks-NonPublished",
default_private_brand_prefix="NOMADNP",
)
job = SimpleNamespace(
is_private=True,
discord_webhook_url="https://discord.com/webhook/123",
)
dist = get_effective_distribution_for_job(job)
assert dist.discord_webhook_url == "https://discord.com/webhook/123"
def test_non_private_job_uses_job_settings(self):
"""Non-private job should use its own distribution settings."""
job = SimpleNamespace(
is_private=False,
dropbox_path="/Tracks-Organized",
brand_prefix="NOMAD",
enable_youtube_upload=True,
gdrive_folder_id="folder-id",
discord_webhook_url="https://discord.com/hook",
youtube_description_template="description",
)
dist = get_effective_distribution_for_job(job)
assert dist.dropbox_path == "/Tracks-Organized"
assert dist.brand_prefix == "NOMAD"
assert dist.enable_youtube_upload is True
assert dist.gdrive_folder_id == "folder-id"
assert dist.discord_webhook_url == "https://discord.com/hook"
assert dist.youtube_description == "description"
def test_job_without_is_private_treated_as_non_private(self):
"""Job object without is_private attribute should be treated as non-private."""
job = SimpleNamespace(
dropbox_path="/Tracks-Organized",
brand_prefix="NOMAD",
enable_youtube_upload=False,
gdrive_folder_id=None,
discord_webhook_url=None,
youtube_description_template=None,
)
dist = get_effective_distribution_for_job(job)
assert dist.dropbox_path == "/Tracks-Organized"
assert dist.brand_prefix == "NOMAD"
assert dist.enable_youtube_upload is False
class TestTemplateServicePrivate:
"""Tests for email template rendering with is_private flag."""
def test_private_job_removes_youtube_section(self):
"""Private job completion email should not contain YouTube section."""
service = TemplateService()
service._storage_client = Mock()
service._bucket = Mock()
# Force use of default template
service._bucket.blob.return_value.exists.return_value = False
result = service.render_job_completion(
name="Test User",
youtube_url=None,
dropbox_url="https://dropbox.com/link",
artist="Artist",
title="Song",
job_id="job-123",
is_private=True,
)
assert "YouTube" not in result
assert "youtube_url" not in result
assert "https://dropbox.com/link" in result
def test_non_private_job_includes_youtube_section(self):
"""Non-private job completion email should include YouTube section."""
service = TemplateService()
service._storage_client = Mock()
service._bucket = Mock()
service._bucket.blob.return_value.exists.return_value = False
result = service.render_job_completion(
name="Test User",
youtube_url="https://youtube.com/watch?v=123",
dropbox_url="https://dropbox.com/link",
artist="Artist",
title="Song",
job_id="job-123",
is_private=False,
)
assert "https://youtube.com/watch?v=123" in result
assert "YouTube" in result
def test_private_job_still_has_dropbox_link(self):
"""Private job email should still contain the Dropbox link."""
service = TemplateService()
service._storage_client = Mock()
service._bucket = Mock()
service._bucket.blob.return_value.exists.return_value = False
result = service.render_job_completion(
name="Test User",
dropbox_url="https://dropbox.com/sh/abc123",
artist="Artist",
title="Song",
job_id="job-123",
is_private=True,
)
assert "https://dropbox.com/sh/abc123" in result
assert "dropbox" in result.lower()
def test_private_job_still_has_greeting(self):
"""Private job email should still contain the greeting."""
service = TemplateService()
service._storage_client = Mock()
service._bucket = Mock()
service._bucket.blob.return_value.exists.return_value = False
result = service.render_job_completion(
name="Jane",
dropbox_url="https://dropbox.com/link",
is_private=True,
)
assert "Hi Jane" in result
class TestNotificationServicePrivate:
"""Tests for notification service passing is_private to template."""
@pytest.mark.asyncio
async def test_send_completion_email_passes_is_private(self):
"""send_job_completion_email should pass is_private to template renderer."""
from backend.services.job_notification_service import JobNotificationService
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_job_completion.return_value = True
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
is_private=True,
)
# Verify is_private=True was passed to template renderer
call_kwargs = service.template_service.render_job_completion.call_args
assert call_kwargs[1].get('is_private') is True or \
(call_kwargs[0] and True in call_kwargs[0]) # positional or keyword
@pytest.mark.asyncio
async def test_send_completion_email_defaults_is_private_false(self):
"""send_job_completion_email should default is_private to False."""
from backend.services.job_notification_service import JobNotificationService
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_job_completion.return_value = True
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
)
call_kwargs = service.template_service.render_job_completion.call_args
assert call_kwargs[1].get('is_private') is False
def test_get_completion_message_passes_is_private(self):
"""get_completion_message should pass is_private to template renderer."""
from backend.services.job_notification_service import JobNotificationService
service = JobNotificationService()
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
service.get_completion_message(job_id="job-123", is_private=True)
call_kwargs = service.template_service.render_job_completion.call_args
assert call_kwargs[1].get('is_private') is True
class TestRequestModelIsPrivate:
"""Tests for is_private on API request models."""
def test_url_submission_request_has_is_private(self):
"""URLSubmissionRequest should accept is_private field."""
from backend.models.requests import URLSubmissionRequest
req = URLSubmissionRequest(url="https://youtube.com/watch?v=123", is_private=True)
assert req.is_private is True
def test_url_submission_request_is_private_defaults_none(self):
"""URLSubmissionRequest.is_private should default to None."""
from backend.models.requests import URLSubmissionRequest
req = URLSubmissionRequest(url="https://youtube.com/watch?v=123")
assert req.is_private is None
class TestAdminPrivateToggle:
"""Tests for admin API handling of is_private toggle."""
def test_is_private_in_editable_fields(self):
"""is_private should be in the EDITABLE_JOB_FIELDS set."""
from backend.api.routes.admin import EDITABLE_JOB_FIELDS
assert "is_private" in EDITABLE_JOB_FIELDS
def test_job_update_request_has_is_private(self):
"""JobUpdateRequest model should accept is_private field."""
from backend.api.routes.admin import JobUpdateRequest
req = JobUpdateRequest(is_private=True)
assert req.is_private is True
def test_job_update_request_is_private_optional(self):
"""JobUpdateRequest.is_private should be optional."""
from backend.api.routes.admin import JobUpdateRequest
req = JobUpdateRequest()
assert req.is_private is None
class TestOrchestratorConfigPrivate:
"""Tests for orchestrator config with private jobs."""
@patch('backend.services.job_defaults_service.get_settings')
def test_private_job_config_disables_youtube_and_gdrive(self, mock_settings):
"""Private job should produce config with no YouTube and no GDrive."""
mock_settings.return_value = SimpleNamespace(
default_private_dropbox_path="/Tracks-NonPublished",
default_private_brand_prefix="NOMADNP",
)
from backend.workers.video_worker_orchestrator import create_orchestrator_config_from_job
job = MagicMock()
job.job_id = "test-private"
job.artist = "Artist"
job.title = "Title"
job.state_data = {"instrumental_selection": "clean"}
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = True # Would normally enable YouTube
job.is_private = True # But private overrides it
job.brand_prefix = "NOMAD"
job.discord_webhook_url = None
job.youtube_description_template = "desc"
job.dropbox_path = "/Tracks-Organized"
job.gdrive_folder_id = "folder-123"
job.keep_brand_code = None
job.existing_instrumental_gcs_path = None
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
youtube_credentials={"token": "test"},
)
assert config.enable_youtube_upload is False
assert config.gdrive_folder_id is None
assert config.brand_prefix == "NOMADNP"
assert config.youtube_description_template is None
+162
View File
@@ -374,6 +374,168 @@ class TestCompleteReviewWithInstrumentalSelection:
assert selection in ['clean', 'with_backing', 'custom']
class TestSummaryEndpoint:
"""Tests for the fields=summary query parameter on list_jobs.
When fields=summary, the endpoint uses Firestore field projection and
returns pruned dicts instead of full Job models. This dramatically
reduces payload size for the dashboard polling use case.
"""
def test_prune_state_data_keeps_allowed_keys(self):
"""Verify _prune_state_data keeps only dashboard-required keys."""
from backend.api.routes.jobs import _prune_state_data
data = {
'state_data': {
'brand_code': 'NKP',
'youtube_url': 'https://youtu.be/abc',
'audio_progress': {'stage': 'separating'},
'lyrics_progress': {'stage': 'transcribing'},
'audio_complete': True,
'lyrics_complete': True,
'backing_vocals_analysis': {'recommended_selection': 'clean'},
'dropbox_link': 'https://dropbox.com/s/...',
# These should be stripped
'corrected_lyrics': {'lines': [1, 2, 3]},
'render_progress': {'stage': 'done'},
'blocking_state_entered_at': '2026-01-01',
}
}
result = _prune_state_data(data)
sd = result['state_data']
assert 'brand_code' in sd
assert 'youtube_url' in sd
assert 'audio_progress' in sd
assert 'lyrics_progress' in sd
assert 'audio_complete' in sd
assert 'lyrics_complete' in sd
assert 'backing_vocals_analysis' in sd
assert 'dropbox_link' in sd
# Stripped keys
assert 'corrected_lyrics' not in sd
assert 'render_progress' not in sd
assert 'blocking_state_entered_at' not in sd
def test_prune_state_data_handles_missing(self):
"""Verify _prune_state_data is safe when state_data is absent."""
from backend.api.routes.jobs import _prune_state_data
assert _prune_state_data({}) == {}
assert _prune_state_data({'state_data': None}) == {'state_data': None}
def test_prune_file_urls_keeps_allowed_keys(self):
"""Verify _prune_file_urls keeps only dashboard-required keys."""
from backend.api.routes.jobs import _prune_file_urls
data = {
'file_urls': {
'finals': {'lossless_4k_mp4': 'gs://bucket/finals/4k.mp4'},
'videos': {'with_vocals': 'gs://bucket/videos/wv.mkv'},
'packages': {'cdg_zip': 'gs://bucket/packages/cdg.zip'},
# These should be stripped
'stems': {'clean': 'gs://bucket/stems/clean.flac'},
'lyrics': {'corrections': 'gs://bucket/lyrics/c.json'},
'screens': {'title': 'gs://bucket/screens/title.mov'},
}
}
result = _prune_file_urls(data)
fu = result['file_urls']
assert 'finals' in fu
assert 'videos' in fu
assert 'packages' in fu
assert 'stems' not in fu
assert 'lyrics' not in fu
assert 'screens' not in fu
def test_prune_file_urls_handles_missing(self):
"""Verify _prune_file_urls is safe when file_urls is absent."""
from backend.api.routes.jobs import _prune_file_urls
assert _prune_file_urls({}) == {}
assert _prune_file_urls({'file_urls': None}) == {'file_urls': None}
def test_summary_returns_only_expected_fields(self):
"""Verify a pruned summary dict only contains dashboard fields."""
from backend.api.routes.jobs import _prune_state_data, _prune_file_urls
raw = {
'job_id': 'abc123',
'status': 'pending',
'progress': 0,
'created_at': '2026-01-01T00:00:00',
'artist': 'Test',
'title': 'Song',
'error_message': None,
'non_interactive': False,
'outputs_deleted_at': None,
'user_email': 'test@example.com',
'state_data': {
'brand_code': 'NKP',
'corrected_lyrics': 'should be removed',
},
'file_urls': {
'finals': {'lossless_4k_mp4': 'gs://...'},
'stems': {'clean': 'gs://...'},
},
}
result = _prune_file_urls(_prune_state_data(raw))
# state_data should only have brand_code
assert list(result['state_data'].keys()) == ['brand_code']
# file_urls should only have finals
assert list(result['file_urls'].keys()) == ['finals']
def test_hide_completed_statuses(self):
"""Verify _HIDE_COMPLETED_STATUSES includes the expected terminal statuses."""
from backend.api.routes.jobs import _HIDE_COMPLETED_STATUSES
assert 'complete' in _HIDE_COMPLETED_STATUSES
assert 'prep_complete' in _HIDE_COMPLETED_STATUSES
assert 'failed' in _HIDE_COMPLETED_STATUSES
assert 'cancelled' in _HIDE_COMPLETED_STATUSES
# Active statuses should NOT be in the list
assert 'pending' not in _HIDE_COMPLETED_STATUSES
assert 'downloading' not in _HIDE_COMPLETED_STATUSES
def test_exclude_test_works_with_summary_dicts(self):
"""Verify test email filtering works on raw dicts (not Job models)."""
from backend.utils.test_data import is_test_email
# Summary mode returns dicts, so exclude_test filters using dict access
job_dicts = [
{'job_id': '1', 'user_email': 'real@example.com'},
{'job_id': '2', 'user_email': 'test@inbox.testmail.app'},
{'job_id': '3', 'user_email': None},
]
filtered = [j for j in job_dicts if not is_test_email(j.get('user_email') or "")]
assert len(filtered) == 2
assert filtered[0]['job_id'] == '1'
assert filtered[1]['job_id'] == '3'
def test_full_response_unchanged_without_fields_param(self):
"""Verify the endpoint returns full Job models when fields is not set.
This is a backward-compatibility test: the admin jobs page (adminApi.listAllJobs)
does NOT send fields=summary, so it must continue receiving full Job objects.
"""
# When fields is None, the route falls through to the full Job path.
# We verify this by checking that the route accepts fields=None gracefully.
# (The actual HTTP test would require a TestClient, so here we verify
# the route function signature accepts None.)
import inspect
from backend.api.routes.jobs import list_jobs
sig = inspect.signature(list_jobs)
fields_param = sig.parameters.get('fields')
assert fields_param is not None
assert fields_param.default is None # Default should be None (full mode)
class TestScreensWorkerBackingVocalsAnalysis:
"""Tests documenting that backing vocals analysis now runs in screens_worker.
+37
View File
@@ -613,3 +613,40 @@ class TestCorrectionFlowIntegration:
# The corrections_updated key should be added
assert "corrections_updated" not in file_urls_before["lyrics"]
assert "corrections_updated" in file_urls_after["lyrics"]
class TestInstrumentalSelectionValidation:
"""Tests for instrumental_selection validation in complete_review."""
def test_custom_allowed_with_custom_instrumental_stem(self):
"""'custom' should be valid when custom_instrumental stem exists (mute-region editing)."""
from unittest.mock import Mock, patch, AsyncMock
valid_selections = ["clean", "with_backing"]
# Simulate the validation logic from review.py
stems = {"custom_instrumental": "jobs/test/stems/custom_instrumental.flac"}
existing_instrumental_gcs_path = None
if existing_instrumental_gcs_path or stems.get("custom_instrumental"):
valid_selections.append("custom")
assert "custom" in valid_selections
def test_custom_allowed_with_uploaded_instrumental(self):
"""'custom' should be valid when existing_instrumental_gcs_path is set."""
valid_selections = ["clean", "with_backing"]
stems = {}
existing_instrumental_gcs_path = "uploads/test/audio/instrumental.flac"
if existing_instrumental_gcs_path or stems.get("custom_instrumental"):
valid_selections.append("custom")
assert "custom" in valid_selections
def test_custom_rejected_without_custom_stem_or_upload(self):
"""'custom' should NOT be valid when neither custom stem nor upload exists."""
valid_selections = ["clean", "with_backing"]
stems = {"instrumental_clean": "jobs/test/stems/clean.flac"}
existing_instrumental_gcs_path = None
if existing_instrumental_gcs_path or stems.get("custom_instrumental"):
valid_selections.append("custom")
assert "custom" not in valid_selections
+12 -2
View File
@@ -184,7 +184,12 @@ class TestWorkerService:
mock_run_v2.RunJobRequest.Overrides = MagicMock()
mock_run_v2.RunJobRequest.Overrides.ContainerOverride = MagicMock()
with patch.dict('sys.modules', {'google.cloud.run_v2': mock_run_v2}):
# Patch both sys.modules and the parent package attribute to ensure
# 'from google.cloud import run_v2' resolves to our mock regardless
# of whether the real package is installed (namespace package resolution)
import google.cloud
with patch.dict('sys.modules', {'google.cloud.run_v2': mock_run_v2}), \
patch.object(google.cloud, 'run_v2', mock_run_v2, create=True):
service = WorkerService()
result = await service.trigger_audio_worker("test123")
@@ -222,7 +227,12 @@ class TestWorkerService:
mock_run_v2.RunJobRequest.Overrides = MagicMock()
mock_run_v2.RunJobRequest.Overrides.ContainerOverride = MagicMock()
with patch.dict('sys.modules', {'google.cloud.run_v2': mock_run_v2}):
# Patch both sys.modules and the parent package attribute to ensure
# 'from google.cloud import run_v2' resolves to our mock regardless
# of whether the real package is installed (namespace package resolution)
import google.cloud
with patch.dict('sys.modules', {'google.cloud.run_v2': mock_run_v2}), \
patch.object(google.cloud, 'run_v2', mock_run_v2, create=True):
service = WorkerService()
result = await service.trigger_lyrics_worker("test123")
@@ -310,6 +310,63 @@ class TestVideoWorkerOrchestratorPackaging:
mock_service.create_cdg_package.assert_called_once()
assert orchestrator.result.final_karaoke_cdg_zip == f"{temp_dir}/cdg.zip"
@pytest.mark.asyncio
async def test_run_packaging_cdg_with_countdown_padding(self):
"""Test CDG packaging passes countdown params when LRC has countdown padding.
This test prevents regression of the CDG sync bug where:
- Video rendering adds 3s countdown and shifts LRC timestamps by +3s
- CDG generation was using shifted timestamps but instrumental audio without padding
- Result: CDG lyrics appeared 3 seconds late
The fix: When lrc_has_countdown_padding=True, the packaging service must pass
this flag to CDGGenerator so it can strip the countdown and shift timestamps back.
"""
with tempfile.TemporaryDirectory() as temp_dir:
lrc_file = os.path.join(temp_dir, "test.lrc")
audio_file = os.path.join(temp_dir, "test.flac")
# Create LRC with countdown-shifted timestamps (as video rendering produces)
with open(lrc_file, "w") as f:
f.write("[00:00.10]1:/3... 2... 1...\n") # Countdown at 0.1s
f.write("[00:03.60]1:/First\n") # First word at 3.6s (shifted from 0.6s)
with open(audio_file, "w") as f:
f.write("dummy audio")
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path=audio_file,
lrc_file_path=lrc_file,
output_dir=temp_dir,
enable_cdg=True,
cdg_styles={"background_color": "black"},
lrc_has_countdown_padding=True, # Critical: flag must be set
)
orchestrator = VideoWorkerOrchestrator(config)
with patch.object(orchestrator, "_get_packaging_service") as mock_get:
mock_service = MagicMock()
mock_service.create_cdg_package.return_value = (
f"{temp_dir}/cdg.zip",
f"{temp_dir}/test.mp3",
f"{temp_dir}/test.cdg",
)
mock_get.return_value = mock_service
await orchestrator._run_packaging()
# CRITICAL ASSERTION: countdown padding params must be passed
mock_service.create_cdg_package.assert_called_once()
call_kwargs = mock_service.create_cdg_package.call_args[1]
assert call_kwargs.get("lrc_has_countdown_padding") is True, \
"lrc_has_countdown_padding must be passed to packaging service for CDG sync fix"
assert call_kwargs.get("countdown_padding_seconds") == 3.0, \
"countdown_padding_seconds must be passed for CDG timestamp adjustment"
class TestVideoWorkerOrchestratorEncoding:
"""Test encoding stage."""
@@ -554,7 +611,7 @@ class TestVideoWorkerOrchestratorOrganization:
@pytest.mark.asyncio
async def test_run_organization_generate_brand_code(self):
"""Test brand code generation from Dropbox."""
"""Test brand code generation via atomic BrandCodeService."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
@@ -567,15 +624,15 @@ class TestVideoWorkerOrchestratorOrganization:
)
orchestrator = VideoWorkerOrchestrator(config)
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
mock_dropbox = MagicMock()
mock_dropbox.is_configured = True
mock_dropbox.get_next_brand_code.return_value = "TEST-0001"
mock_get.return_value = mock_dropbox
with patch("backend.services.brand_code_service.get_brand_code_service") as mock_get:
mock_service = MagicMock()
mock_service.allocate_brand_code.return_value = "TEST-0001"
mock_get.return_value = mock_service
await orchestrator._run_organization()
assert orchestrator.result.brand_code == "TEST-0001"
mock_service.allocate_brand_code.assert_called_once_with("TEST", "/Karaoke/Tracks")
class TestVideoWorkerOrchestratorDistribution:
@@ -798,6 +855,7 @@ class TestCreateOrchestratorConfigFromJob:
job.enable_cdg = True
job.enable_txt = False
job.enable_youtube_upload = True
job.is_private = False
job.brand_prefix = "NOMAD"
job.discord_webhook_url = "https://discord.com/api/webhooks/123/abc"
job.youtube_description_template = "Test description"
@@ -831,6 +889,7 @@ class TestCreateOrchestratorConfigFromJob:
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = False
job.is_private = False
job.brand_prefix = None
job.discord_webhook_url = None
job.youtube_description_template = None
@@ -869,6 +928,7 @@ class TestCreateOrchestratorConfigFromJob:
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = False
job.is_private = False
job.brand_prefix = None
job.discord_webhook_url = None
job.youtube_description_template = None
+23 -1
View File
@@ -24,6 +24,7 @@ from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_health_service import validate_worker_can_run
from backend.config import get_settings
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.workers.registry import worker_registry
@@ -288,7 +289,15 @@ async def process_audio_separation(job_id: str) -> bool:
logger.error(f"[job:{job_id}] Job not found in Firestore")
job_log.error(f"Job {job_id} not found in Firestore!")
return False
# Validate job status is appropriate for audio worker
# This helps catch bugs where the worker is triggered incorrectly
status_error = validate_worker_can_run("audio_worker", job)
if status_error:
logger.warning(f"[job:{job_id}] {status_error}")
job_log.warning(status_error)
# Continue anyway - this is a safety net warning, not a hard failure
# Create temporary working directory
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_{job_id}_")
job_log.info(f"Created temp directory: {temp_dir}")
@@ -476,6 +485,19 @@ async def download_audio(
local_path = os.path.join(temp_dir, safe_filename)
storage.download_file(job.input_media_gcs_path, local_path)
logger.info(f"Job {job_id}: Downloaded uploaded file to {local_path}")
# Persist uploaded audio to jobs/ prefix (uploads/ has 7-day lifecycle)
if job.input_media_gcs_path.startswith("uploads/"):
try:
persistent_path = f"jobs/{job_id}/input/{safe_filename}"
storage.upload_file(local_path, persistent_path)
jm = job_manager_instance or JobManager()
jm.update_job(job_id, {'input_media_gcs_path': persistent_path})
jm.update_file_url(job_id, 'input', 'audio', persistent_path)
logger.info(f"Job {job_id}: Persisted upload to {persistent_path}")
except Exception as e:
logger.warning(f"Job {job_id}: Failed to persist upload to jobs/ prefix: {e}")
return local_path
# Case 2: URL download (from file_urls if already downloaded, or from job.url)
+10 -1
View File
@@ -32,6 +32,7 @@ from backend.models.job import JobStatus
from karaoke_gen.utils import sanitize_filename
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_health_service import validate_worker_can_run
from backend.services.lyrics_cache_service import LyricsCacheService
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.workers.style_helper import load_style_config
@@ -190,7 +191,15 @@ async def process_lyrics_transcription(job_id: str) -> bool:
logger.error(f"[job:{job_id}] Job not found in Firestore")
job_log.error(f"Job {job_id} not found in Firestore!")
return False
# Validate job status is appropriate for lyrics worker
# This helps catch bugs where the worker is triggered incorrectly
status_error = validate_worker_can_run("lyrics_worker", job)
if status_error:
logger.warning(f"[job:{job_id}] {status_error}")
job_log.warning(status_error)
# Continue anyway - this is a safety net warning, not a hard failure
add_span_attribute("artist", job.artist)
add_span_attribute("title", job.title)
+10 -1
View File
@@ -36,6 +36,7 @@ from dataclasses import dataclass
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_health_service import validate_worker_can_run
from backend.config import get_settings
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.services.tracing import job_span, add_span_event, add_span_attribute
@@ -98,7 +99,15 @@ async def process_render_video(job_id: str) -> bool:
logger.error(f"[job:{job_id}] Job not found in Firestore")
job_log.error(f"Job {job_id} not found in Firestore!")
return False
# Validate job status is appropriate for render video worker
# This helps catch bugs where the worker is triggered incorrectly
status_error = validate_worker_can_run("render_video_worker", job)
if status_error:
logger.warning(f"[job:{job_id}] {status_error}")
job_log.warning(status_error)
# Continue anyway - this is a safety net warning, not a hard failure
job_log.info(f"Starting video render for {job.artist} - {job.title}")
logger.info(f"[job:{job_id}] Starting video render (post-review)")
+47 -1
View File
@@ -33,6 +33,7 @@ from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_health_service import validate_worker_can_run
from backend.config import get_settings
from backend.workers.style_helper import load_style_config, StyleConfig
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
@@ -183,6 +184,12 @@ async def generate_screens(job_id: str) -> bool:
job_log.info("Analyzing backing vocals for instrumental selection...")
await _analyze_backing_vocals(job_id, job_manager, storage, job_log)
# Transcode review audio to OGG Opus for fast browser playback
# This runs BEFORE review so all files are cached when the user opens the UI
with job_span("transcode-review-audio", job_id):
job_log.info("Transcoding review audio to OGG Opus...")
await _transcode_review_audio(job_id, job_manager, job_log)
# Transition to AWAITING_REVIEW (combined lyrics + instrumental review)
# Human must review lyrics AND select instrumental before video can be rendered
logger.info(f"[job:{job_id}] Screens generated, awaiting combined review")
@@ -238,6 +245,14 @@ def _validate_prerequisites(job) -> bool:
Returns:
True if prerequisites met, False otherwise
"""
# Validate job status is appropriate for screens worker
# This helps catch bugs where the worker is triggered incorrectly
status_error = validate_worker_can_run("screens_worker", job)
if status_error:
logger.warning(f"Job {job.job_id}: {status_error}")
# Log warning but don't fail - status validation is a safety net,
# the state_data flags below are the authoritative check
# SAFETY NET: Enforce theme requirement at processing time
# This catches any jobs that somehow bypassed JobManager.create_job() validation
if not job.theme_id:
@@ -262,7 +277,7 @@ def _validate_prerequisites(job) -> bool:
if not job.artist or not job.title:
logger.error(f"Job {job.job_id}: Missing artist or title")
return False
return True
@@ -606,6 +621,37 @@ async def _analyze_backing_vocals(
})
async def _transcode_review_audio(
job_id: str,
job_manager: JobManager,
job_log: logging.Logger,
) -> None:
"""
Transcode all review audio files to OGG Opus for fast browser playback.
This eagerly transcodes input audio + stems so the review UI serves
~3 MB OGG files instead of 35+ MB FLACs. Non-fatal: if transcoding
fails, the review UI falls back to original FLAC signed URLs.
"""
from backend.services.audio_transcoding_service import AudioTranscodingService
try:
job = job_manager.get_job(job_id)
if not job:
job_log.warning(f"Could not get job {job_id} for review audio transcoding")
return
transcoding = AudioTranscodingService()
transcoded = transcoding.prepare_review_audio_for_job(job)
job_log.info(f"Review audio transcoding complete: {len(transcoded)} files")
except Exception as e:
# Non-fatal: review UI will fall back to FLAC signed URLs
job_log.warning(f"Review audio transcoding failed (non-fatal): {e}")
logger.warning(f"Job {job_id}: Review audio transcoding failed: {e}")
async def _apply_countdown_padding_if_needed(
job_id: str,
job_manager: JobManager,
+40 -14
View File
@@ -34,6 +34,7 @@ from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_health_service import validate_worker_can_run
from backend.services.rclone_service import get_rclone_service
from backend.services.youtube_service import get_youtube_service
from backend.services.encoding_service import get_encoding_service
@@ -236,15 +237,17 @@ async def generate_video_orchestrated(job_id: str) -> bool:
if rclone_service.setup_rclone_config():
job_log.info("Rclone config loaded for Dropbox upload")
# Load YouTube credentials if needed
# Load YouTube credentials if needed (skip for private/non-published tracks)
youtube_credentials = None
if getattr(job, 'enable_youtube_upload', False):
if getattr(job, 'enable_youtube_upload', False) and not getattr(job, 'is_private', False):
youtube_service = get_youtube_service()
if youtube_service.is_configured:
youtube_credentials = youtube_service.get_credentials_dict()
job_log.info("YouTube credentials loaded for video upload")
else:
job_log.warning("YouTube credentials not available - upload will be skipped")
elif getattr(job, 'is_private', False):
job_log.info("Private track - skipping YouTube upload")
try:
# Wrap entire worker in a tracing span
@@ -466,16 +469,18 @@ async def generate_video_legacy(job_id: str) -> bool:
else:
job_log.warning("Rclone config not available - Dropbox upload will be skipped")
# Load YouTube credentials if needed
# Load YouTube credentials if needed (skip for private/non-published tracks)
youtube_credentials = None
if getattr(job, 'enable_youtube_upload', False):
if getattr(job, 'enable_youtube_upload', False) and not getattr(job, 'is_private', False):
youtube_service = get_youtube_service()
if youtube_service.is_configured:
youtube_credentials = youtube_service.get_credentials_dict()
job_log.info("YouTube credentials loaded for video upload")
else:
job_log.warning("YouTube credentials not available - upload will be skipped")
elif getattr(job, 'is_private', False):
job_log.info("Private track - skipping YouTube upload")
try:
# Wrap entire worker in a tracing span
with job_span("video-worker", job_id, {"artist": job.artist, "title": job.title}) as root_span:
@@ -775,10 +780,14 @@ async def _handle_native_distribution(
brand_code = keep_brand_code
result['brand_code'] = brand_code
job_log.info(f"Using preserved brand code: {brand_code}")
# Get effective distribution settings (applies private track overrides if is_private=True)
from backend.services.job_defaults_service import get_effective_distribution_for_job
dist = get_effective_distribution_for_job(job)
# Upload to Dropbox using native SDK
dropbox_path = getattr(job, 'dropbox_path', None)
brand_prefix = getattr(job, 'brand_prefix', None)
dropbox_path = dist.dropbox_path
brand_prefix = dist.brand_prefix
if dropbox_path and brand_prefix:
try:
@@ -791,9 +800,11 @@ async def _handle_native_distribution(
else:
job_log.info(f"Starting native Dropbox upload to {dropbox_path}")
# Calculate brand code from existing folders (if not already generated)
# Calculate brand code atomically (if not already generated)
if not brand_code:
brand_code = dropbox.get_next_brand_code(dropbox_path, brand_prefix)
from backend.services.brand_code_service import get_brand_code_service
brand_code_service = get_brand_code_service()
brand_code = brand_code_service.allocate_brand_code(brand_prefix, dropbox_path)
result['brand_code'] = brand_code
job_log.info(f"Generated brand code: {brand_code}")
@@ -835,7 +846,7 @@ async def _handle_native_distribution(
# Upload to Google Drive using native API
# Skip if orchestrator already uploaded (gdrive_files already populated)
# This prevents duplicate uploads when using the orchestrator path
gdrive_folder_id = getattr(job, 'gdrive_folder_id', None)
gdrive_folder_id = dist.gdrive_folder_id
existing_gdrive_files = result.get('gdrive_files')
if gdrive_folder_id and not existing_gdrive_files:
@@ -894,6 +905,13 @@ async def _handle_native_distribution(
def _validate_prerequisites(job) -> bool:
"""Validate that all prerequisites are met for video generation."""
# Validate job status is appropriate for video worker
# This helps catch bugs where the worker is triggered incorrectly
status_error = validate_worker_can_run("video_worker", job)
if status_error:
logger.warning(f"Job {job.job_id}: {status_error}")
# Continue anyway - this is a safety net warning, not a hard failure
# Check instrumental selection
instrumental_selection = job.state_data.get('instrumental_selection')
if not instrumental_selection:
@@ -917,11 +935,12 @@ def _validate_prerequisites(job) -> bool:
logger.error(f"Job {job.job_id}: Missing lyrics video")
return False
# Check instrumental exists - for 'custom', check existing_instrumental_gcs_path instead of stems
# Check instrumental exists - for 'custom', check uploaded file or mute-region stem
if instrumental_selection == 'custom':
existing_instrumental_path = getattr(job, 'existing_instrumental_gcs_path', None)
if not existing_instrumental_path:
logger.error(f"Job {job.job_id}: Custom instrumental selected but no existing_instrumental_gcs_path")
stems = job.file_urls.get('stems', {})
if not existing_instrumental_path and not stems.get('custom_instrumental'):
logger.error(f"Job {job.job_id}: Custom instrumental selected but no existing_instrumental_gcs_path or custom_instrumental stem")
return False
return True # Other stem checks not needed for custom instrumental
@@ -992,6 +1011,13 @@ async def _setup_working_directory(
log_progress(f"Downloading user-provided existing instrumental...")
storage.download_file(existing_instrumental_path, instrumental_path)
log_progress("Downloaded user-provided instrumental")
elif instrumental_selection == 'custom':
# Custom instrumental from mute-region editing
custom_url = job.file_urls['stems']['custom_instrumental']
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental Custom).flac")
log_progress("Downloading custom (mute-region) instrumental...")
storage.download_file(custom_url, instrumental_path)
log_progress("Downloaded custom instrumental")
else:
# Use AI-separated instrumental based on selection
instrumental_key = 'instrumental_clean' if instrumental_selection == 'clean' else 'instrumental_with_backing'
+40 -21
View File
@@ -75,6 +75,10 @@ class OrchestratorConfig:
# Audio synchronization - pad instrumental to match countdown-padded vocals
countdown_padding_seconds: Optional[float] = None
# CDG sync fix: track if LRC already has countdown timestamps
# When True, CDG generation will strip the countdown segment and adjust timestamps
lrc_has_countdown_padding: bool = False
# Encoding backend preference
encoding_backend: str = "auto" # "auto", "local", "gce"
@@ -263,7 +267,12 @@ class VideoWorkerOrchestrator:
# Generate CDG package
if self.config.enable_cdg:
self.job_log.info("Generating CDG package")
if self.config.lrc_has_countdown_padding:
self.job_log.info(
f"Generating CDG package (compensating for {self.config.countdown_padding_seconds}s countdown in LRC)"
)
else:
self.job_log.info("Generating CDG package")
try:
cdg_zip_path = os.path.join(
self.config.output_dir,
@@ -286,6 +295,8 @@ class VideoWorkerOrchestrator:
title=self.config.title,
output_mp3_path=mp3_path,
output_cdg_path=cdg_path,
lrc_has_countdown_padding=self.config.lrc_has_countdown_padding,
countdown_padding_seconds=self.config.countdown_padding_seconds or 3.0,
)
self.result.final_karaoke_cdg_zip = zip_file
@@ -457,18 +468,15 @@ class VideoWorkerOrchestrator:
# Generate brand code from Dropbox if configured
if self.config.dropbox_path and self.config.brand_prefix:
try:
from backend.services.dropbox_service import get_dropbox_service
from backend.services.brand_code_service import get_brand_code_service
dropbox = get_dropbox_service()
if dropbox.is_configured:
brand_code = dropbox.get_next_brand_code(
self.config.dropbox_path,
self.config.brand_prefix
)
self.result.brand_code = brand_code
self.job_log.info(f"Generated brand code: {brand_code}")
else:
self.job_log.warning("Dropbox not configured, skipping brand code generation")
brand_code_service = get_brand_code_service()
brand_code = brand_code_service.allocate_brand_code(
self.config.brand_prefix,
self.config.dropbox_path
)
self.result.brand_code = brand_code
self.job_log.info(f"Generated brand code: {brand_code}")
except Exception as e:
self.job_log.error(f"Brand code generation failed: {e}")
@@ -706,18 +714,26 @@ def create_orchestrator_config_from_job(
# Get countdown padding info from lyrics metadata
# This ensures instrumental is padded to match vocals if countdown was added
# Also tracks if LRC has countdown padding so CDG can compensate
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
countdown_padding_seconds = None
if lyrics_metadata.get('has_countdown_padding'):
lrc_has_countdown_padding = lyrics_metadata.get('has_countdown_padding', False)
if lrc_has_countdown_padding:
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
if existing_instrumental:
ext = Path(existing_instrumental).suffix.lower()
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
elif instrumental_selection == 'custom':
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental Custom).flac")
else:
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
# Get effective distribution settings (applies private track overrides if is_private=True)
from backend.services.job_defaults_service import get_effective_distribution_for_job
dist = get_effective_distribution_for_job(job)
return OrchestratorConfig(
job_id=job.job_id,
artist=job.artist,
@@ -737,18 +753,18 @@ def create_orchestrator_config_from_job(
# Feature flags
enable_cdg=getattr(job, 'enable_cdg', False),
enable_txt=getattr(job, 'enable_txt', False),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
enable_youtube_upload=dist.enable_youtube_upload,
# Service configurations
brand_prefix=getattr(job, 'brand_prefix', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
# Service configurations (distribution settings from helper, respects is_private)
brand_prefix=dist.brand_prefix,
discord_webhook_url=dist.discord_webhook_url,
youtube_credentials=youtube_credentials,
youtube_description_template=getattr(job, 'youtube_description_template', None),
youtube_description_template=dist.youtube_description,
cdg_styles=cdg_styles,
# Dropbox/GDrive
dropbox_path=getattr(job, 'dropbox_path', None),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
# Dropbox/GDrive (overridden for private tracks)
dropbox_path=dist.dropbox_path,
gdrive_folder_id=dist.gdrive_folder_id,
# Keep existing brand code
keep_brand_code=getattr(job, 'keep_brand_code', None),
@@ -759,6 +775,9 @@ def create_orchestrator_config_from_job(
# Audio synchronization - pad instrumental to match countdown-padded vocals
countdown_padding_seconds=countdown_padding_seconds,
# CDG sync fix: track if LRC already has countdown timestamps
lrc_has_countdown_padding=lrc_has_countdown_padding,
# Encoding backend - auto selects GCE if available
encoding_backend="auto",
+235 -7
View File
@@ -77,6 +77,7 @@ Content-Type: multipart/form-data
file: <audio file>
artist: "Artist Name"
title: "Song Title"
is_private: false # Optional. If true, Dropbox uses Tracks-NonPublished/NOMADNP, no YouTube/GDrive
```
Returns job ID. Triggers async processing.
@@ -100,12 +101,15 @@ GET /api/jobs
GET /api/jobs?status=complete
GET /api/jobs?limit=10&offset=0
GET /api/jobs?exclude_test=false
GET /api/jobs?fields=summary&hide_completed=true
```
Query parameters:
- `exclude_test` (bool, default: true) - Admin only: filter out jobs from test users
- `status` - Filter by job status
- `limit` / `offset` - Pagination
- `fields` (string, optional) - Set to `summary` to return only dashboard-required fields using Firestore field projection. Reduces payload from ~16MB to <500KB for large job lists.
- `hide_completed` (bool, default: false) - Exclude terminal statuses (complete, prep_complete, failed, cancelled) server-side
#### Delete Job
@@ -166,7 +170,60 @@ Generates preview video during review. The `use_background_image` option control
GET /api/review/{job_id}/audio/{stem_type}
```
Streams audio for review playback.
Redirects to a signed GCS URL for review playback. Audio is served as OGG Opus (~3 MB) transcoded from the original FLAC (~35 MB). Transcoding happens eagerly during screen generation; if the transcoded file is missing, falls back to the original FLAC.
#### Create Custom Instrumental
```http
POST /api/jobs/{job_id}/create-custom-instrumental
Content-Type: application/json
{
"mute_regions": [
{"start_seconds": 17.0, "end_seconds": 42.0},
{"start_seconds": 53.8, "end_seconds": 71.4}
]
}
```
Creates a custom instrumental by muting selected regions of backing vocals, combining with the clean instrumental, and uploading to GCS. Returns a signed audio URL for immediate playback.
Response:
```json
{
"status": "success",
"message": "Custom instrumental created with 2 mute regions",
"audio_url": "https://storage.googleapis.com/...",
"muted_duration_seconds": 42.6
}
```
#### Upload Custom Instrumental
```http
POST /api/jobs/{job_id}/upload-instrumental
Content-Type: multipart/form-data
file: <audio file (mp3, wav, flac, ogg, aac, m4a)>
```
Uploads an external instrumental audio file for use during review. The uploaded file's duration is validated against the original audio — must match within ±0.5 seconds. Requires job to be in `awaiting_review` or `in_review` state.
Response:
```json
{
"status": "success",
"duration_seconds": 240.0,
"message": "Custom instrumental uploaded (240.0s)"
}
```
Error (duration mismatch):
```json
{
"detail": "Duration mismatch: uploaded file is 235.0s but original audio is 240.0s. The instrumental must be exactly 240.0s (±0.5s)."
}
```
### Instrumental Selection (Finalise-Only Jobs)
@@ -348,7 +405,9 @@ POST /api/internal/jobs/{job_id}/trigger-video
Common status codes:
- `400` - Bad request
- `401` - Unauthorized
- `402` - Insufficient credits (see [Credit Enforcement](#credit-enforcement))
- `404` - Job not found
- `429` - Rate limit exceeded
- `500` - Server error
## User Authentication
@@ -448,6 +507,30 @@ After payment completes via Stripe webhook:
**Email suppression**: Intermediate reminder emails (lyrics review, instrumental selection) are suppressed for made-for-you jobs since admin handles these directly. Only order confirmation and final delivery emails go to customer
### Credit Enforcement
Credits are checked and deducted at job creation time. The flow:
1. **Check** - `user_service.has_credits()` verifies the user has >= 1 credit
2. **Create job** - Job is persisted to Firestore
3. **Deduct** - `user_service.deduct_credit()` atomically deducts 1 credit (with job_id for audit trail)
4. **Refund on failure** - If the job fails, 1 credit is automatically refunded
Admin users bypass credit checks entirely. New users receive 2 welcome credits.
#### 402 Response
When a user has no credits, job creation returns HTTP 402:
```json
{
"detail": "You're out of credits. Buy more to continue creating karaoke videos.",
"credits_available": 0,
"credits_required": 1,
"buy_url": "/#pricing"
}
```
### Stripe Webhook
```http
@@ -858,7 +941,9 @@ Content-Type: application/json
}
```
Updates editable job fields. Allowed fields: `artist`, `title`, `user_email`, `theme_id`, `brand_prefix`, `discord_webhook_url`, `youtube_description`, `youtube_description_template`, `customer_email`, `customer_notes`, `enable_cdg`, `enable_txt`, `enable_youtube_upload`, `non_interactive`, `prep_only`.
Updates editable job fields. Allowed fields: `artist`, `title`, `user_email`, `theme_id`, `brand_prefix`, `discord_webhook_url`, `youtube_description`, `youtube_description_template`, `customer_email`, `customer_notes`, `enable_cdg`, `enable_txt`, `enable_youtube_upload`, `non_interactive`, `prep_only`, `is_private`.
**Note:** Setting `is_private=true` on a completed job with existing outputs triggers automatic output deletion.
Response:
```json
@@ -958,15 +1043,158 @@ Response:
}
```
If triggering fails:
#### Regenerate Screens
```http
POST /api/admin/jobs/{job_id}/regenerate-screens
Authorization: Bearer ADMIN_TOKEN
```
Regenerates title and end screen videos using the **current** artist/title metadata. Use this after editing artist or title fields to update the screens without full reprocessing.
Requirements:
- Job must have audio and lyrics processing complete
- Job must be in an allowed state: `complete`, `failed`, `awaiting_review`, `awaiting_instrumental_selection`, `instrumental_selected`, `prep_complete`
The endpoint:
1. Deletes existing screen files from GCS
2. Triggers the screens worker to regenerate with current metadata
3. Returns immediately (does not wait for completion)
Response:
```json
{
"status": "success",
"job_id": "abc123",
"message": "Screens regeneration started",
"previous_screens_deleted": true,
"worker_triggered": true,
"error": null
}
```
Screen generation takes 30-60 seconds. Monitor progress via job timeline or logs.
#### Prepare Review Audio
```http
POST /api/admin/jobs/{job_id}/prepare-review-audio
Authorization: Bearer ADMIN_TOKEN
```
Transcodes all review audio files (input + stems) to OGG Opus 128kbps for fast browser playback. Idempotent — skips files already transcoded.
Use this to backfill existing jobs created before eager transcoding was deployed, or to re-transcode after stems are regenerated.
Response:
```json
{
"status": "success",
"job_id": "abc123",
"transcoded_files": [
"jobs/abc123/review-audio/song.ogg",
"jobs/abc123/review-audio/instrumental_clean.ogg"
],
"message": "Transcoded 2 files to OGG Opus"
}
```
#### Restart Job
```http
POST /api/admin/jobs/{job_id}/restart
Authorization: Bearer ADMIN_TOKEN
Content-Type: application/json
{
"preserve_audio_stems": true,
"delete_outputs": true
}
```
Fully restarts a job from the beginning. Unlike the reset endpoint (which just changes state), restart actually **triggers workers** to begin processing.
Options:
- `preserve_audio_stems` (default: false) - If true, keeps existing audio separation and lyrics. Only regenerates screens and video with current metadata. Good for fixing title/artist typos.
- `delete_outputs` (default: true) - Delete existing output files from GCS.
Behavior by job type:
- **YouTube URL jobs**: Re-downloads audio, then triggers audio/lyrics workers
- **Audio search jobs**: Transitions to `awaiting_audio_selection` for admin to re-select
- **File upload jobs**: Triggers audio/lyrics workers directly
When `preserve_audio_stems=true`:
- Validates audio and lyrics processing completed previously
- Clears screens/video/encoding state only
- Triggers screens worker immediately
Allowed states: `pending`, `complete`, `failed`, `awaiting_review`, `awaiting_audio_selection`, `awaiting_instrumental_selection`, `instrumental_selected`, `prep_complete`
Response:
```json
{
"status": "success",
"job_id": "abc123",
"message": "Job restarted from complete to downloading",
"previous_status": "complete",
"new_status": "downloading",
"cleared_data": ["audio_complete", "lyrics_complete", "screens_progress", "..."],
"deleted_gcs_paths": ["jobs/abc123/screens/title.mov", "..."],
"workers_triggered": ["download", "audio", "lyrics"],
"error": null
}
```
#### Override Audio Source
```http
POST /api/admin/jobs/{job_id}/override-audio-source
Authorization: Bearer ADMIN_TOKEN
Content-Type: application/json
{
"source_type": "audio_search"
}
```
Switches a job's audio source from YouTube URL to audio search mode. Use this when a Made-For-You order was submitted with a YouTube URL but you want to find higher quality audio.
Currently only supports switching to `audio_search` mode, which:
1. Clears existing audio-related state (URL, downloaded file, stems, transcription)
2. Performs an audio search using the job's artist/title
3. Stores search results in state_data
4. Transitions job to `awaiting_audio_selection` with results ready
5. Admin can then select an audio source from the search results in the UI
Allowed states: `pending`, `complete`, `failed`, `awaiting_review`, `awaiting_audio_selection`, `instrumental_selected`, `prep_complete`
Response:
```json
{
"status": "success",
"job_id": "abc123",
"message": "Found 5 audio sources - select one in the admin panel.",
"previous_source": "youtube",
"new_source": "audio_search",
"cleared_data": ["audio_complete", "lyrics_complete", "url", "input_media_gcs_path", "..."],
"new_status": "awaiting_audio_selection",
"search_results_count": 5,
"error": null
}
```
If no audio sources found:
```json
{
"status": "error",
"job_id": "abc123",
"worker_type": "video",
"triggered": false,
"message": "Failed to trigger video worker for job abc123",
"error": "Job is in wrong state for video processing"
"message": "No audio sources found for: Artist - Title",
"previous_source": "youtube",
"new_source": "audio_search",
"cleared_data": ["..."],
"new_status": "failed",
"search_results_count": null,
"error": "No audio sources found for: Artist - Title"
}
```
+5 -1
View File
@@ -55,10 +55,11 @@
└────────┬────────┘ └────────┬────────┘
└──────────┬───────────┘
▼
3. SCREENS GENERATION + BACKING VOCALS ANALYSIS
3. SCREENS GENERATION + BACKING VOCALS ANALYSIS + AUDIO TRANSCODING
• Title screen (5s video)
• End screen (5s video)
• Analyze backing vocals track (for instrumental selection)
• Transcode audio to OGG Opus for review UI (~3 MB vs 35 MB FLAC)
▼
4. COMBINED HUMAN REVIEW (AWAITING_REVIEW)
• React UI for lyrics correction
@@ -102,6 +103,8 @@
**Note**: `awaiting_instrumental_selection` exists for backwards compatibility with historical jobs but is no longer used. Instrumental selection is now part of the combined review (`awaiting_review` → `in_review` → `review_complete`).
**State Machine Robustness (Feb 2026)**: State transitions are enforced by `STATE_TRANSITIONS` in `models/job.py`. Invalid transitions raise `InvalidStateTransitionError` by default. Use `scripts/generate_state_diagram.py` to generate a Mermaid diagram of valid transitions. The `/health/job-consistency` endpoint detects jobs stuck in invalid states. See `docs/archive/2026-02-02-state-machine-robustness-plan.md` for implementation details.
## GCS File Structure
```text
@@ -226,6 +229,7 @@ The Video Worker uses an orchestrator pattern to ensure all features work regard
| `email_service.py` | SendGrid email delivery with CC support |
| `template_service.py` | GCS-backed email templates |
| `job_notification_service.py` | Email orchestration (completion, reminders) |
| `audio_transcoding_service.py` | Transcode FLAC → OGG Opus for review UI playback |
## Audio Source Download Flow
+162
View File
@@ -0,0 +1,162 @@
# GCP Cost Optimization Analysis
**Date:** February 10, 2026
**Current Spend:** ~$6,000/month (~$200/day)
**Target:** Sustainable burn rate
## Executive Summary
Analyzed GCP billing data and identified $1,440+/month in immediate savings through GitHub runner auto-scaling (implemented). Additional opportunities identified for Cloud Run optimization (~$3,200/month potential savings).
## Cost Breakdown (Before Optimization)
| Service | Monthly Cost | % of Total | Notes |
|---------|--------------|------------|-------|
| **Cloud Run (karaoke-backend)** | ~$3,200 | 53% | concurrency=1, 4 min instances, 8 CPU, 16GB RAM |
| **GitHub Actions Runners** | ~$1,440 | 24% | 20 VMs running 24/7 |
| **GCE Encoding Worker** | ~$700 | 12% | c4d-highcpu-32 running 24/7 |
| **Cloud Storage** | ~$400 | 7% | 2.3TB storage |
| **Artifact Registry** | ~$78 | 1% | 390GB Docker images |
| **Other Services** | ~$182 | 3% | NAT, Load Balancers, etc. |
| **Total** | **~$6,000** | 100% | |
## Implemented Optimizations
### ✅ GitHub Actions Runners Auto-Scaling
**Status:** Deployed and operational (February 10, 2026)
**Changes Made:**
- Reduced from 20 VMs to 3 Spot/Preemptible instances
- Implemented auto-start on CI job queue (via GitHub webhook)
- Implemented auto-stop after 1 hour idle (via Cloud Scheduler)
- Removed external IPs (using Cloud NAT instead)
**Cost Impact:**
- **Before:** 20 VMs × $72/month = $1,440/month
- **After:** 3 Spot VMs × $25/month × ~20% utilization = $50-100/month
- **Savings:** ~$1,340-1,390/month (93% reduction)
**Implementation:**
- PRs: [#376](https://github.com/nomadkaraoke/karaoke-gen/pull/376), [#383](https://github.com/nomadkaraoke/karaoke-gen/pull/383), [#385](https://github.com/nomadkaraoke/karaoke-gen/pull/385)
- Infrastructure: `infrastructure/modules/runner_manager.py`, `infrastructure/compute/github_runners.py`
- Webhook: Configured at org level for all repos
- Monitoring: Cloud Function logs, Cloud Scheduler jobs
**Technical Details:**
```
Function: github-runner-manager
Memory: 512M
Idle Timeout: 1 hour
Check Frequency: Every 15 minutes
Webhook URL: https://us-central1-nomadkaraoke.cloudfunctions.net/github-runner-manager
```
## Pending Optimization Opportunities
### 🔍 Cloud Run Configuration (High Priority)
**Current Issue:**
- `concurrency=1` forces one request per instance
- `min-instances=4` keeps 4 instances always running
- High CPU/RAM allocation (8 CPU, 16GB) per instance
**Analysis Required:**
The `concurrency=1` setting may have been added deliberately for performance reasons. Investigation needed to determine:
1. Why was `concurrency=1` set? (performance issue, or unnecessary?)
2. Does `render_video_worker` run FFmpeg locally in Cloud Run? (CPU-intensive)
3. Can we increase concurrency without causing OOM or CPU contention?
**Potential Savings:** ~$2,400-3,000/month (if we can increase concurrency and reduce min-instances)
**Next Steps:**
1. Review git history for when `concurrency=1` was added and why
2. Profile render_video_worker memory/CPU usage during FFmpeg operations
3. Test with `concurrency=2-4` in staging environment
4. Monitor for OOM errors or performance degradation
### 🔍 GCE Encoding Worker
**Current State:**
- c4d-highcpu-32 VM running 24/7
- Cost: ~$700/month
**Opportunities:**
1. **Spot Instance:** Could save 60-91% (~$420-630/month savings)
- Risk: May be preempted during video encoding jobs
- Mitigation: Implement job restart logic
2. **Auto-scaling:** Start/stop based on video queue depth
- Could scale to zero when no jobs queued
- Startup time: ~2-3 minutes (acceptable for async processing)
**Recommendation:** Test Spot instance first (lower risk), then consider auto-scaling
### 🔧 Artifact Registry Cleanup (Quick Win)
**Current State:**
- 390GB of Docker images
- Cost: ~$78/month
- Many old images likely unused
**Action Items:**
```bash
# List images and their sizes
gcloud artifacts docker images list \
us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend \
--include-tags --format="table(package,version,create_time,size)"
# Set lifecycle policy to delete images older than 30 days (keep 10 most recent)
gcloud artifacts repositories update karaoke-repo \
--location=us-central1 \
--cleanup-policy-dry-run \
--cleanup-policies='tagState=tagged,olderThan=30d,keep=10'
```
**Potential Savings:** ~$40-60/month
### 🔧 Billing Alerts (Risk Management)
**Status:** No billing alerts configured
**Recommendation:** Set up budget alerts
```bash
# Alert at 50%, 80%, 100% of $5,000/month budget
gcloud billing budgets create \
--billing-account=BILLING_ACCOUNT_ID \
--display-name="Monthly Budget Alert" \
--budget-amount=5000 \
--threshold-rule=percent=50 \
--threshold-rule=percent=80 \
--threshold-rule=percent=100
```
## Summary
### Immediate Results (Implemented)
- **$1,390/month saved** through GitHub runner auto-scaling
- **New monthly spend:** ~$4,600/month (24% reduction)
### Potential Additional Savings
- Cloud Run optimization: $2,400-3,000/month (requires investigation)
- Encoding Worker Spot: $420-630/month (moderate risk)
- Artifact Registry cleanup: $40-60/month (quick win)
- **Total potential:** $2,860-3,690/month additional savings
### Target State
If all optimizations implemented:
- **Current:** $6,000/month
- **After runner optimization:** $4,600/month
- **After all optimizations:** $1,200-1,800/month
- **Total reduction:** 70-80%
## References
- [GCP Pricing Calculator](https://cloud.google.com/products/calculator)
- [Cloud Run Pricing](https://cloud.google.com/run/pricing)
- [Spot VM Pricing](https://cloud.google.com/compute/docs/instances/spot)
- [GitHub Self-Hosted Runners](https://docs.github.com/en/actions/hosting-your-own-runners)
## Change Log
- **2026-02-10:** Initial analysis and GitHub runner optimization implemented
- **2026-02-10:** Runners reduced from 20 → 3 with auto-scaling, saving $1,390/month
+257 -3
View File
@@ -45,6 +45,19 @@ When UI removes options, backend must apply sensible defaults. Don't rely on eac
### Centralize Job Creation Logic
When multiple code paths create jobs (file upload, audio search, webhooks), use a shared service for default resolution. The made-for-you webhook handler diverged from regular job creation, missing CDG/TXT defaults because it didn't call the shared `resolve_cdg_txt_defaults()` function. Fix: Create `job_defaults_service.py` with centralized helpers used by ALL job creation paths.
### Make State Transitions Loud, Not Silent (Feb 2026)
**What happened**: Jobs 06cfea29 and 984da08b got stuck at `pending` forever. The Made-For-You webhook handler triggered workers without first transitioning from PENDING to DOWNLOADING. The state machine returned `False` silently, workers ran, but job status never advanced.
**Root cause**: `transition_to_state()` returned `False` on invalid transitions, but callers didn't check the return value. Silent failures let bugs hide in production.
**Fix implemented**:
1. **Make failures loud**: `transition_to_state()` now raises `InvalidStateTransitionError` by default (use `raise_on_invalid=False` for soft failures)
2. **Centralize the pattern**: Created `start_job_processing()` helper that atomically transitions + triggers workers - all handlers use this instead of implementing independently
3. **Runtime consistency checks**: Added `job_health_service.py` with `check_job_consistency()` to detect stuck jobs (audio_complete but status=pending)
4. **Worker validation**: Workers now log warnings if triggered when job is in unexpected status
**Pattern**: When a function can fail, prefer exceptions over boolean returns. Silent failures accumulate into production bugs. Use centralized helpers for multi-step operations (state change + side effects) to prevent divergent implementations.
### Fix Both Sides of Dual Code Paths
When fixing a bug in a system with multiple code paths (e.g., legacy vs orchestrator, local vs cloud), verify ALL paths are fixed. PR #271 fixed the GCE worker to READ `instrumental_selection` but only checked the legacy path which was already SENDING it. The orchestrator path (production default) wasn't sending it. **Pattern**: If a component receives config from multiple callers, check ALL callers when fixing the receiving side. Write integration tests that cover each path.
@@ -80,17 +93,98 @@ await run_video_worker(job_id, job_manager, storage)
**Example 2:** The countdown audio sync fix (PR #328) only fixed the writer side (`render_video_worker` updating `lyrics_metadata.has_countdown_padding`). The orchestrator path that READS that state and pads the instrumental wasn't updated. PR #338 added the reader-side fix: `OrchestratorConfig` now has `countdown_padding_seconds`, `create_orchestrator_config_from_job` reads from lyrics_metadata, and GCE worker pads instrumental audio. **Lesson**: When adding cross-worker state, trace the data flow end-to-end through ALL code paths (legacy KaraokeFinalise path AND orchestrator path).
**Example 3:** CDG sync fix (Feb 2026). Video rendering adds 3s countdown padding to audio AND shifts LRC timestamps by +3s. But CDG generation uses instrumental audio (no padding) while still using the countdown-shifted LRC timestamps, causing CDG lyrics to appear 3s late. **Root cause**: The LRC file was designed for countdown-padded video audio, but CDG uses unpadded instrumental. **Fix**: Pass `lrc_has_countdown_padding` flag to CDG generator, which strips the countdown segment and shifts timestamps back. **Pattern**: When one output format (video) transforms shared data (LRC), check if other consumers (CDG) need the original or transformed version.
### Worker Idempotency Must Complete the Lifecycle
When implementing idempotency checks that set `stage='running'` at start, workers MUST also set `stage='complete'` on success. Without the completion update, retries or reprocessing attempts will be blocked because the stage is permanently stuck at `'running'`. The fix in v0.108.14 added completion markers to render_video, video, and screens workers.
### pip Wheel Filename Validation (Feb 2026)
**What happened**: Job 749141f8 failed with "Cannot connect to host 136.119.50.148:8080" during encoding. Investigation revealed the GCE encoding worker service had crashed and was restarting. The crash was caused by pip rejecting `karaoke_gen-current.whl` with "Invalid wheel filename (wrong number of parts)".
**Root cause**: The `ensure_latest_wheel()` function in `backend/services/gce_encoding/main.py` downloaded all wheels from GCS using a wildcard pattern (`karaoke_gen-*.whl`), which included `karaoke_gen-current.whl`. This file is a convenience symlink/copy for CI deployment but lacks the version and platform tags required by PEP 427 wheel naming standard (`{distribution}-{version}-{python}-{abi}-{platform}.whl`). When pip tried to install it, it failed, causing the worker to crash.
**Why it was confusing**: The actual error ("Cannot connect") masked the real problem (wheel installation failure). The service was healthy but kept restarting due to the pip error, making it temporarily unavailable when Cloud Run tried to connect.
**Fix**: Filter out wheels containing '-current' before selecting which to install:
```python
wheels = glob.glob("/tmp/karaoke_gen-*.whl")
# Filter out karaoke_gen-current.whl (not a valid PEP 427 wheel name)
wheels = [w for w in wheels if '-current' not in w]
```
**Pattern**: When downloading artifacts with wildcards, filter results to exclude non-standard naming patterns before processing. CI convenience files (symlinks, "latest", "current") often don't follow the same naming conventions as versioned releases. Always validate artifact names match expected patterns before attempting to use them.
### Semantic Version Sorting (Feb 2026)
**What happened**: PR #384 fixed the invalid wheel filename crash but exposed a version sorting bug. Production encoder started showing "Encoder: v0.99.9" instead of the latest v0.116.1. The GCE encoding worker was running an extremely old version from months ago.
**Root cause**: After filtering out invalid wheels, the code used `sorted(wheels)[-1]` to select the "latest" wheel. This does **alphabetical sorting**, not semantic version sorting. String comparison puts "karaoke_gen-0.99.9-..." AFTER "karaoke_gen-0.116.1-..." because '9' > '1' in the second character position.
**Why it's tricky**: For some version ranges, alphabetical sorting happens to work correctly (0.100.0 > 0.99.0), but fails for others (0.99.9 > 0.116.1). This makes the bug intermittent and hard to notice until a specific version combination triggers it.
**Fix**: Extract version numbers from filenames and sort using `packaging.version.Version`:
```python
from packaging.version import Version
import re
def extract_version(wheel_path):
"""Extract version from wheel filename like karaoke_gen-0.116.1-py3-none-any.whl"""
match = re.search(r'karaoke_gen-([0-9.]+)-', wheel_path)
if match:
return Version(match.group(1))
return Version("0.0.0") # Fallback for unparseable filenames
wheels.sort(key=extract_version, reverse=True)
latest_wheel = wheels[0]
```
**Pattern**: When selecting "latest" from a list of versioned artifacts (wheels, releases, tags), always use semantic version comparison, not string/alphabetical sorting. The `packaging` library is part of Python's standard packaging infrastructure and handles all semantic versioning edge cases (major.minor.patch, pre-releases, etc.). String sorting only works accidentally for some version ranges.
### Clear Worker Progress Keys When Reprocessing
When resetting a job or re-reviewing a completed job, all worker progress keys (`*_progress`) must be cleared from `state_data`. Workers check `state_data.{worker}_progress.stage == 'complete'` for idempotency - if stale keys exist from a previous run, workers will skip execution even though the job needs reprocessing. **Pattern**: Any operation that intends to re-run workers (admin reset, review resubmission) must explicitly clear progress keys using `job_manager.delete_state_data_keys()`. See `backend/api/routes/review.py:complete_review()` and `backend/api/routes/admin.py:clear_worker_state()`.
### Defense in Depth
Enforce critical requirements at multiple layers (e.g., reject at creation in JobManager + safety net at processing time).
### Retry Transient Failures
HTTP calls to services that can restart (VMs, containers) need retry logic with exponential backoff (2s → 4s → 8s) for connection errors.
### Retry Transient Failures (Feb 2026)
HTTP calls to services that can restart (VMs, containers) need retry logic with exponential backoff for connection errors.
**Implementation patterns:**
- Use exponential backoff with cap: 10s → 20s → 40s → 60s (capped)
- Retry on network errors (`httpx.RequestError`) and 5xx server errors
- Do NOT retry on 4xx client errors (bad request, auth failure, not found)
- Make retry configuration runtime-adjustable via environment variables
- Let exceptions bubble through retry logic before wrapping in custom error types
- Log each retry attempt with attempt number and wait time
**State persistence for retry capability:**
When operations can fail before completion, persist operation parameters BEFORE attempting the operation. This enables retry even if the operation fails. For example, when downloading audio, save the source info (provider, source_id, target_file, download_url) to the job document before calling the download service. If download fails, the retry endpoint can use these saved parameters to retry.
**Example (flacfetch client):**
```python
# Manual retry logic with exponential backoff
async def with_retry(func, *args, **kwargs):
max_attempts = 9
for attempt in range(1, max_attempts + 1):
try:
return await func(*args, **kwargs)
except Exception as e:
if not should_retry(e): # Don't retry 4xx errors
raise
if attempt >= max_attempts:
raise
wait_time = min(10 * (2 ** (attempt - 1)), 60)
await asyncio.sleep(wait_time)
raise last_exception
# Wrap implementation, handle errors after retry logic
async def download_by_id(...):
try:
return await with_retry(self._download_by_id_impl, ...)
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download failed after retries: {e}")
```
See `backend/services/flacfetch_client.py` and `backend/tests/test_flacfetch_client.py` for full implementation.
### Cross-Domain localStorage
Auth tokens in localStorage are domain-isolated. Keep auth on a single domain or use cookies with `domain=.example.com`.
@@ -125,6 +219,68 @@ For subprocess without shell, FFmpeg filter paths need: apostrophes escaped as `
### Unicode in HTTP Headers
Sanitize user input (artist/title) to ASCII for HTTP headers. Smart quotes, em dashes from copy/paste cause encoding errors.
### Unicode in File Paths (Feb 2026)
**What happened**: Job `1eab1172` with Russian title "я куплю тебе дом" (Cyrillic) failed with FFmpeg exit code 183. The temp subtitle file path contained Unicode characters: `temp_subtitles_zivert___я_куплю_тебе_дом_1770150285852.ass`.
**Root cause**: Python's `str.isalnum()` returns `True` for ALL Unicode alphanumeric characters (Cyrillic, Arabic, Chinese, etc.), not just ASCII. The code used `"".join(c if c.isalnum() else "_" for c in output_prefix)` to sanitize filenames, which allowed Cyrillic letters through because they're "alphanumeric" in Unicode. FFmpeg cannot handle Unicode characters in filter argument paths, causing file I/O errors.
**Fix**: Replace `c.isalnum()` with `c.isascii() and c.isalnum()` to only allow ASCII alphanumeric characters (a-z, A-Z, 0-9). This ensures temp file paths are safe for FFmpeg and other external tools.
**Locations fixed** (v0.115.2):
- `karaoke_gen/lyrics_transcriber/output/video.py:222` - `generate_video()` temp subtitle path
- `karaoke_gen/lyrics_transcriber/output/video.py:282` - `generate_preview_video()` temp subtitle path
- `karaoke_gen/utils/remote_cli.py:1970` - output folder name sanitization
**Pattern**: When sanitizing filenames or paths for external tools, always use ASCII-only checks. Don't assume `isalnum()`, `isalpha()`, or `isdigit()` are ASCII-only - they match all Unicode categories.
**Test coverage**: Added 10 comprehensive tests covering Cyrillic, Chinese, Arabic, emoji, and accented characters to prevent regression.
### Admin Job Restart Must Use Valid State Transitions (Feb 2026)
**What happened**: After fixing the Unicode bug (v0.115.2), job `1eab1172` was restarted using the admin "Full Restart" button. The restart triggered successfully, but then immediately failed with: "Invalid state transition for job 1eab1172: transcribing -> JobStatus.GENERATING_SCREENS. Valid transitions: ['correcting', 'failed']"
**Root cause**: The admin restart endpoint (`backend/api/routes/admin.py:2161`) was setting the job status to `transcribing` when `preserve_audio_stems=True`, then triggering the screens worker. The screens worker immediately tries to transition to `GENERATING_SCREENS` state. However, the state machine (`backend/models/job.py:STATE_TRANSITIONS`) doesn't allow `transcribing -> generating_screens` - only `transcribing -> correcting` or `transcribing -> failed`.
**Why this happened**: The code was written intuitively ("we're preserving audio stems and want to restart from screens, so set to transcribing") without checking the state machine's allowed transitions. The comment even said "Allows screens worker to run" but that was incorrect.
**Fix** (v0.115.3): Change restart endpoint to set status to `downloading` instead of `transcribing`. Looking at STATE_TRANSITIONS:
- `downloading` → `generating_screens` ✓ (valid)
- `audio_complete` → `generating_screens` ✓ (valid)
- `lyrics_complete` → `generating_screens` ✓ (valid)
- `transcribing` → `generating_screens` ✗ (invalid)
Using `downloading` is the safest choice because it's the most flexible entry state that allows transitioning to screens generation.
**Pattern**: When writing admin operations or restart logic that sets job status and triggers workers, always check `backend/models/job.py:STATE_TRANSITIONS` to ensure the target state allows transitioning to whatever state the worker will try to enter. Don't make assumptions about valid transitions - verify against the state machine definition.
### AudioShake Language Auto-Detection (Feb 2026)
**What happened**: Job `1eab1172` with Russian lyrics ("я куплю тебе дом") was transcribed by AudioShake, but the lyrics were translated to English instead of being transcribed in Russian. The transcription showed English words instead of Cyrillic characters.
**Root cause**: `karaoke_gen/lyrics_transcriber/transcribers/audioshake.py:152` was hardcoding `"language": "en"` in the AudioShake API request. This told AudioShake to transcribe/translate everything as English, regardless of the actual audio language.
**Fix** (v0.115.4): Remove the hardcoded language parameter from the API request. AudioShake supports language auto-detection when the `language` field is omitted. Tested with API call - omitting the language field allows AudioShake to auto-detect from 100+ supported languages (Russian, Chinese, Spanish, etc.).
```python
# Before (forced English):
data = {
"targets": [{
"model": "alignment",
"formats": ["json"],
"language": "en" # ← Hardcoded
}]
}
# After (auto-detect):
data = {
"targets": [{
"model": "alignment",
"formats": ["json"]
# Language omitted - AudioShake auto-detects
}]
}
```
**Pattern**: When integrating with transcription/translation APIs, check if they support language auto-detection before hardcoding a language. Many modern APIs (Whisper, AudioShake, etc.) can auto-detect language, which makes the system work globally without configuration. Only hardcode a language when you need to force a specific output language or when the API requires it.
### Fonts in Docker
Base Docker images have no fonts. Install `fonts-noto-core`, `fonts-noto-cjk` for video rendering.
@@ -165,6 +321,38 @@ admin_email = auth_data.user_email or "unknown" # Use attribute access
**Key insight**: When a class has `__iter__` for backward-compatible tuple unpacking but not `__getitem__`, prefer attribute access over index access. Type hints can be misleading if the codebase hasn't been updated consistently.
### Field Mapping Gaps in Manual Constructors (Feb 2026)
**What happened**: The `is_private` field was added to the `JobCreate` Pydantic model and the `Job` model, but `job_manager.create_job()` manually maps fields from `JobCreate` to `Job()` — and the `is_private` field was omitted from that mapping. The API accepted the field, the model had it, but it silently defaulted to `False` in Firestore because the constructor never included it.
**Root cause**: The constructor uses explicit field-by-field mapping (`Job(title=data.title, artist=data.artist, ...)`) rather than something like `data.model_dump()`. When a new field is added to the input model, it's easy to forget to add it to the manual mapping.
**Fix**: Add `is_private=data.is_private` to the `Job()` constructor in `create_job()`.
**Pattern**: When adding new fields to Pydantic models, trace the full path from API request -> model -> persistence to ensure the field is explicitly mapped at every step. Automated regression tests that verify round-trip persistence are essential — a test that creates a job with `is_private=True` and reads it back would have caught this immediately.
### Return Value Contract Mismatches (Feb 2026)
**What happened**: `job_manager.update_job()` returns `None` (no explicit return), but `admin.py` checked `if not success:` treating `None` as failure. Since `not None` is `True`, this caused ALL admin PATCH updates to return 500 errors — not just the private toggle.
**Root cause**: The calling code assumed `update_job()` returned a boolean success indicator, but the actual contract is void (returns `None`, raises on error). `success = await job_manager.update_job(...)` assigns `None`, and `if not success:` always evaluates to `True`.
**Fix**: Use `try/except` for void functions rather than checking return values:
```python
# ❌ WRONG - update_job returns None, not a boolean
success = await job_manager.update_job(job_id, updates)
if not success:
raise HTTPException(status_code=500, ...)
# ✅ CORRECT - catch exceptions from void functions
try:
await job_manager.update_job(job_id, updates)
except Exception as e:
raise HTTPException(status_code=500, ...)
```
**Pattern**: When calling a function and checking its return value, verify what it actually returns. Use `try/except` for void functions rather than checking return values. This is closely related to the "Mocks Must Match Real Return Values" lesson — mocks that return `True` for a void function will hide this class of bug entirely.
---
## Testing Insights
### Test Webhook Handlers with Unit Tests
@@ -214,6 +402,37 @@ worker_index = call_order.index('audio_worker')
assert download_index < worker_index, "Download must happen BEFORE workers"
```
### Made-For-You Jobs Must Transition to DOWNLOADING Before Workers (Feb 2026)
**Problem**: Made-for-you orders with YouTube URLs (jobs 06cfea29, 984da08b) got stuck at `pending` status even though all processing completed successfully. Workers ran, `state_data` showed `audio_complete: true`, `lyrics_complete: true`, `screens_progress.stage: complete`, but the `status` field remained `pending`.
**Root cause**: The `_handle_made_for_you_order` function in `users.py` triggered workers for YouTube URL orders **without first transitioning the job from PENDING to DOWNLOADING**. The job state machine defines strict valid transitions:
- `PENDING` → `DOWNLOADING`, `SEARCHING_AUDIO`, `FAILED`, `CANCELLED` (only these are valid)
- Workers try: `PENDING` → `GENERATING_SCREENS` → `AWAITING_REVIEW` (INVALID!)
All status transitions after workers complete fail silently because they're invalid from `PENDING`.
**Why it was subtle**:
1. Workers completed successfully (no errors in logs)
2. `state_data` updates worked (they bypass state machine validation)
3. Only the `status` field failed to update (uses `transition_to_state` which validates)
**Fix**:
1. Immediate: Add `transition_to_state(job_id, JobStatus.DOWNLOADING)` before triggering workers
2. Robustness: Make `transition_to_state()` raise `InvalidStateTransitionError` by default instead of returning `False` silently
3. Pattern: Create centralized `job_manager.start_job_processing(job_id)` helper that handles transition + worker triggers atomically
**Robustness changes** (see `docs/archive/2026-02-02-state-machine-robustness-plan.md`):
- `InvalidStateTransitionError` exception with full context (job_id, from_status, to_status, valid_transitions)
- `validate_state_transition()` and `transition_to_state()` now have `raise_on_invalid=True` default
- `start_job_processing()` centralizes the pattern: validate prerequisites → transition → trigger workers
**Pattern**: Always verify state machine flow in webhook handlers. Use centralized helpers like `start_job_processing()` instead of duplicating transition + worker logic.
**Tests added**:
- `test_youtube_url_calls_start_job_processing_after_download` - verifies proper sequence
- `TestInvalidStateTransitionError`, `TestValidateStateTransitionRaisesBehavior`, `TestStartJobProcessing` in `test_job_manager.py`
### Use data-testid for E2E
Prefer `data-testid` over label/text selectors. They're immune to label changes and won't break when similar fields are added.
@@ -268,6 +487,38 @@ const debouncedUpdate = useCallback((val: string) => {
**Testing**: Verify (1) typing multi-digit values doesn't jump cursor, (2) debounce fires after delay, (3) blur syncs immediately, (4) external updates respected when not focused. See `frontend/components/lyrics-review/__tests__/TimeInput.test.tsx` for comprehensive test suite.
### Mocks Must Match Real Return Values (Feb 2026)
**Problem**: PR #371 introduced a bug where admin reset always returned 500. The code checked `if not success:` after calling `update_job()`, but `update_job()` returns `None`, not a boolean. Since `not None` is `True`, the error was always raised.
**Why tests didn't catch it**: All 22 test mocks had:
```python
mock_jm.update_job.return_value = True # WRONG - real method returns None
```
The mock returned `True`, so `if not True:` was `False` and tests passed. But in production, `if not None:` is `True`, causing 500 errors.
**Root cause**: Mocks were written based on an assumed API contract (boolean return) rather than the actual contract (void return, raises on error).
**Fix**:
1. Updated all mocks to `return_value = None` with comment explaining why
2. Added emulator integration test (`test_admin_reset_integration.py`) that uses real `JobManager`
3. Added explicit test `test_update_job_returns_none_not_boolean` to document the API contract
**Pattern for preventing this**:
1. **Mocks should match real signatures**: Check the actual method before setting `return_value`. Void methods should mock `return_value = None`.
2. **Add emulator tests for critical paths**: Integration tests with real services catch contract mismatches mocks hide.
3. **Document API contracts in tests**: Add explicit tests for non-obvious behaviors (`update_job` returns None, raises on failure).
**Code pattern**:
```python
# ❌ WRONG - assumes boolean return
mock_jm.update_job.return_value = True
# ✅ CORRECT - matches real API
mock_jm.update_job.return_value = None # Real API: returns None, raises on error
```
### Emulator Tests Catch Real Bugs
Firestore emulator tests catch issues (like missing indexes) that unit tests with mocks miss.
@@ -391,7 +642,7 @@ Set `min-instances > 0` for Cloud Run services with heavy initialization.
## Data & Storage
### Firestore 1MB Limit
Use subcollections for unbounded data (logs, events). Embedded arrays hit 1MB limit.
Use subcollections for unbounded data (logs, events). Embedded arrays hit 1MB limit. When saving large objects to Firestore `state_data`, strip fields not needed for Firestore queries (e.g. `reference_lyrics`) and store full data in GCS instead. The Genius API occasionally returns non-lyrics content (screenplays, articles) that can bloat documents past 1MB.
### Firestore Timezone-Aware
Firestore returns timezone-aware datetimes. Compare with `datetime.now(timezone.utc)`, not naive `datetime.now()`.
@@ -424,6 +675,9 @@ When adding alternative implementations (GCE vs local encoding), audit ALL side
### External Response Validation
Add defensive type checking for all external service responses. Lists vs dicts, missing fields cause cryptic failures.
### Brand Code Allocation Must Be Atomic (Feb 2026)
Two concurrent jobs both got brand code `NOMADNP-0012` because `get_next_brand_code()` scanned Dropbox folders to find the next number — a classic TOCTOU race condition. Jobs processed 0.86s apart both saw the same state. **Fix**: `brand_code_service.py` uses Firestore transactions to atomically allocate codes. Counter doc per prefix (`NOMAD`, `NOMADNP`) with `next_number` and `recycled` pool. E2E cleanup recycles numbers via `recycle_brand_code()`.
---
## Frontend Patterns
+19
View File
@@ -30,6 +30,7 @@
| CI/CD self-hosted runner | Working (GCP) |
| E2E happy path test | Working (~20-25 min full pipeline) |
| **White-label B2B portals** | Working (Vocal Star first tenant) |
| **Private (non-published) tracks** | Working |
## Known Issues
@@ -41,6 +42,24 @@
## Recent Changes
- **Private (Non-Published) Tracks** (2026-02-24): Added `is_private` flag on jobs for private/non-published track production. Private jobs upload to Dropbox `Tracks-NonPublished` folder with `NOMADNP` brand prefix, skip YouTube and Google Drive entirely, and send completion emails without the YouTube section. Available as a checkbox on all job creation tabs and as an admin toggle (auto-deletes outputs when toggling a completed job to private). See [archive/2026-02-24-private-tracks-plan.md](archive/2026-02-24-private-tracks-plan.md).
- **GCE Encoding Worker Version Sorting Fix** (2026-02-11): Fixed critical regression from PR #384 where alphabetical sorting caused encoder to use v0.99.9 instead of v0.116.1. **Root cause**: `sorted(wheels)[-1]` uses string comparison, not semantic versioning. String comparison puts "0.99.9" > "0.116.1" (because '9' > '1' in second position). **Impact**: Production encoder was running an extremely old version (v0.99.9 from months ago) instead of the latest (v0.116.1). **Fix**: Extract version from wheel filename using regex, sort by `packaging.version.Version` objects instead of strings. Added 10 comprehensive unit tests covering the exact regression case (0.99.9 vs 0.116.1), edge cases (empty list, invalid filenames), and complex version comparisons (major/minor/patch). Production now correctly selects the latest semantic version. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#semantic-version-sorting-feb-2026).
- **GCE Encoding Worker Wheel Fix** (2026-02-10): Fixed encoding worker crash caused by attempting to install `karaoke_gen-current.whl`, which is not a valid PEP 427 wheel filename (missing version and platform tags). **Root cause**: The `ensure_latest_wheel()` function downloaded all wheels including `karaoke_gen-current.whl`, then tried to install it. pip rejected it with "Invalid wheel filename (wrong number of parts)", causing encoding to fail and the service to restart, resulting in "Cannot connect" errors from Cloud Run. **Fix**: Filter out wheels containing '-current' before selecting which wheel to install. Only properly versioned wheels (e.g., `karaoke_gen-0.116.0-py3-none-any.whl`) are now considered. Added 5 comprehensive unit tests covering current wheel exclusion, empty results, version sorting, and path-agnostic filtering. Fixes job 749141f8 and prevents future encoding failures. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#pip-wheel-filename-validation-feb-2026).
- **Flacfetch Retry Logic & Audio Search Recovery** (2026-02-10): Added comprehensive retry logic for flacfetch download failures to handle transient network issues and service restarts. Jobs now retry downloads up to 9 times over ~6 minutes with exponential backoff (10s → 20s → 40s → 60s capped), significantly reducing failures from temporary network hiccups. **Key improvements**: (1) Retry logic with selective error handling (retries on connection errors and 5xx, immediate fail on 4xx), (2) State persistence - download parameters (source_name, source_id, target_file, download_url) now saved BEFORE download attempt, enabling retry even if download fails, (3) Enhanced retry endpoint - `/api/jobs/{id}/retry` now handles audio search jobs by re-downloading using saved params, (4) User feedback - retry button now shows error toast with actual API error message instead of silent failures. Added 5 comprehensive retry tests covering network errors, 4xx/5xx responses, max retries, and exponential backoff timing. Job 749141f8 investigation revealed the root cause: no source info was saved when flacfetch calls failed, making retry impossible. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#retry-transient-failures-feb-2026) for retry implementation patterns.
- **CDG Sync Fix for Countdown-Padded Songs** (2026-02-05): Fixed CDG files being 3 seconds out of sync with audio when songs have countdown padding applied. **Root cause**: Video rendering adds 3s countdown padding to audio AND shifts LRC timestamps by +3s. CDG generation was using these countdown-shifted LRC timestamps but with instrumental audio (which has no countdown padding), causing lyrics to appear 3s late. **Fix**: CDG generator now receives `lrc_has_countdown_padding` flag; when True, it strips the countdown segment and shifts timestamps back before generating the CDG. Added 11 unit tests and updated orchestrator/packaging pipeline. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fix-both-sides-of-dual-code-paths).
- **Custom Instrumental Upload Fix** (2026-02-26): Fixed critical bug where uploaded custom instrumentals were ignored during encoding — GCE encoding worker and orchestrator had no `"custom"` case, falling back to clean instrumental instead. Upload endpoint now converts all formats to FLAC for pipeline consistency. Three fixes: (1) GCE encoding worker `find_file` now searches for `custom_instrumental*.flac` when selection is `"custom"`, (2) orchestrator config constructs correct `(Instrumental Custom).flac` path, (3) upload endpoint converts mp3/m4a/wav/etc. to FLAC before storing. Added 10 new regression tests for GCE resolution and orchestrator paths. See [API.md](API.md#upload-custom-instrumental).
- **Custom Instrumental Upload** (2026-02-25): Added `POST /api/jobs/{job_id}/upload-instrumental` endpoint — was missing from backend, causing 404 in production when users tried to upload custom instrumental files during review. Validates uploaded audio duration matches original (±0.5s tolerance) using ffprobe on GCS signed URLs. Fixed frontend bug where "uploaded" selection type was sent to API instead of "custom" (the only value the backend accepts). Added 22 tests covering upload validation, duration checks, and frontend/backend contract. See [API.md](API.md#upload-custom-instrumental).
- **Unicode Filename Fix** (2026-02-04): Fixed FFmpeg failures (exit code 183) when processing songs with non-ASCII characters in titles. Job `1eab1172` with Russian title "я куплю тебе дом" failed because Python's `str.isalnum()` returns True for ALL Unicode alphanumeric characters (Cyrillic, Arabic, Chinese, emoji, etc.), not just ASCII. These Unicode characters in temp subtitle file paths (`temp_subtitles_zivert___я_куплю_тебе_дом_1770150285852.ass`) caused FFmpeg to fail. **Fix**: Changed filename sanitization from `c.isalnum()` to `c.isascii() and c.isalnum()` in three locations: `video.py:222` (generate_video), `video.py:282` (generate_preview_video), and `remote_cli.py:1970` (output folders). Added comprehensive test suite covering Cyrillic, Chinese, Arabic, emoji, and accented characters. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#unicode-in-file-paths-feb-2026).
- **Admin MFY Restart & Override Tools** (2026-02-04): Added three new admin endpoints for better Made-For-You order management: (1) **Regenerate Screens** (`POST /api/admin/jobs/{job_id}/regenerate-screens`) - regenerates title/end screens with current artist/title metadata without full reprocessing; (2) **Full Restart** (`POST /api/admin/jobs/{job_id}/restart`) - unlike "Reset to Start", this actually triggers workers to run, with option to preserve audio stems for faster restart; (3) **Override Audio Source** (`POST /api/admin/jobs/{job_id}/override-audio-source`) - switch from YouTube URL to audio search mode for higher quality audio. Frontend admin toolbar updated with new buttons (Regen Screens, Full Restart, Audio Search) with confirmation dialogs. **Use case**: Fix MFY orders where customer provided wrong artist/title or low-quality YouTube URL. See [API.md](API.md#regenerate-screens).
- **Made-For-You YouTube URL Download Fix** (2026-01-28): Fixed bug where made-for-you orders with YouTube URLs failed with "Failed to download audio file". **Root cause**: The webhook handler triggered workers directly without downloading the audio first via `YouTubeDownloadService`. Workers require `input_media_gcs_path` to be set, which only happens after download completes. **Fix**: Handler now downloads YouTube audio to GCS before triggering workers, matching the pattern in `file_upload.py` and `audio_search.py`. Added 4 new contract tests with call-order tracking and AST import validation to prevent regression. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#youtube-url-downloads-must-happen-before-workers-jan-2026).
- **Lyrics Corrections Now Applied to Video** (2026-01-26): Fixed critical bug where user's lyrics corrections made during human review were being ignored. Videos were rendering with raw transcription instead of corrected lyrics. **Root cause**: The `GET /api/review/{job_id}/correction-data` endpoint was returning the original `corrections.json` instead of `corrections_updated.json` (where user edits are saved). The combined review flow would then overwrite the user's edits when completing the review. **Fix**: The endpoint now checks for `corrections_updated.json` first, matching the behavior of `render_video_worker.py`. Added 35 new tests (backend unit + integration + frontend E2E) to prevent regression. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#always-test-the-full-data-flow-not-just-individual-components).
+24
View File
@@ -158,6 +158,30 @@ def test_create_job_applies_default_theme(mock_firestore, mock_gcs):
assert job.theme_id == "nomad" # Default theme applied
```
**Critical: Mocks Must Match Real API Contracts**
When mocking methods, ensure the mock's `return_value` matches what the real method returns. A common bug pattern:
```python
# ❌ BAD: Assumes method returns boolean
mock_service.update_job.return_value = True
# Code under test:
success = job_manager.update_job(job_id, updates)
if not success: # BUG: if real method returns None, this is always True!
raise HTTPException(500)
```
The fix:
```python
# ✅ GOOD: Check real method signature first
# def update_job(self, job_id: str, updates: Dict) -> None:
mock_service.update_job.return_value = None # Match real API
```
**Before writing a mock**, check the actual method's return type. Void methods should mock `return_value = None`. Methods that raise on error (rather than returning False) should have mocks that raise exceptions for error cases. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#mocks-must-match-real-return-values-feb-2026) for a production bug caused by this.
### Integration Tests (`tests/integration/`)
**Purpose:** Test CLI workflows and component interactions.
@@ -0,0 +1,307 @@
# State Machine Robustness Plan
**Date**: 2026-02-02
**Status**: ✓ Complete (Weeks 1-3 implemented)
**Context**: Jobs 06cfea29 and 984da08b got stuck at `pending` because state transitions failed silently.
## Problem Analysis
The root issues that allowed this bug:
1. **Silent failures**: `transition_to_state()` returns `False` on invalid transitions, but callers don't check it
2. **No centralized worker trigger pattern**: Each handler (file_upload.py, users.py, audio_search.py) implements "transition + trigger workers" independently
3. **No runtime consistency checks**: Nothing detects when `state_data` says complete but `status` is wrong
4. **Workers don't validate prerequisites**: Workers run regardless of job status
## Proposed Improvements
### 1. Make State Transition Failures Loud (High Priority)
**Option A**: Raise exception by default (breaking change, but safest)
```python
class InvalidStateTransitionError(Exception):
"""Raised when an invalid state transition is attempted."""
pass
def transition_to_state(self, job_id: str, new_status: JobStatus, ...,
raise_on_invalid: bool = True) -> bool:
if not self.validate_state_transition(job_id, new_status):
if raise_on_invalid:
raise InvalidStateTransitionError(
f"Invalid transition for job {job_id}: {current} -> {new_status}"
)
return False
...
```
**Option B**: Always log + metric, return bool (non-breaking)
```python
def transition_to_state(...) -> bool:
if not self.validate_state_transition(job_id, new_status):
# Log with structured data for alerting
logger.error(
f"STATE_TRANSITION_FAILED job={job_id} from={current} to={new_status}",
extra={"job_id": job_id, "from_status": current, "to_status": new_status}
)
# Emit metric for monitoring
emit_metric("state_transition_failed", 1, {"from": current, "to": new_status})
return False
...
```
**Recommendation**: Option A with `raise_on_invalid=True` default. Forces callers to handle failures explicitly.
### 2. Centralize "Start Processing" Pattern (High Priority)
Create a single function that handles the transition + worker trigger pattern:
```python
# backend/services/job_manager.py
async def start_job_processing(self, job_id: str, worker_service) -> None:
"""
Transition job to DOWNLOADING and trigger workers.
This is the single entry point for starting job processing.
All handlers (file_upload, webhook, audio_search) should use this.
Raises:
InvalidStateTransitionError: If job can't transition to DOWNLOADING
ValueError: If job not found or missing required fields
"""
job = self.get_job(job_id)
if not job:
raise ValueError(f"Job {job_id} not found")
if not job.input_media_gcs_path:
raise ValueError(f"Job {job_id} missing input_media_gcs_path")
# This will raise if transition is invalid
self.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=15,
message="Starting audio and lyrics processing"
)
# Only trigger workers if transition succeeded
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
```
Then update all handlers to use this:
```python
# In _handle_made_for_you_order, file_upload, audio_search:
await job_manager.start_job_processing(job_id, worker_service)
```
### 3. Add State Consistency Checks (Medium Priority)
Add a health check that detects inconsistent state:
```python
# backend/services/job_health_service.py
def check_job_consistency(job: Job) -> List[str]:
"""
Check for state inconsistencies in a job.
Returns list of issues found.
"""
issues = []
state_data = job.state_data or {}
# Check: audio_complete but status still pending
if state_data.get('audio_complete') and job.status == JobStatus.PENDING:
issues.append(f"audio_complete=True but status=pending")
# Check: lyrics_complete but status still pending
if state_data.get('lyrics_complete') and job.status == JobStatus.PENDING:
issues.append(f"lyrics_complete=True but status=pending")
# Check: screens_progress.stage=complete but status not awaiting_review
screens = state_data.get('screens_progress', {})
if screens.get('stage') == 'complete' and job.status not in [
JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE,
JobStatus.RENDERING_VIDEO, JobStatus.INSTRUMENTAL_SELECTED, ...
]:
issues.append(f"screens complete but status={job.status}")
return issues
```
Run this:
- In a daily Cloud Scheduler job
- After each worker completes
- Expose via `/health/job-consistency` endpoint
### 4. Workers Validate Prerequisites (Medium Priority)
Workers should verify job is in expected state before processing:
```python
# backend/workers/screens_worker.py
async def generate_screens(job_id: str) -> bool:
job = job_manager.get_job(job_id)
# Validate job is in expected state for this worker
valid_statuses = [JobStatus.DOWNLOADING, JobStatus.AUDIO_COMPLETE, JobStatus.LYRICS_COMPLETE]
if job.status not in [s.value for s in valid_statuses]:
logger.error(
f"[job:{job_id}] screens_worker called but job status is {job.status}, "
f"expected one of {valid_statuses}. This indicates a bug in the trigger logic."
)
# Option: return False, or try to recover, or alert
return False
# ... rest of worker
```
### 5. Add Integration Tests for State Flow (Medium Priority)
Add tests that verify the complete state machine flow without mocking transitions:
```python
# backend/tests/integration/test_state_machine_flow.py
@pytest.mark.integration
async def test_made_for_you_youtube_flow_state_transitions():
"""
Verify complete state flow for made-for-you YouTube orders.
Uses real JobManager (mocked Firestore) to validate state machine.
"""
# Create job
job = job_manager.create_job(...)
assert job.status == JobStatus.PENDING
# Simulate YouTube download complete
job_manager.update_job(job_id, {'input_media_gcs_path': 'test/path'})
# Start processing (this should transition to DOWNLOADING)
await job_manager.start_job_processing(job_id, mock_worker_service)
job = job_manager.get_job(job_id)
assert job.status == JobStatus.DOWNLOADING, \
f"Expected DOWNLOADING after start_job_processing, got {job.status}"
# ... continue through full flow
```
### 6. State Machine Diagram Generation (Low Priority)
Auto-generate a Mermaid diagram from `STATE_TRANSITIONS`:
```python
# scripts/generate_state_diagram.py
def generate_mermaid():
lines = ["stateDiagram-v2"]
for from_state, to_states in STATE_TRANSITIONS.items():
for to_state in to_states:
lines.append(f" {from_state} --> {to_state}")
return "\n".join(lines)
```
Add to `docs/ARCHITECTURE.md` and regenerate on changes.
### 7. Fix Admin Reset Buttons (High Priority)
The admin dashboard has reset buttons that bypass state machine validation and have several issues:
**Current buttons:**
- "Start" → reset to `pending`
- "Audio" → reset to `awaiting_audio_selection`
- "Lyrics" → reset to `awaiting_review`
- "Inst." → reset to `awaiting_instrumental_selection`
- "Reprocess" → reset to `instrumental_selected`
**Issues identified:**
1. **"Start" button leaves job stuck**: Resets to `pending` but doesn't trigger workers or clear `audio_complete`/`lyrics_complete` flags. Job sits indefinitely.
2. **"Lyrics" and "Inst." are obsolete**: Combined review flow (Jan 2026) merged these into a single step. Should be replaced with single "Review" button that resets to `awaiting_review`.
3. **Missing state_data keys in clear list for `pending`**:
```python
# Should also clear:
"audio_complete", "lyrics_complete"
```
4. **State machine bypass**: Admin resets use `update_job()` directly, not `transition_to_state()`. This is intentional for admin overrides but should be documented.
**Fixes needed:**
```python
# backend/api/routes/admin.py
# 1. Update STATE_DATA_CLEAR_KEYS for "pending"
STATE_DATA_CLEAR_KEYS = {
"pending": [
# ... existing keys ...
"audio_complete", # NEW - clear parallel processing flag
"lyrics_complete", # NEW - clear parallel processing flag
],
# ...
}
# 2. Remove awaiting_instrumental_selection from ALLOWED_RESET_STATES
ALLOWED_RESET_STATES = {
"pending",
"awaiting_audio_selection",
"awaiting_review", # Combined review (lyrics + instrumental)
# "awaiting_instrumental_selection", # REMOVED - deprecated
"instrumental_selected",
}
# 3. After reset to pending, trigger audio search if job has artist/title
# or provide clear guidance to admin about next steps
```
```tsx
// frontend/app/admin/jobs/page.tsx
// Replace "Lyrics" and "Inst." buttons with single "Review" button:
{[
{ state: "pending", icon: RotateCcw, label: "Start" },
{ state: "awaiting_audio_selection", icon: Music, label: "Audio" },
{ state: "awaiting_review", icon: Mic, label: "Review" }, // Combined
// REMOVED: { state: "awaiting_instrumental_selection", ... }
{ state: "instrumental_selected", icon: RefreshCw, label: "Reprocess" },
]}
```
## Implementation Priority
1. **Week 1** ✓ DONE:
- ✓ Make `transition_to_state` raise by default (`InvalidStateTransitionError`, `raise_on_invalid=True`)
- ✓ Create `start_job_processing()` helper in `job_manager.py`
- ✓ Update `_handle_made_for_you_order` to use it
- ✓ Added 15 unit tests covering exception behavior and new helper
2. **Week 1.5** ✓ DONE (Admin Button Fixes):
- ✓ Add `audio_complete`, `lyrics_complete` to STATE_DATA_CLEAR_KEYS for "pending"
- ✓ Remove `awaiting_instrumental_selection` from ALLOWED_RESET_STATES
- ✓ Update frontend to replace "Lyrics"/"Inst." with single "Review" button
- ✓ Add tests for admin reset endpoint (`test_reset_to_pending_clears_parallel_processing_flags`, `test_rejects_deprecated_awaiting_instrumental_selection`)
3. **Week 2** ✓ DONE:
- ✓ Update `file_upload.py` and `audio_search.py` to use `start_job_processing()`
- ✓ Fixed double-transition bug in YouTube URL upload flow
- ✓ Removed unused `_trigger_workers_parallel` helper from file_upload.py and audio_search.py
- ✓ Add worker prerequisite validation (`validate_worker_can_run()` in all workers)
- ✓ Add state consistency health check (`job_health_service.py` with `check_job_consistency()`)
- ✓ Added 26 tests for job_health_service
4. **Week 3** ✓ DONE:
- ✓ Add integration tests for state flow (`tests/integration/test_state_machine_flow.py` - 11 tests)
- ✓ Generate state machine diagram script (`scripts/generate_state_diagram.py`)
- ✓ Expose `/health/job-consistency` endpoint for admin dashboard
- TODO: Add monitoring/alerting on `STATE_TRANSITION_FAILED` logs (requires Cloud Logging alert setup)
## Quick Wins (Can Do Now)
1. Add explicit error logging when transition fails (already exists but add job_id to structured logging)
2. Create Cloud Logging alert for "Invalid state transition" errors
3. Add consistency check to admin dashboard "stuck jobs" view
@@ -0,0 +1,343 @@
# Plan: Add Retry Logic for Flacfetch Download Failures
**Created:** 2026-02-10
**Branch:** feat/sess-20260210-1158-investigate-job-749141f8
**Status:** ✅ Completed (2026-02-10)
## Implementation Summary
Successfully implemented all four related fixes in a single PR:
1. **Retry Logic** - Added manual exponential backoff retry logic to flacfetch_client.py (9 attempts over ~6 minutes)
2. **Source Persistence** - Modified audio_search.py to save download params BEFORE attempting download
3. **Retry Endpoint** - Enhanced jobs.py retry endpoint to handle audio search jobs using saved params
4. **UI Feedback** - Added error toast notifications in JobActions.tsx
**Key Implementation Decision:** Used manual retry logic with `asyncio.sleep` instead of tenacity decorator to ensure runtime configuration flexibility and simpler test mocking.
**Tests:** Added 5 comprehensive retry tests - all passing. Backend test suite: 103 passed.
---
## Overview
Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"
Investigation revealed **multiple related issues** when flacfetch network calls fail:
1. **No retry logic** - A single network hiccup causes immediate job failure
2. **Missing source info** - When download fails, source_id/source_name aren't saved to job
3. **Broken retry button** - Can't retry because no source info was persisted
4. **Silent UI failures** - Frontend doesn't show retry error messages to user
### Root Cause
Audio search flow saves selection info to `state_data`, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.
### Solution
This plan fixes all four issues:
- Add retry logic to flacfetch client (network resilience)
- Persist download info before attempting download (enables retry)
- Update retry endpoint to handle audio search jobs
- Show error messages in frontend when retry fails
## Requirements
### Functional Requirements - Flacfetch Retry Logic
- [x] Retry `download_by_id()` on `httpx.RequestError` (network failures)
- [x] Use exponential backoff spread over ~5 minutes
- [x] Configurable max retries via environment variable
- [x] Log each retry attempt with context
- [x] Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
- [x] Do retry on HTTP 5xx errors (server errors - transient)
### Functional Requirements - Persist Download Info
- [x] Save source_id, source_name, target_file, download_url to job document
- [x] Save BEFORE attempting download (not after)
- [x] Store in top-level fields for easy retry access
- [x] Preserve existing state_data fields (selected_audio_index, etc.)
### Functional Requirements - Retry Endpoint
- [x] Add case for audio search jobs in `/jobs/{id}/retry` endpoint
- [x] Check for saved download params (source_id, source_name)
- [x] Re-download using saved params
- [x] Clear error state before retrying
- [x] Trigger appropriate workers after successful download
### Functional Requirements - Frontend Error Handling
- [x] Show toast notification when retry fails
- [x] Display actual error message from API
- [x] Don't just console.error - make it visible to user
### Non-Functional Requirements
- [x] Minimal code changes (use existing patterns)
- [x] Clear error messages that include retry context
- [x] Backwards compatible (no breaking changes)
- [x] Should not significantly increase total timeout duration
## Technical Approach
### Architecture Decisions
**Option 1: Tenacity library**
- ✅ Industry standard, well-tested
- ✅ Declarative retry configuration
- ✅ Built-in exponential backoff
- ❌ Module-load-time decorator evaluation made runtime config difficult
- ❌ Mocking complexity in tests
**Option 2: Manual retry loop (CHOSEN)**
- ✅ No new dependencies
- ✅ Full control over logic
- ✅ Runtime configuration works perfectly
- ✅ Easy to test with mocking
- ❌ More code to maintain (mitigated by comprehensive tests)
**Decision:** Implemented **manual retry logic** with `asyncio.sleep` after discovering tenacity decorators evaluate settings at module load time, not runtime.
### Retry Strategy
```python
# Retry on network errors only (RequestError)
# Do NOT retry on:
# - HTTP 4xx (client errors like 400, 401, 404)
# - Already completed downloads
# Do retry on:
# - httpx.RequestError (connection timeout, refused, DNS failure)
# - HTTP 5xx (server errors - service temporarily down)
Max retries: 8 (total 9 attempts)
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
- Attempt 1: immediate
- Attempt 2: wait 10s
- Attempt 3: wait 20s
- Attempt 4: wait 40s
- Attempt 5: wait 60s (capped)
- Attempt 6: wait 60s (capped)
- Attempt 7: wait 60s (capped)
- Attempt 8: wait 60s (capped)
- Attempt 9: wait 60s (capped)
Total max time: ~370s (6+ minutes) of backoff + request timeouts
This gives the flacfetch service time to recover from restarts,
network issues, or temporary VM problems.
```
### Configuration
Added to `backend/config.py`:
```python
# Flacfetch client retry settings
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
```
### Error Message Improvements
Change from:
```
Download by ID request failed:
```
To:
```
Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out
```
## Implementation Steps
### Phase 1: Check Dependencies
1. [x] Check if `tenacity` is in `pyproject.toml` dependencies
2. [x] Add `tenacity = "^8.2.0"` to dependencies (for future use)
3. [x] Run `poetry lock` if dependencies changed
### Phase 2: Add Configuration
4. [x] Add retry configuration to `backend/config.py`
- `flacfetch_retry_max_attempts = 9`
- `flacfetch_retry_min_wait = 10.0`
- `flacfetch_retry_max_wait = 60.0`
### Phase 3: Implement Flacfetch Retry Logic
5. [x] Implement manual retry logic in `backend/services/flacfetch_client.py`
6. [x] Create `with_retry()` wrapper with:
- Retry on `httpx.RequestError`
- Retry on `httpx.HTTPStatusError` with 5xx status
- Stop on `httpx.HTTPStatusError` with 4xx status
- Exponential backoff with cap at 60s
- Retry attempt logging
7. [x] Wrap `_download_by_id_impl()` with retry logic
8. [x] Move exception wrapping OUTSIDE retry logic so httpx exceptions can bubble through
9. [x] Improve error messages to include:
- Number of attempts made
- Original exception type and message
### Phase 4: Persist Download Info Before Download
10. [x] Update `audio_search.py:_download_and_start_processing()`
11. [x] Save to job document BEFORE download attempt:
```python
job_manager.update_job(job_id, {
'audio_source_type': 'audio_search',
'source_name': source_name,
'source_id': source_id,
'target_file': target_file,
'download_url': download_url,
})
```
12. [x] Add fields to Job model (audio_source_type, source_name, source_id, target_file, download_url)
### Phase 5: Update Retry Endpoint for Audio Search
13. [x] Update `backend/api/routes/jobs.py:retry_job()`
14. [x] Add comprehensive audio search retry case (~150 lines)
15. [x] Check if job has saved download params
16. [x] Re-download based on source_name (YouTube, RED, OPS, Spotify)
17. [x] Clear error state and transition to DOWNLOADING
18. [x] Trigger appropriate workers
### Phase 6: Frontend Error Handling
19. [x] Update `frontend/components/job/JobActions.tsx`
20. [x] Use existing `useToast` hook
21. [x] Show toast on retry success and failure
22. [x] Extract error message from API response
### Phase 7: Testing - Flacfetch Retry
23. [x] Add 5 comprehensive tests to `backend/tests/test_flacfetch_client.py`:
- `test_download_by_id_retries_on_request_error` - Retry on network errors
- `test_download_by_id_no_retry_on_4xx` - No retry on client errors
- `test_download_by_id_retries_on_5xx` - Retry on server errors
- `test_download_by_id_fails_after_max_retries` - Max retries exhausted
- `test_download_retries_same_as_download_by_id` - `download()` wrapper works
24. [x] All tests passing (5 passed, 28 warnings)
### Phase 8: Testing - End-to-End
25. [x] Run full backend test suite: 103 passed ✅
26. [ ] ~~Manual production testing~~ (will be done post-deployment)
### Phase 9: Documentation
27. [ ] Update `docs/LESSONS-LEARNED.md` (will do in separate commit if needed)
28. [ ] This plan document serves as implementation record
## Files Created/Modified
| File | Action | Description |
|------|--------|-------------|
| `backend/config.py` | ✅ Modified | Added retry configuration variables |
| `backend/services/flacfetch_client.py` | ✅ Modified | Added manual retry logic with exponential backoff |
| `backend/api/routes/audio_search.py` | ✅ Modified | Save download info before attempting download |
| `backend/api/routes/jobs.py` | ✅ Modified | Added comprehensive audio search retry case (~150 lines) |
| `backend/models/job.py` | ✅ Modified | Added fields: audio_source_type, source_name, source_id, target_file, download_url |
| `frontend/components/job/JobActions.tsx` | ✅ Modified | Show error toast when retry fails |
| `backend/tests/test_flacfetch_client.py` | ✅ Modified | Added 5 retry tests - all passing |
| `pyproject.toml` | ✅ Modified | Added tenacity dependency (for potential future use) |
## Testing Strategy
### Unit Tests (`backend/tests/test_flacfetch_client.py`)
All tests implemented and passing ✅:
1. ✅ **test_download_by_id_retries_on_request_error** - Verifies retry on RequestError with exponential backoff
2. ✅ **test_download_by_id_no_retry_on_4xx** - Verifies NO retry on HTTP 404
3. ✅ **test_download_by_id_retries_on_5xx** - Verifies retry on HTTP 503
4. ✅ **test_download_by_id_fails_after_max_retries** - Verifies failure after max attempts
5. ✅ **test_download_retries_same_as_download_by_id** - Verifies `download()` wrapper has same behavior
### Integration Tests
Backend test suite: **103 passed, 900 warnings** ✅
### Manual Testing (Post-Deployment)
Still to do:
- Test with actual flacfetch service restart
- Verify retry button works on failed audio search jobs
- Verify toast notifications appear correctly
## Open Questions
### Resolved During Implementation
- [x] ~~Is tenacity already in dependencies?~~ - Added to pyproject.toml
- [x] ~~What's the appropriate total timeout budget?~~ - 9 attempts over ~6 minutes
- [x] ~~Should we fix the Retry button UX in this PR?~~ - YES, included in same PR
- [x] ~~Why did job 749141f8 fail before saving audio source info?~~ - Download info saved AFTER download, not before
- [x] ~~Which toast library should frontend use?~~ - Used existing `useToast` hook
- [x] ~~Should we save download params to state_data or top-level fields?~~ - Top-level fields
### Deferred for Future Work
- [ ] Should we also add retries to `wait_for_download()`?
- Probably NO - that already has its own polling/timeout logic
- [ ] Should we add circuit breaker pattern?
- Probably NO for now - keep it simple
- Could add in future if we see cascading failures
- [ ] Should retry logic be applied to other flacfetch methods?
- `start_download()` - probably YES (same pattern)
- `search()` - maybe YES (search can also have network issues)
- `get_download_status()` - probably NO (polling already handles this)
- [ ] Apply similar pattern to other external service calls:
- AudioShake API (lyrics transcription)
- Modal API (audio separation)
- YouTube direct API calls (not via flacfetch)
- Email sending (SendGrid)
## Rollback Plan
If this causes issues:
1. **Quick rollback**: Set env var `FLACFETCH_RETRY_MAX_ATTEMPTS=1` in Cloud Run
- Disables retries without code change
- Can deploy in ~2 minutes
2. **Full rollback**: Revert the PR
- Git revert the merge commit
- Deploy previous version
3. **Monitoring**: Watch for:
- Increased request duration to flacfetch
- Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
- Duplicate download attempts causing issues
## Success Criteria
- [x] Jobs with transient flacfetch failures succeed instead of failing ✅
- [x] Error messages are more informative (show attempts and root cause) ✅
- [x] No increase in false positives (jobs that should fail, now hang) ✅
- [x] All tests pass ✅
- [x] No impact on successful jobs (retry overhead is minimal) ✅
## Related Issues
### Job 749141f8 - The Incident
**What happened:**
- User searched for audio via audio search
- Selected "Elizabeth Moen - Songbird"
- Backend tried to download via flacfetch
- Flacfetch call failed with network error (connection timeout/refused)
- Job immediately failed
- No source info saved → Retry button broken
- User sees "Retry" button but clicking does nothing (silent 400 error)
**How this fix helps:**
1. ✅ Flacfetch retry logic → Download would have succeeded after retry
2. ✅ Persist download info → Even if all retries fail, user can retry later
3. ✅ Retry endpoint fix → Retry button will actually work
4. ✅ Frontend toast → User sees helpful error if retry still fails
**Note:** Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.
### Future Work (Not in This PR)
Similar pattern should be applied to other external service calls:
- AudioShake API (lyrics transcription)
- Modal API (audio separation)
- YouTube direct API calls (not via flacfetch)
- Email sending (SendGrid)
## References
- [Tenacity documentation](https://tenacity.readthedocs.io/)
- [LESSONS-LEARNED.md](docs/LESSONS-LEARNED.md) - External service response format mismatches
- [flacfetch README](../../flacfetch/README.md) - Understanding the service
@@ -0,0 +1,279 @@
# Plan: Private (Non-Published) Tracks
**Created:** 2026-02-24
**Branch:** feat/sess-20260224-1149-private-tracks
**Status:** Draft
## Overview
Add a `is_private` flag to jobs that changes distribution behavior:
- **Dropbox**: Uses `Tracks-NonPublished` folder (not `Tracks-Organized`) with brand code `NOMADNP` (not `NOMAD`)
- **YouTube**: Skipped entirely
- **Google Drive**: Skipped entirely
- **Email**: Completion email includes Dropbox link but no YouTube section
- **Discord**: Still notified (for internal tracking)
Available to all users at job creation and toggleable by admin on existing jobs.
## Requirements
- [ ] All users can mark a job as "Private (Non-Published)" when creating it
- [ ] Admin can toggle `is_private` on any existing job via admin dashboard
- [ ] Private jobs upload to Dropbox at `Tracks-NonPublished` with `NOMADNP-XXXX` brand codes
- [ ] Private jobs skip YouTube and Google Drive uploads entirely
- [ ] Completion emails omit YouTube section when no YouTube URL (private jobs + any other edge case)
- [ ] When admin toggles a completed job to private, existing outputs are auto-deleted
- [ ] Discord notifications still fire for private jobs
- [ ] Existing (non-private) jobs are completely unaffected (backward compatible)
## Technical Approach
### Core Design: Single Boolean Flag with Runtime Overrides
Store `is_private: bool = False` on the job model. At distribution time, a helper function returns effective distribution parameters based on this flag. This means:
- **Toggling is_private automatically changes all distribution behavior** - no need to manually update dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id
- **Job fields still store the "normal" defaults** - the override happens at the config-to-orchestrator boundary
- **Admin can flip one field** and everything adapts
### Key Helper Function
```python
# In job_defaults_service.py
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
"""Return distribution settings, applying private overrides if is_private=True."""
if getattr(job, 'is_private', False):
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=settings.default_private_dropbox_path,
brand_prefix=settings.default_private_brand_prefix,
enable_youtube_upload=False,
gdrive_folder_id=None,
# Keep other settings from job (discord, youtube_description, etc.)
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
# Non-private: return job's own settings
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=getattr(job, 'youtube_description_template', None),
)
```
This helper is used in both distribution paths (orchestrator and native fallback).
### Distribution Paths That Need Changes
Two code paths handle distribution (see LESSONS-LEARNED: "Fix Both Sides of Dual Code Paths"):
1. **Orchestrator path** (`video_worker_orchestrator.py`): Primary path. `create_orchestrator_config_from_job()` uses the helper to build config.
2. **Native distribution path** (`video_worker.py` → `_handle_native_distribution()`): Fallback path. Also uses the helper instead of reading job fields directly.
### Email Template Change
Add conditional YouTube section removal in `render_template()`, matching the existing pattern for `feedback_url`:
```python
# Remove YouTube section if no URL provided
if not variables.get("youtube_url"):
result = re.sub(
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
'\n',
result,
flags=re.DOTALL
)
variables["youtube_url"] = ""
```
And change `render_job_completion()` to pass `None` instead of `"[YouTube URL not available]"` when youtube_url is empty.
### Admin Toggle Auto-Delete
When admin PATCH updates `is_private` from False to True on a job that has existing outputs, reuse the existing delete-outputs logic to remove YouTube + GDrive + Dropbox outputs. Admin can then "Reprocess" to re-distribute to the private Dropbox path.
## Implementation Steps
### Phase 1: Backend Model & Config (foundation)
1. [ ] **Add `is_private` to Job model** (`backend/models/job.py`)
- Add `is_private: bool = False` field after `made_for_you` field
- Add to `JobCreate` model as well
2. [ ] **Add private distribution config** (`backend/config.py`)
- Add `default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")`
- Add `default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")`
3. [ ] **Add helper function** (`backend/services/job_defaults_service.py`)
- Add `get_effective_distribution_for_job(job)` function that returns distribution settings with private overrides applied
### Phase 2: Backend API (job creation + admin)
4. [ ] **Add `is_private` to request models** (`backend/models/requests.py`)
- Add `is_private: Optional[bool] = None` to `URLSubmissionRequest`
- Add `is_private: Optional[bool] = None` to `AudioSearchRequest`
5. [ ] **Update URL creation endpoint** (`backend/api/routes/jobs.py`)
- Accept and pass through `is_private` field to `JobCreate`
6. [ ] **Update file upload endpoint** (`backend/api/routes/file_upload.py`)
- Accept `is_private` Form parameter
- Pass through to `JobCreate`
7. [ ] **Update audio search endpoint** (`backend/api/routes/audio_search.py`)
- Accept `is_private` in `AudioSearchRequest`
- Pass through to `JobCreate`
8. [ ] **Update admin endpoints** (`backend/api/routes/admin.py`)
- Add `is_private: Optional[bool] = None` to `JobUpdateRequest`
- Add `"is_private"` to `EDITABLE_JOB_FIELDS`
- In the PATCH handler: detect when `is_private` changes from False→True on a completed job, auto-delete existing outputs (call existing delete-outputs service logic)
### Phase 3: Distribution Pipeline
9. [ ] **Update orchestrator config creation** (`backend/workers/video_worker_orchestrator.py`)
- In `create_orchestrator_config_from_job()`: Use `get_effective_distribution_for_job()` helper to determine dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id instead of reading directly from job
10. [ ] **Update native distribution fallback** (`backend/workers/video_worker.py`)
- In `_handle_native_distribution()`: Use `get_effective_distribution_for_job()` helper instead of reading directly from job fields
### Phase 4: Email Template
11. [ ] **Add YouTube section conditional removal** (`backend/services/template_service.py`)
- In `render_template()`: Add regex to remove YouTube paragraph when `youtube_url` is empty (matching existing `feedback_url` pattern)
- In `render_job_completion()`: Pass `None` for youtube_url when empty (instead of `"[YouTube URL not available]"`)
### Phase 5: Frontend
12. [ ] **Add private checkbox to job creation form** (`frontend/components/job/JobSubmission.tsx`)
- Add "Private (Non-Published)" checkbox below the existing "Skip lyrics review" checkbox
- Show for all users (not just admin)
- Include brief description text: "Private tracks go to Dropbox only, not YouTube or Google Drive"
13. [ ] **Update API interfaces** (`frontend/lib/api.ts`)
- Add `is_private?: boolean` to all job creation option interfaces
- Add `is_private` to `JobUpdateRequest` interface
- Pass through in `uploadJob()`, `createJobFromUrl()`, `searchAudio()` methods
14. [ ] **Add private toggle to admin job detail** (`frontend/app/admin/jobs/page.tsx`)
- Add `is_private` as an inline-editable field in the job overview section (use Switch component)
- Show current state with visual indicator (e.g., lock icon for private jobs)
- When toggling to private on a completed job, show confirmation dialog about auto-deleting outputs
### Phase 6: Environment Configuration
15. [ ] **Update CI/deployment config** (`.github/workflows/ci.yml` or Cloud Run env)
- Add `DEFAULT_PRIVATE_DROPBOX_PATH` environment variable (e.g., `/Karaoke/Tracks-NonPublished`)
- Add `DEFAULT_PRIVATE_BRAND_PREFIX` environment variable (e.g., `NOMADNP`)
- These should be configured in Cloud Run service and CI test environment
### Phase 7: Testing
16. [ ] **Unit tests for private distribution logic** (`backend/tests/test_private_tracks.py`)
- Test `get_effective_distribution_for_job()` with private and non-private jobs
- Test `create_orchestrator_config_from_job()` applies private overrides correctly
- Test `_run_distribution()` skips YouTube and GDrive when private config applied
- Test `_upload_to_dropbox()` uses private path and brand prefix
- Test brand code generation with NOMADNP prefix in Tracks-NonPublished
- Test `_handle_native_distribution()` respects private overrides
- Test non-private jobs are completely unaffected (regression tests)
17. [ ] **Unit tests for email changes** (add to `backend/tests/test_email_service.py` or `test_template_service.py`)
- Test YouTube section removed from email when youtube_url is None
- Test YouTube section present when youtube_url is provided
- Test Dropbox section still present for private jobs
- Test full email rendering for private job scenario
18. [ ] **Unit tests for admin toggle** (add to `backend/tests/test_admin.py` or new file)
- Test admin can update is_private field
- Test toggling is_private to True on completed job triggers output deletion
- Test toggling is_private to True on non-completed job does NOT delete outputs
- Test toggling is_private to False does nothing special
19. [ ] **Integration test** (`backend/tests/emulator/` or `test_distribution_services.py`)
- Test full distribution pipeline with is_private=True: verify Dropbox path, brand prefix, YouTube skip, GDrive skip
20. [ ] **Frontend E2E regression test** (`frontend/e2e/regression/`)
- Test private checkbox appears in job creation form
- Test private toggle in admin dashboard
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `is_private: bool = False` to Job and JobCreate |
| `backend/config.py` | Modify | Add `default_private_dropbox_path` and `default_private_brand_prefix` |
| `backend/services/job_defaults_service.py` | Modify | Add `get_effective_distribution_for_job()` helper |
| `backend/models/requests.py` | Modify | Add `is_private` to URL and AudioSearch request models |
| `backend/api/routes/jobs.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/file_upload.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/audio_search.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/admin.py` | Modify | Add `is_private` to editable fields + auto-delete logic |
| `backend/workers/video_worker_orchestrator.py` | Modify | Use helper in `create_orchestrator_config_from_job()` |
| `backend/workers/video_worker.py` | Modify | Use helper in `_handle_native_distribution()` |
| `backend/services/template_service.py` | Modify | Conditional YouTube section removal |
| `frontend/components/job/JobSubmission.tsx` | Modify | Add private checkbox |
| `frontend/lib/api.ts` | Modify | Add `is_private` to interfaces and API calls |
| `frontend/app/admin/jobs/page.tsx` | Modify | Add private toggle to admin detail |
| `backend/tests/test_private_tracks.py` | Create | Comprehensive unit tests for private track logic |
## Testing Strategy
### Unit Tests (~25-30 tests in `test_private_tracks.py`)
- `get_effective_distribution_for_job()` with private=True and False
- `create_orchestrator_config_from_job()` private overrides (YouTube=False, GDrive=None, private Dropbox path, NOMADNP brand)
- Distribution stage skips: YouTube not called, GDrive not called, Dropbox IS called with correct path
- Brand code generation uses NOMADNP prefix and Tracks-NonPublished path
- Native distribution fallback applies same overrides
- **Regression tests**: Non-private jobs have unchanged behavior (same paths, same brand code, YouTube+GDrive+Dropbox all fire)
### Email Template Tests (~5 tests)
- YouTube section stripped when youtube_url is None
- YouTube section preserved when youtube_url has value
- Dropbox section always present
- Full private job email rendering (Dropbox + no YouTube)
- Full non-private job email rendering (both present) - regression
### Admin Toggle Tests (~5 tests)
- is_private editable via PATCH
- Toggle to private on completed job deletes outputs
- Toggle to private on pending/in-progress job does NOT delete
- Toggle to non-private does nothing special
### Frontend Tests
- E2E: Private checkbox visible and functional in job creation
- E2E: Admin toggle visible and includes confirmation dialog
### Manual Verification
- Create a private job end-to-end, verify Dropbox folder in Tracks-NonPublished with NOMADNP code
- Verify completion email has Dropbox link but no YouTube section
- Toggle existing completed job to private, verify outputs deleted
## Regression Risk Analysis
| Risk | Mitigation |
|------|-----------|
| Non-private jobs affected | `is_private` defaults to `False`; helper returns unchanged values for non-private. Explicit regression tests. |
| Dual code paths (orchestrator vs native) | Both paths use same helper function. Tests cover both paths. (Lesson: "Fix Both Sides of Dual Code Paths") |
| Email template regex breaks other content | Regex targets specific YouTube paragraph text. Tests verify non-YouTube content untouched. |
| Brand code namespace collision | NOMADNP codes are in a completely separate Dropbox folder (Tracks-NonPublished), no overlap with NOMAD codes in Tracks-Organized |
| Admin auto-delete on wrong job | Only triggers when `is_private` changes from False→True AND job has terminal status (complete). Confirmation dialog in frontend. |
| Made-for-you orders | MFY webhook doesn't set is_private (defaults False). Admin can toggle after. No change to MFY flow. |
| Existing Firestore documents | Missing `is_private` field defaults to `False` via model default. No migration needed. |
## Open Questions
- [x] Who sees private toggle? → **Everyone** (all users at creation, admin on existing)
- [x] Auto-delete on toggle? → **Yes**, auto-delete all outputs when toggling to private on completed job
- [x] Email format? → **Remove YouTube section** when no YouTube URL (improves all edge cases)
- [x] Discord? → **Still notify** (useful for internal tracking)
## Rollback Plan
1. **Feature flag**: The `is_private` field defaults to `False`, so setting `DEFAULT_PRIVATE_DROPBOX_PATH` to empty/None effectively disables private distribution
2. **No schema migration needed**: Missing `is_private` field on existing Firestore documents defaults to `False`
3. **Email change is backward-compatible**: The YouTube section removal only happens when youtube_url is None, which is already an edge case showing "[YouTube URL not available]"
4. **Frontend**: Private checkbox is additive UI, removing it just means users can't create new private jobs
+136 -1
View File
@@ -2,7 +2,7 @@
* Unit tests for job-status.ts utility functions
*/
import { getJobStep, formatStepIndicator, isBlockingStatus, getJobProgressPercent, getJobPriority, sortJobsByPriority, JobStep } from '../lib/job-status';
import { getJobStep, formatStepIndicator, isBlockingStatus, getJobProgressPercent, getJobPriority, sortJobsByPriority, getDisplayJobs, JobStep } from '../lib/job-status';
import type { Job } from '../lib/api';
// Helper to create a minimal Job object for testing
@@ -595,3 +595,138 @@ describe('sortJobsByPriority', () => {
expect(sorted[4].status).toBe('failed'); // failed
});
});
describe('getDisplayJobs', () => {
// Helper: pre-sorted jobs (blocking, processing, complete, failed)
function createSortedJobs(counts: { blocking?: number; processing?: number; complete?: number; failed?: number }): Job[] {
const jobs: Job[] = [];
let t = 20; // descending timestamp within each priority group
for (let i = 0; i < (counts.blocking ?? 0); i++) {
jobs.push({ job_id: `blocking-${i}`, status: 'awaiting_review', progress: 0, created_at: `2024-01-01T${t--}:00:00Z`, updated_at: '2024-01-01T00:00:00Z' });
}
for (let i = 0; i < (counts.processing ?? 0); i++) {
jobs.push({ job_id: `processing-${i}`, status: 'rendering_video', progress: 0, created_at: `2024-01-01T${t--}:00:00Z`, updated_at: '2024-01-01T00:00:00Z' });
}
for (let i = 0; i < (counts.complete ?? 0); i++) {
jobs.push({ job_id: `complete-${i}`, status: 'complete', progress: 100, created_at: `2024-01-01T${t--}:00:00Z`, updated_at: '2024-01-01T00:00:00Z' });
}
for (let i = 0; i < (counts.failed ?? 0); i++) {
jobs.push({ job_id: `failed-${i}`, status: 'failed', progress: 0, created_at: `2024-01-01T${t--}:00:00Z`, updated_at: '2024-01-01T00:00:00Z' });
}
return jobs;
}
it('returns all jobs when fewer than display limit', () => {
const jobs = createSortedJobs({ complete: 3 });
const { displayedJobs, totalFetched } = getDisplayJobs(jobs, 5);
expect(displayedJobs).toHaveLength(3);
expect(totalFetched).toBe(3);
});
it('returns empty array for empty input', () => {
const { displayedJobs, totalFetched } = getDisplayJobs([], 5);
expect(displayedJobs).toHaveLength(0);
expect(totalFetched).toBe(0);
});
it('limits to displayLimit when no incomplete jobs', () => {
const jobs = createSortedJobs({ complete: 10 });
const { displayedJobs, totalFetched } = getDisplayJobs(jobs, 5);
expect(displayedJobs).toHaveLength(5);
expect(totalFetched).toBe(10);
});
it('expands beyond limit when more incomplete than limit', () => {
const jobs = createSortedJobs({ blocking: 3, processing: 4, complete: 5 });
const { displayedJobs } = getDisplayJobs(jobs, 5);
// 7 incomplete > limit of 5, so show all 7
expect(displayedJobs).toHaveLength(7);
expect(displayedJobs.filter(j => j.status === 'awaiting_review')).toHaveLength(3);
expect(displayedJobs.filter(j => j.status === 'rendering_video')).toHaveLength(4);
});
it('fills remaining slots with completed when fewer incomplete than limit', () => {
const jobs = createSortedJobs({ blocking: 1, processing: 1, complete: 10 });
const { displayedJobs } = getDisplayJobs(jobs, 5);
// 2 incomplete < 5 limit, so show 5 total (2 incomplete + 3 completed)
expect(displayedJobs).toHaveLength(5);
expect(displayedJobs[0].status).toBe('awaiting_review');
expect(displayedJobs[1].status).toBe('rendering_video');
expect(displayedJobs[2].status).toBe('complete');
});
it('returns all jobs when displayLimit is -1 (All mode)', () => {
const jobs = createSortedJobs({ blocking: 2, complete: 20 });
const { displayedJobs, totalFetched } = getDisplayJobs(jobs, -1);
expect(displayedJobs).toHaveLength(22);
expect(totalFetched).toBe(22);
});
it('preserves sort order from input', () => {
const jobs = createSortedJobs({ blocking: 1, processing: 1, complete: 3 });
const { displayedJobs } = getDisplayJobs(jobs, 5);
expect(displayedJobs[0].status).toBe('awaiting_review');
expect(displayedJobs[1].status).toBe('rendering_video');
expect(displayedJobs[2].status).toBe('complete');
expect(displayedJobs[3].status).toBe('complete');
expect(displayedJobs[4].status).toBe('complete');
});
describe('hideCompleted filter', () => {
it('filters out complete and failed jobs when hideCompleted is true', () => {
const jobs = createSortedJobs({ blocking: 1, processing: 2, complete: 5, failed: 3 });
const { displayedJobs, totalFetched } = getDisplayJobs(jobs, -1, true);
expect(totalFetched).toBe(11);
expect(displayedJobs).toHaveLength(3);
expect(displayedJobs.every(j => j.status === 'awaiting_review' || j.status === 'rendering_video')).toBe(true);
});
it('respects display limit after filtering', () => {
const jobs = createSortedJobs({ blocking: 2, processing: 5, complete: 10 });
const { displayedJobs } = getDisplayJobs(jobs, 5, true);
// 7 incomplete jobs, all visible since incomplete count (7) > limit (5)
expect(displayedJobs).toHaveLength(7);
});
it('returns empty when all jobs are complete and hideCompleted is true', () => {
const jobs = createSortedJobs({ complete: 10 });
const { displayedJobs, totalFetched } = getDisplayJobs(jobs, 5, true);
expect(displayedJobs).toHaveLength(0);
expect(totalFetched).toBe(10);
});
it('does not filter when hideCompleted is false', () => {
const jobs = createSortedJobs({ blocking: 1, complete: 5, failed: 2 });
const { displayedJobs } = getDisplayJobs(jobs, -1, false);
expect(displayedJobs).toHaveLength(8);
});
it('defaults to showing all when hideCompleted is omitted', () => {
const jobs = createSortedJobs({ complete: 5, failed: 2 });
const { displayedJobs } = getDisplayJobs(jobs, -1);
expect(displayedJobs).toHaveLength(7);
});
});
it('core bug: old incomplete job visible even when recent jobs are all completed', () => {
// Simulate the bug scenario: 5 recent completed jobs + 1 older blocking job
// With old behavior (LIMIT 5 at query), the blocking job would never be fetched.
// With new behavior, all 6 are fetched and display limit of 5 expands to show blocking.
const jobs = sortJobsByPriority([
{ job_id: 'old-blocking', status: 'awaiting_review', progress: 0, created_at: '2024-01-01T01:00:00Z', updated_at: '2024-01-01T01:00:00Z' },
{ job_id: 'new-1', status: 'complete', progress: 100, created_at: '2024-01-10T01:00:00Z', updated_at: '2024-01-10T01:00:00Z' },
{ job_id: 'new-2', status: 'complete', progress: 100, created_at: '2024-01-09T01:00:00Z', updated_at: '2024-01-09T01:00:00Z' },
{ job_id: 'new-3', status: 'complete', progress: 100, created_at: '2024-01-08T01:00:00Z', updated_at: '2024-01-08T01:00:00Z' },
{ job_id: 'new-4', status: 'complete', progress: 100, created_at: '2024-01-07T01:00:00Z', updated_at: '2024-01-07T01:00:00Z' },
{ job_id: 'new-5', status: 'complete', progress: 100, created_at: '2024-01-06T01:00:00Z', updated_at: '2024-01-06T01:00:00Z' },
]);
const { displayedJobs } = getDisplayJobs(jobs, 5);
// The blocking job must be visible
const blockingJobs = displayedJobs.filter(j => j.status === 'awaiting_review');
expect(blockingJobs).toHaveLength(1);
expect(blockingJobs[0].job_id).toBe('old-blocking');
// Should show 5 total (1 incomplete + 4 completed, since limit is 5 and max(5, 1) = 5)
expect(displayedJobs).toHaveLength(5);
});
});
+318 -78
View File
@@ -2,7 +2,7 @@
import { useEffect, useState, useCallback, Suspense } from "react"
import { useRouter, useSearchParams } from "next/navigation"
import { adminApi, api, Job, FileInfo, JobFilesResponse, JobUpdateRequest, JobResetResponse, DeleteOutputsResponse, ClearWorkersResponse, TriggerWorkerResponse } from "@/lib/api"
import { adminApi, api, Job, FileInfo, JobFilesResponse, JobUpdateRequest, JobResetResponse, DeleteOutputsResponse, RegenerateScreensResponse, RestartJobResponse, OverrideAudioSourceResponse } from "@/lib/api"
import { useAdminSettings } from "@/lib/admin-settings"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
@@ -182,11 +182,21 @@ function AdminJobsPageContent() {
const [deleteOutputsResult, setDeleteOutputsResult] = useState<DeleteOutputsResponse | null>(null)
const [deleteOutputsResultOpen, setDeleteOutputsResultOpen] = useState(false)
// Clear workers state
const [clearingWorkers, setClearingWorkers] = useState(false)
// Note: clearingWorkers and triggeringWorker states removed
// These low-level escape hatches are replaced by Full Restart and Regen Screens
// Trigger worker state
const [triggeringWorker, setTriggeringWorker] = useState(false)
// Regenerate screens state
const [regeneratingScreens, setRegeneratingScreens] = useState(false)
const [regenerateScreensDialogOpen, setRegenerateScreensDialogOpen] = useState(false)
// Restart job state
const [restartDialogOpen, setRestartDialogOpen] = useState(false)
const [restarting, setRestarting] = useState(false)
const [restartPreserveStems, setRestartPreserveStems] = useState(true)
// Override audio source state
const [overrideAudioDialogOpen, setOverrideAudioDialogOpen] = useState(false)
const [overridingAudio, setOverridingAudio] = useState(false)
const loadJobs = useCallback(async () => {
try {
@@ -420,74 +430,113 @@ function AdminJobsPageContent() {
}
}
// Handle clear workers - clears worker progress to allow re-execution
const handleClearWorkers = async () => {
// Note: handleClearWorkers and handleTriggerWorker removed
// These low-level escape hatches are replaced by Full Restart and Regen Screens
// Handle regenerate screens - regenerates title/end screens with current metadata
const handleRegenerateScreens = async () => {
if (!selectedJobId) return
try {
setClearingWorkers(true)
const result = await adminApi.clearWorkers(selectedJobId)
setRegeneratingScreens(true)
const result = await adminApi.regenerateScreens(selectedJobId)
if (result.cleared_keys.length > 0) {
if (result.worker_triggered) {
toast({
title: "Workers Cleared",
description: `${result.message}: ${result.cleared_keys.join(", ")}`,
title: "Screen Regeneration Started",
description: "Title and end screens are being regenerated with current artist/title. This may take 30-60 seconds.",
})
} else {
toast({
title: "No Progress Keys",
description: result.message,
})
}
// Refresh job details
loadJobDetail(selectedJobId)
loadLogs(selectedJobId)
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to clear workers",
variant: "destructive",
})
} finally {
setClearingWorkers(false)
}
}
// Handle trigger worker - manually triggers a worker for processing
const handleTriggerWorker = async (workerType: string = "video") => {
if (!selectedJobId) return
try {
setTriggeringWorker(true)
const result = await adminApi.triggerWorker(selectedJobId, workerType)
if (result.triggered) {
toast({
title: `${workerType.charAt(0).toUpperCase() + workerType.slice(1)} Worker Triggered`,
description: `Job ${selectedJobId} is now processing. The ${workerType} worker will begin generating the video.`,
})
} else {
toast({
title: "Trigger Failed",
description: result.error
? `${result.message}: ${result.error}`
: result.message,
title: "Regeneration Failed",
description: result.error || result.message,
variant: "destructive",
})
}
setRegenerateScreensDialogOpen(false)
// Refresh job details
loadJobDetail(selectedJobId)
loadLogs(selectedJobId)
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to trigger worker",
description: err.message || "Failed to regenerate screens",
variant: "destructive",
})
} finally {
setTriggeringWorker(false)
setRegeneratingScreens(false)
}
}
// Handle restart job - fully restarts a job with worker triggering
const handleRestartJob = async () => {
if (!selectedJobId) return
try {
setRestarting(true)
const result = await adminApi.restartJob(selectedJobId, {
preserve_audio_stems: restartPreserveStems,
delete_outputs: true,
})
if (result.status === "success" || result.status === "partial") {
toast({
title: "Job Restarted",
description: `${result.message}. Workers triggered: ${result.workers_triggered.join(", ") || "none (waiting for action)"}`,
})
}
if (result.error) {
toast({
title: "Warning",
description: result.error,
variant: "destructive",
})
}
setRestartDialogOpen(false)
// Refresh job details
loadJobDetail(selectedJobId)
loadLogs(selectedJobId)
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to restart job",
variant: "destructive",
})
} finally {
setRestarting(false)
}
}
// Handle override audio source - switches to audio search mode
const handleOverrideAudioSource = async () => {
if (!selectedJobId) return
try {
setOverridingAudio(true)
const result = await adminApi.overrideAudioSource(selectedJobId, {
source_type: "audio_search",
})
toast({
title: "Audio Source Changed",
description: result.message,
})
setOverrideAudioDialogOpen(false)
// Refresh job details
loadJobDetail(selectedJobId)
loadLogs(selectedJobId)
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to override audio source",
variant: "destructive",
})
} finally {
setOverridingAudio(false)
}
}
@@ -496,7 +545,7 @@ function AdminJobsPageContent() {
const info: Record<string, { label: string; description: string; icon: any }> = {
pending: {
label: "Pending",
description: "Restart from the beginning. Clears all processing data.",
description: "Restart from the beginning. Clears all processing data including audio/lyrics progress.",
icon: RotateCcw,
},
awaiting_audio_selection: {
@@ -505,15 +554,10 @@ function AdminJobsPageContent() {
icon: Music,
},
awaiting_review: {
label: "Lyrics Review",
description: "Re-review lyrics. Preserves audio stems.",
label: "Combined Review",
description: "Re-review lyrics AND re-select instrumental. Preserves audio stems.",
icon: Mic,
},
awaiting_instrumental_selection: {
label: "Instrumental",
description: "Re-select instrumental. Preserves lyrics review.",
icon: Sliders,
},
instrumental_selected: {
label: "Reprocess Video",
description: "Re-encode and re-distribute with current settings. Triggers video worker automatically.",
@@ -814,10 +858,9 @@ function AdminJobsPageContent() {
<div className="flex items-center gap-2 flex-wrap">
<span className="text-xs font-medium text-muted-foreground uppercase tracking-wide mr-1">Reset to:</span>
{[
{ state: "pending", icon: RotateCcw, label: "Start" },
// Note: "pending" (Start) removed - use "Full Restart" button instead
{ state: "awaiting_audio_selection", icon: Music, label: "Audio" },
{ state: "awaiting_review", icon: Mic, label: "Lyrics" },
{ state: "awaiting_instrumental_selection", icon: Sliders, label: "Inst." },
{ state: "awaiting_review", icon: Mic, label: "Review" }, // Combined lyrics + instrumental review
{ state: "instrumental_selected", icon: RefreshCw, label: "Reprocess" },
].map(({ state, icon: Icon, label }) => (
<Tooltip key={state}>
@@ -864,21 +907,27 @@ function AdminJobsPageContent() {
</TooltipContent>
</Tooltip>
{/* Admin Actions */}
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="outline"
size="sm"
className="h-7 px-2 text-xs"
onClick={handleClearWorkers}
disabled={clearingWorkers}
className="h-7 px-2 text-xs border-blue-500 text-blue-600 hover:bg-blue-50 dark:hover:bg-blue-950"
onClick={() => setRegenerateScreensDialogOpen(true)}
disabled={regeneratingScreens}
>
<RefreshCw className={`w-3 h-3 mr-1 ${clearingWorkers ? "animate-spin" : ""}`} />
Clear Workers
{regeneratingScreens ? (
<Loader2 className="w-3 h-3 mr-1 animate-spin" />
) : (
<RefreshCw className="w-3 h-3 mr-1" />
)}
Regen Screens
</Button>
</TooltipTrigger>
<TooltipContent>
Clear worker progress markers to allow re-execution
<p className="font-medium">Regenerate Title/End Screens</p>
<p className="text-xs text-muted-foreground">Use after editing artist/title to update screens</p>
</TooltipContent>
</Tooltip>
@@ -887,23 +936,52 @@ function AdminJobsPageContent() {
<Button
variant="outline"
size="sm"
className={`h-7 px-2 text-xs ${selectedJob.status === "instrumental_selected" ? "border-green-500 text-green-600 hover:bg-green-50 dark:hover:bg-green-950" : ""}`}
onClick={() => handleTriggerWorker("video")}
disabled={triggeringWorker}
className="h-7 px-2 text-xs border-purple-500 text-purple-600 hover:bg-purple-50 dark:hover:bg-purple-950"
onClick={() => {
setRestartPreserveStems(true)
setRestartDialogOpen(true)
}}
disabled={restarting}
>
{triggeringWorker ? (
{restarting ? (
<Loader2 className="w-3 h-3 mr-1 animate-spin" />
) : (
<Play className="w-3 h-3 mr-1" />
<RotateCcw className="w-3 h-3 mr-1" />
)}
Trigger
Full Restart
</Button>
</TooltipTrigger>
<TooltipContent>
Manually trigger video worker processing
<p className="font-medium">Full Restart with Worker Triggering</p>
<p className="text-xs text-muted-foreground">Unlike Reset, this actually triggers workers to run</p>
</TooltipContent>
</Tooltip>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="outline"
size="sm"
className="h-7 px-2 text-xs border-amber-500 text-amber-600 hover:bg-amber-50 dark:hover:bg-amber-950"
onClick={() => setOverrideAudioDialogOpen(true)}
disabled={overridingAudio}
>
{overridingAudio ? (
<Loader2 className="w-3 h-3 mr-1 animate-spin" />
) : (
<Music className="w-3 h-3 mr-1" />
)}
Audio Search
</Button>
</TooltipTrigger>
<TooltipContent>
<p className="font-medium">Switch to Audio Search</p>
<p className="text-xs text-muted-foreground">Override YouTube URL with audio search for better quality</p>
</TooltipContent>
</Tooltip>
<Separator orientation="vertical" className="h-5 mx-1" />
<Tooltip>
<TooltipTrigger asChild>
<Button
@@ -1132,6 +1210,41 @@ function AdminJobsPageContent() {
)}
</div>
))}
{/* Private (non-published) toggle */}
<div className="flex items-center gap-2 pt-2 border-t">
<span className="text-[10px] font-medium text-muted-foreground uppercase w-20 shrink-0">Private</span>
<div className="flex items-center gap-2 flex-1">
<input
type="checkbox"
id="admin-is-private"
checked={selectedJob.is_private || false}
onChange={async (e) => {
const newValue = e.target.checked
try {
await adminApi.updateJob(selectedJobId!, { is_private: newValue })
setSelectedJob({ ...selectedJob, is_private: newValue })
toast({
title: "Updated",
description: newValue
? "Job set to private (outputs will be auto-deleted if completed)"
: "Job set to public",
})
} catch (err: any) {
toast({
title: "Error",
description: err.message || "Failed to update private flag",
variant: "destructive",
})
}
}}
className="w-3.5 h-3.5 rounded border-border accent-orange-500"
/>
<label htmlFor="admin-is-private" className="text-xs cursor-pointer">
Non-published (Dropbox only, no YouTube/GDrive)
</label>
</div>
</div>
</CardContent>
</Card>
@@ -1638,8 +1751,7 @@ function AdminJobsPageContent() {
<span className="font-medium">What happens next:</span>{" "}
{resetResult.new_status === "pending" && "Job will need to go through audio search, download, and all processing steps."}
{resetResult.new_status === "awaiting_audio_selection" && "User can select from the cached audio search results."}
{resetResult.new_status === "awaiting_review" && "User can review and correct the lyrics."}
{resetResult.new_status === "awaiting_instrumental_selection" && "User can select the instrumental track."}
{resetResult.new_status === "awaiting_review" && "User can review lyrics and select instrumental track."}
{resetResult.new_status === "instrumental_selected" && (
resetResult.worker_triggered
? "Video generation is now in progress."
@@ -1667,6 +1779,134 @@ function AdminJobsPageContent() {
</div>
</DialogContent>
</Dialog>
{/* Regenerate Screens Confirmation Dialog */}
<AlertDialog open={regenerateScreensDialogOpen} onOpenChange={setRegenerateScreensDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle className="flex items-center gap-2">
<RefreshCw className="w-5 h-5 text-blue-500" />
Regenerate Title/End Screens
</AlertDialogTitle>
<AlertDialogDescription>
This will regenerate the title and end screen videos using the <strong>current</strong> artist and title metadata.
<br /><br />
<strong>Current values:</strong>
<ul className="list-disc list-inside mt-2 space-y-1">
<li>Artist: <code className="bg-muted px-1 rounded">{selectedJob?.artist || "(not set)"}</code></li>
<li>Title: <code className="bg-muted px-1 rounded">{selectedJob?.title || "(not set)"}</code></li>
</ul>
<br />
Screen generation takes 30-60 seconds. Progress will be visible in the job timeline.
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleRegenerateScreens}
disabled={regeneratingScreens}
className="bg-blue-500 hover:bg-blue-600"
>
{regeneratingScreens && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
Regenerate Screens
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
{/* Restart Job Confirmation Dialog */}
<AlertDialog open={restartDialogOpen} onOpenChange={setRestartDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle className="flex items-center gap-2">
<RotateCcw className="w-5 h-5 text-purple-500" />
Full Restart Job
</AlertDialogTitle>
<AlertDialogDescription asChild>
<div>
<p>Unlike &quot;Reset to Start&quot;, this will actually <strong>trigger workers</strong> to begin processing.</p>
<br />
<div className="flex items-center space-x-2 p-3 bg-muted rounded-lg">
<input
type="checkbox"
id="preserveStems"
checked={restartPreserveStems}
onChange={(e) => setRestartPreserveStems(e.target.checked)}
className="h-4 w-4 rounded border-gray-300"
/>
<label htmlFor="preserveStems" className="text-sm font-medium">
Preserve audio stems (faster)
</label>
</div>
<br />
{restartPreserveStems ? (
<p className="text-sm text-muted-foreground">
<strong>Quick restart:</strong> Keeps existing audio separation and lyrics. Only regenerates screens and video with current metadata. Good for fixing title/artist typos.
</p>
) : (
<p className="text-sm text-muted-foreground">
<strong>Full restart:</strong> Re-downloads audio (if YouTube), re-separates stems, re-transcribes lyrics. Use when audio source needs to change.
</p>
)}
</div>
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleRestartJob}
disabled={restarting}
className="bg-purple-500 hover:bg-purple-600"
>
{restarting && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
Restart Job
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
{/* Override Audio Source Confirmation Dialog */}
<AlertDialog open={overrideAudioDialogOpen} onOpenChange={setOverrideAudioDialogOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle className="flex items-center gap-2">
<Music className="w-5 h-5 text-amber-500" />
Switch to Audio Search
</AlertDialogTitle>
<AlertDialogDescription>
This will clear the current audio source and switch to audio search mode.
<br /><br />
<strong>What happens:</strong>
<ul className="list-disc list-inside mt-2 space-y-1">
<li>YouTube URL and downloaded audio will be cleared</li>
<li>All processing state will be reset</li>
<li>Job will move to &quot;Awaiting Audio Selection&quot;</li>
<li>You can then search for and select higher quality audio</li>
</ul>
<br />
<strong>Current source:</strong>{" "}
{selectedJob?.url ? (
<span className="text-muted-foreground">YouTube URL</span>
) : selectedJob?.input_media_gcs_path ? (
<span className="text-muted-foreground">Uploaded file</span>
) : (
<span className="text-muted-foreground">Audio search</span>
)}
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleOverrideAudioSource}
disabled={overridingAudio}
className="bg-amber-500 hover:bg-amber-600"
>
{overridingAudio && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
Switch to Audio Search
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</div>
</TooltipProvider>
)
+60 -14
View File
@@ -1,6 +1,6 @@
"use client"
import { useState, useEffect, useCallback, Suspense, useRef } from "react"
import { useState, useEffect, useCallback, useMemo, Suspense, useRef } from "react"
import { useRouter, useSearchParams } from "next/navigation"
import { api, Job, getAccessToken } from "@/lib/api"
import { useAuth } from "@/lib/auth"
@@ -15,8 +15,8 @@ import {
SelectTrigger,
SelectValue,
} from "@/components/ui/select"
import { Music2, RefreshCw, Loader2, Moon, Sun } from "lucide-react"
import { sortJobsByPriority } from "@/lib/job-status"
import { Music2, RefreshCw, Loader2, Moon, Sun, Eye, EyeOff } from "lucide-react"
import { sortJobsByPriority, getDisplayJobs } from "@/lib/job-status"
import { WarmingUpLoader } from "@/components/WarmingUpLoader"
import { JobCard } from "@/components/job"
import { JobSubmission } from "@/components/job/JobSubmission"
@@ -35,7 +35,7 @@ import {
function AppPageContent() {
const router = useRouter()
const searchParams = useSearchParams()
const [jobs, setJobs] = useState<Job[]>([])
const [allJobs, setAllJobs] = useState<Job[]>([])
const [isLoadingJobs, setIsLoadingJobs] = useState(true)
const [isInitialLoad, setIsInitialLoad] = useState(true)
const [isAuthenticated, setIsAuthenticated] = useState<boolean | null>(null)
@@ -44,11 +44,25 @@ function AppPageContent() {
const { isDarkMode, toggleTheme, mounted } = useTheme()
const { user, fetchUser, verifyMagicLink } = useAuth()
const { showTestData } = useAdminSettings()
const [jobLimit, setJobLimit] = useState<number>(5)
const [jobLimit, setJobLimit] = useState<number>(() => {
if (typeof window === "undefined") return 10
const saved = localStorage.getItem("nomad-karaoke-job-limit")
return saved ? Number(saved) : 10
})
const [hideCompleted, setHideCompleted] = useState<boolean>(() => {
if (typeof window === "undefined") return false
return localStorage.getItem("nomad-karaoke-hide-completed") === "true"
})
// Check if user is admin (for exclude_test parameter)
const isAdmin = user?.role === "admin" || user?.email?.endsWith("@nomadkaraoke.com")
// Derive displayed jobs from allJobs + display limit + filter (instant, no re-fetch)
const { displayedJobs: jobs, totalFetched } = useMemo(
() => getDisplayJobs(allJobs, jobLimit, hideCompleted),
[allJobs, jobLimit, hideCompleted]
)
// Memoize loadJobs for use with visibility refresh
const loadJobs = useCallback(async () => {
if (!getAccessToken()) {
@@ -57,13 +71,15 @@ function AppPageContent() {
return
}
try {
// Only admins can filter test jobs, and by default we hide them
// Use summary mode for reduced payload (~12 fields instead of 50+)
const data = await api.listJobs({
limit: jobLimit === -1 ? 100 : jobLimit,
exclude_test: isAdmin ? !showTestData : undefined
limit: 100,
exclude_test: isAdmin ? !showTestData : undefined,
fields: 'summary',
hide_completed: hideCompleted,
})
// Sort: blocking jobs first, then processing, then completed
setJobs(sortJobsByPriority(data))
setAllJobs(sortJobsByPriority(data))
} catch (err: any) {
// Don't log auth errors - user just needs to authenticate
if (err?.status !== 401) {
@@ -73,10 +89,10 @@ function AppPageContent() {
setIsLoadingJobs(false)
setIsInitialLoad(false)
}
}, [isAdmin, showTestData, jobLimit])
}, [isAdmin, showTestData, hideCompleted])
// Enable notifications for job status changes (sound + title animation)
useJobNotifications(jobs)
useJobNotifications(allJobs)
// Refresh jobs immediately when tab becomes visible (after returning from review UIs)
useVisibilityRefresh(loadJobs, isAuthenticated === true)
@@ -158,7 +174,7 @@ function AppPageContent() {
return (
<div className="min-h-screen animated-gradient">
{/* AutoProcessor - handles non-interactive mode for jobs with that flag */}
<AutoProcessor jobs={jobs} onJobsChanged={loadJobs} />
<AutoProcessor jobs={allJobs} onJobsChanged={loadJobs} />
{/* Header */}
<header className="fixed top-0 left-0 right-0 z-50 bg-dark-900/80 backdrop-blur-md border-b border-dark-700">
@@ -235,7 +251,33 @@ function AppPageContent() {
<CardTitle style={{ color: 'var(--text)' }}>Recent Jobs</CardTitle>
<div className="flex items-center gap-2">
{isLoadingJobs && <Loader2 className="w-4 h-4 animate-spin" style={{ color: 'var(--text-muted)' }} />}
<Select value={String(jobLimit)} onValueChange={(v) => setJobLimit(Number(v))}>
<TooltipProvider>
<Tooltip>
<TooltipTrigger asChild>
<Button
variant="ghost"
size="sm"
className="h-7 w-7 p-0"
style={{ color: 'var(--text-muted)' }}
onClick={() => {
const next = !hideCompleted
setHideCompleted(next)
localStorage.setItem("nomad-karaoke-hide-completed", String(next))
}}
>
{hideCompleted ? <EyeOff className="w-3.5 h-3.5" /> : <Eye className="w-3.5 h-3.5" />}
</Button>
</TooltipTrigger>
<TooltipContent side="bottom">
<p>{hideCompleted ? "Show all jobs" : "Hide completed jobs"}</p>
</TooltipContent>
</Tooltip>
</TooltipProvider>
<Select value={String(jobLimit)} onValueChange={(v) => {
const val = Number(v)
setJobLimit(val)
localStorage.setItem("nomad-karaoke-job-limit", String(val))
}}>
<SelectTrigger className="h-7 w-[80px] text-xs">
<SelectValue />
</SelectTrigger>
@@ -249,7 +291,11 @@ function AppPageContent() {
</div>
</div>
<CardDescription style={{ color: 'var(--text-muted)' }}>
{jobs.length} job{jobs.length !== 1 ? 's' : ''} shown
{hideCompleted
? `${jobs.length} incomplete of ${totalFetched} total`
: jobs.length < totalFetched
? `Showing ${jobs.length} of ${totalFetched} jobs`
: `${jobs.length} job${jobs.length !== 1 ? 's' : ''}`}
</CardDescription>
</CardHeader>
<CardContent className="px-3 sm:px-6">
@@ -71,6 +71,7 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
// UI state
const [zoomLevel, setZoomLevel] = useState<ZoomLevel>(1)
const [customAudioUrl, setCustomAudioUrl] = useState<string | null>(null)
const [isCreatingCustom, setIsCreatingCustom] = useState(false)
const [isUploading, setIsUploading] = useState(false)
const [isSubmitting, setIsSubmitting] = useState(false)
@@ -222,6 +223,10 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
// Get audio URL for current selection
const getAudioUrl = useCallback(() => {
// Custom instrumental uses the signed URL from creation response
if (activeAudio === "custom" && customAudioUrl) {
return customAudioUrl
}
// For cloud mode, use signed URLs from analysis data
// For local mode, use the API stream endpoint
if (!isLocalMode && analysisData?.audio_urls) {
@@ -237,7 +242,7 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
// Local mode: use API stream endpoint
const stemType = STEM_TYPE_MAP[activeAudio]
return api.getAudioStreamUrl(job.job_id, stemType)
}, [job.job_id, activeAudio, isLocalMode, analysisData])
}, [job.job_id, activeAudio, isLocalMode, analysisData, customAudioUrl])
// Handle audio type change
const handleAudioChange = useCallback(
@@ -253,9 +258,12 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
setIsAudioLoading(true)
// Update audio source
// For cloud mode, use signed URLs from analysis data
let audioUrl: string
if (!isLocalMode && analysisData?.audio_urls) {
// Custom instrumental uses the signed URL from creation response
if (type === "custom" && customAudioUrl) {
audioUrl = customAudioUrl
} else if (!isLocalMode && analysisData?.audio_urls) {
// Cloud mode: use signed URLs from analysis data
const urls = analysisData.audio_urls
switch (type) {
case "clean": audioUrl = urls.clean || ""; break
@@ -290,7 +298,7 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
audio.addEventListener("loadeddata", handleLoaded, { once: true })
audio.addEventListener("error", handleError, { once: true })
},
[job.job_id, isLocalMode, analysisData]
[job.job_id, isLocalMode, analysisData, customAudioUrl]
)
// Seek to time
@@ -351,14 +359,16 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
setIsCreatingCustom(true)
try {
await api.createCustomInstrumental(job.job_id, muteRegions)
const result = await api.createCustomInstrumental(job.job_id, muteRegions)
const audioUrl = result.audio_url || api.getAudioStreamUrl(job.job_id, "custom_instrumental")
setCustomAudioUrl(audioUrl)
setHasCustom(true)
setSelectedOption("custom")
setActiveAudio("custom")
// Update audio source
// Update audio source with signed URL from response
if (audio) {
audio.src = api.getAudioStreamUrl(job.job_id, "custom_instrumental")
audio.src = audioUrl
audio.addEventListener(
"loadeddata",
() => {
@@ -428,8 +438,10 @@ export function InstrumentalSelector({ job, isLocalMode = false }: InstrumentalS
try {
// Fetch current correction data and submit with instrumental selection
// Using the same review API for both local and cloud mode
// Map "uploaded" to "custom" for the API — both use stems/custom_instrumental in GCS
const apiSelection = selectedOption === "uploaded" ? "custom" : selectedOption
const correctionData = await lyricsReviewApi.getCorrectionData(job.job_id)
await lyricsReviewApi.completeReview(job.job_id, correctionData, selectedOption)
await lyricsReviewApi.completeReview(job.job_id, correctionData, apiSelection)
// Show success screen in both modes
setShowSuccess(true)
+19 -1
View File
@@ -3,6 +3,7 @@
import { useState } from "react"
import { Job, api } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { useToast } from "@/hooks/use-toast"
import {
Loader2, RotateCcw, XCircle
} from "lucide-react"
@@ -15,6 +16,7 @@ interface JobActionsProps {
export function JobActions({ job, onRefresh }: JobActionsProps) {
const [isRetrying, setIsRetrying] = useState(false)
const [isCancelling, setIsCancelling] = useState(false)
const { toast } = useToast()
const canRetry = job.status === "failed" || job.status === "cancelled"
const canCancel = !["complete", "failed", "cancelled"].includes(job.status)
@@ -23,9 +25,25 @@ export function JobActions({ job, onRefresh }: JobActionsProps) {
setIsRetrying(true)
try {
await api.retryJob(job.job_id)
toast({
title: "Job retry started",
description: "The job is being retried. Refreshing status...",
variant: "default",
})
onRefresh()
} catch (error) {
} catch (error: any) {
console.error("Failed to retry job:", error)
// Extract error message from API response
const errorMessage = error?.response?.data?.detail ||
error?.message ||
"Failed to retry job. Please try again."
toast({
title: "Retry failed",
description: errorMessage,
variant: "destructive",
})
} finally {
setIsRetrying(false)
}
+131 -10
View File
@@ -8,7 +8,8 @@ import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Label } from "@/components/ui/label"
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"
import { Upload, Youtube, Music, Loader2, ChevronDown, ChevronRight } from "lucide-react"
import { Upload, Youtube, Music, Loader2, ChevronDown, ChevronRight, AlertTriangle } from "lucide-react"
import Link from "next/link"
interface JobSubmissionProps {
onJobCreated: () => void
@@ -41,6 +42,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
}, [availableTabs, activeTab])
const [error, setError] = useState("")
const [isCreditError, setIsCreditError] = useState(false)
const [isSubmitting, setIsSubmitting] = useState(false)
// Upload form
@@ -64,9 +66,16 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
// Non-interactive mode (shared across all tabs)
const [nonInteractive, setNonInteractive] = useState(false)
// Private (non-published) mode - Dropbox only, no YouTube/GDrive
const [isPrivate, setIsPrivate] = useState(false)
// Credit enforcement
const noCredits = !isAdmin && user?.credits === 0
async function handleUploadSubmit(e: React.FormEvent) {
e.preventDefault()
setError("")
setIsCreditError(false)
if (!uploadFile) {
setError("Please select an audio file")
@@ -81,13 +90,17 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
try {
await api.uploadJob(uploadFile, uploadArtist.trim(), uploadTitle.trim(), {
non_interactive: nonInteractive,
is_private: isPrivate,
})
setUploadFile(null)
setUploadArtist("")
setUploadTitle("")
onJobCreated()
} catch (err) {
if (err instanceof ApiError) {
if (err instanceof ApiError && err.status === 402) {
setIsCreditError(true)
setError("You're out of credits. Buy more to continue creating karaoke videos.")
} else if (err instanceof ApiError) {
setError(err.message)
} else {
setError("Failed to create job")
@@ -100,6 +113,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
async function handleUrlSubmit(e: React.FormEvent) {
e.preventDefault()
setError("")
setIsCreditError(false)
if (!youtubeUrl.trim()) {
setError("Please enter a URL")
@@ -118,6 +132,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
youtubeTitle.trim(),
{
non_interactive: nonInteractive,
is_private: isPrivate,
}
)
setYoutubeUrl("")
@@ -125,7 +140,10 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
setYoutubeTitle("")
onJobCreated()
} catch (err) {
if (err instanceof ApiError) {
if (err instanceof ApiError && err.status === 402) {
setIsCreditError(true)
setError("You're out of credits. Buy more to continue creating karaoke videos.")
} else if (err instanceof ApiError) {
setError(err.message)
} else {
setError("Failed to create job")
@@ -138,6 +156,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
async function handleSearchSubmit(e: React.FormEvent) {
e.preventDefault()
setError("")
setIsCreditError(false)
if (!searchArtist.trim() || !searchTitle.trim()) {
setError("Please enter both artist and title")
@@ -148,6 +167,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
try {
await api.searchAudio(searchArtist.trim(), searchTitle.trim(), false, {
non_interactive: nonInteractive,
is_private: isPrivate,
// Display overrides (empty string means use search values)
display_artist: displayArtist.trim() || undefined,
display_title: displayTitle.trim() || undefined,
@@ -158,7 +178,10 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
setDisplayTitle("")
onJobCreated()
} catch (err) {
if (err instanceof ApiError) {
if (err instanceof ApiError && err.status === 402) {
setIsCreditError(true)
setError("You're out of credits. Buy more to continue creating karaoke videos.")
} else if (err instanceof ApiError) {
setError(err.message)
} else {
setError("Failed to search for audio")
@@ -178,6 +201,23 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
return (
<Tabs value={activeTab} onValueChange={setActiveTab}>
{/* Zero-credit warning banner */}
{noCredits && (
<div className="flex items-start gap-2 rounded-lg p-3 mb-4 border border-amber-500/30 bg-amber-500/10">
<AlertTriangle className="w-5 h-5 text-amber-500 shrink-0 mt-0.5" />
<div className="text-sm" style={{ color: 'var(--text)' }}>
<p>You have no credits remaining. Buy credits to create new karaoke videos.</p>
<Link
href="/#pricing"
className="inline-block mt-1 text-sm font-medium underline"
style={{ color: 'var(--brand-pink)' }}
>
Buy Credits
</Link>
</div>
</div>
)}
{/* Only show tabs bar when there are multiple options */}
{availableTabs.length > 1 && (
<TabsList
@@ -307,14 +347,41 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
</div>
)}
{/* Private (non-published) mode */}
<div className="space-y-2 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
<div className="flex items-start gap-3">
<input
type="checkbox"
id="private-upload"
checked={isPrivate}
onChange={(e) => setIsPrivate(e.target.checked)}
disabled={isSubmitting}
className="mt-1 w-4 h-4 rounded border-border bg-secondary accent-[var(--brand-pink)] focus:ring-[var(--brand-pink)] focus:ring-offset-background"
/>
<div>
<Label htmlFor="private-upload" className="cursor-pointer" style={{ color: 'var(--text)' }}>
Private (non-published)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
Files delivered via Dropbox only. No YouTube or Google Drive upload.
</p>
</div>
</div>
</div>
{error && activeTab === "upload" && (
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
<div className="text-sm text-red-400 bg-red-500/10 rounded p-2">
<p>{error}</p>
{isCreditError && (
<Link href="/#pricing" className="inline-block mt-1 font-medium underline" style={{ color: 'var(--brand-pink)' }}>Buy Credits</Link>
)}
</div>
)}
<Button
type="submit"
className="w-full bg-[var(--brand-pink)] hover:bg-[var(--brand-pink-hover)] text-white shadow-[0_0_15px_rgba(255,122,204,0.3)] hover:shadow-[0_0_20px_rgba(255,122,204,0.5)]"
disabled={isSubmitting}
disabled={isSubmitting || noCredits}
>
{isSubmitting ? (
<>
@@ -402,14 +469,41 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
</div>
)}
{/* Private (non-published) mode */}
<div className="space-y-2 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
<div className="flex items-start gap-3">
<input
type="checkbox"
id="private-url"
checked={isPrivate}
onChange={(e) => setIsPrivate(e.target.checked)}
disabled={isSubmitting}
className="mt-1 w-4 h-4 rounded border-border bg-secondary accent-[var(--brand-pink)] focus:ring-[var(--brand-pink)] focus:ring-offset-background"
/>
<div>
<Label htmlFor="private-url" className="cursor-pointer" style={{ color: 'var(--text)' }}>
Private (non-published)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
Files delivered via Dropbox only. No YouTube or Google Drive upload.
</p>
</div>
</div>
</div>
{error && activeTab === "url" && (
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
<div className="text-sm text-red-400 bg-red-500/10 rounded p-2">
<p>{error}</p>
{isCreditError && (
<Link href="/#pricing" className="inline-block mt-1 font-medium underline" style={{ color: 'var(--brand-pink)' }}>Buy Credits</Link>
)}
</div>
)}
<Button
type="submit"
className="w-full bg-[var(--brand-pink)] hover:bg-[var(--brand-pink-hover)] text-white shadow-[0_0_15px_rgba(255,122,204,0.3)] hover:shadow-[0_0_20px_rgba(255,122,204,0.5)]"
disabled={isSubmitting}
disabled={isSubmitting || noCredits}
>
{isSubmitting ? (
<>
@@ -535,14 +629,41 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
</div>
)}
{/* Private (non-published) mode */}
<div className="space-y-2 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
<div className="flex items-start gap-3">
<input
type="checkbox"
id="private-search"
checked={isPrivate}
onChange={(e) => setIsPrivate(e.target.checked)}
disabled={isSubmitting}
className="mt-1 w-4 h-4 rounded border-border bg-secondary accent-[var(--brand-pink)] focus:ring-[var(--brand-pink)] focus:ring-offset-background"
/>
<div>
<Label htmlFor="private-search" className="cursor-pointer" style={{ color: 'var(--text)' }}>
Private (non-published)
</Label>
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
Files delivered via Dropbox only. No YouTube or Google Drive upload.
</p>
</div>
</div>
</div>
{error && activeTab === "search" && (
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
<div className="text-sm text-red-400 bg-red-500/10 rounded p-2">
<p>{error}</p>
{isCreditError && (
<Link href="/#pricing" className="inline-block mt-1 font-medium underline" style={{ color: 'var(--brand-pink)' }}>Buy Credits</Link>
)}
</div>
)}
<Button
type="submit"
className="w-full bg-[var(--brand-pink)] hover:bg-[var(--brand-pink-hover)] text-white shadow-[0_0_15px_rgba(255,122,204,0.3)] hover:shadow-[0_0_20px_rgba(255,122,204,0.5)]"
disabled={isSubmitting}
disabled={isSubmitting || noCredits}
>
{isSubmitting ? (
<>
@@ -7,7 +7,7 @@ import {
TooltipProvider,
TooltipTrigger,
} from '@/components/ui/tooltip'
import { Highlighter, Edit, Trash2 } from 'lucide-react'
import { Highlighter, Pencil, Trash2 } from 'lucide-react'
import { InteractionMode } from '@/lib/lyrics-review/types'
interface ModeSelectorProps {
@@ -23,13 +23,16 @@ export default function ModeSelector({ effectiveMode, onChange }: ModeSelectorPr
<ToggleGroup
type="single"
value={effectiveMode}
onValueChange={(value) => value === 'edit' && onChange(value as InteractionMode)}
onValueChange={(value) => value && onChange(value as InteractionMode)}
className="h-8"
>
<Tooltip>
<TooltipTrigger asChild>
<ToggleGroupItem value="edit" className="h-8 px-2 text-[0.75rem]">
<Edit className="h-3.5 w-3.5 mr-1" />
<ToggleGroupItem
value="edit"
className="h-8 px-2.5 text-[0.75rem] !flex-none aria-checked:font-semibold aria-checked:bg-blue-600 aria-checked:text-white dark:aria-checked:bg-blue-500"
>
<Pencil className="h-3.5 w-3.5 mr-1" />
Edit
</ToggleGroupItem>
</TooltipTrigger>
@@ -40,7 +43,10 @@ export default function ModeSelector({ effectiveMode, onChange }: ModeSelectorPr
<Tooltip>
<TooltipTrigger asChild>
<ToggleGroupItem value="highlight" className="h-8 px-2 text-[0.75rem]" disabled>
<ToggleGroupItem
value="highlight"
className="h-8 px-2.5 text-[0.75rem] !flex-none aria-checked:font-semibold aria-checked:bg-blue-600 aria-checked:text-white dark:aria-checked:bg-blue-500"
>
<Highlighter className="h-3.5 w-3.5 mr-1" />
Highlight
</ToggleGroupItem>
@@ -52,7 +58,10 @@ export default function ModeSelector({ effectiveMode, onChange }: ModeSelectorPr
<Tooltip>
<TooltipTrigger asChild>
<ToggleGroupItem value="delete_word" className="h-8 px-2 text-[0.75rem]" disabled>
<ToggleGroupItem
value="delete_word"
className="h-8 px-2.5 text-[0.75rem] !flex-none aria-checked:font-semibold aria-checked:bg-blue-600 aria-checked:text-white dark:aria-checked:bg-blue-500"
>
<Trash2 className="h-3.5 w-3.5 mr-1" />
Delete
</ToggleGroupItem>
@@ -1,6 +1,6 @@
'use client'
import { useState, useRef, useEffect, useCallback, memo } from 'react'
import { useState, useRef, useEffect, useCallback, useId, memo } from 'react'
import { Input } from '@/components/ui/input'
import { Label } from '@/components/ui/label'
@@ -19,6 +19,7 @@ const TimeInput = memo(function TimeInput({
widthClass = 'w-24',
debounceMs = 300
}: TimeInputProps) {
const inputId = useId()
const [localValue, setLocalValue] = useState<string>('')
const [isFocused, setIsFocused] = useState(false)
const debounceRef = useRef<NodeJS.Timeout | null>(null)
@@ -65,8 +66,9 @@ const TimeInput = memo(function TimeInput({
return (
<div className={widthClass}>
<Label className="text-xs text-muted-foreground mb-1">{label}</Label>
<Label htmlFor={inputId} className="text-xs text-muted-foreground mb-1">{label}</Label>
<Input
id={inputId}
type="number"
value={isFocused ? localValue : (value?.toFixed(2) ?? '')}
onFocus={handleFocus}
@@ -0,0 +1,63 @@
import { render, screen } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import ModeSelector from '../ModeSelector'
import { InteractionMode } from '@/lib/lyrics-review/types'
describe('ModeSelector', () => {
it('renders all three mode buttons', () => {
const mockOnChange = jest.fn()
render(<ModeSelector effectiveMode="edit" onChange={mockOnChange} />)
expect(screen.getByRole('radio', { name: /edit/i })).toBeInTheDocument()
expect(screen.getByRole('radio', { name: /highlight/i })).toBeInTheDocument()
expect(screen.getByRole('radio', { name: /delete/i })).toBeInTheDocument()
})
it('all mode buttons are enabled (not disabled)', () => {
const mockOnChange = jest.fn()
render(<ModeSelector effectiveMode="edit" onChange={mockOnChange} />)
const editBtn = screen.getByRole('radio', { name: /edit/i })
const highlightBtn = screen.getByRole('radio', { name: /highlight/i })
const deleteBtn = screen.getByRole('radio', { name: /delete/i })
// Verify buttons are not disabled
expect(editBtn).not.toBeDisabled()
expect(highlightBtn).not.toBeDisabled()
expect(deleteBtn).not.toBeDisabled()
})
it('calls onChange when clicking highlight button', async () => {
const user = userEvent.setup()
const mockOnChange = jest.fn()
render(<ModeSelector effectiveMode="edit" onChange={mockOnChange} />)
const highlightBtn = screen.getByRole('radio', { name: /highlight/i })
await user.click(highlightBtn)
expect(mockOnChange).toHaveBeenCalledWith('highlight')
})
it('calls onChange when clicking delete button', async () => {
const user = userEvent.setup()
const mockOnChange = jest.fn()
render(<ModeSelector effectiveMode="edit" onChange={mockOnChange} />)
const deleteBtn = screen.getByRole('radio', { name: /delete/i })
await user.click(deleteBtn)
expect(mockOnChange).toHaveBeenCalledWith('delete_word')
})
it('shows correct active state for current mode', () => {
const mockOnChange = jest.fn()
const { rerender } = render(<ModeSelector effectiveMode="highlight" onChange={mockOnChange} />)
const highlightBtn = screen.getByRole('radio', { name: /highlight/i })
expect(highlightBtn).toHaveAttribute('aria-checked', 'true')
rerender(<ModeSelector effectiveMode="delete_word" onChange={mockOnChange} />)
const deleteBtn = screen.getByRole('radio', { name: /delete/i })
expect(deleteBtn).toHaveAttribute('aria-checked', 'true')
})
})
@@ -0,0 +1,313 @@
import { render, screen, within } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import ReplaceAllLyricsModal from '../modals/ReplaceAllLyricsModal'
import { LyricsSegment } from '@/lib/lyrics-review/types'
// Mock the LyricsSynchronizer to avoid complex dependencies
jest.mock('../synchronizer/LyricsSynchronizer', () => {
return function MockLyricsSynchronizer() {
return <div data-testid="lyrics-synchronizer">Synchronizer</div>
}
})
const mockSegments: LyricsSegment[] = [
{
id: 'seg1',
text: 'Hello world',
start_time: 0,
end_time: 2,
words: [
{ id: 'w1', text: 'Hello', start_time: 0, end_time: 1, confidence: 0.9 },
{ id: 'w2', text: 'world', start_time: 1, end_time: 2, confidence: 0.95 },
],
},
{
id: 'seg2',
text: 'Goodbye moon',
start_time: 3,
end_time: 5,
words: [
{ id: 'w3', text: 'Goodbye', start_time: 3, end_time: 4, confidence: 0.88 },
{ id: 'w4', text: 'moon', start_time: 4, end_time: 5, confidence: 0.92 },
],
},
{
id: 'seg3',
text: 'Third line here',
start_time: 6,
end_time: 9,
words: [
{ id: 'w5', text: 'Third', start_time: 6, end_time: 7, confidence: 0.9 },
{ id: 'w6', text: 'line', start_time: 7, end_time: 8, confidence: 0.9 },
{ id: 'w7', text: 'here', start_time: 8, end_time: 9, confidence: 0.9 },
],
},
]
const defaultProps = {
open: true,
onClose: jest.fn(),
onSave: jest.fn(),
setModalSpacebarHandler: jest.fn(),
existingSegments: mockSegments,
}
async function openReplaceSegmentsMode() {
const user = userEvent.setup()
render(<ReplaceAllLyricsModal {...defaultProps} />)
// Click "Replace Segment Lyrics" in mode selection
const replaceSegmentsCard = screen.getByText('Replace Segment Lyrics')
await user.click(replaceSegmentsCard)
}
describe('Replace Segments Mode', () => {
beforeEach(() => {
jest.clearAllMocks()
})
it('shows Replace Segment Lyrics option in mode selection when existing lyrics exist', () => {
render(<ReplaceAllLyricsModal {...defaultProps} />)
expect(screen.getByText('Replace Segment Lyrics')).toBeInTheDocument()
expect(screen.getByText('Recommended for custom lyrics')).toBeInTheDocument()
})
it('does not show Replace Segment Lyrics when no existing lyrics', () => {
render(<ReplaceAllLyricsModal {...defaultProps} existingSegments={[]} />)
expect(screen.queryByText('Replace Segment Lyrics')).not.toBeInTheDocument()
})
it('renders textarea pre-filled with existing segment text', async () => {
await openReplaceSegmentsMode()
const textarea = screen.getByRole('textbox')
expect(textarea).toHaveValue('Hello world\nGoodbye moon\nThird line here')
})
it('shows correct matching line count', async () => {
await openReplaceSegmentsMode()
expect(screen.getByText('3/3 lines')).toBeInTheDocument()
})
it('Apply button is enabled when line count matches', async () => {
await openReplaceSegmentsMode()
const applyButton = screen.getByRole('button', { name: 'Apply' })
expect(applyButton).not.toBeDisabled()
})
it('disables Apply when line count differs (too many)', async () => {
const user = userEvent.setup()
await openReplaceSegmentsMode()
const textarea = screen.getByRole('textbox')
await user.clear(textarea)
await user.type(textarea, 'line 1\nline 2\nline 3\nline 4')
const applyButton = screen.getByRole('button', { name: 'Apply' })
expect(applyButton).toBeDisabled()
expect(screen.getByText(/1 too many/)).toBeInTheDocument()
})
it('disables Apply when line count differs (too few)', async () => {
const user = userEvent.setup()
await openReplaceSegmentsMode()
const textarea = screen.getByRole('textbox')
await user.clear(textarea)
await user.type(textarea, 'line 1\nline 2')
const applyButton = screen.getByRole('button', { name: 'Apply' })
expect(applyButton).toBeDisabled()
expect(screen.getByText(/1 too few/)).toBeInTheDocument()
})
it('shows mismatch error message with correct counts', async () => {
const user = userEvent.setup()
await openReplaceSegmentsMode()
const textarea = screen.getByRole('textbox')
await user.clear(textarea)
await user.type(textarea, 'only one line')
expect(screen.getByText(/expected 3, got 1/)).toBeInTheDocument()
})
it('calls onSave with unchanged segments when no text is modified', async () => {
const user = userEvent.setup()
await openReplaceSegmentsMode()
const applyButton = screen.getByRole('button', { name: 'Apply' })
await user.click(applyButton)
expect(defaultProps.onSave).toHaveBeenCalledTimes(1)
const savedSegments = defaultProps.onSave.mock.calls[0][0]
expect(savedSegments).toHaveLength(3)
// Unchanged segments should have same words (deep copy)
expect(savedSegments[0].text).toBe('Hello world')
expect(savedSegments[0].words).toHaveLength(2)
expect(savedSegments[0].words[0].text).toBe('Hello')
expect(savedSegments[0].words[1].text).toBe('world')
// Preserved original timing
expect(savedSegments[0].words[0].start_time).toBe(0)
expect(savedSegments[0].words[0].end_time).toBe(1)
})
it('creates new words with distributed timing for changed lines', async () => {
const user = userEvent.setup()
await openReplaceSegmentsMode()
const textarea = screen.getByRole('textbox')
await user.clear(textarea)
await user.type(textarea, 'Hello world\nNew custom lyrics here\nThird line here')
const applyButton = screen.getByRole('button', { name: 'Apply' })
await user.click(applyButton)
expect(defaultProps.onSave).toHaveBeenCalledTimes(1)
const savedSegments = defaultProps.onSave.mock.calls[0][0]
// First segment unchanged
expect(savedSegments[0].words[0].text).toBe('Hello')
expect(savedSegments[0].words[0].start_time).toBe(0)
// Second segment changed — new words with distributed timing over 3-5s
expect(savedSegments[1].text).toBe('New custom lyrics here')
expect(savedSegments[1].words).toHaveLength(4)
expect(savedSegments[1].words[0].text).toBe('New')
expect(savedSegments[1].words[0].start_time).toBe(3)
expect(savedSegments[1].words[3].text).toBe('here')
expect(savedSegments[1].words[3].end_time).toBe(5)
// Third segment unchanged
expect(savedSegments[2].words[0].text).toBe('Third')
})
it('preserves segment id and timing for changed lines', async () => {
const user = userEvent.setup()
await openReplaceSegmentsMode()
const textarea = screen.getByRole('textbox')
await user.clear(textarea)
await user.type(textarea, 'Hello world\nCompletely different\nThird line here')
const applyButton = screen.getByRole('button', { name: 'Apply' })
await user.click(applyButton)
const savedSegments = defaultProps.onSave.mock.calls[0][0]
// Changed segment keeps original id and timing boundaries
expect(savedSegments[1].id).toBe('seg2')
expect(savedSegments[1].start_time).toBe(3)
expect(savedSegments[1].end_time).toBe(5)
})
it('calls onClose after applying', async () => {
const user = userEvent.setup()
await openReplaceSegmentsMode()
const applyButton = screen.getByRole('button', { name: 'Apply' })
await user.click(applyButton)
expect(defaultProps.onClose).toHaveBeenCalled()
})
it('back button returns to mode selection', async () => {
const user = userEvent.setup()
await openReplaceSegmentsMode()
// Should be in replace segments mode
expect(screen.getByText('Edit lyrics line by line', { exact: false })).toBeInTheDocument()
// Click back arrow
const backButton = screen.getAllByRole('button').find(
(btn) => btn.querySelector('svg.lucide-arrow-left')
)
expect(backButton).toBeTruthy()
await user.click(backButton!)
// Should be back in mode selection
expect(screen.getByText('Edit All Lyrics')).toBeInTheDocument()
expect(screen.getByText('Re-sync Existing Lyrics')).toBeInTheDocument()
expect(screen.getByText('Replace Segment Lyrics')).toBeInTheDocument()
})
it('cancel button closes the modal', async () => {
const user = userEvent.setup()
await openReplaceSegmentsMode()
const cancelButton = screen.getByRole('button', { name: 'Cancel' })
await user.click(cancelButton)
expect(defaultProps.onClose).toHaveBeenCalled()
})
it('does not show Replace Segment Lyrics when segments have no words', () => {
const emptyWordSegments: LyricsSegment[] = [
{ id: 's1', text: 'test', start_time: 0, end_time: 1, words: [] },
]
render(<ReplaceAllLyricsModal {...defaultProps} existingSegments={emptyWordSegments} />)
expect(screen.queryByText('Replace Segment Lyrics')).not.toBeInTheDocument()
})
it('handles changing empty segment text to non-empty', async () => {
const user = userEvent.setup()
const segmentsWithEmpty: LyricsSegment[] = [
{
id: 'seg1',
text: '',
start_time: 0,
end_time: 2,
words: [{ id: 'w1', text: '', start_time: 0, end_time: 2, confidence: 1 }],
},
{
id: 'seg2',
text: 'Hello world',
start_time: 3,
end_time: 5,
words: [
{ id: 'w2', text: 'Hello', start_time: 3, end_time: 4, confidence: 1 },
{ id: 'w3', text: 'world', start_time: 4, end_time: 5, confidence: 1 },
],
},
]
render(<ReplaceAllLyricsModal {...defaultProps} existingSegments={segmentsWithEmpty} />)
const replaceSegmentsCard = screen.getByText('Replace Segment Lyrics')
await user.click(replaceSegmentsCard)
const textarea = screen.getByRole('textbox')
await user.clear(textarea)
await user.type(textarea, 'New first line\nHello world')
const applyButton = screen.getByRole('button', { name: 'Apply' })
await user.click(applyButton)
const savedSegments = defaultProps.onSave.mock.calls[0][0]
expect(savedSegments[0].text).toBe('New first line')
expect(savedSegments[0].words).toHaveLength(3)
expect(savedSegments[0].words[0].start_time).toBe(0)
expect(savedSegments[0].words[2].end_time).toBeCloseTo(2)
})
it('deep copies unchanged segments (no mutation)', async () => {
const user = userEvent.setup()
await openReplaceSegmentsMode()
const applyButton = screen.getByRole('button', { name: 'Apply' })
await user.click(applyButton)
const savedSegments = defaultProps.onSave.mock.calls[0][0]
// Verify deep copy — different object references
expect(savedSegments[0]).not.toBe(mockSegments[0])
expect(savedSegments[0].words).not.toBe(mockSegments[0].words)
expect(savedSegments[0].words[0]).not.toBe(mockSegments[0].words[0])
})
})
@@ -10,7 +10,7 @@ import EditWordList from '../EditWordList'
import EditTimelineSection from '../EditTimelineSection'
import useManualSync from '@/hooks/useManualSync'
import { setModalHandler } from '@/lib/lyrics-review/utils/keyboardHandlers'
import { splitWordWithTiming } from '@/lib/lyrics-review/utils/wordUtils'
import { splitWordWithTiming, createWordsWithDistributedTiming } from '@/lib/lyrics-review/utils/wordUtils'
interface EditModalProps {
open: boolean
@@ -248,22 +248,12 @@ export default function EditModal({
const handleReplaceAllWords = useCallback(
(replacementText: string) => {
if (!editedSegment) return
const wordTexts = replacementText.trim().split(/\s+/).filter((w) => w.length > 0)
if (wordTexts.length === 0) return
const segStart = editedSegment.start_time ?? 0
const segEnd = editedSegment.end_time ?? segStart + 1
const totalDuration = segEnd - segStart
const wordDuration = totalDuration / wordTexts.length
const newWords: Word[] = wordTexts.map((text, index) => ({
id: nanoid(),
text,
start_time: segStart + index * wordDuration,
end_time: segStart + (index + 1) * wordDuration,
confidence: 1.0,
}))
const newWords = createWordsWithDistributedTiming(
replacementText,
editedSegment.start_time,
editedSegment.end_time
)
if (newWords.length === 0) return
updateSegment(newWords)
},
[editedSegment, updateSegment]
@@ -2,7 +2,7 @@
import { Dialog, DialogContent, DialogHeader, DialogTitle, DialogFooter } from '@/components/ui/dialog'
import { Button } from '@/components/ui/button'
import { RefreshCw, ClipboardPaste, X } from 'lucide-react'
import { RefreshCw, ClipboardPaste, TextCursorInput, X } from 'lucide-react'
import { cn } from '@/lib/utils'
interface ModeSelectionModalProps {
@@ -10,6 +10,7 @@ interface ModeSelectionModalProps {
onClose: () => void
onSelectReplace: () => void
onSelectResync: () => void
onSelectReplaceSegments: () => void
hasExistingLyrics: boolean
}
@@ -18,6 +19,7 @@ export default function ModeSelectionModal({
onClose,
onSelectReplace,
onSelectResync,
onSelectReplaceSegments,
hasExistingLyrics,
}: ModeSelectionModalProps) {
return (
@@ -59,6 +61,29 @@ export default function ModeSelectionModal({
</div>
)}
{/* Replace Segment Lyrics option - only show if there are existing lyrics */}
{hasExistingLyrics && (
<div
className={cn(
'p-4 border rounded-lg cursor-pointer',
'hover:bg-muted/30 hover:border-muted-foreground transition-colors'
)}
onClick={onSelectReplaceSegments}
>
<div className="flex items-start gap-3">
<TextCursorInput className="h-10 w-10 text-muted-foreground mt-0.5" />
<div className="flex-1">
<h3 className="text-lg font-semibold">Replace Segment Lyrics</h3>
<p className="text-sm text-muted-foreground mt-1">
Edit or paste new lyrics for each line. Timing is preserved — only the words
change. Best for custom lyrics, translations, or fixing multiple lines at once.
</p>
<p className="text-xs text-green-500 mt-2">Recommended for custom lyrics</p>
</div>
</div>
</div>
)}
{/* Replace All option */}
<div
className={cn(
@@ -4,9 +4,10 @@ import { useState, useEffect, useCallback, useMemo } from 'react'
import { Dialog, DialogContent, DialogHeader, DialogTitle, DialogFooter } from '@/components/ui/dialog'
import { Button } from '@/components/ui/button'
import { Textarea } from '@/components/ui/textarea'
import { X, ArrowLeft, ClipboardPaste, Sparkles } from 'lucide-react'
import { X, ArrowLeft, ClipboardPaste, Sparkles, Info, Check, AlertTriangle } from 'lucide-react'
import { nanoid } from 'nanoid'
import { LyricsSegment, Word } from '@/lib/lyrics-review/types'
import { createWordsWithDistributedTiming } from '@/lib/lyrics-review/utils/wordUtils'
import ModeSelectionModal from './ModeSelectionModal'
import LyricsSynchronizer from '../synchronizer/LyricsSynchronizer'
@@ -18,7 +19,7 @@ declare global {
}
}
type ModalMode = 'selection' | 'replace' | 'resync'
type ModalMode = 'selection' | 'replace' | 'resync' | 'replaceSegments'
interface ReplaceAllLyricsModalProps {
open: boolean
@@ -67,6 +68,15 @@ export default function ReplaceAllLyricsModal({
return { lines: lines.length, words: totalWords }
}, [inputText])
// Line count validation for replaceSegments mode
const replaceSegmentsInfo = useMemo(() => {
const expectedLineCount = existingSegments.length
const lines = inputText.split('\n')
const currentLineCount = lines.length
const lineDiff = currentLineCount - expectedLineCount
return { expectedLineCount, currentLineCount, lineDiff }
}, [inputText, existingSegments])
// Process the input text into segments and words
const processLyrics = useCallback(() => {
if (!inputText.trim()) return
@@ -123,6 +133,36 @@ export default function ReplaceAllLyricsModal({
onClose()
}, [onClose])
// Apply replace segments: build updated segments with new words for changed lines
const handleApplyReplaceSegments = useCallback(() => {
const newLines = inputText.split('\n')
const updatedSegments: LyricsSegment[] = existingSegments.map((segment, i) => {
const newLineText = newLines[i]?.trim() ?? ''
const originalText = segment.text.trim()
if (newLineText === originalText) {
// Unchanged — deep copy to avoid mutation
return JSON.parse(JSON.stringify(segment))
}
// Changed — create new words with distributed timing
const newWords = createWordsWithDistributedTiming(
newLineText,
segment.start_time,
segment.end_time
)
return {
...segment,
text: newLineText,
words: newWords,
}
})
onSave(updatedSegments)
handleClose()
}, [inputText, existingSegments, onSave, handleClose])
// Handle save from synchronizer
const handleSave = useCallback(
(segments: LyricsSegment[]) => {
@@ -141,6 +181,11 @@ export default function ReplaceAllLyricsModal({
setMode('resync')
}, [])
const handleSelectReplaceSegments = useCallback(() => {
setInputText(existingSegments.map((s) => s.text.trim()).join('\n'))
setMode('replaceSegments')
}, [existingSegments])
// Handle back to selection
const handleBackToSelection = useCallback(() => {
setMode('selection')
@@ -163,6 +208,7 @@ export default function ReplaceAllLyricsModal({
onClose={handleClose}
onSelectReplace={handleSelectReplace}
onSelectResync={handleSelectResync}
onSelectReplaceSegments={handleSelectReplaceSegments}
hasExistingLyrics={hasExistingLyrics}
/>
@@ -219,6 +265,86 @@ export default function ReplaceAllLyricsModal({
</DialogContent>
</Dialog>
{/* Replace Segment Lyrics Modal */}
<Dialog open={open && mode === 'replaceSegments'} onOpenChange={(isOpen) => !isOpen && handleClose()}>
<DialogContent className="max-w-2xl h-[80vh] max-h-[80vh] flex flex-col">
<DialogHeader>
<DialogTitle className="flex items-center gap-2">
<Button variant="ghost" size="icon" onClick={handleBackToSelection} className="h-8 w-8">
<ArrowLeft className="h-4 w-4" />
</Button>
<span className="flex-1">Replace Segment Lyrics</span>
<Button variant="ghost" size="icon" onClick={handleClose} className="h-8 w-8">
<X className="h-4 w-4" />
</Button>
</DialogTitle>
</DialogHeader>
<div className="flex-1 overflow-hidden flex flex-col gap-4">
<div className="flex items-start gap-2 p-3 rounded-md bg-muted/50 text-sm text-muted-foreground">
<Info className="h-4 w-4 mt-0.5 shrink-0" />
<p>
Edit lyrics line by line. Each line corresponds to a timed segment.
The number of lines must stay the same (currently: {replaceSegmentsInfo.expectedLineCount} lines).
Changed lines will get new word timing estimated from the segment duration.
Unchanged lines are left as-is.
</p>
</div>
<div className="flex items-center gap-4">
<Button variant="outline" size="sm" onClick={handlePasteFromClipboard}>
<ClipboardPaste className="h-4 w-4 mr-2" />
Paste from Clipboard
</Button>
<span className={`text-sm font-medium ${replaceSegmentsInfo.lineDiff === 0 ? 'text-green-500' : 'text-destructive'}`}>
{replaceSegmentsInfo.lineDiff === 0 ? (
<span className="flex items-center gap-1">
<Check className="h-4 w-4" />
{replaceSegmentsInfo.currentLineCount}/{replaceSegmentsInfo.expectedLineCount} lines
</span>
) : (
<span className="flex items-center gap-1">
<AlertTriangle className="h-4 w-4" />
{replaceSegmentsInfo.currentLineCount}/{replaceSegmentsInfo.expectedLineCount} lines
</span>
)}
</span>
</div>
<Textarea
value={inputText}
onChange={(e) => setInputText(e.target.value)}
placeholder="Edit lyrics line by line..."
className="flex-1 resize-none font-mono text-sm min-h-[300px]"
/>
{replaceSegmentsInfo.lineDiff !== 0 && (
<div className="flex items-start gap-2 p-3 rounded-md bg-destructive/10 text-sm text-destructive">
<AlertTriangle className="h-4 w-4 mt-0.5 shrink-0" />
<p>
You have {Math.abs(replaceSegmentsInfo.lineDiff)} too{' '}
{replaceSegmentsInfo.lineDiff > 0 ? 'many' : 'few'} lines (expected{' '}
{replaceSegmentsInfo.expectedLineCount}, got {replaceSegmentsInfo.currentLineCount}).
Each line maps to a timed segment, so the count must match exactly.
</p>
</div>
)}
</div>
<DialogFooter>
<Button variant="outline" onClick={handleClose}>
Cancel
</Button>
<Button
onClick={handleApplyReplaceSegments}
disabled={replaceSegmentsInfo.lineDiff !== 0}
>
Apply
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
{/* Synchronizer Modal */}
<Dialog open={open && mode === 'resync'} onOpenChange={(isOpen) => !isOpen && handleClose()}>
<DialogContent className="max-w-[calc(100vw-4vw)] w-[calc(100vw-4vw)] h-[90vh] max-h-[90vh] flex flex-col">
@@ -16,8 +16,8 @@ import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-tes
* 4. Create Job - Search for test song via UI
* 5. Audio Selection - Select audio source via UI dialog
* 6. Wait for Processing - Monitor via UI status updates
* 7. Lyrics Review - Open review UI, interact, submit via UI
* 8. Instrumental Selection - Open selection UI, interact, submit via UI
* 7. Combined Review - Open review UI, preview video, proceed to instrumental
* 8. Instrumental Selection - Select instrumental and submit (same page via hash nav)
* 9. Wait for Completion - Monitor via UI
* 10. Verify Downloads - Check download links work
* 11. Verify Distribution - Check YouTube/Dropbox/GDrive indicators
@@ -540,11 +540,15 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
console.log('STEP 6 COMPLETE: Lyrics processing finished');
// =========================================================================
// STEP 7: Lyrics Review (via UI - REAL USER FLOW)
// The minimal path: scroll to bottom -> click "Preview Video" -> wait for video -> click "Complete Review"
// STEP 7: Combined Lyrics Review & Instrumental Selection (via UI - REAL USER FLOW)
// After the UI redesign, lyrics review and instrumental selection are combined:
// 1. User opens lyrics review page
// 2. User clicks "Preview Video" to open modal
// 3. User clicks "Proceed to Instrumental Review" which navigates to instrumental selection
// 4. User selects instrumental and clicks "Confirm & Continue"
// =========================================================================
console.log('\n========================================');
console.log('STEP 7: Lyrics Review (UI Interaction)');
console.log('STEP 7: Combined Lyrics Review & Instrumental Selection');
console.log('========================================');
// Click the Review Lyrics button/link
@@ -621,7 +625,7 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
const errorText = await errorAlert.textContent();
console.log(` WARNING: Preview error: ${errorText}`);
await reviewPage.screenshot({ path: 'test-results/07e-preview-error.png', fullPage: true });
// Continue anyway - we can still click Complete Review even if preview failed
// Continue anyway - we can still proceed to instrumental even if preview failed
} else if (await videoElement.isVisible({ timeout: 10000 }).catch(() => false)) {
console.log(' Video element visible in modal');
// Give the video a moment to buffer/load
@@ -632,31 +636,93 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
await reviewPage.screenshot({ path: 'test-results/07f-video-state.png', fullPage: true });
// Click "Complete Review" button in the modal
// This button has variant="contained", color="success", and endIcon={<CloudUpload />}
const completeReviewBtn = reviewPage.getByRole('button', { name: /complete review/i });
await expect(completeReviewBtn).toBeVisible({ timeout: TIMEOUTS.action });
console.log(' Found "Complete Review" button');
// Click "Proceed to Instrumental Review" button in the modal
// This button saves corrections and navigates to the instrumental selection UI
const proceedBtn = reviewPage.getByRole('button', { name: /proceed to instrumental/i });
await expect(proceedBtn).toBeVisible({ timeout: TIMEOUTS.action });
console.log(' Found "Proceed to Instrumental Review" button');
await completeReviewBtn.click();
console.log(' Clicked "Complete Review" button');
await proceedBtn.click();
console.log(' Clicked "Proceed to Instrumental Review" button');
// Wait for the submission to process
// The handleSubmitToServer function calls apiClient.submitCorrections() then window.close()
// Wait for the navigation to instrumental selection (happens via hash change)
// The page navigates to #/{jobId}/instrumental
await reviewPage.waitForTimeout(3000);
await reviewPage.screenshot({ path: 'test-results/07g-after-proceed.png', fullPage: true });
// =========================================================================
// STEP 8: Instrumental Selection (continuation of combined flow)
// The page has navigated to the instrumental selection UI via hash routing
// =========================================================================
console.log('\n========================================');
console.log('STEP 8: Instrumental Selection');
console.log('========================================');
// Wait for the instrumental selection UI to load
// The InstrumentalSelector component renders selection options
console.log(' Waiting for instrumental selection UI to load...');
// Wait for either the selection options or the loading indicator to appear
try {
await reviewPage.waitForSelector('.selection-option, .selection-panel, [class*="selection"]', { timeout: 30000 });
console.log(' Instrumental selection UI loaded');
} catch {
// If selector not found, check for loading state
const loadingState = reviewPage.getByText(/loading instrumental/i);
if (await loadingState.isVisible({ timeout: 5000 }).catch(() => false)) {
console.log(' Waiting for instrumental analysis to complete...');
await expect(loadingState).not.toBeVisible({ timeout: 60000 });
}
}
await reviewPage.screenshot({ path: 'test-results/08a-instrumental-opened.png', fullPage: true });
console.log(' Instrumental selection UI opened');
// Select "Clean" instrumental option
// The options have class "selection-option" and contain labels with "Clean" or "With Backing"
const cleanOption = reviewPage.locator('.selection-option:has-text("Clean")').first();
if (await cleanOption.isVisible({ timeout: 5000 }).catch(() => false)) {
await cleanOption.click();
console.log(' Selected "Clean" instrumental option');
} else {
// If we can't find "Clean" specifically, the first option might already be selected
console.log(' Clean option not found - using default selection');
}
await reviewPage.screenshot({ path: 'test-results/08b-clean-selected.png', fullPage: true });
// Click the submit button: "✓ Confirm & Continue" (id="submit-btn")
const submitBtn = reviewPage.locator('#submit-btn');
await expect(submitBtn).toBeVisible({ timeout: TIMEOUTS.action });
console.log(' Found submit button');
await submitBtn.click();
console.log(' Clicked "Confirm & Continue" button');
// Wait for submission to complete - the page shows a success screen then redirects
// In cloud mode, after success it redirects to /app after a countdown
await reviewPage.waitForTimeout(5000);
await reviewPage.screenshot({ path: 'test-results/07g-after-complete-review.png', fullPage: true });
await reviewPage.screenshot({ path: 'test-results/08c-instrumental-submitted.png', fullPage: true });
// The review page may close automatically after submission (window.close())
// If it's still open, close it manually
// The page redirects to /app after completion, or may close
// Wait a bit for the redirect/close to happen
if (!reviewPage.isClosed()) {
await reviewPage.close();
console.log(' Closed review UI');
// Wait for potential redirect or success screen
try {
await reviewPage.waitForURL(/\/app/, { timeout: 10000 });
console.log(' Redirected to /app after instrumental selection');
} catch {
// If no redirect, close manually
await reviewPage.close();
console.log(' Closed review UI');
}
} else {
console.log(' Review UI closed automatically after submission');
}
// Refresh main app to see updated status
// Refresh main app
await page.bringToFront();
const refreshBtn = page.getByRole('button', { name: /refresh/i });
if (await refreshBtn.isVisible().catch(() => false)) {
@@ -664,121 +730,10 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
await page.waitForTimeout(3000);
}
await page.screenshot({ path: 'test-results/07h-after-review-refresh.png' });
console.log('STEP 7 COMPLETE: Lyrics review submitted via UI');
// =========================================================================
// STEP 8: Wait for Video Rendering & Instrumental Selection (via UI - REAL USER FLOW)
// The instrumental UI has: selection-option elements, submit button with id="submit-btn"
// =========================================================================
console.log('\n========================================');
console.log('STEP 8: Wait for Video Rendering & Instrumental Selection');
console.log('========================================');
// Wait for instrumental selection to become available
console.log(' Waiting for video rendering and instrumental selection...');
let foundInstrumentalStatus = false;
const renderStartTime = Date.now();
while (Date.now() - renderStartTime < TIMEOUTS.videoRendering) {
// Refresh
if (await refreshBtn.isVisible().catch(() => false)) {
await refreshBtn.click();
await page.waitForTimeout(2000);
}
// Check job card status
const statusText = await jobCard.textContent() || '';
console.log(` Status: ${statusText.substring(0, 80)}...`);
// Look for instrumental selection UI availability
// Users ALWAYS have to go through the instrumental review UI - there's no skip
if (statusText.toLowerCase().includes('select instrumental') ||
statusText.toLowerCase().includes('awaiting_instrumental') ||
(statusText.toLowerCase().includes('action needed') && statusText.toLowerCase().includes('instrumental'))) {
foundInstrumentalStatus = true;
console.log(' Job ready for instrumental selection');
break;
}
if (statusText.toLowerCase().includes('failed')) {
throw new Error('Job failed during video rendering');
}
await page.waitForTimeout(10000);
}
if (!foundInstrumentalStatus) {
throw new Error('Timeout waiting for instrumental selection - users must always complete this step');
}
{
// Click the Select Instrumental button/link
const instrumentalLink = jobCard.getByRole('link', { name: /select instrumental/i });
await expect(instrumentalLink).toBeVisible({ timeout: TIMEOUTS.action });
const instrumentalUrl = await instrumentalLink.getAttribute('href');
console.log(` Instrumental URL: ${instrumentalUrl}`);
// Open instrumental UI in new page
const instrumentalPage = await context.newPage();
// Handle both absolute and relative URLs
const fullUrl = instrumentalUrl?.startsWith('http') ? instrumentalUrl : `${PROD_URL}${instrumentalUrl}`;
// Use 'load' instead of 'networkidle' - instrumental page has ongoing audio requests
// that prevent networkidle from ever being reached
await instrumentalPage.goto(fullUrl, { waitUntil: 'load' });
await instrumentalPage.waitForTimeout(3000);
await instrumentalPage.screenshot({ path: 'test-results/08a-instrumental-opened.png', fullPage: true });
console.log(' Instrumental selection UI opened');
// Wait for the page to fully load - the instrumental UI renders selection options
await instrumentalPage.waitForSelector('.selection-option, .selection-panel', { timeout: 30000 });
// Select "Clean" instrumental option
// The options have class "selection-option" and contain labels with "Clean" or "With Backing"
const cleanOption = instrumentalPage.locator('.selection-option:has-text("Clean")').first();
if (await cleanOption.isVisible({ timeout: 5000 }).catch(() => false)) {
await cleanOption.click();
console.log(' Selected "Clean" instrumental option');
} else {
// If we can't find "Clean" specifically, the first option might already be selected
console.log(' Clean option not found - using default selection');
}
await instrumentalPage.screenshot({ path: 'test-results/08b-clean-selected.png', fullPage: true });
// Click the submit button: "✓ Confirm & Continue" (id="submit-btn")
const submitBtn = instrumentalPage.locator('#submit-btn');
await expect(submitBtn).toBeVisible({ timeout: TIMEOUTS.action });
console.log(' Found submit button');
await submitBtn.click();
console.log(' Clicked "Confirm & Continue" button');
// Wait for submission to complete - the page shows a success screen then auto-closes
await instrumentalPage.waitForTimeout(5000);
await instrumentalPage.screenshot({ path: 'test-results/08c-instrumental-submitted.png', fullPage: true });
// The page may auto-close after submission
if (!instrumentalPage.isClosed()) {
await instrumentalPage.close();
console.log(' Closed instrumental UI');
} else {
console.log(' Instrumental UI closed automatically after submission');
}
}
// Refresh main app
await page.bringToFront();
if (await refreshBtn.isVisible().catch(() => false)) {
await refreshBtn.click();
await page.waitForTimeout(3000);
}
await page.screenshot({ path: 'test-results/08d-after-instrumental.png' });
await reviewPage.screenshot({ path: 'test-results/08d-after-instrumental.png', fullPage: true }).catch(() => {
// reviewPage may be closed already, that's OK
});
await page.screenshot({ path: 'test-results/08d-main-after-instrumental.png' });
console.log('STEP 8 COMPLETE: Instrumental selection handled via UI');
// =========================================================================
@@ -190,43 +190,51 @@ test.describe('Lyrics Review Only - Focused Test', () => {
await page.screenshot({ path: 'test-results/focused-06-video-state.png', fullPage: true });
// Click "Complete Review" button in the modal
console.log('6. Looking for "Complete Review" button...');
// Click "Proceed to Instrumental Review" button in the modal
// (Previously "Complete Review" - changed with combined review UI)
console.log('6. Looking for "Proceed to Instrumental Review" button...');
// List all buttons in the modal
const modalButtons = await previewModal.getByRole('button').allTextContents();
console.log(` Modal buttons: ${modalButtons.join(', ')}`);
const completeReviewBtn = page.getByRole('button', { name: /complete review/i });
const proceedBtn = page.getByRole('button', { name: /proceed to instrumental/i });
if (await completeReviewBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
console.log(' Found "Complete Review" button - clicking...');
await completeReviewBtn.click();
console.log(' Clicked "Complete Review" button');
if (await proceedBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
console.log(' Found "Proceed to Instrumental Review" button - clicking...');
await proceedBtn.click();
console.log(' Clicked "Proceed to Instrumental Review" button');
} else {
// Try to find any submit-like button
const submitBtn = previewModal.getByRole('button', { name: /submit|confirm|done|save/i });
// Try to find any submit-like button (for backwards compatibility)
const submitBtn = previewModal.getByRole('button', { name: /submit|confirm|done|save|complete/i });
if (await submitBtn.isVisible().catch(() => false)) {
const btnText = await submitBtn.textContent();
console.log(` Found alternative submit button: "${btnText}" - clicking...`);
await submitBtn.click();
} else {
console.log(' Could not find Complete Review or similar button');
await page.screenshot({ path: 'test-results/focused-07-no-complete-btn.png', fullPage: true });
throw new Error('Could not find Complete Review button');
console.log(' Could not find Proceed to Instrumental or similar button');
await page.screenshot({ path: 'test-results/focused-07-no-proceed-btn.png', fullPage: true });
throw new Error('Could not find Proceed to Instrumental Review button');
}
}
// Wait for the submission to process
console.log('7. Waiting for submission to complete...');
// Wait for navigation to instrumental selection
console.log('7. Waiting for navigation to instrumental selection...');
await page.waitForTimeout(5000);
await page.screenshot({ path: 'test-results/focused-08-after-submit.png', fullPage: true });
await page.screenshot({ path: 'test-results/focused-08-after-proceed.png', fullPage: true });
// Note: This focused test only covers the lyrics review portion.
// The instrumental selection happens next on the same page via hash navigation.
// For full flow testing, use happy-path-real-user.spec.ts
console.log('');
console.log('========================================');
console.log('FOCUSED TEST COMPLETE');
console.log('========================================');
console.log('NOTE: This test completes the lyrics review step only.');
console.log(' The instrumental selection UI should now be visible on this page.');
console.log(' For the full flow, use happy-path-real-user.spec.ts');
});
test('Add Reference Lyrics via API', async ({ page, context, request }) => {
@@ -256,27 +256,28 @@ async function testLyricsReview(page: any, context: any, jobId: string) {
console.log(' WARNING: No video element found');
}
// Click "Complete Review" button
console.log('10. Clicking "Complete Review" button...');
// Click "Proceed to Instrumental Review" button (previously "Complete Review")
console.log('10. Clicking "Proceed to Instrumental Review" button...');
const modalButtons = await previewModal.getByRole('button').allTextContents();
console.log(` Modal buttons: ${modalButtons.join(', ')}`);
const completeReviewBtn = reviewPage.getByRole('button', { name: /complete review/i });
const proceedBtn = reviewPage.getByRole('button', { name: /proceed to instrumental/i });
if (await completeReviewBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
await completeReviewBtn.click();
console.log(' Clicked Complete Review button');
if (await proceedBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
await proceedBtn.click();
console.log(' Clicked Proceed to Instrumental Review button');
} else {
await reviewPage.screenshot({ path: 'test-results/existing-09-no-complete-btn.png', fullPage: true });
throw new Error('Could not find Complete Review button');
await reviewPage.screenshot({ path: 'test-results/existing-09-no-proceed-btn.png', fullPage: true });
throw new Error('Could not find Proceed to Instrumental Review button');
}
// Wait for submission
console.log('11. Waiting for submission...');
// Wait for navigation to instrumental selection (happens via hash change on same page)
console.log('11. Waiting for navigation to instrumental selection...');
await reviewPage.waitForTimeout(5000);
await reviewPage.screenshot({ path: 'test-results/existing-10-after-submit.png', fullPage: true });
console.log(' Lyrics review completed successfully!');
await reviewPage.screenshot({ path: 'test-results/existing-10-after-proceed.png', fullPage: true });
console.log(' Lyrics review step completed - now on instrumental selection!');
console.log(' NOTE: With combined UI, instrumental selection is on the same page.');
await reviewPage.close();
}
@@ -418,10 +418,9 @@ test.describe('Admin Job Detail Page', () => {
// Reset buttons are now in a toolbar (not accordion) labeled "Reset to:"
await expect(page.getByText('Reset to:')).toBeVisible();
// Check the reset buttons are visible in the toolbar
await expect(page.locator('button:has-text("Start")').first()).toBeVisible();
// Note: "Start" and "Inst." were removed; "Lyrics" renamed to "Review"
await expect(page.locator('button:has-text("Audio")').first()).toBeVisible();
await expect(page.locator('button:has-text("Lyrics")').first()).toBeVisible();
await expect(page.locator('button:has-text("Inst.")').first()).toBeVisible();
await expect(page.locator('button:has-text("Review")').first()).toBeVisible();
await expect(page.locator('button:has-text("Reprocess")').first()).toBeVisible();
});
@@ -438,8 +437,8 @@ test.describe('Admin Job Detail Page', () => {
await page.goto('/admin/jobs?id=test-job-123');
await page.waitForLoadState('networkidle');
// Click a reset button (Start = reset to pending)
await page.locator('button:has-text("Start")').first().click();
// Click a reset button (Audio = reset to awaiting_audio_selection)
await page.locator('button:has-text("Audio")').first().click();
// Should show confirmation dialog
await expect(page.getByRole('alertdialog')).toBeVisible();
@@ -448,121 +447,7 @@ test.describe('Admin Job Detail Page', () => {
await expect(page.getByRole('button', { name: 'Reset' })).toBeVisible();
});
test('clear workers button is visible in toolbar', async ({ page }) => {
// Mock job with complete worker progress (typical state when workers need clearing)
const jobWithWorkerProgress = {
...mockJob,
state_data: {
...mockJob.state_data,
render_progress: { stage: 'complete' },
video_progress: { stage: 'complete' },
encoding_progress: { stage: 'complete' },
},
};
await setupApiFixtures(page, {
mocks: [
{ method: 'GET', path: '/api/users/me', response: { body: mockAdminUser } },
{ method: 'GET', path: '/api/jobs', response: { body: [] } },
{ method: 'GET', path: '/api/jobs/test-job-123', response: { body: jobWithWorkerProgress } },
{ method: 'GET', path: '/api/jobs/test-job-123/logs', response: { body: mockLogs } },
],
});
await page.goto('/admin/jobs?id=test-job-123');
await page.waitForLoadState('networkidle');
// Clear Workers button should be visible in the toolbar (near Del Outputs, Delete Job)
await expect(page.locator('button:has-text("Clear Workers")')).toBeVisible();
});
test('clear workers button calls API endpoint', async ({ page }) => {
// Mock successful clear workers response
const clearWorkersResponse = {
status: 'success',
job_id: 'test-job-123',
message: 'Cleared 3 worker progress keys',
cleared_keys: ['render_progress', 'video_progress', 'encoding_progress'],
};
// Track API calls
let clearWorkersApiCalled = false;
await setupApiFixtures(page, {
mocks: [
{ method: 'GET', path: '/api/users/me', response: { body: mockAdminUser } },
{ method: 'GET', path: '/api/jobs', response: { body: [] } },
{ method: 'GET', path: '/api/jobs/test-job-123', response: { body: mockJob } },
{ method: 'GET', path: '/api/jobs/test-job-123/logs', response: { body: mockLogs } },
{ method: 'POST', path: '/api/admin/jobs/test-job-123/clear-workers', response: { body: clearWorkersResponse } },
],
});
// Also track when the clear-workers endpoint is called
await page.route('**/api/admin/jobs/*/clear-workers', async (route) => {
clearWorkersApiCalled = true;
await route.fulfill({
status: 200,
contentType: 'application/json',
body: JSON.stringify(clearWorkersResponse),
});
});
await page.goto('/admin/jobs?id=test-job-123');
await page.waitForLoadState('networkidle');
// Click Clear Workers button in toolbar
await page.locator('button:has-text("Clear Workers")').click();
// Wait a bit for the API call to complete
await page.waitForTimeout(1000);
// Verify the API was called
expect(clearWorkersApiCalled).toBe(true);
});
test('clear workers button shows loading state while processing', async ({ page }) => {
// Mock slow clear workers response
const clearWorkersResponse = {
status: 'success',
job_id: 'test-job-123',
message: 'Cleared 3 worker progress keys',
cleared_keys: ['render_progress', 'video_progress', 'encoding_progress'],
};
await setupApiFixtures(page, {
mocks: [
{ method: 'GET', path: '/api/users/me', response: { body: mockAdminUser } },
{ method: 'GET', path: '/api/jobs', response: { body: [] } },
{ method: 'GET', path: '/api/jobs/test-job-123', response: { body: mockJob } },
{ method: 'GET', path: '/api/jobs/test-job-123/logs', response: { body: mockLogs } },
],
});
// Add delayed response for clear-workers
await page.route('**/api/admin/jobs/*/clear-workers', async (route) => {
// Delay response to test loading state
await new Promise(resolve => setTimeout(resolve, 500));
await route.fulfill({
status: 200,
contentType: 'application/json',
body: JSON.stringify(clearWorkersResponse),
});
});
await page.goto('/admin/jobs?id=test-job-123');
await page.waitForLoadState('networkidle');
// Get the Clear Workers button
const clearWorkersButton = page.locator('button:has-text("Clear Workers")');
// Click it and immediately check for disabled state
await clearWorkersButton.click();
// Button should be disabled during processing (has spinning icon)
// The button contains a RefreshCw icon that animates when clearingWorkers is true
await expect(clearWorkersButton).toBeDisabled({ timeout: 500 });
});
// Note: "Clear Workers" tests removed - feature replaced by Full Restart + Regen Screens
test('delete outputs button shows confirmation dialog', async ({ page }) => {
await setupApiFixtures(page, {
@@ -103,9 +103,16 @@ const mockJobData = {
};
const mockUserData = {
email: 'test@example.com',
name: 'Test User',
role: 'user',
user: {
email: 'test@example.com',
name: 'Test User',
role: 'user',
credits: 10,
display_name: 'Test User',
total_jobs_created: 5,
total_jobs_completed: 3,
},
has_session: true,
};
// Setup function to configure mocks for cloud mode
@@ -147,6 +154,45 @@ async function setupCloudModeMocks(page: Page) {
path: `/api/review/${TEST_JOB_ID}/audio/test_audio_hash_cloud`,
response: { status: 200, body: '' },
},
// Instrumental analysis endpoint (for instrumental route)
{
method: 'GET',
path: `/api/review/${TEST_JOB_ID}/instrumental-analysis`,
response: {
body: {
job_id: TEST_JOB_ID,
artist: 'Test Artist',
title: 'Test Song',
duration_seconds: 180,
analysis: {
has_audible_content: false,
total_duration_seconds: 180,
audible_segments: [],
recommended_selection: 'clean',
total_audible_duration_seconds: 0,
audible_percentage: 0,
silence_threshold_db: -40,
},
audio_urls: {
clean: 'https://storage.example.com/clean.flac',
with_backing: 'https://storage.example.com/with_backing.flac',
},
has_original: true,
has_uploaded_instrumental: false,
},
},
},
// Waveform data endpoint (for instrumental route)
{
method: 'GET',
path: `/api/review/${TEST_JOB_ID}/waveform-data`,
response: {
body: {
amplitudes: Array(100).fill(0.5),
duration_seconds: 180,
},
},
},
// User endpoint (for auth)
{
method: 'GET',
@@ -179,7 +225,7 @@ test.describe('Cloud Mode - Hash Route Parsing', () => {
});
test('parses hash-based instrumental route correctly', async ({ page }) => {
// Update mock to include instrumental options for the job in instrumental state
// Update mock to include instrumental analysis endpoints
await setupApiFixtures(page, {
mocks: [
{
@@ -192,6 +238,43 @@ test.describe('Cloud Mode - Hash Route Parsing', () => {
path: `/api/review/${TEST_JOB_ID}/correction-data`,
response: { body: mockCorrectionData },
},
{
method: 'GET',
path: `/api/review/${TEST_JOB_ID}/instrumental-analysis`,
response: {
body: {
job_id: TEST_JOB_ID,
artist: 'Test Artist',
title: 'Test Song',
duration_seconds: 180,
analysis: {
has_audible_content: false,
total_duration_seconds: 180,
audible_segments: [],
recommended_selection: 'clean',
total_audible_duration_seconds: 0,
audible_percentage: 0,
silence_threshold_db: -40,
},
audio_urls: {
clean: 'https://storage.example.com/clean.flac',
with_backing: 'https://storage.example.com/with_backing.flac',
},
has_original: true,
has_uploaded_instrumental: false,
},
},
},
{
method: 'GET',
path: `/api/review/${TEST_JOB_ID}/waveform-data`,
response: {
body: {
amplitudes: Array(100).fill(0.5),
duration_seconds: 180,
},
},
},
{
method: 'GET',
path: '/api/users/me',
@@ -210,7 +293,7 @@ test.describe('Cloud Mode - Hash Route Parsing', () => {
await page.waitForLoadState('networkidle');
// Should show the instrumental selection UI (not "Page not found")
await expect(page.locator('h1')).toContainText('Instrumental Selection', { timeout: 10000 });
await expect(page.getByText('Instrumental Review')).toBeVisible({ timeout: 10000 });
});
test('invalid hash route shows page not found', async ({ page }) => {
@@ -297,7 +380,7 @@ test.describe('Cloud Mode - Navigation Flow', () => {
await page.waitForTimeout(500);
// Should now show instrumental UI
await expect(page.locator('h1')).toContainText('Instrumental Selection', { timeout: 10000 });
await expect(page.getByText('Instrumental Review')).toBeVisible({ timeout: 10000 });
});
});
@@ -416,7 +499,7 @@ test.describe('Cloud Mode - Instrumental Audio Loading', () => {
await page.waitForLoadState('networkidle');
// Should show the instrumental selection UI
await expect(page.locator('h1')).toContainText('Instrumental Selection', { timeout: 10000 });
await expect(page.getByText('Instrumental Review')).toBeVisible({ timeout: 10000 });
// Wait for UI to fully load
await page.waitForTimeout(1000);
@@ -507,7 +590,7 @@ test.describe('Cloud Mode - Instrumental Audio Loading', () => {
await page.waitForLoadState('networkidle');
// Should show the instrumental selection UI
await expect(page.locator('h1')).toContainText('Instrumental Selection', { timeout: 10000 });
await expect(page.getByText('Instrumental Review')).toBeVisible({ timeout: 10000 });
// Find audio tabs
const originalTab = page.getByRole('button', { name: /^original$/i });
@@ -102,9 +102,16 @@ const mockJobData = {
};
const mockUserData = {
email: 'test@example.com',
name: 'Test User',
role: 'user',
user: {
email: 'test@example.com',
name: 'Test User',
role: 'user',
credits: 10,
display_name: 'Test User',
total_jobs_created: 5,
total_jobs_completed: 3,
},
has_session: true,
};
const mockInstrumentalAnalysis = {
@@ -166,7 +173,7 @@ test.describe('Combined Review Corrections Preservation', () => {
}
});
await setAuthToken(page);
await setAuthToken(page, 'test-token-123');
});
test.afterEach(async ({ page }) => {
@@ -289,7 +296,7 @@ test.describe('Combined Review Corrections Preservation', () => {
// Audio endpoint
{
method: 'GET',
path: new RegExp(`/api/review/${TEST_JOB_ID}/audio`),
path: `/api/review/${TEST_JOB_ID}/audio/:audioHash`,
response: { status: 200, body: '' },
},
],
@@ -381,7 +388,7 @@ test.describe('Combined Review Corrections Preservation', () => {
test.describe('Regression Prevention', () => {
test.beforeEach(async ({ page }) => {
await setAuthToken(page);
await setAuthToken(page, 'test-token-123');
});
test.afterEach(async ({ page }) => {
@@ -0,0 +1,135 @@
import { test, expect } from '@playwright/test';
import { setupApiFixtures, setAuthToken } from '../fixtures/test-helper';
/**
* Regression Tests - Credit Enforcement
*
* Tests that the credit system properly gates job creation:
* - Zero-credit users see warning and disabled submit
* - 402 errors display "out of credits" message with buy link
* - Users with credits can submit normally
*/
const BASE_MOCKS = [
{
method: 'GET',
path: '/api/themes',
response: { body: { themes: [] } },
},
{
method: 'GET',
path: '/api/users/credits/packages',
response: { body: { packages: [] } },
},
];
function userMeMock(credits: number, role: string = 'user') {
return {
method: 'GET',
path: '/api/users/me',
response: {
body: {
user: { email: 'test@example.com', credits, role },
has_session: true,
},
},
};
}
test.describe('Credit Enforcement', () => {
test.beforeEach(async ({ page }) => {
await setAuthToken(page, 'test-token');
});
test('zero-credit user sees warning banner and disabled submit', async ({ page }) => {
await setupApiFixtures(page, {
mocks: [
userMeMock(0),
...BASE_MOCKS,
{
method: 'GET',
path: '/api/jobs',
response: { body: [] },
},
],
});
await page.goto('/app');
await page.waitForLoadState('networkidle');
// Warning banner should be visible
await expect(page.getByText('You have no credits remaining')).toBeVisible();
await expect(page.getByRole('link', { name: 'Buy Credits' })).toBeVisible();
// Submit button should be disabled
const submitButton = page.getByRole('button', { name: /Create Karaoke Video|Search & Create Job/i });
await expect(submitButton).toBeDisabled();
});
test('402 error displays out of credits message with buy link', async ({ page }) => {
await setupApiFixtures(page, {
mocks: [
userMeMock(1), // Has credits initially (form enabled)
...BASE_MOCKS,
{
method: 'GET',
path: '/api/jobs',
response: { body: [] },
},
{
method: 'POST',
path: '/api/audio-search',
response: {
status: 402,
body: {
detail: "You're out of credits. Buy more to continue creating karaoke videos.",
credits_available: 0,
credits_required: 1,
buy_url: '/#pricing',
},
},
},
],
});
await page.goto('/app');
await page.waitForLoadState('networkidle');
// Fill in the search form
await page.getByTestId('search-artist-input').fill('Test Artist');
await page.getByTestId('search-title-input').fill('Test Song');
// Submit
await page.getByRole('button', { name: /Search & Create Job/i }).click();
// Should show credit error message
await expect(page.getByText(/out of credits/i)).toBeVisible();
// Should show buy credits link
await expect(page.getByRole('link', { name: 'Buy Credits' })).toBeVisible();
});
test('user with credits can submit normally', async ({ page }) => {
await setupApiFixtures(page, {
mocks: [
userMeMock(5),
...BASE_MOCKS,
{
method: 'GET',
path: '/api/jobs',
response: { body: [] },
},
],
});
await page.goto('/app');
await page.waitForLoadState('networkidle');
// No warning banner
await expect(page.getByText('You have no credits remaining')).not.toBeVisible();
// Submit button should be enabled
const submitButton = page.getByRole('button', { name: /Create Karaoke Video|Search & Create Job/i });
await expect(submitButton).toBeEnabled();
});
});
+32 -34
View File
@@ -463,8 +463,8 @@ test.describe('Lyrics Review - Edit Operations', () => {
const dialog = page.locator('[role="dialog"]');
await expect(dialog).toBeVisible({ timeout: 5000 });
// Find the start time input (labeled "Start")
const startInput = dialog.getByLabel('Start');
// Find the first start time input (labeled "Start") - multiple segments may have one
const startInput = dialog.getByLabel('Start').first();
await expect(startInput).toBeVisible();
// Click into the input to focus it
@@ -477,7 +477,7 @@ test.describe('Lyrics Review - Edit Operations', () => {
await expect(startInput).toHaveValue('12.34');
// Test end time input as well
const endInput = dialog.getByLabel('End');
const endInput = dialog.getByLabel('End').first();
await expect(endInput).toBeVisible();
await endInput.click();
@@ -566,7 +566,7 @@ test.describe('Lyrics Review - Complete Review Flow', () => {
await expect(dialog).toBeVisible({ timeout: 10000 });
// Look for Complete Review button in modal
const completeBtn = dialog.getByRole('button', { name: /complete review/i });
const completeBtn = dialog.getByRole('button', { name: /proceed to instrumental review/i });
if (await completeBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
await completeBtn.click();
// Should submit and close
@@ -655,27 +655,7 @@ test.describe('Lyrics Review - Submit UX', () => {
await setupSubmitFlowMocks(page);
});
test('success screen shows after submission in local mode', async ({ page }) => {
await page.goto('/app/jobs/local/review');
await page.waitForLoadState('networkidle');
// Open preview modal and submit
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
const previewBtn = page.getByRole('button', { name: /preview video/i });
await expect(previewBtn).toBeVisible({ timeout: 10000 });
await previewBtn.click();
const dialog = page.locator('[role="dialog"]');
await expect(dialog).toBeVisible({ timeout: 10000 });
await dialog.getByRole('button', { name: /complete review/i }).click();
// Wait for success screen
await expect(page.getByText('Review Submitted')).toBeVisible({ timeout: 10000 });
await expect(page.getByText(/Closing in \d+s/)).toBeVisible();
});
test('submit button becomes disabled when clicked', async ({ page }) => {
test('proceed button navigates to instrumental review in local mode', async ({ page }) => {
await page.goto('/app/jobs/local/review');
await page.waitForLoadState('networkidle');
@@ -688,18 +668,36 @@ test.describe('Lyrics Review - Submit UX', () => {
const dialog = page.locator('[role="dialog"]');
await expect(dialog).toBeVisible({ timeout: 10000 });
const completeBtn = dialog.getByRole('button', { name: /complete review/i });
await expect(completeBtn).toBeVisible({ timeout: 5000 });
await dialog.getByRole('button', { name: /proceed to instrumental review/i }).click();
// Should navigate to instrumental review page
await page.waitForURL('**/instrumental**', { timeout: 10000 });
expect(page.url()).toContain('/instrumental');
});
test('proceed button is enabled and clickable in preview modal', async ({ page }) => {
await page.goto('/app/jobs/local/review');
await page.waitForLoadState('networkidle');
// Open preview modal
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
const previewBtn = page.getByRole('button', { name: /preview video/i });
await expect(previewBtn).toBeVisible({ timeout: 10000 });
await previewBtn.click();
const dialog = page.locator('[role="dialog"]');
await expect(dialog).toBeVisible({ timeout: 10000 });
const proceedBtn = dialog.getByRole('button', { name: /proceed to instrumental review/i });
await expect(proceedBtn).toBeVisible({ timeout: 5000 });
// Verify button is enabled before click
await expect(completeBtn).toBeEnabled();
await expect(proceedBtn).toBeEnabled();
// Click the button
await completeBtn.click();
// Transition should occur - either loading state or success screen
// We can verify we reach the success screen
await expect(page.getByText('Review Submitted')).toBeVisible({ timeout: 10000 });
// Click navigates to instrumental review
await proceedBtn.click();
await page.waitForURL('**/instrumental**', { timeout: 10000 });
expect(page.url()).toContain('/instrumental');
});
});
@@ -0,0 +1,270 @@
import { test, expect, Page } from '@playwright/test';
import { setupApiFixtures, clearAuthToken } from '../fixtures/test-helper';
/**
* Replace Segment Lyrics E2E Tests
*
* Tests the "Replace Segment Lyrics" mode in Edit All Lyrics, which lets users
* swap in custom lyrics line-by-line while preserving existing segment timing.
*
* To run these tests:
* npx playwright test e2e/regression/replace-segments.spec.ts
*/
const mockCorrectionData = {
original_segments: [
{
id: 'seg_1',
text: 'Hello world from the test',
start_time: 0.0,
end_time: 2.5,
words: [
{ id: 'word_1', text: 'Hello', start_time: 0.0, end_time: 0.5, confidence: 0.95 },
{ id: 'word_2', text: 'world', start_time: 0.5, end_time: 1.0, confidence: 0.92 },
{ id: 'word_3', text: 'from', start_time: 1.0, end_time: 1.5, confidence: 0.88 },
{ id: 'word_4', text: 'the', start_time: 1.5, end_time: 2.0, confidence: 0.90 },
{ id: 'word_5', text: 'test', start_time: 2.0, end_time: 2.5, confidence: 0.94 },
],
},
{
id: 'seg_2',
text: 'This is another line',
start_time: 3.0,
end_time: 5.0,
words: [
{ id: 'word_6', text: 'This', start_time: 3.0, end_time: 3.5, confidence: 0.96 },
{ id: 'word_7', text: 'is', start_time: 3.5, end_time: 4.0, confidence: 0.97 },
{ id: 'word_8', text: 'another', start_time: 4.0, end_time: 4.5, confidence: 0.93 },
{ id: 'word_9', text: 'line', start_time: 4.5, end_time: 5.0, confidence: 0.91 },
],
},
{
id: 'seg_3',
text: 'Final segment here',
start_time: 6.0,
end_time: 8.0,
words: [
{ id: 'word_10', text: 'Final', start_time: 6.0, end_time: 6.7, confidence: 0.90 },
{ id: 'word_11', text: 'segment', start_time: 6.7, end_time: 7.4, confidence: 0.91 },
{ id: 'word_12', text: 'here', start_time: 7.4, end_time: 8.0, confidence: 0.92 },
],
},
],
corrected_segments: [
{
id: 'seg_1',
text: 'Hello world from the test',
start_time: 0.0,
end_time: 2.5,
words: [
{ id: 'word_1', text: 'Hello', start_time: 0.0, end_time: 0.5, confidence: 0.95 },
{ id: 'word_2', text: 'world', start_time: 0.5, end_time: 1.0, confidence: 0.92 },
{ id: 'word_3', text: 'from', start_time: 1.0, end_time: 1.5, confidence: 0.88 },
{ id: 'word_4', text: 'the', start_time: 1.5, end_time: 2.0, confidence: 0.90 },
{ id: 'word_5', text: 'test', start_time: 2.0, end_time: 2.5, confidence: 0.94 },
],
},
{
id: 'seg_2',
text: 'This is another line',
start_time: 3.0,
end_time: 5.0,
words: [
{ id: 'word_6', text: 'This', start_time: 3.0, end_time: 3.5, confidence: 0.96 },
{ id: 'word_7', text: 'is', start_time: 3.5, end_time: 4.0, confidence: 0.97 },
{ id: 'word_8', text: 'another', start_time: 4.0, end_time: 4.5, confidence: 0.93 },
{ id: 'word_9', text: 'line', start_time: 4.5, end_time: 5.0, confidence: 0.91 },
],
},
{
id: 'seg_3',
text: 'Final segment here',
start_time: 6.0,
end_time: 8.0,
words: [
{ id: 'word_10', text: 'Final', start_time: 6.0, end_time: 6.7, confidence: 0.90 },
{ id: 'word_11', text: 'segment', start_time: 6.7, end_time: 7.4, confidence: 0.91 },
{ id: 'word_12', text: 'here', start_time: 7.4, end_time: 8.0, confidence: 0.92 },
],
},
],
reference_lyrics: {},
anchor_sequences: [],
gap_sequences: [],
corrections: [],
corrections_made: 0,
confidence: 0.9,
metadata: {
anchor_sequences_count: 0,
gap_sequences_count: 0,
total_words: 12,
correction_ratio: 0,
audio_hash: 'test_audio_hash_123',
artist: 'Test Artist',
title: 'Test Song',
available_handlers: [],
enabled_handlers: [],
},
correction_steps: [],
word_id_map: {},
segment_id_map: {},
resized_segments: [],
};
const mockJobData = {
job_id: 'local',
status: 'awaiting_review',
progress: 50,
created_at: null,
updated_at: null,
artist: 'Test Artist',
title: 'Test Song',
user_email: 'local@localhost',
audio_hash: 'test_audio_hash_123',
};
async function setupLocalModeMocks(page: Page) {
return setupApiFixtures(page, {
mocks: [
{ method: 'GET', path: '/api/jobs/local', response: { body: mockJobData } },
{ method: 'GET', path: '/api/jobs/local/corrections', response: { body: mockCorrectionData } },
{ method: 'GET', path: '/api/review/local/correction-data', response: { body: mockCorrectionData } },
{ method: 'POST', path: '/api/jobs/local/corrections', response: { body: { status: 'success' } } },
{ method: 'GET', path: '/api/audio/test_audio_hash_123', response: { status: 200, body: '' } },
{ method: 'GET', path: '/api/review/local/audio/test_audio_hash_123', response: { status: 200, body: '' } },
{ method: 'GET', path: '/api/tenant/config', response: { body: { tenant: null, is_default: true } } },
],
});
}
async function openReplaceSegmentsMode(page: Page) {
await page.goto('/app/jobs/local/review');
await page.waitForLoadState('networkidle');
// Click "Edit All" button
const editAllBtn = page.getByRole('button', { name: /edit all/i });
await expect(editAllBtn).toBeVisible({ timeout: 10000 });
await editAllBtn.click();
// Mode selection should appear
await expect(page.getByText('Edit All Lyrics')).toBeVisible({ timeout: 5000 });
// Click "Replace Segment Lyrics"
const replaceSegmentsCard = page.getByText('Replace Segment Lyrics');
await expect(replaceSegmentsCard).toBeVisible();
await replaceSegmentsCard.click();
// Wait for replace segments dialog
await expect(page.getByText('Edit lyrics line by line')).toBeVisible({ timeout: 5000 });
}
test.describe('Replace Segment Lyrics', () => {
test.beforeEach(async ({ page }) => {
await clearAuthToken(page);
await setupLocalModeMocks(page);
});
test('shows Replace Segment Lyrics option in mode selection', async ({ page }) => {
await page.goto('/app/jobs/local/review');
await page.waitForLoadState('networkidle');
const editAllBtn = page.getByRole('button', { name: /edit all/i });
await expect(editAllBtn).toBeVisible({ timeout: 10000 });
await editAllBtn.click();
// Should see all three options
await expect(page.getByText('Re-sync Existing Lyrics')).toBeVisible();
await expect(page.getByText('Replace Segment Lyrics')).toBeVisible();
await expect(page.getByText('Replace All Lyrics')).toBeVisible();
await expect(page.getByText('Recommended for custom lyrics')).toBeVisible();
});
test('textarea shows existing lyrics pre-filled', async ({ page }) => {
await openReplaceSegmentsMode(page);
const textarea = page.getByRole('textbox');
await expect(textarea).toHaveValue(
'Hello world from the test\nThis is another line\nFinal segment here'
);
});
test('shows matching line count when correct', async ({ page }) => {
await openReplaceSegmentsMode(page);
await expect(page.getByText('3/3 lines')).toBeVisible();
});
test('Apply is enabled when line count matches', async ({ page }) => {
await openReplaceSegmentsMode(page);
const applyBtn = page.getByRole('button', { name: 'Apply' });
await expect(applyBtn).toBeEnabled();
});
test('disables Apply and shows error when extra line added', async ({ page }) => {
await openReplaceSegmentsMode(page);
const textarea = page.getByRole('textbox');
// Add an extra line
await textarea.fill(
'Hello world from the test\nThis is another line\nFinal segment here\nExtra line'
);
const applyBtn = page.getByRole('button', { name: 'Apply' });
await expect(applyBtn).toBeDisabled();
await expect(page.getByText(/1 too many/)).toBeVisible();
await expect(page.getByText('4/3 lines')).toBeVisible();
});
test('disables Apply and shows error when line removed', async ({ page }) => {
await openReplaceSegmentsMode(page);
const textarea = page.getByRole('textbox');
await textarea.fill('Hello world from the test\nThis is another line');
const applyBtn = page.getByRole('button', { name: 'Apply' });
await expect(applyBtn).toBeDisabled();
await expect(page.getByText(/1 too few/)).toBeVisible();
});
test('Apply updates modified segment lyrics in the UI', async ({ page }) => {
await openReplaceSegmentsMode(page);
const textarea = page.getByRole('textbox');
await textarea.fill(
'Hello world from the test\nCustom parody lyrics now\nFinal segment here'
);
const applyBtn = page.getByRole('button', { name: 'Apply' });
await expect(applyBtn).toBeEnabled();
await applyBtn.click();
// Dialog should close
await expect(page.getByText('Edit lyrics line by line')).not.toBeVisible();
// The modified line should appear in the lyrics view
await expect(page.getByText('Custom')).toBeVisible({ timeout: 5000 });
await expect(page.getByText('parody')).toBeVisible();
// Unchanged lines should still be there
await expect(page.getByText('Hello')).toBeVisible();
await expect(page.getByText('Final')).toBeVisible();
});
test('re-enables Apply after fixing line count mismatch', async ({ page }) => {
await openReplaceSegmentsMode(page);
const textarea = page.getByRole('textbox');
// Add extra line
await textarea.fill(
'Hello world from the test\nThis is another line\nFinal segment here\nExtra'
);
await expect(page.getByRole('button', { name: 'Apply' })).toBeDisabled();
// Fix it back to 3 lines
await textarea.fill(
'Hello world from the test\nThis is another line\nFinal segment here'
);
await expect(page.getByRole('button', { name: 'Apply' })).toBeEnabled();
});
});
+1
View File
@@ -11,6 +11,7 @@ const customJestConfig = {
testEnvironment: 'jest-environment-jsdom',
moduleNameMapper: {
'^@/(.*)$': '<rootDir>/$1',
'^nanoid$': '<rootDir>/node_modules/nanoid/index.cjs',
},
testMatch: [
'**/__tests__/**/*.test.[jt]s?(x)',
+8
View File
@@ -27,3 +27,11 @@ global.IntersectionObserver = class IntersectionObserver {
unobserve() {}
}
// Mock ResizeObserver
global.ResizeObserver = class ResizeObserver {
constructor() {}
disconnect() {}
observe() {}
unobserve() {}
}
Loaded 100 of 574 files, more files were not shown because too many files have changed in this diff. Show more