mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 21:17:56 +09:00
Compare commits
41
Commits
No files matched your search
@@ -1,118 +0,0 @@
|
||||
---
|
||||
description: Review and address CodeRabbit comments on current PR
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write, Bash, mcp__github__pull_request_read, mcp__github__get_me, mcp__github__list_pull_requests, mcp__github__add_issue_comment
|
||||
---
|
||||
|
||||
# CodeRabbit Comment Review
|
||||
|
||||
You are reviewing the current PR to address all CodeRabbit automated review comments.
|
||||
|
||||
## Your Task
|
||||
|
||||
### 1. Identify the Current PR
|
||||
|
||||
First, determine the current branch and find the associated PR:
|
||||
```bash
|
||||
git branch --show-current
|
||||
```
|
||||
|
||||
Then use GitHub tools to find the PR for this branch in the `nomadkaraoke/karaoke-gen` repository.
|
||||
|
||||
### 2. Fetch CodeRabbit Comments
|
||||
|
||||
Get all reviews, review comments, and regular comments on the PR:
|
||||
|
||||
**PR Reviews (MOST IMPORTANT - Contains Nitpicks and Issues)**
|
||||
- Use `pull_request_read` with `method: "get_reviews"` to get all PR reviews
|
||||
- Look for reviews from the `coderabbitai[bot]` user
|
||||
- The review `body` field contains the detailed analysis with:
|
||||
- "Nitpick comments" section with specific code suggestions
|
||||
- "Potential issues" flagged in the diff
|
||||
- Code improvement suggestions with file paths and line numbers
|
||||
- **This is the primary source of CodeRabbit feedback** - parse this carefully
|
||||
|
||||
**Review Comments (Inline Code Comments)**
|
||||
- Use `pull_request_read` with `method: "get_review_comments"` to get inline code comments
|
||||
- These are comments attached to specific lines in the diff
|
||||
- Also check for `coderabbitai[bot]` comments here
|
||||
|
||||
**PR Comments (Summary)**
|
||||
- Use `pull_request_read` with `method: "get_comments"` to get regular PR comments
|
||||
- CodeRabbit posts a walkthrough/summary comment here
|
||||
- Less actionable than the review body, but may contain additional context
|
||||
|
||||
### 3. Analyze Each Comment
|
||||
|
||||
For each CodeRabbit comment:
|
||||
1. Read the comment carefully to understand the concern
|
||||
2. Look at the referenced code location
|
||||
3. Decide if a code change is warranted:
|
||||
- **Make changes** for: valid bugs, security issues, clear improvements, style violations
|
||||
- **Skip changes** for: subjective preferences, false positives, already addressed, not applicable
|
||||
4. If skipping, note the reason
|
||||
|
||||
### 4. Make Code Changes
|
||||
|
||||
For each comment requiring action:
|
||||
1. Navigate to the file and line mentioned
|
||||
2. Understand the surrounding context
|
||||
3. Make the minimal change needed to address the concern
|
||||
4. Verify the change doesn't break anything obvious
|
||||
|
||||
### 5. Commit and Push
|
||||
|
||||
After making all changes:
|
||||
```bash
|
||||
# Stage all changes
|
||||
git add -A
|
||||
|
||||
# Commit with descriptive message
|
||||
git commit -m "fix: address CodeRabbit review comments
|
||||
|
||||
- [list each fix briefly]
|
||||
|
||||
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||||
|
||||
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>"
|
||||
|
||||
# Push to the PR branch
|
||||
git push
|
||||
```
|
||||
|
||||
### 6. Resolve Threads
|
||||
|
||||
After pushing:
|
||||
- Reply to each CodeRabbit review thread indicating what was done
|
||||
- If a comment was addressed, reply with what was changed
|
||||
- If a comment was intentionally skipped, explain why
|
||||
- This helps mark threads as resolved so the PR can be merged
|
||||
|
||||
## Output Format
|
||||
|
||||
Provide a summary report:
|
||||
|
||||
```
|
||||
## CodeRabbit Review Summary
|
||||
|
||||
### Comments Addressed
|
||||
- [ ] File:line - Brief description of issue → What was changed
|
||||
|
||||
### Comments Skipped (with reason)
|
||||
- [ ] File:line - Issue → Reason for skipping
|
||||
|
||||
### Actions Taken
|
||||
- Committed: [commit hash]
|
||||
- Pushed to: [branch name]
|
||||
- Threads replied to: [count]
|
||||
|
||||
### Status
|
||||
Ready for merge: Yes/No (if No, explain why)
|
||||
```
|
||||
|
||||
## Guidelines
|
||||
|
||||
- Focus on code quality improvements, not just making CodeRabbit happy
|
||||
- Don't blindly accept every suggestion - use judgment
|
||||
- Group related fixes into logical commits if needed
|
||||
- Keep changes minimal and focused on the specific issues raised
|
||||
- If a suggestion would require significant refactoring, note it for future work instead
|
||||
@@ -1,74 +0,0 @@
|
||||
---
|
||||
description: Periodic documentation maintenance and organization check
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write, Bash
|
||||
---
|
||||
|
||||
# Documentation Maintenance Check
|
||||
|
||||
You are performing periodic maintenance on the project documentation.
|
||||
|
||||
## Checks to Perform
|
||||
|
||||
### 1. Structure Compliance
|
||||
Verify docs/ follows the correct structure:
|
||||
```
|
||||
docs/
|
||||
├── README.md # Current status + navigation
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Accumulated wisdom
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
Report any files that don't belong or are misplaced.
|
||||
|
||||
### 2. Freshness Check
|
||||
- Read `docs/README.md` - is the status section current?
|
||||
- Check recent git commits - do they suggest docs should be updated?
|
||||
- Look for stale information that contradicts current code
|
||||
|
||||
### 3. Archive Organization
|
||||
- Verify all archive files have YYYY-MM-DD prefix
|
||||
- Check for any docs that should be archived (completed work, old plans)
|
||||
- Remove truly obsolete content (empty files, duplicate info)
|
||||
|
||||
### 4. Cross-Reference Check
|
||||
- Verify CLAUDE.md points to current doc locations
|
||||
- Check that docs reference each other correctly
|
||||
- Ensure no broken internal links
|
||||
|
||||
### 5. Content Quality
|
||||
- Flag any docs over 500 lines (may need splitting)
|
||||
- Identify duplicate information across docs
|
||||
- Note sections that seem outdated
|
||||
|
||||
## Actions to Take
|
||||
|
||||
1. **Fix structural issues** - Move/rename misplaced files
|
||||
2. **Update stale content** - Refresh outdated information
|
||||
3. **Archive completed work** - Move old docs to archive/
|
||||
4. **Report findings** - Summarize what was found and fixed
|
||||
|
||||
## Output Format
|
||||
|
||||
Provide a maintenance report:
|
||||
```
|
||||
## Documentation Maintenance Report - YYYY-MM-DD
|
||||
|
||||
### Issues Found
|
||||
- [list of issues]
|
||||
|
||||
### Actions Taken
|
||||
- [list of fixes made]
|
||||
|
||||
### Recommendations
|
||||
- [suggestions for future attention]
|
||||
|
||||
### Status
|
||||
- [ ] Structure: OK/Fixed
|
||||
- [ ] Freshness: OK/Updated
|
||||
- [ ] Archive: OK/Organized
|
||||
- [ ] Cross-refs: OK/Fixed
|
||||
- [ ] Quality: OK/Flagged
|
||||
```
|
||||
@@ -1,44 +0,0 @@
|
||||
---
|
||||
description: Review current session for documentation updates before merging PR
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write
|
||||
---
|
||||
|
||||
# Pre-Merge Documentation Review
|
||||
|
||||
You are reviewing this chat session to identify documentation updates needed before merging the current PR.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. **Analyze the current session** - What work was done? What was learned? What changed?
|
||||
|
||||
2. **Check if docs need updates:**
|
||||
- Does `docs/README.md` status section need updating?
|
||||
- Should anything be added to `docs/LESSONS-LEARNED.md`?
|
||||
- Does `docs/ARCHITECTURE.md` need changes (if architecture changed)?
|
||||
- Does `docs/DEVELOPMENT.md` need changes (if dev workflow changed)?
|
||||
- Does `docs/API.md` need changes (if API changed)?
|
||||
|
||||
3. **For significant completed work:**
|
||||
- Create `docs/archive/YYYY-MM-DD-topic.md` summarizing the work
|
||||
- Use today's date and a descriptive topic name
|
||||
|
||||
4. **Update docs/README.md** with:
|
||||
- Current status if it changed
|
||||
- Any new known issues discovered
|
||||
- Link to new archive doc if created
|
||||
|
||||
## Guidelines
|
||||
|
||||
- **IMPORTANT**: Make all documentation changes on the current feature branch (git worktree), NOT on main. Check `git branch` or the worktree path to ensure you're editing files in the correct location.
|
||||
- Keep updates concise and factual
|
||||
- Focus on information useful for future AI agents
|
||||
- Don't duplicate information that's in the code
|
||||
- Archive docs should capture "what was done and why" not implementation details
|
||||
|
||||
## Output
|
||||
|
||||
After reviewing, either:
|
||||
- Make the necessary documentation updates
|
||||
- Report "No documentation updates needed" with brief explanation why
|
||||
|
||||
Remember: Good documentation helps the next agent continue where you left off.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Plan: Migrate from MailSlurp to testmail.app
|
||||
|
||||
**Created:** 2026-01-02
|
||||
**Branch:** feat/sess-20260102-2109-migrate-mailslurp-testmail
|
||||
**Status:** Complete
|
||||
|
||||
## Overview
|
||||
|
||||
Replace MailSlurp email testing service with testmail.app for E2E testing of magic link authentication. testmail.app uses a simpler API with namespace-based email addresses that require no inbox creation/deletion - emails just arrive at `{namespace}.{tag}@inbox.testmail.app`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Remove all code referencing MailSlurp
|
||||
- [x] Remove `mailslurp-client` npm dependency
|
||||
- [x] Implement testmail.app API integration
|
||||
- [x] Update environment variable names (`TESTMAIL_API_KEY`, `TESTMAIL_NAMESPACE`)
|
||||
- [x] Maintain same `EmailHelper` interface for minimal test changes
|
||||
- [x] Update archive docs to note migration (optional - historical accuracy)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### API Differences
|
||||
|
||||
| Feature | MailSlurp | testmail.app |
|
||||
|---------|-----------|--------------|
|
||||
| Email format | Random `xxx@mailslurp.com` | `{namespace}.{tag}@inbox.testmail.app` |
|
||||
| Inbox creation | Required (`createInbox()`) | Not needed - use any tag |
|
||||
| Inbox deletion | Required (`deleteInbox()`) | Not needed |
|
||||
| Get emails | `waitForLatestEmail(inboxId)` | GET `/api/json?apikey=X&namespace=Y&tag=Z&livequery=true` |
|
||||
| Auth | API key in constructor | API key as query param |
|
||||
| Email body | `email.body` | `email.html` or `email.text` |
|
||||
|
||||
### Implementation Strategy
|
||||
|
||||
1. **Keep same `EmailHelper` interface** - `createInbox()`, `waitForEmail()`, `extractMagicLink()`, `deleteInbox()`
|
||||
2. **Use testmail.app's live query feature** - `livequery=true` waits for email to arrive (up to timeout)
|
||||
3. **Generate unique tags** - Use timestamp + random suffix for each test run to avoid collisions
|
||||
4. **Inbox becomes a simple object** - `{ id: tag, emailAddress: '{namespace}.{tag}@inbox.testmail.app' }`
|
||||
5. **Delete becomes no-op** - testmail.app emails auto-expire, no cleanup needed
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Old | New |
|
||||
|-----|-----|
|
||||
| `MAILSLURP_API_KEY` | `TESTMAIL_API_KEY` |
|
||||
| (none) | `TESTMAIL_NAMESPACE` |
|
||||
|
||||
Values: Set via environment variables (see `frontend/.env.local.example`)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] **Update `email-testing.ts`** - Replace MailSlurp implementation with testmail.app
|
||||
- Remove `mailslurp-client` import
|
||||
- Add testmail.app API fetch implementation
|
||||
- Use native `fetch` (no new dependencies)
|
||||
- Generate unique tag for each "inbox"
|
||||
- Implement `livequery=true` for waiting
|
||||
- Adapt email body extraction (`html`/`text` instead of `body`)
|
||||
|
||||
2. [x] **Update `full-user-journey.spec.ts`** - Change references
|
||||
- Update doc comments (env var name)
|
||||
- Update error messages
|
||||
- Update console logs
|
||||
|
||||
3. [x] **Update `playwright.record.config.ts`** - Change env var comment
|
||||
|
||||
4. [x] **Update `package.json`** - Remove `mailslurp-client` dependency
|
||||
|
||||
5. [x] **Run `npm install`** - Update package-lock.json
|
||||
|
||||
6. [x] **Create `.env.local.example`** - Document required env vars for local testing
|
||||
|
||||
7. [x] **Update archive docs** - Skipped intentionally; historical docs preserved for accuracy
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/e2e/helpers/email-testing.ts` | Modify | Replace MailSlurp with testmail.app |
|
||||
| `frontend/e2e/production/full-user-journey.spec.ts` | Modify | Update env var refs and messages |
|
||||
| `frontend/playwright.record.config.ts` | Modify | Update env var comment |
|
||||
| `frontend/package.json` | Modify | Remove `mailslurp-client` |
|
||||
| `frontend/package-lock.json` | Modify | Auto-updated by npm install |
|
||||
| `frontend/.env.local.example` | Create | Document env vars for local testing |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Manual testing**: Run production E2E test with testmail.app credentials
|
||||
```bash
|
||||
cd frontend
|
||||
TESTMAIL_API_KEY=your-api-key \
|
||||
TESTMAIL_NAMESPACE=your-namespace \
|
||||
npm run test:e2e:prod:headed
|
||||
```
|
||||
- **Verify**: Email received, magic link extracted, auth flow completes
|
||||
- **CI testing**: Env vars already configured in GitHub Actions org secrets
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - requirements are clear.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Revert the commit
|
||||
2. Run `npm install` to restore mailslurp-client
|
||||
3. Re-add MAILSLURP_API_KEY to secrets if removed
|
||||
+215
-3
@@ -4,6 +4,7 @@ on:
|
||||
push:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
types: [opened, synchronize, reopened, labeled]
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
@@ -143,6 +144,23 @@ jobs:
|
||||
needs.changes.outputs.frontend == 'true' &&
|
||||
contains(github.event.pull_request.labels.*.name, 'e2e')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js with caching
|
||||
@@ -172,6 +190,64 @@ jobs:
|
||||
path: frontend/playwright-report/
|
||||
retention-days: 7
|
||||
|
||||
# Happy Path Production Test - runs the comprehensive real-user test against prod
|
||||
frontend-e2e-happy-path:
|
||||
name: "Frontend - E2E Happy Path (Production)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
# Only runs when PR has 'e2e-happy-path' label
|
||||
if: >-
|
||||
github.ref != 'refs/heads/main' &&
|
||||
contains(github.event.pull_request.labels.*.name, 'e2e-happy-path')
|
||||
timeout-minutes: 90
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps chromium
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Run Happy Path Production Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ secrets.E2E_TEST_TOKEN }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test against Production"
|
||||
echo "Time: $(date)"
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=chromium \
|
||||
--reporter=list \
|
||||
--timeout=3600000
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
retention-days: 30
|
||||
|
||||
frontend-build:
|
||||
name: "Frontend - Build"
|
||||
runs-on: ubuntu-latest
|
||||
@@ -207,6 +283,23 @@ jobs:
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
@@ -265,6 +358,23 @@ jobs:
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
@@ -318,6 +428,23 @@ jobs:
|
||||
(needs.package-integration-tests.result == 'success' || needs.package-integration-tests.result == 'skipped') &&
|
||||
(needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
@@ -351,6 +478,23 @@ jobs:
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -388,6 +532,23 @@ jobs:
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -435,6 +596,23 @@ jobs:
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -510,6 +688,23 @@ jobs:
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
@@ -720,6 +915,23 @@ jobs:
|
||||
BASE_IMAGE_NAME: karaoke-backend-base
|
||||
APP_IMAGE_NAME: karaoke-backend
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -872,9 +1084,9 @@ jobs:
|
||||
--timeout 1800 \
|
||||
--concurrency 1 \
|
||||
--max-instances 50 \
|
||||
--min-instances 0 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest"
|
||||
--min-instances 4 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest"
|
||||
|
||||
- name: Verify deployment and version
|
||||
run: |
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
name: E2E Happy Path Test
|
||||
|
||||
# Comprehensive end-to-end test that validates the full karaoke generation journey
|
||||
# using ONLY browser interactions (no API shortcuts)
|
||||
#
|
||||
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
|
||||
|
||||
on:
|
||||
# Daily run at 6 AM UTC
|
||||
schedule:
|
||||
- cron: '0 6 * * *'
|
||||
|
||||
# Manual trigger with optional parameters
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
debug_mode:
|
||||
description: 'Enable debug mode (slower, more screenshots)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
browser:
|
||||
description: 'Browser to test with'
|
||||
required: false
|
||||
default: 'chromium'
|
||||
type: choice
|
||||
options:
|
||||
- chromium
|
||||
- firefox
|
||||
- webkit
|
||||
use_test_token:
|
||||
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
e2e-happy-path:
|
||||
name: "Full Happy Path (Real User)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: |
|
||||
BROWSER=${{ inputs.browser || 'chromium' }}
|
||||
npx playwright install --with-deps $BROWSER
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Run E2E Happy Path Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test"
|
||||
echo "Browser: ${{ inputs.browser || 'chromium' }}"
|
||||
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
|
||||
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
|
||||
echo "Time: $(date)"
|
||||
echo ""
|
||||
|
||||
# Run the real user happy path test
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
echo "Test PASSED"
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
echo "Test FAILED - see artifacts for details"
|
||||
fi
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload video recordings
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-videos-${{ github.run_number }}
|
||||
path: frontend/test-results/**/*.webm
|
||||
retention-days: 30
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Create summary
|
||||
if: always()
|
||||
run: |
|
||||
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
if [ -f frontend/test-output.log ]; then
|
||||
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
|
||||
- name: Send email notification
|
||||
if: always() && github.event_name == 'schedule'
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.sendgrid.net
|
||||
server_port: 587
|
||||
username: apikey
|
||||
password: ${{ secrets.SENDGRID_API_KEY }}
|
||||
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
E2E Happy Path Test Results
|
||||
===========================
|
||||
|
||||
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
|
||||
Run: #${{ github.run_number }}
|
||||
Trigger: ${{ github.event_name }}
|
||||
Time: ${{ github.event.repository.updated_at }}
|
||||
|
||||
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
|
||||
This is an automated notification from the daily E2E happy path test.
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
@@ -29,11 +29,8 @@
|
||||
## Essential Rules
|
||||
|
||||
### Git Workflow
|
||||
- **Never commit directly to `main`** - always use feature branches
|
||||
- **Use git worktrees** for all changes:
|
||||
```bash
|
||||
git worktree add -b feature/xyz ../karaoke-gen-feat-xyz main
|
||||
```
|
||||
- **Never commit directly to `main`** - use `/new-worktree <description>` to start
|
||||
- **Follow global workflow** - see `~/.claude/CLAUDE.md` for command sequence
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
@@ -90,9 +87,12 @@ Location: `/Users/andrew/Projects/flacfetch`
|
||||
|
||||
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
|
||||
|
||||
## Pre-Commit Checklist
|
||||
## Pre-PR Checklist
|
||||
|
||||
- [ ] Tests pass (`make test` / `npm run test:all`)
|
||||
Follow global workflow (`~/.claude/CLAUDE.md`):
|
||||
|
||||
- [ ] Tests pass: `/test` (or `make test` / `npm run test:all`)
|
||||
- [ ] Version bumped (if code changed)
|
||||
- [ ] `/docs-review` run (check for doc updates)
|
||||
- [ ] PR created with clear description
|
||||
- [ ] Docs checked: `/docs-review`
|
||||
- [ ] Code reviewed: `/review` (CodeRabbit CLI)
|
||||
- [ ] PR created: `/pr` (adds @coderabbitai ignore)
|
||||
+13
-1
@@ -10,14 +10,26 @@ FROM python:3.11-slim
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
build-essential \
|
||||
curl \
|
||||
xz-utils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install optimized static FFmpeg build (7.0.2) instead of Debian package (5.1.8)
|
||||
# John Van Sickle's builds are compiled with better optimizations for x86_64
|
||||
# This provides ~20-40% performance improvement for encoding tasks
|
||||
# Source: https://johnvansickle.com/ffmpeg/
|
||||
RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz && \
|
||||
tar -xf /tmp/ffmpeg.tar.xz -C /tmp && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/ && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/ && \
|
||||
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe && \
|
||||
rm -rf /tmp/ffmpeg* && \
|
||||
ffmpeg -version
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
"""
|
||||
Admin API routes for dashboard statistics and management.
|
||||
|
||||
Handles:
|
||||
- Dashboard overview statistics
|
||||
- System-wide metrics
|
||||
- Admin-only operations
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Tuple
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.models.job import JobStatus
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/admin", tags=["admin"])
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Response Models
|
||||
# =============================================================================
|
||||
|
||||
class JobsByStatusResponse(BaseModel):
|
||||
"""Breakdown of jobs by status."""
|
||||
pending: int = 0
|
||||
processing: int = 0
|
||||
awaiting_review: int = 0
|
||||
awaiting_instrumental: int = 0
|
||||
complete: int = 0
|
||||
failed: int = 0
|
||||
cancelled: int = 0
|
||||
|
||||
|
||||
class AdminStatsOverview(BaseModel):
|
||||
"""Overview statistics for admin dashboard."""
|
||||
total_users: int
|
||||
active_users_7d: int
|
||||
active_users_30d: int
|
||||
total_jobs: int
|
||||
jobs_last_7d: int
|
||||
jobs_last_30d: int
|
||||
jobs_by_status: JobsByStatusResponse
|
||||
total_credits_issued_30d: int
|
||||
total_beta_testers: int
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Admin Stats Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/stats/overview", response_model=AdminStatsOverview)
|
||||
async def get_admin_stats_overview(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
Get overview statistics for admin dashboard.
|
||||
|
||||
Includes:
|
||||
- User counts (total, active in 7d, active in 30d)
|
||||
- Job counts (total, by status, recent)
|
||||
- Credit statistics
|
||||
- Beta program stats
|
||||
"""
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from google.cloud.firestore_v1 import aggregation
|
||||
|
||||
db = user_service.db
|
||||
now = datetime.utcnow()
|
||||
seven_days_ago = now - timedelta(days=7)
|
||||
thirty_days_ago = now - timedelta(days=30)
|
||||
|
||||
# Helper function to get count using aggregation
|
||||
def get_count(query) -> int:
|
||||
try:
|
||||
agg_query = aggregation.AggregationQuery(query)
|
||||
agg_query.count(alias="count")
|
||||
results = agg_query.get()
|
||||
return results[0][0].value if results else 0
|
||||
except Exception as e:
|
||||
logger.warning(f"Aggregation query failed: {e}")
|
||||
return 0
|
||||
|
||||
# User statistics
|
||||
users_collection = db.collection(USERS_COLLECTION)
|
||||
|
||||
total_users = get_count(users_collection)
|
||||
|
||||
active_users_7d = get_count(
|
||||
users_collection.where(filter=FieldFilter("last_login_at", ">=", seven_days_ago))
|
||||
)
|
||||
|
||||
active_users_30d = get_count(
|
||||
users_collection.where(filter=FieldFilter("last_login_at", ">=", thirty_days_ago))
|
||||
)
|
||||
|
||||
total_beta_testers = get_count(
|
||||
users_collection.where(filter=FieldFilter("is_beta_tester", "==", True))
|
||||
)
|
||||
|
||||
# Job statistics
|
||||
jobs_collection = db.collection("jobs")
|
||||
|
||||
total_jobs = get_count(jobs_collection)
|
||||
|
||||
jobs_last_7d = get_count(
|
||||
jobs_collection.where(filter=FieldFilter("created_at", ">=", seven_days_ago))
|
||||
)
|
||||
|
||||
jobs_last_30d = get_count(
|
||||
jobs_collection.where(filter=FieldFilter("created_at", ">=", thirty_days_ago))
|
||||
)
|
||||
|
||||
# Jobs by status - map multiple statuses to simplified categories
|
||||
processing_statuses = [
|
||||
"downloading", "downloading_audio", "searching_audio", "awaiting_audio_selection",
|
||||
"separating_stage1", "separating_stage2", "transcribing", "correcting",
|
||||
"generating_screens", "applying_padding", "rendering_video",
|
||||
"instrumental_selected", "generating_video", "encoding", "packaging",
|
||||
"uploading", "notifying"
|
||||
]
|
||||
|
||||
jobs_by_status = JobsByStatusResponse(
|
||||
pending=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "pending"))
|
||||
),
|
||||
processing=sum(
|
||||
get_count(jobs_collection.where(filter=FieldFilter("status", "==", status)))
|
||||
for status in processing_statuses
|
||||
),
|
||||
awaiting_review=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "awaiting_review"))
|
||||
) + get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "in_review"))
|
||||
),
|
||||
awaiting_instrumental=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "awaiting_instrumental_selection"))
|
||||
),
|
||||
complete=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "complete"))
|
||||
) + get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "prep_complete"))
|
||||
),
|
||||
failed=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "failed"))
|
||||
),
|
||||
cancelled=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "cancelled"))
|
||||
),
|
||||
)
|
||||
|
||||
# Credit statistics - sum credits added in last 30 days
|
||||
# This is more expensive, so we'll just estimate from users
|
||||
total_credits_issued_30d = 0
|
||||
try:
|
||||
# Get all users and sum recent credit transactions
|
||||
users_docs = users_collection.limit(500).stream()
|
||||
for user_doc in users_docs:
|
||||
user_data = user_doc.to_dict()
|
||||
transactions = user_data.get("credit_transactions", [])
|
||||
for txn in transactions:
|
||||
txn_date = txn.get("created_at")
|
||||
if txn_date:
|
||||
# Handle both datetime and string formats
|
||||
if isinstance(txn_date, str):
|
||||
try:
|
||||
txn_date = datetime.fromisoformat(txn_date.replace("Z", "+00:00"))
|
||||
except Exception:
|
||||
continue
|
||||
if isinstance(txn_date, datetime):
|
||||
txn_date = txn_date.replace(tzinfo=None)
|
||||
else:
|
||||
continue
|
||||
if txn_date >= thirty_days_ago:
|
||||
amount = txn.get("amount", 0)
|
||||
if amount > 0: # Only count additions, not deductions
|
||||
total_credits_issued_30d += amount
|
||||
except Exception as e:
|
||||
logger.warning(f"Error calculating credits: {e}")
|
||||
|
||||
return AdminStatsOverview(
|
||||
total_users=total_users,
|
||||
active_users_7d=active_users_7d,
|
||||
active_users_30d=active_users_30d,
|
||||
total_jobs=total_jobs,
|
||||
jobs_last_7d=jobs_last_7d,
|
||||
jobs_last_30d=jobs_last_30d,
|
||||
jobs_by_status=jobs_by_status,
|
||||
total_credits_issued_30d=total_credits_issued_30d,
|
||||
total_beta_testers=total_beta_testers,
|
||||
)
|
||||
@@ -35,7 +35,7 @@ from backend.services.audio_search_service import (
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -117,6 +117,9 @@ class AudioSearchRequest(BaseModel):
|
||||
# Style params JSON and related assets (background images, fonts)
|
||||
style_files: Optional[List[StyleFileRequest]] = Field(None, description="Style files to upload (style_params JSON, backgrounds, fonts)")
|
||||
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
|
||||
class AudioSearchResultResponse(BaseModel):
|
||||
"""A single audio search result.
|
||||
@@ -372,8 +375,31 @@ async def _download_and_start_processing(
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
)
|
||||
elif source_name == 'YouTube' and download_url:
|
||||
# For YouTube without remote, download directly using URL from state_data
|
||||
# This avoids relying on in-memory cache which doesn't persist across instances
|
||||
logger.info(f"Downloading YouTube audio directly from URL: {download_url}")
|
||||
from backend.workers.audio_worker import download_from_url
|
||||
|
||||
local_path = await download_from_url(
|
||||
download_url,
|
||||
temp_dir,
|
||||
selected.get('artist'),
|
||||
selected.get('title')
|
||||
)
|
||||
|
||||
if not local_path or not os.path.exists(local_path):
|
||||
raise DownloadError(f"Failed to download from YouTube: {download_url}")
|
||||
|
||||
# Create a result-like object
|
||||
class DownloadResult:
|
||||
def __init__(self, filepath):
|
||||
self.filepath = filepath
|
||||
|
||||
result = DownloadResult(local_path)
|
||||
else:
|
||||
# Fallback to search-based download
|
||||
# Fallback to search-based download (may fail if cache is empty)
|
||||
logger.warning(f"Falling back to cache-based download for {source_name} - this may fail on multi-instance deployments")
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir=temp_dir,
|
||||
@@ -436,7 +462,7 @@ async def search_audio(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSearchRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Search for audio by artist and title, creating a new job.
|
||||
@@ -500,6 +526,9 @@ async def search_audio(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Use authenticated user's email
|
||||
effective_user_email = auth_result.user_email
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
@@ -515,6 +544,7 @@ async def search_audio(
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
user_email=effective_user_email,
|
||||
lyrics_artist=body.lyrics_artist,
|
||||
lyrics_title=body.lyrics_title,
|
||||
subtitle_offset_ms=body.subtitle_offset_ms,
|
||||
@@ -525,10 +555,11 @@ async def search_audio(
|
||||
audio_search_title=body.title,
|
||||
auto_download=body.auto_download,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
|
||||
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
|
||||
|
||||
# Update job with audio search fields
|
||||
@@ -766,7 +797,7 @@ async def search_audio(
|
||||
@router.get("/audio-search/{job_id}/results")
|
||||
async def get_audio_search_results(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Get audio search results for a job.
|
||||
@@ -801,7 +832,7 @@ async def select_audio_source(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSelectRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Select an audio source and start job processing.
|
||||
|
||||
@@ -33,7 +33,7 @@ from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.services.metrics import metrics
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["jobs"])
|
||||
@@ -92,6 +92,9 @@ class CreateJobFromUrlRequest(BaseModel):
|
||||
backing_vocals_models: Optional[List[str]] = Field(None, description="Models for backing vocals separation")
|
||||
other_stems_models: Optional[List[str]] = Field(None, description="Models for other stems")
|
||||
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
|
||||
class CreateJobFromUrlResponse(BaseModel):
|
||||
"""Response from creating a job from URL."""
|
||||
@@ -150,6 +153,9 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
|
||||
prep_only: bool = Field(False, description="Stop after review phase, don't run finalisation")
|
||||
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code instead of generating new one")
|
||||
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
|
||||
class SignedUploadUrl(BaseModel):
|
||||
"""Signed URL for uploading a file."""
|
||||
@@ -247,6 +253,16 @@ async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
async def _trigger_audio_worker_only(job_id: str) -> None:
|
||||
"""
|
||||
Trigger only the audio worker.
|
||||
|
||||
Used for URL jobs where audio needs to be downloaded first.
|
||||
The audio worker will trigger the lyrics worker after download completes.
|
||||
"""
|
||||
await worker_service.trigger_audio_worker(job_id)
|
||||
|
||||
|
||||
def _resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool],
|
||||
@@ -368,7 +384,7 @@ def _prepare_theme_for_job(
|
||||
async def upload_and_create_job(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
# Required fields
|
||||
file: UploadFile = File(..., description="Audio file to process"),
|
||||
artist: str = Form(..., description="Artist name"),
|
||||
@@ -409,6 +425,8 @@ async def upload_and_create_job(
|
||||
clean_instrumental_model: Optional[str] = Form(None, description="Model for clean instrumental separation (e.g., model_bs_roformer_ep_317_sdr_12.9755.ckpt)"),
|
||||
backing_vocals_models: Optional[str] = Form(None, description="Comma-separated list of models for backing vocals separation"),
|
||||
other_stems_models: Optional[str] = Form(None, description="Comma-separated list of models for other stems (bass, drums, guitar, etc.)"),
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Form(False, description="Skip interactive steps (lyrics review, instrumental selection)"),
|
||||
):
|
||||
"""
|
||||
Upload an audio file and create a karaoke generation job with full style configuration.
|
||||
@@ -563,6 +581,9 @@ async def upload_and_create_job(
|
||||
if other_stems_models:
|
||||
parsed_other_stems_models = [m.strip() for m in other_stems_models.split(',') if m.strip()]
|
||||
|
||||
# Prefer authenticated user's email over form parameter
|
||||
effective_user_email = auth_result.user_email or user_email
|
||||
|
||||
# Create job first to get job_id
|
||||
job_create = JobCreate(
|
||||
artist=artist,
|
||||
@@ -577,7 +598,7 @@ async def upload_and_create_job(
|
||||
youtube_description=youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=webhook_url,
|
||||
user_email=user_email,
|
||||
user_email=effective_user_email,
|
||||
# Native API distribution (preferred for remote CLI)
|
||||
dropbox_path=dist.dropbox_path,
|
||||
gdrive_folder_id=dist.gdrive_folder_id,
|
||||
@@ -593,6 +614,8 @@ async def upload_and_create_job(
|
||||
other_stems_models=parsed_other_stems_models,
|
||||
# Request metadata for tracking and filtering
|
||||
request_metadata=request_metadata,
|
||||
# Non-interactive mode
|
||||
non_interactive=non_interactive,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
@@ -973,7 +996,7 @@ def _get_gcs_path_for_file(job_id: str, file_type: str, filename: str) -> str:
|
||||
async def create_job_with_upload_urls(
|
||||
request: Request,
|
||||
body: CreateJobWithUploadUrlsRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Create a karaoke generation job and return signed URLs for direct file upload to GCS.
|
||||
@@ -1069,6 +1092,9 @@ async def create_job_with_upload_urls(
|
||||
# Check if style_params is being uploaded (overrides theme)
|
||||
has_style_params_upload = any(f.file_type == 'style_params' for f in body.files)
|
||||
|
||||
# Prefer authenticated user's email over request body
|
||||
effective_user_email = auth_result.user_email or body.user_email
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
@@ -1084,7 +1110,7 @@ async def create_job_with_upload_urls(
|
||||
youtube_description_template=body.youtube_description, # video_worker reads this field
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=body.webhook_url,
|
||||
user_email=body.user_email,
|
||||
user_email=effective_user_email,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
gdrive_folder_id=dist.gdrive_folder_id,
|
||||
organised_dir_rclone_root=body.organised_dir_rclone_root,
|
||||
@@ -1095,6 +1121,7 @@ async def create_job_with_upload_urls(
|
||||
backing_vocals_models=body.backing_vocals_models,
|
||||
other_stems_models=body.other_stems_models,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
@@ -1160,7 +1187,7 @@ async def mark_uploads_complete(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: UploadsCompleteRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Mark file uploads as complete and start job processing.
|
||||
@@ -1406,7 +1433,7 @@ async def create_job_from_url(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: CreateJobFromUrlRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Create a karaoke generation job from a YouTube or other online video URL.
|
||||
@@ -1481,6 +1508,9 @@ async def create_job_from_url(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Prefer authenticated user's email over request body
|
||||
effective_user_email = auth_result.user_email or body.user_email
|
||||
|
||||
# Create job with URL
|
||||
job_create = JobCreate(
|
||||
url=body.url,
|
||||
@@ -1496,7 +1526,7 @@ async def create_job_from_url(
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=body.webhook_url,
|
||||
user_email=body.user_email,
|
||||
user_email=effective_user_email,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
gdrive_folder_id=dist.gdrive_folder_id,
|
||||
organised_dir_rclone_root=body.organised_dir_rclone_root,
|
||||
@@ -1507,6 +1537,7 @@ async def create_job_from_url(
|
||||
backing_vocals_models=body.backing_vocals_models,
|
||||
other_stems_models=body.other_stems_models,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
@@ -1542,10 +1573,11 @@ async def create_job_from_url(
|
||||
progress=5,
|
||||
message="Starting audio download from URL"
|
||||
)
|
||||
|
||||
# Trigger workers in parallel
|
||||
# The audio worker will handle downloading from URL
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
|
||||
# For URL jobs, trigger ONLY audio worker first
|
||||
# The audio worker will trigger lyrics worker after download completes
|
||||
# This prevents race condition where lyrics worker times out waiting for audio
|
||||
background_tasks.add_task(_trigger_audio_worker_only, job_id)
|
||||
|
||||
return CreateJobFromUrlResponse(
|
||||
status="success",
|
||||
@@ -1608,7 +1640,7 @@ def _get_gcs_path_for_finalise_file(job_id: str, file_type: str, filename: str)
|
||||
async def create_finalise_only_job(
|
||||
request: Request,
|
||||
body: CreateFinaliseOnlyJobRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Create a finalise-only job for continuing from a local prep phase.
|
||||
@@ -1721,6 +1753,9 @@ async def create_finalise_only_job(
|
||||
# Check if style_params is being uploaded (overrides theme)
|
||||
has_style_params_upload = any(f.file_type == 'style_params' for f in body.files)
|
||||
|
||||
# Use authenticated user's email
|
||||
effective_user_email = auth_result.user_email
|
||||
|
||||
# Create job with finalise_only=True
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
@@ -1736,6 +1771,7 @@ async def create_finalise_only_job(
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
gdrive_folder_id=dist.gdrive_folder_id,
|
||||
user_email=effective_user_email,
|
||||
finalise_only=True,
|
||||
keep_brand_code=body.keep_brand_code,
|
||||
request_metadata=request_metadata,
|
||||
@@ -1804,7 +1840,7 @@ async def mark_finalise_uploads_complete(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: UploadsCompleteRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Mark finalise-only file uploads as complete and start video generation.
|
||||
|
||||
@@ -3,6 +3,8 @@ Health check routes.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
import subprocess
|
||||
import platform
|
||||
from fastapi import APIRouter
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
@@ -10,11 +12,114 @@ from backend.version import VERSION
|
||||
from backend.services.flacfetch_client import get_flacfetch_client
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_system_info() -> Dict[str, Any]:
|
||||
"""Get system information for performance analysis."""
|
||||
info = {
|
||||
"platform": platform.system(),
|
||||
"platform_release": platform.release(),
|
||||
"python_version": platform.python_version(),
|
||||
}
|
||||
|
||||
# Get CPU info
|
||||
try:
|
||||
cpu_count = os.cpu_count() or 0
|
||||
info["cpu_count"] = cpu_count
|
||||
|
||||
# Try to get more detailed CPU info on Linux
|
||||
if platform.system() == "Linux":
|
||||
try:
|
||||
with open("/proc/cpuinfo", "r") as f:
|
||||
for line in f:
|
||||
if "model name" in line:
|
||||
info["cpu_model"] = line.split(":")[1].strip()
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
info["cpu_error"] = str(e)
|
||||
|
||||
# Get memory info
|
||||
try:
|
||||
if platform.system() == "Linux":
|
||||
with open("/proc/meminfo", "r") as f:
|
||||
for line in f:
|
||||
if line.startswith("MemTotal:"):
|
||||
mem_kb = int(line.split()[1])
|
||||
info["memory_gb"] = round(mem_kb / (1024 * 1024), 1)
|
||||
break
|
||||
except Exception as e:
|
||||
info["memory_error"] = str(e)
|
||||
|
||||
return info
|
||||
|
||||
|
||||
def get_ffmpeg_info() -> Dict[str, Any]:
|
||||
"""Get FFmpeg version and configuration for debugging encoding performance."""
|
||||
try:
|
||||
# Get FFmpeg version
|
||||
result = subprocess.run(
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if result.returncode == 0:
|
||||
lines = result.stdout.split("\n")
|
||||
version_line = lines[0] if lines else "unknown"
|
||||
|
||||
# Check for hardware acceleration support
|
||||
hw_accel = {}
|
||||
config_result = subprocess.run(
|
||||
["ffmpeg", "-hwaccels"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if config_result.returncode == 0:
|
||||
hw_lines = config_result.stdout.strip().split("\n")
|
||||
hw_accel["available"] = [h.strip() for h in hw_lines[1:] if h.strip()]
|
||||
|
||||
# Check encoder availability
|
||||
encoders = {}
|
||||
encoder_result = subprocess.run(
|
||||
["ffmpeg", "-encoders"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if encoder_result.returncode == 0:
|
||||
encoder_output = encoder_result.stdout
|
||||
# Check for specific encoders we care about
|
||||
encoders["libx264"] = "libx264" in encoder_output
|
||||
encoders["h264_nvenc"] = "h264_nvenc" in encoder_output
|
||||
encoders["h264_vaapi"] = "h264_vaapi" in encoder_output
|
||||
encoders["hevc_nvenc"] = "hevc_nvenc" in encoder_output
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"version": version_line,
|
||||
"hw_acceleration": hw_accel,
|
||||
"encoders": encoders,
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"available": False,
|
||||
"error": result.stderr[:500] if result.stderr else "Unknown error",
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"available": False, "error": "Timeout getting FFmpeg info"}
|
||||
except FileNotFoundError:
|
||||
return {"available": False, "error": "FFmpeg not found"}
|
||||
except Exception as e:
|
||||
return {"available": False, "error": str(e)}
|
||||
|
||||
|
||||
def check_transmission_status() -> Dict[str, Any]:
|
||||
"""Check if Transmission daemon is available and responsive."""
|
||||
try:
|
||||
@@ -101,6 +206,37 @@ async def check_flacfetch_service_status() -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
"""Check if GCE encoding worker is available and healthy."""
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
if not encoding_service.is_configured:
|
||||
return {
|
||||
"configured": False,
|
||||
"enabled": encoding_service.settings.use_gce_encoding,
|
||||
"message": "GCE encoding worker not configured (ENCODING_WORKER_URL or API key not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await encoding_service.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": health.get("status") == "ok",
|
||||
"status": health.get("status"),
|
||||
"active_jobs": health.get("active_jobs", 0),
|
||||
"queue_length": health.get("queue_length", 0),
|
||||
"ffmpeg_version": health.get("ffmpeg_version"),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
@@ -110,6 +246,7 @@ async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
transmission_status = check_transmission_status()
|
||||
flacfetch_status = await check_flacfetch_service_status()
|
||||
encoding_status = await check_encoding_worker_status()
|
||||
|
||||
# Check email service
|
||||
email_service = get_email_service()
|
||||
@@ -141,13 +278,20 @@ async def detailed_health_check() -> Dict[str, Any]:
|
||||
if not stripe_status["configured"]:
|
||||
logger.warning("Stripe service not configured - payments will not work")
|
||||
|
||||
# Get system and FFmpeg info for performance analysis
|
||||
system_info = get_system_info()
|
||||
ffmpeg_info = get_ffmpeg_info()
|
||||
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"system": system_info,
|
||||
"ffmpeg": ffmpeg_info,
|
||||
"dependencies": {
|
||||
"transmission_local": transmission_status,
|
||||
"flacfetch_remote": flacfetch_status,
|
||||
"encoding_worker": encoding_status,
|
||||
},
|
||||
"services": {
|
||||
"email": email_status,
|
||||
|
||||
+162
-2
@@ -1082,6 +1082,18 @@ async def get_download_urls(
|
||||
}
|
||||
|
||||
|
||||
# Map file keys to human-readable suffixes (matching Dropbox naming from karaoke_finalise.py)
|
||||
DOWNLOAD_FILENAME_SUFFIXES = {
|
||||
"lossless_4k_mp4": " (Final Karaoke Lossless 4k).mp4",
|
||||
"lossless_4k_mkv": " (Final Karaoke Lossless 4k).mkv",
|
||||
"lossy_4k_mp4": " (Final Karaoke Lossy 4k).mp4",
|
||||
"lossy_720p_mp4": " (Final Karaoke Lossy 720p).mp4",
|
||||
"cdg_zip": " (Final Karaoke CDG).zip",
|
||||
"txt_zip": " (Final Karaoke TXT).zip",
|
||||
"with_vocals": " (With Vocals).mkv",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{job_id}/download/{category}/{file_key}")
|
||||
async def download_file(
|
||||
job_id: str,
|
||||
@@ -1146,8 +1158,28 @@ async def download_file(
|
||||
}
|
||||
content_type = content_types.get(ext, 'application/octet-stream')
|
||||
|
||||
# Get filename for Content-Disposition
|
||||
filename = gcs_path.split('/')[-1]
|
||||
# Build proper filename: "Artist - Title (Final Karaoke Lossy 4k).mp4"
|
||||
# Sanitize artist/title to remove filesystem-unsafe characters
|
||||
def sanitize_filename_part(s: str) -> str:
|
||||
if not s:
|
||||
return ""
|
||||
# Remove/replace characters unsafe for filesystems: / \ : * ? " < > |
|
||||
unsafe_chars = r'[/\\:*?"<>|]'
|
||||
import re
|
||||
sanitized = re.sub(unsafe_chars, '', s)
|
||||
# Collapse multiple spaces and trim
|
||||
sanitized = ' '.join(sanitized.split())
|
||||
# Limit length to prevent overly long filenames
|
||||
return sanitized[:100].strip()
|
||||
|
||||
artist_clean = sanitize_filename_part(job.artist) if job.artist else None
|
||||
title_clean = sanitize_filename_part(job.title) if job.title else None
|
||||
base_name = f"{artist_clean} - {title_clean}" if artist_clean and title_clean else None
|
||||
|
||||
if base_name and file_key in DOWNLOAD_FILENAME_SUFFIXES:
|
||||
filename = f"{base_name}{DOWNLOAD_FILENAME_SUFFIXES[file_key]}"
|
||||
else:
|
||||
filename = gcs_path.split('/')[-1] # Fallback to original
|
||||
|
||||
try:
|
||||
# Download to temp file and stream
|
||||
@@ -1446,3 +1478,131 @@ async def get_worker_logs(
|
||||
"total_logs": total
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{job_id}/cleanup-distribution")
|
||||
async def cleanup_distribution(
|
||||
job_id: str,
|
||||
delete_job: bool = True,
|
||||
auth_result: AuthResult = Depends(require_admin)
|
||||
) -> dict:
|
||||
"""
|
||||
Clean up all distributed content for a job (YouTube, Dropbox, Google Drive).
|
||||
|
||||
This admin-only endpoint is designed for E2E test cleanup. It:
|
||||
1. Deletes YouTube video (if uploaded)
|
||||
2. Deletes Dropbox folder (if uploaded)
|
||||
3. Deletes Google Drive files (if uploaded)
|
||||
4. Optionally deletes the job itself
|
||||
|
||||
Args:
|
||||
job_id: Job ID to clean up
|
||||
delete_job: If True, also delete the job after cleaning up distribution
|
||||
|
||||
Returns:
|
||||
Cleanup results for each service
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
state_data = job.state_data or {}
|
||||
results = {
|
||||
"job_id": job_id,
|
||||
"youtube": {"status": "skipped", "reason": "no youtube_url in state_data"},
|
||||
"dropbox": {"status": "skipped", "reason": "no brand_code or dropbox_path"},
|
||||
"gdrive": {"status": "skipped", "reason": "no gdrive_files in state_data"},
|
||||
"job_deleted": False
|
||||
}
|
||||
|
||||
# Clean up YouTube
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
if youtube_url:
|
||||
try:
|
||||
# Extract video ID from URL (format: https://youtu.be/VIDEO_ID or https://www.youtube.com/watch?v=VIDEO_ID)
|
||||
import re
|
||||
video_id_match = re.search(r'(?:youtu\.be/|youtube\.com/watch\?v=)([^&\s]+)', youtube_url)
|
||||
if video_id_match:
|
||||
video_id = video_id_match.group(1)
|
||||
|
||||
# Import and use karaoke_finalise for YouTube deletion
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
from backend.services.youtube_service import get_youtube_service
|
||||
|
||||
youtube_service = get_youtube_service()
|
||||
if youtube_service.is_configured:
|
||||
# Create minimal KaraokeFinalise instance for deletion
|
||||
finalise = KaraokeFinalise(
|
||||
dry_run=False,
|
||||
non_interactive=True,
|
||||
enable_youtube=True,
|
||||
user_youtube_credentials=youtube_service.get_credentials_dict()
|
||||
)
|
||||
success = finalise.delete_youtube_video(video_id)
|
||||
results["youtube"] = {
|
||||
"status": "success" if success else "failed",
|
||||
"video_id": video_id
|
||||
}
|
||||
else:
|
||||
results["youtube"] = {"status": "failed", "reason": "YouTube credentials not configured"}
|
||||
else:
|
||||
results["youtube"] = {"status": "failed", "reason": f"Could not extract video ID from {youtube_url}"}
|
||||
except Exception as e:
|
||||
logger.error(f"Error cleaning up YouTube for job {job_id}: {e}", exc_info=True)
|
||||
results["youtube"] = {"status": "error", "error": str(e)}
|
||||
|
||||
# Clean up Dropbox
|
||||
brand_code = state_data.get('brand_code')
|
||||
dropbox_path = getattr(job, 'dropbox_path', None)
|
||||
if brand_code and dropbox_path:
|
||||
try:
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
dropbox = get_dropbox_service()
|
||||
if dropbox.is_configured:
|
||||
base_name = f"{job.artist} - {job.title}"
|
||||
folder_name = f"{brand_code} - {base_name}"
|
||||
full_path = f"{dropbox_path}/{folder_name}"
|
||||
success = dropbox.delete_folder(full_path)
|
||||
results["dropbox"] = {
|
||||
"status": "success" if success else "failed",
|
||||
"path": full_path
|
||||
}
|
||||
else:
|
||||
results["dropbox"] = {"status": "failed", "reason": "Dropbox credentials not configured"}
|
||||
except Exception as e:
|
||||
logger.error(f"Error cleaning up Dropbox for job {job_id}: {e}", exc_info=True)
|
||||
results["dropbox"] = {"status": "error", "error": str(e)}
|
||||
|
||||
# Clean up Google Drive
|
||||
gdrive_files = state_data.get('gdrive_files')
|
||||
if gdrive_files:
|
||||
try:
|
||||
from backend.services.gdrive_service import get_gdrive_service
|
||||
gdrive = get_gdrive_service()
|
||||
if gdrive.is_configured:
|
||||
# gdrive_files is a dict like {"mp4": "file_id", "mp4_720p": "file_id", "cdg": "file_id"}
|
||||
file_ids = list(gdrive_files.values()) if isinstance(gdrive_files, dict) else []
|
||||
delete_results = gdrive.delete_files(file_ids)
|
||||
all_success = all(delete_results.values())
|
||||
results["gdrive"] = {
|
||||
"status": "success" if all_success else "partial",
|
||||
"files": delete_results
|
||||
}
|
||||
else:
|
||||
results["gdrive"] = {"status": "failed", "reason": "Google Drive credentials not configured"}
|
||||
except Exception as e:
|
||||
logger.error(f"Error cleaning up Google Drive for job {job_id}: {e}", exc_info=True)
|
||||
results["gdrive"] = {"status": "error", "error": str(e)}
|
||||
|
||||
# Delete the job if requested
|
||||
if delete_job:
|
||||
try:
|
||||
job_manager.delete_job(job_id, delete_files=True)
|
||||
results["job_deleted"] = True
|
||||
logger.info(f"Deleted job {job_id} after distribution cleanup")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting job {job_id}: {e}", exc_info=True)
|
||||
results["job_deleted"] = False
|
||||
results["job_delete_error"] = str(e)
|
||||
|
||||
return results
|
||||
|
||||
+199
-20
@@ -29,7 +29,7 @@ from backend.models.user import (
|
||||
BetaFeedbackResponse,
|
||||
BetaTesterFeedback,
|
||||
)
|
||||
from backend.services.user_service import get_user_service, UserService
|
||||
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
|
||||
from backend.services.email_service import get_email_service, EmailService
|
||||
from backend.services.stripe_service import get_stripe_service, StripeService, CREDIT_PACKAGES
|
||||
from backend.api.dependencies import require_admin
|
||||
@@ -228,15 +228,34 @@ async def get_current_user(
|
||||
"""
|
||||
Get the current user's profile.
|
||||
|
||||
Requires a valid session token.
|
||||
Requires a valid session token or admin token.
|
||||
"""
|
||||
from backend.services.auth_service import get_auth_service
|
||||
|
||||
if not authorization:
|
||||
raise HTTPException(status_code=401, detail="Not authenticated")
|
||||
|
||||
# Extract token
|
||||
token = authorization.replace("Bearer ", "") if authorization.startswith("Bearer ") else authorization
|
||||
|
||||
# Validate session
|
||||
# First try auth service (handles admin tokens and auth_tokens)
|
||||
auth_service = get_auth_service()
|
||||
auth_result = auth_service.validate_token_full(token)
|
||||
|
||||
if auth_result.is_valid and auth_result.user_email:
|
||||
# Get or create user record for the authenticated email
|
||||
user = user_service.get_or_create_user(auth_result.user_email)
|
||||
user_public = UserPublic(
|
||||
email=user.email,
|
||||
role=user.role if not auth_result.is_admin else UserRole.ADMIN,
|
||||
credits=user.credits,
|
||||
display_name=user.display_name,
|
||||
total_jobs_created=user.total_jobs_created,
|
||||
total_jobs_completed=user.total_jobs_completed,
|
||||
)
|
||||
return UserProfileResponse(user=user_public, has_session=True)
|
||||
|
||||
# Fall back to session validation (for magic link sessions)
|
||||
valid, user, message = user_service.validate_session(token)
|
||||
|
||||
if not valid or not user:
|
||||
@@ -607,31 +626,191 @@ async def get_feedback_form_data(
|
||||
# Admin Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/admin/users", response_model=UserListResponse)
|
||||
class UserListResponsePaginated(BaseModel):
|
||||
"""Paginated response for user list."""
|
||||
users: list[UserPublic]
|
||||
total: int
|
||||
offset: int
|
||||
limit: int
|
||||
has_more: bool
|
||||
|
||||
|
||||
class UserDetailResponse(BaseModel):
|
||||
"""Detailed user information for admin view."""
|
||||
email: str
|
||||
role: UserRole
|
||||
credits: int
|
||||
display_name: Optional[str] = None
|
||||
is_active: bool = True
|
||||
email_verified: bool = False
|
||||
created_at: Optional[str] = None
|
||||
updated_at: Optional[str] = None
|
||||
last_login_at: Optional[str] = None
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
is_beta_tester: bool = False
|
||||
beta_tester_status: Optional[str] = None
|
||||
credit_transactions: list[dict] = []
|
||||
recent_jobs: list[dict] = []
|
||||
active_sessions_count: int = 0
|
||||
|
||||
|
||||
@router.get("/admin/users", response_model=UserListResponsePaginated)
|
||||
async def list_users(
|
||||
limit: int = 100,
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
search: Optional[str] = None,
|
||||
sort_by: str = "created_at",
|
||||
sort_order: str = "desc",
|
||||
include_inactive: bool = False,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
List all users (admin only).
|
||||
List all users with search, pagination, and sorting (admin only).
|
||||
|
||||
Args:
|
||||
limit: Maximum users to return (default 50, max 100)
|
||||
offset: Number of users to skip for pagination
|
||||
search: Search by email (case-insensitive prefix match)
|
||||
sort_by: Field to sort by (created_at, last_login_at, credits, email)
|
||||
sort_order: Sort direction (asc, desc)
|
||||
include_inactive: Include disabled users
|
||||
"""
|
||||
users = user_service.list_users(limit=limit, include_inactive=include_inactive)
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
users_public = [
|
||||
UserPublic(
|
||||
email=u.email,
|
||||
role=u.role,
|
||||
credits=u.credits,
|
||||
display_name=u.display_name,
|
||||
total_jobs_created=u.total_jobs_created,
|
||||
total_jobs_completed=u.total_jobs_completed,
|
||||
)
|
||||
for u in users
|
||||
]
|
||||
# Validate and cap limit
|
||||
limit = min(limit, 100)
|
||||
|
||||
return UserListResponse(users=users_public, total=len(users_public))
|
||||
db = user_service.db
|
||||
query = db.collection(USERS_COLLECTION)
|
||||
|
||||
# Filter inactive users
|
||||
if not include_inactive:
|
||||
query = query.where(filter=FieldFilter('is_active', '==', True))
|
||||
|
||||
# Search by email prefix (case-insensitive via range query)
|
||||
if search:
|
||||
search_lower = search.lower()
|
||||
# Use range query for prefix matching
|
||||
query = query.where(filter=FieldFilter('email', '>=', search_lower))
|
||||
query = query.where(filter=FieldFilter('email', '<', search_lower + '\uffff'))
|
||||
|
||||
# Sorting
|
||||
direction = firestore.Query.DESCENDING if sort_order == "desc" else firestore.Query.ASCENDING
|
||||
if sort_by in ["created_at", "last_login_at", "credits", "email"]:
|
||||
query = query.order_by(sort_by, direction=direction)
|
||||
else:
|
||||
query = query.order_by("created_at", direction=direction)
|
||||
|
||||
# Get total count (without pagination) for has_more calculation
|
||||
# Note: This is expensive for large datasets, consider caching
|
||||
all_docs = list(query.stream())
|
||||
total_count = len(all_docs)
|
||||
|
||||
# Apply pagination manually (Firestore doesn't support offset well)
|
||||
paginated_docs = all_docs[offset:offset + limit]
|
||||
|
||||
users_public = []
|
||||
for doc in paginated_docs:
|
||||
data = doc.to_dict()
|
||||
users_public.append(UserPublic(
|
||||
email=data.get("email", ""),
|
||||
role=data.get("role", UserRole.USER),
|
||||
credits=data.get("credits", 0),
|
||||
display_name=data.get("display_name"),
|
||||
total_jobs_created=data.get("total_jobs_created", 0),
|
||||
total_jobs_completed=data.get("total_jobs_completed", 0),
|
||||
))
|
||||
|
||||
return UserListResponsePaginated(
|
||||
users=users_public,
|
||||
total=total_count,
|
||||
offset=offset,
|
||||
limit=limit,
|
||||
has_more=(offset + limit) < total_count,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/admin/users/{email}/detail", response_model=UserDetailResponse)
|
||||
async def get_user_detail(
|
||||
email: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
Get detailed user information including credit history and recent jobs (admin only).
|
||||
"""
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from urllib.parse import unquote
|
||||
|
||||
# URL decode the email (handles @ and other special chars)
|
||||
email = unquote(email).lower()
|
||||
|
||||
user = user_service.get_user(email)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
db = user_service.db
|
||||
|
||||
# Get recent jobs for this user
|
||||
jobs_query = db.collection("jobs").where(
|
||||
filter=FieldFilter("user_email", "==", email)
|
||||
).order_by("created_at", direction=firestore.Query.DESCENDING).limit(10)
|
||||
|
||||
recent_jobs = []
|
||||
for job_doc in jobs_query.stream():
|
||||
job_data = job_doc.to_dict()
|
||||
created_at = job_data.get("created_at")
|
||||
# Handle both datetime objects and ISO strings
|
||||
if created_at:
|
||||
created_at_str = created_at.isoformat() if hasattr(created_at, 'isoformat') else str(created_at)
|
||||
else:
|
||||
created_at_str = None
|
||||
recent_jobs.append({
|
||||
"job_id": job_data.get("job_id"),
|
||||
"status": job_data.get("status"),
|
||||
"artist": job_data.get("artist"),
|
||||
"title": job_data.get("title"),
|
||||
"created_at": created_at_str,
|
||||
})
|
||||
|
||||
# Count active sessions
|
||||
sessions_query = db.collection("sessions").where(
|
||||
filter=FieldFilter("user_email", "==", email)
|
||||
).where(
|
||||
filter=FieldFilter("is_active", "==", True)
|
||||
)
|
||||
active_sessions_count = sum(1 for _ in sessions_query.stream())
|
||||
|
||||
# Format credit transactions
|
||||
credit_transactions = []
|
||||
for txn in user.credit_transactions[-20:]: # Last 20 transactions
|
||||
if hasattr(txn, 'model_dump'):
|
||||
credit_transactions.append(txn.model_dump(mode='json'))
|
||||
elif isinstance(txn, dict):
|
||||
credit_transactions.append(txn)
|
||||
|
||||
return UserDetailResponse(
|
||||
email=user.email,
|
||||
role=user.role,
|
||||
credits=user.credits,
|
||||
display_name=user.display_name,
|
||||
is_active=user.is_active,
|
||||
email_verified=user.email_verified,
|
||||
created_at=user.created_at.isoformat() if user.created_at else None,
|
||||
updated_at=user.updated_at.isoformat() if user.updated_at else None,
|
||||
last_login_at=user.last_login_at.isoformat() if user.last_login_at else None,
|
||||
total_jobs_created=user.total_jobs_created,
|
||||
total_jobs_completed=user.total_jobs_completed,
|
||||
is_beta_tester=user.is_beta_tester,
|
||||
beta_tester_status=user.beta_tester_status,
|
||||
credit_transactions=credit_transactions,
|
||||
recent_jobs=recent_jobs,
|
||||
active_sessions_count=active_sessions_count,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/admin/credits", response_model=AddCreditsResponse)
|
||||
@@ -773,7 +952,7 @@ async def get_beta_stats(
|
||||
from google.cloud.firestore_v1 import aggregation
|
||||
|
||||
# Count beta testers by status using efficient aggregation queries
|
||||
users_collection = user_service.db.collection("users")
|
||||
users_collection = user_service.db.collection(USERS_COLLECTION)
|
||||
|
||||
# Helper function to get count using aggregation
|
||||
def get_count(query) -> int:
|
||||
|
||||
+12
-1
@@ -45,7 +45,10 @@ class Settings(BaseSettings):
|
||||
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
|
||||
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
|
||||
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
|
||||
|
||||
# Timeout for agentic correction in seconds. If correction takes longer, abort and
|
||||
# use uncorrected transcription - human review will fix any issues.
|
||||
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
|
||||
|
||||
# Cloud Tasks (for scalable worker coordination)
|
||||
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
|
||||
# When disabled (default), workers are triggered via direct HTTP (for development)
|
||||
@@ -57,6 +60,14 @@ class Settings(BaseSettings):
|
||||
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
|
||||
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
|
||||
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# GCE Encoding Worker (for high-performance video encoding)
|
||||
# When enabled, video encoding is offloaded to a dedicated C4 GCE instance
|
||||
# with faster CPU (Intel Granite Rapids 3.9 GHz) instead of Cloud Run.
|
||||
# This provides 2-3x faster encoding times for CPU-bound FFmpeg libx264 encoding.
|
||||
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
|
||||
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
+2
-1
@@ -7,7 +7,7 @@ from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
@@ -114,6 +114,7 @@ app.include_router(auth.router, prefix="/api") # OAuth credential management
|
||||
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
|
||||
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
||||
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
|
||||
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
|
||||
|
||||
|
||||
@app.get("/")
|
||||
|
||||
@@ -204,6 +204,7 @@ class Job(BaseModel):
|
||||
youtube_description: Optional[str] = None # YouTube video description
|
||||
webhook_url: Optional[str] = None # Webhook for notifications
|
||||
user_email: Optional[str] = None # Email for notifications
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
@@ -419,6 +420,7 @@ class JobCreate(BaseModel):
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
|
||||
@@ -276,20 +276,35 @@ class AudioSearchService:
|
||||
Raises:
|
||||
DownloadError: If download fails or no cached result for index
|
||||
"""
|
||||
if result_index < 0 or result_index >= len(self._cached_results):
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. "
|
||||
"Run search() first."
|
||||
)
|
||||
# Use provided search_id or fall back to service-level cache
|
||||
effective_search_id = remote_search_id or self._remote_search_id
|
||||
|
||||
# Check if we have local cached results
|
||||
has_local_cache = result_index >= 0 and result_index < len(self._cached_results)
|
||||
|
||||
# If no local cache but we have remote search_id, use remote download
|
||||
# This handles multi-instance deployments where the cache doesn't persist
|
||||
if not has_local_cache:
|
||||
if effective_search_id and self._remote_client:
|
||||
logger.info(f"No local cache, using remote download with search_id={effective_search_id}, index={result_index}")
|
||||
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
|
||||
else:
|
||||
# Provide clear error message based on whether cache is empty or index out of bounds
|
||||
if len(self._cached_results) == 0:
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
"No cached results available. Run search() first."
|
||||
)
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. "
|
||||
"Run search() first."
|
||||
)
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
logger.info(f"Downloading: {result.artist} - {result.title} from {result.provider}")
|
||||
|
||||
# Use provided search_id or fall back to service-level cache
|
||||
effective_search_id = remote_search_id or self._remote_search_id
|
||||
|
||||
# Check if this was a remote search (torrent sources need remote download)
|
||||
if effective_search_id and self._remote_client:
|
||||
# Check if this is a torrent source that needs remote handling
|
||||
@@ -481,8 +496,12 @@ class AudioSearchService:
|
||||
if not effective_search_id:
|
||||
raise DownloadError("No remote search ID - run search() first")
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
# Try to get local result for metadata, but it's optional
|
||||
# The remote service maintains its own cache by search_id
|
||||
result = None
|
||||
if result_index >= 0 and result_index < len(self._cached_results):
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
try:
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
try:
|
||||
@@ -529,13 +548,15 @@ class AudioSearchService:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download complete: {filepath}")
|
||||
|
||||
|
||||
# Use local result metadata if available, otherwise use empty strings
|
||||
# (the actual download is handled by the remote service using search_id)
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist=result.artist,
|
||||
title=result.title,
|
||||
provider=result.provider,
|
||||
quality=result.quality,
|
||||
artist=result.artist if result else "",
|
||||
title=result.title if result else "",
|
||||
provider=result.provider if result else "remote",
|
||||
quality=result.quality if result else "",
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
|
||||
@@ -263,6 +263,38 @@ class DropboxService:
|
||||
return links.links[0].url
|
||||
raise
|
||||
|
||||
def delete_folder(self, path: str) -> bool:
|
||||
"""
|
||||
Delete a folder and all its contents from Dropbox.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to delete (e.g., "/Karaoke/Tracks-Organized/NOMAD-1234 - Artist - Title")
|
||||
|
||||
Returns:
|
||||
True if deleted successfully, False otherwise
|
||||
"""
|
||||
from dropbox.exceptions import ApiError
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Deleting Dropbox folder: {path}")
|
||||
|
||||
try:
|
||||
self.client.files_delete_v2(path)
|
||||
logger.info(f"Successfully deleted: {path}")
|
||||
return True
|
||||
except ApiError as e:
|
||||
if e.error.is_path_lookup() and e.error.get_path_lookup().is_not_found():
|
||||
logger.warning(f"Folder not found (already deleted?): {path}")
|
||||
return True # Consider it success if already gone
|
||||
logger.error(f"Failed to delete Dropbox folder: {e}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting Dropbox folder: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def get_dropbox_service() -> DropboxService:
|
||||
"""Factory function to get a DropboxService instance."""
|
||||
|
||||
@@ -0,0 +1,270 @@
|
||||
"""
|
||||
GCE Encoding Worker Service.
|
||||
|
||||
This service dispatches video encoding jobs to a dedicated high-performance
|
||||
GCE instance (C4-standard with Intel Granite Rapids CPU) for faster encoding.
|
||||
|
||||
The GCE worker provides:
|
||||
- 3.9 GHz all-core frequency (vs 3.7 GHz on Cloud Run)
|
||||
- Dedicated vCPUs (no contention)
|
||||
- 2-3x faster FFmpeg libx264 encoding
|
||||
|
||||
Usage:
|
||||
encoding_service = get_encoding_service()
|
||||
if encoding_service.is_configured:
|
||||
result = await encoding_service.encode_videos(job_id, input_gcs_path, config)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
import aiohttp
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EncodingService:
|
||||
"""Service for dispatching encoding jobs to GCE worker."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._url = None
|
||||
self._api_key = None
|
||||
self._initialized = False
|
||||
|
||||
def _load_credentials(self):
|
||||
"""Load encoding worker URL and API key from config/secrets."""
|
||||
if self._initialized:
|
||||
return
|
||||
|
||||
# Try environment variables first, then Secret Manager
|
||||
self._url = self.settings.encoding_worker_url
|
||||
self._api_key = self.settings.encoding_worker_api_key
|
||||
|
||||
# Fall back to Secret Manager
|
||||
if not self._url:
|
||||
self._url = self.settings.get_secret("encoding-worker-url")
|
||||
if not self._api_key:
|
||||
self._api_key = self.settings.get_secret("encoding-worker-api-key")
|
||||
|
||||
self._initialized = True
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if encoding service is configured with URL and API key."""
|
||||
self._load_credentials()
|
||||
return bool(self._url and self._api_key)
|
||||
|
||||
@property
|
||||
def is_enabled(self) -> bool:
|
||||
"""Check if GCE encoding is enabled and configured."""
|
||||
return self.settings.use_gce_encoding and self.is_configured
|
||||
|
||||
async def submit_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
input_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
encoding_config: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit an encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
input_gcs_path: GCS path to input files (gs://bucket/path/)
|
||||
output_gcs_path: GCS path for output files (gs://bucket/path/)
|
||||
encoding_config: Configuration for encoding (formats, quality, etc.)
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"input_gcs_path": input_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"encoding_config": encoding_config,
|
||||
}
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting encoding job to GCE worker: {url}")
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
|
||||
if resp.status == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp.status == 409:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists")
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
raise RuntimeError(f"Failed to submit encoding job: {resp.status} - {text}")
|
||||
|
||||
return await resp.json()
|
||||
|
||||
async def get_job_status(self, job_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the status of an encoding job.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
|
||||
Returns:
|
||||
Job status including: status, progress, error, output_files
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/status/{job_id}"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=30) as resp:
|
||||
if resp.status == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp.status == 404:
|
||||
raise RuntimeError(f"Encoding job {job_id} not found")
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
raise RuntimeError(f"Failed to get job status: {resp.status} - {text}")
|
||||
|
||||
return await resp.json()
|
||||
|
||||
async def wait_for_completion(
|
||||
self,
|
||||
job_id: str,
|
||||
poll_interval: float = 10.0,
|
||||
timeout: float = 3600.0,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Poll for encoding job completion.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
poll_interval: Seconds between status checks
|
||||
timeout: Maximum time to wait (default 1 hour)
|
||||
progress_callback: Optional callback(progress: int) for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
|
||||
Raises:
|
||||
TimeoutError: If job doesn't complete within timeout
|
||||
RuntimeError: If job fails
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
|
||||
|
||||
start_time = asyncio.get_event_loop().time()
|
||||
last_progress = 0
|
||||
|
||||
while True:
|
||||
elapsed = asyncio.get_event_loop().time() - start_time
|
||||
if elapsed > timeout:
|
||||
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
|
||||
|
||||
status = await self.get_job_status(job_id)
|
||||
job_status = status.get("status", "unknown")
|
||||
progress = status.get("progress", 0)
|
||||
|
||||
# Report progress
|
||||
if progress != last_progress:
|
||||
logger.info(f"[job:{job_id}] Encoding progress: {progress}%")
|
||||
last_progress = progress
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(progress)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback failed: {e}")
|
||||
|
||||
if job_status == "complete":
|
||||
logger.info(f"[job:{job_id}] GCE encoding complete in {elapsed:.1f}s")
|
||||
return status
|
||||
|
||||
if job_status == "failed":
|
||||
error = status.get("error", "Unknown error")
|
||||
raise RuntimeError(f"Encoding job {job_id} failed: {error}")
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
async def encode_videos(
|
||||
self,
|
||||
job_id: str,
|
||||
input_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
encoding_config: Optional[Dict[str, Any]] = None,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
input_gcs_path: GCS path to input files
|
||||
output_gcs_path: GCS path for output files
|
||||
encoding_config: Optional encoding configuration
|
||||
progress_callback: Optional callback for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
|
||||
|
||||
# Submit the job
|
||||
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
|
||||
|
||||
# Wait for completion
|
||||
return await self.wait_for_completion(
|
||||
job_id, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check the health of the encoding worker.
|
||||
|
||||
Returns:
|
||||
Health status including active jobs and FFmpeg version
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
return {"status": "not_configured"}
|
||||
|
||||
url = f"{self._url}/health"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=10) as resp:
|
||||
if resp.status == 200:
|
||||
return await resp.json()
|
||||
return {"status": "error", "code": resp.status}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_encoding_service: Optional[EncodingService] = None
|
||||
|
||||
|
||||
def get_encoding_service() -> EncodingService:
|
||||
"""Get the singleton encoding service instance."""
|
||||
global _encoding_service
|
||||
if _encoding_service is None:
|
||||
_encoding_service = EncodingService()
|
||||
return _encoding_service
|
||||
@@ -300,6 +300,45 @@ class GoogleDriveService:
|
||||
logger.info(f"Public share upload complete: {len(uploaded_files)} files uploaded")
|
||||
return uploaded_files
|
||||
|
||||
def delete_file(self, file_id: str) -> bool:
|
||||
"""
|
||||
Delete a file from Google Drive.
|
||||
|
||||
Args:
|
||||
file_id: Google Drive file ID to delete
|
||||
|
||||
Returns:
|
||||
True if deleted successfully, False otherwise
|
||||
"""
|
||||
logger.info(f"Deleting Google Drive file: {file_id}")
|
||||
|
||||
try:
|
||||
self.service.files().delete(fileId=file_id).execute()
|
||||
logger.info(f"Successfully deleted file: {file_id}")
|
||||
return True
|
||||
except Exception as e:
|
||||
# Check if it's a 404 (already deleted)
|
||||
if hasattr(e, 'resp') and e.resp.status == 404:
|
||||
logger.warning(f"File not found (already deleted?): {file_id}")
|
||||
return True
|
||||
logger.error(f"Failed to delete Google Drive file: {e}")
|
||||
return False
|
||||
|
||||
def delete_files(self, file_ids: list[str]) -> dict[str, bool]:
|
||||
"""
|
||||
Delete multiple files from Google Drive.
|
||||
|
||||
Args:
|
||||
file_ids: List of Google Drive file IDs to delete
|
||||
|
||||
Returns:
|
||||
Dictionary mapping file_id to success status
|
||||
"""
|
||||
results = {}
|
||||
for file_id in file_ids:
|
||||
results[file_id] = self.delete_file(file_id)
|
||||
return results
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_gdrive_service: Optional[GoogleDriveService] = None
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
"""
|
||||
LyricsTranscriber cache synchronization with GCS.
|
||||
|
||||
This service persists LyricsTranscriber's cache files to GCS so that
|
||||
cloud workers (Cloud Run instances) can share cache across containers.
|
||||
|
||||
Cache files are stored flat in GCS under lyrics-transcriber-cache/ prefix:
|
||||
- Transcription: {provider}_{audio_hash}_raw.json, {provider}_{audio_hash}_converted.json
|
||||
- Lyrics: {provider}_{artist_title_hash}_raw.json, {provider}_{artist_title_hash}_converted.json
|
||||
|
||||
Hash computation matches LyricsTranscriber's implementation exactly:
|
||||
- Audio hash: MD5 of audio file bytes
|
||||
- Lyrics hash: MD5 of "{artist.lower()}_{title.lower()}"
|
||||
"""
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Providers that use audio file hash as cache key
|
||||
TRANSCRIPTION_PROVIDERS = ["audioshake", "whisper", "localwhisper"]
|
||||
|
||||
# Providers that use artist+title hash as cache key
|
||||
LYRICS_PROVIDERS = ["genius", "spotify", "lrclib", "musixmatch"]
|
||||
|
||||
# Cache file suffixes
|
||||
CACHE_SUFFIXES = ["raw", "converted"]
|
||||
|
||||
|
||||
class LyricsCacheService:
|
||||
"""Service to sync LyricsTranscriber cache with GCS."""
|
||||
|
||||
GCS_CACHE_PREFIX = "lyrics-transcriber-cache/"
|
||||
|
||||
def __init__(self, storage: Optional[StorageService] = None):
|
||||
"""Initialize the cache service.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance. If None, creates a new one.
|
||||
"""
|
||||
self.storage = storage or StorageService()
|
||||
|
||||
def compute_audio_hash(self, audio_path: str) -> str:
|
||||
"""Compute MD5 hash of audio file bytes.
|
||||
|
||||
This matches LyricsTranscriber's _get_file_hash() method exactly.
|
||||
|
||||
Args:
|
||||
audio_path: Path to the audio file.
|
||||
|
||||
Returns:
|
||||
MD5 hex digest of the file contents.
|
||||
"""
|
||||
md5_hash = hashlib.md5()
|
||||
with open(audio_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
md5_hash.update(chunk)
|
||||
return md5_hash.hexdigest()
|
||||
|
||||
def compute_lyrics_hash(self, artist: str, title: str) -> str:
|
||||
"""Compute MD5 hash of artist and title.
|
||||
|
||||
This matches LyricsTranscriber's _get_artist_title_hash() method exactly.
|
||||
|
||||
Args:
|
||||
artist: Artist name.
|
||||
title: Track title.
|
||||
|
||||
Returns:
|
||||
MD5 hex digest of "{artist.lower()}_{title.lower()}".
|
||||
"""
|
||||
combined = f"{artist.lower()}_{title.lower()}"
|
||||
return hashlib.md5(combined.encode()).hexdigest()
|
||||
|
||||
def _get_cache_filenames(
|
||||
self, providers: List[str], hash_value: str
|
||||
) -> List[str]:
|
||||
"""Generate list of possible cache filenames for given providers and hash.
|
||||
|
||||
Args:
|
||||
providers: List of provider names (e.g., ["audioshake", "whisper"]).
|
||||
hash_value: The hash to use in filenames.
|
||||
|
||||
Returns:
|
||||
List of filenames like ["audioshake_abc123_raw.json", ...].
|
||||
"""
|
||||
filenames = []
|
||||
for provider in providers:
|
||||
for suffix in CACHE_SUFFIXES:
|
||||
filenames.append(f"{provider}_{hash_value}_{suffix}.json")
|
||||
return filenames
|
||||
|
||||
def sync_cache_from_gcs(
|
||||
self,
|
||||
local_cache_dir: str,
|
||||
audio_hash: str,
|
||||
lyrics_hash: str,
|
||||
) -> Dict[str, int]:
|
||||
"""Download relevant cache files from GCS to local directory.
|
||||
|
||||
Downloads cache files for both transcription (audio hash) and
|
||||
lyrics (artist+title hash) providers.
|
||||
|
||||
Args:
|
||||
local_cache_dir: Local directory to download cache files to.
|
||||
audio_hash: MD5 hash of audio file.
|
||||
lyrics_hash: MD5 hash of artist+title.
|
||||
|
||||
Returns:
|
||||
Dict with counts: {"downloaded": N, "not_found": M, "errors": E}
|
||||
"""
|
||||
os.makedirs(local_cache_dir, exist_ok=True)
|
||||
|
||||
stats = {"downloaded": 0, "not_found": 0, "errors": 0}
|
||||
|
||||
# Get all possible cache filenames
|
||||
transcription_files = self._get_cache_filenames(
|
||||
TRANSCRIPTION_PROVIDERS, audio_hash
|
||||
)
|
||||
lyrics_files = self._get_cache_filenames(LYRICS_PROVIDERS, lyrics_hash)
|
||||
all_files = transcription_files + lyrics_files
|
||||
|
||||
for filename in all_files:
|
||||
gcs_path = f"{self.GCS_CACHE_PREFIX}{filename}"
|
||||
local_path = os.path.join(local_cache_dir, filename)
|
||||
|
||||
try:
|
||||
if self.storage.file_exists(gcs_path):
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
logger.info(f"Cache hit: downloaded {filename}")
|
||||
stats["downloaded"] += 1
|
||||
else:
|
||||
logger.debug(f"Cache miss: {filename} not in GCS")
|
||||
stats["not_found"] += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error downloading cache file {filename}: {e}")
|
||||
stats["errors"] += 1
|
||||
|
||||
logger.info(
|
||||
f"Cache sync from GCS complete: "
|
||||
f"{stats['downloaded']} downloaded, "
|
||||
f"{stats['not_found']} not found, "
|
||||
f"{stats['errors']} errors"
|
||||
)
|
||||
return stats
|
||||
|
||||
def sync_cache_to_gcs(
|
||||
self,
|
||||
local_cache_dir: str,
|
||||
audio_hash: str,
|
||||
lyrics_hash: str,
|
||||
) -> Dict[str, int]:
|
||||
"""Upload new cache files from local directory to GCS.
|
||||
|
||||
Only uploads files that match expected cache patterns for the given
|
||||
hashes and don't already exist in GCS.
|
||||
|
||||
Args:
|
||||
local_cache_dir: Local directory with cache files.
|
||||
audio_hash: MD5 hash of audio file.
|
||||
lyrics_hash: MD5 hash of artist+title.
|
||||
|
||||
Returns:
|
||||
Dict with counts: {"uploaded": N, "skipped": M, "errors": E}
|
||||
"""
|
||||
stats = {"uploaded": 0, "skipped": 0, "errors": 0}
|
||||
|
||||
if not os.path.exists(local_cache_dir):
|
||||
logger.warning(f"Local cache dir does not exist: {local_cache_dir}")
|
||||
return stats
|
||||
|
||||
# Get all possible cache filenames we're interested in
|
||||
transcription_files = self._get_cache_filenames(
|
||||
TRANSCRIPTION_PROVIDERS, audio_hash
|
||||
)
|
||||
lyrics_files = self._get_cache_filenames(LYRICS_PROVIDERS, lyrics_hash)
|
||||
expected_files = set(transcription_files + lyrics_files)
|
||||
|
||||
# Check each file in local cache dir
|
||||
for filename in os.listdir(local_cache_dir):
|
||||
# Only process files we expect (matching our hash patterns)
|
||||
if filename not in expected_files:
|
||||
continue
|
||||
|
||||
local_path = os.path.join(local_cache_dir, filename)
|
||||
if not os.path.isfile(local_path):
|
||||
continue
|
||||
|
||||
gcs_path = f"{self.GCS_CACHE_PREFIX}{filename}"
|
||||
|
||||
try:
|
||||
# Skip if already exists in GCS (same hash = same content)
|
||||
if self.storage.file_exists(gcs_path):
|
||||
logger.debug(f"Cache file already in GCS: {filename}")
|
||||
stats["skipped"] += 1
|
||||
continue
|
||||
|
||||
self.storage.upload_file(local_path, gcs_path)
|
||||
logger.info(f"Uploaded cache file: {filename}")
|
||||
stats["uploaded"] += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error uploading cache file {filename}: {e}")
|
||||
stats["errors"] += 1
|
||||
|
||||
logger.info(
|
||||
f"Cache sync to GCS complete: "
|
||||
f"{stats['uploaded']} uploaded, "
|
||||
f"{stats['skipped']} skipped (already exist), "
|
||||
f"{stats['errors']} errors"
|
||||
)
|
||||
return stats
|
||||
@@ -32,7 +32,7 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Collection names
|
||||
USERS_COLLECTION = "users"
|
||||
USERS_COLLECTION = "gen_users"
|
||||
MAGIC_LINKS_COLLECTION = "magic_links"
|
||||
SESSIONS_COLLECTION = "sessions"
|
||||
PROCESSED_STRIPE_SESSIONS_COLLECTION = "processed_stripe_sessions"
|
||||
|
||||
@@ -20,6 +20,7 @@ import os
|
||||
import json
|
||||
from typing import Optional
|
||||
import httpx
|
||||
from google.protobuf import duration_pb2
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.services.tracing import inject_trace_context
|
||||
@@ -38,6 +39,18 @@ WORKER_QUEUES = {
|
||||
"video": "video-worker-queue",
|
||||
}
|
||||
|
||||
# Dispatch deadlines for each worker type (in seconds)
|
||||
# Cloud Tasks max is 1800s (30 min). These set how long Cloud Tasks waits
|
||||
# for the HTTP handler to respond before considering it failed.
|
||||
# Must be >= actual worker timeout + buffer for startup/upload.
|
||||
WORKER_DISPATCH_DEADLINES = {
|
||||
"audio": 1800, # 30 min - Modal separation can take 15-20 min
|
||||
"lyrics": 1500, # 25 min - TRANSCRIPTION_TIMEOUT_SECONDS is 1200s (20 min)
|
||||
"screens": 600, # 10 min - Screen generation is fast
|
||||
"render-video": 1800, # 30 min - Video encoding can be slow
|
||||
"video": 1800, # 30 min - Video encoding can be slow
|
||||
}
|
||||
|
||||
|
||||
class WorkerService:
|
||||
"""
|
||||
@@ -234,6 +247,9 @@ class WorkerService:
|
||||
# This allows worker spans to link back to the original request trace
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Get dispatch deadline for this worker type (how long Cloud Tasks waits for response)
|
||||
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get(worker_type, 600)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
@@ -247,6 +263,10 @@ class WorkerService:
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
# Set dispatch_deadline - how long Cloud Tasks waits for the handler to respond
|
||||
# Default is 10 min, but audio separation can take 30+ min
|
||||
# Max is 1800s (30 min) for Cloud Tasks
|
||||
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
|
||||
}
|
||||
|
||||
# Create task
|
||||
@@ -254,7 +274,8 @@ class WorkerService:
|
||||
# This prevents duplicate task errors on retries
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name}"
|
||||
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name} "
|
||||
f"(dispatch_deadline={dispatch_deadline_seconds}s)"
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
@@ -460,16 +460,18 @@ class TestAudioSearchServiceDownload:
|
||||
url="https://youtube.com/watch?v=abc123",
|
||||
index=0,
|
||||
)
|
||||
|
||||
|
||||
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
|
||||
service._fetcher = Mock()
|
||||
service._fetcher.search.return_value = [mock_result]
|
||||
|
||||
# Ensure no remote client so we test the local cache path
|
||||
service._remote_client = None
|
||||
|
||||
service.search("ABBA", "Waterloo")
|
||||
|
||||
|
||||
with pytest.raises(DownloadError) as exc_info:
|
||||
service.download(99, "/tmp")
|
||||
|
||||
|
||||
assert "No cached result for index 99" in str(exc_info.value)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,583 @@
|
||||
"""
|
||||
Regression tests for job creation issues discovered 2024-12-31.
|
||||
|
||||
These tests ensure:
|
||||
1. user_email is properly extracted from AuthResult and set on jobs
|
||||
2. URL jobs trigger workers sequentially (audio first, then lyrics)
|
||||
3. Audio search downloads work without relying on in-memory cache
|
||||
4. Transcription has proper timeout handling
|
||||
|
||||
See docs/archive/2024-12-31-job-failure-investigation.md for details.
|
||||
"""
|
||||
import asyncio
|
||||
import pytest
|
||||
from unittest.mock import Mock, AsyncMock, patch, MagicMock
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from backend.services.auth_service import AuthResult, UserType
|
||||
|
||||
|
||||
class TestUserEmailExtraction:
|
||||
"""
|
||||
Issue 1: user_email not being set on jobs.
|
||||
|
||||
Root cause: Job creation endpoints used require_auth but didn't extract
|
||||
auth_result.user_email and set it on the job.
|
||||
|
||||
These tests verify that authenticated user's email is properly set on jobs.
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_auth_result_with_email(self):
|
||||
"""AuthResult for a logged-in user with email."""
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.UNLIMITED, # Session-based users typically have UNLIMITED
|
||||
remaining_uses=-1,
|
||||
message="Valid session",
|
||||
user_email="testuser@example.com",
|
||||
is_admin=False,
|
||||
)
|
||||
|
||||
@pytest.fixture
|
||||
def mock_auth_result_admin_token(self):
|
||||
"""AuthResult for admin token without email."""
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Valid admin token",
|
||||
user_email=None,
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
def test_auth_result_has_user_email_field(self):
|
||||
"""AuthResult must have user_email field for job association."""
|
||||
result = AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.UNLIMITED,
|
||||
remaining_uses=-1,
|
||||
message="test",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
assert hasattr(result, 'user_email')
|
||||
assert result.user_email == "user@example.com"
|
||||
|
||||
def test_auth_result_user_email_can_be_none(self):
|
||||
"""AuthResult user_email can be None for token-based auth."""
|
||||
result = AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="test",
|
||||
user_email=None,
|
||||
)
|
||||
assert result.user_email is None
|
||||
|
||||
def test_upload_endpoint_extracts_user_email_from_auth(self):
|
||||
"""
|
||||
/jobs/upload endpoint must set user_email from authenticated user.
|
||||
|
||||
This verifies the code pattern by inspecting the source.
|
||||
Integration tests verify the full behavior.
|
||||
"""
|
||||
from backend.api.routes import file_upload
|
||||
import inspect
|
||||
|
||||
# Get the source code of upload_and_create_job
|
||||
source = inspect.getsource(file_upload.upload_and_create_job)
|
||||
|
||||
# Verify the endpoint extracts user_email from auth_result
|
||||
assert 'auth_result.user_email' in source, \
|
||||
"upload_and_create_job must extract user_email from auth_result"
|
||||
assert 'effective_user_email' in source, \
|
||||
"upload_and_create_job must use effective_user_email pattern"
|
||||
assert 'user_email=effective_user_email' in source or 'user_email=' in source, \
|
||||
"upload_and_create_job must pass user_email to JobCreate"
|
||||
|
||||
def test_all_job_creation_endpoints_extract_user_email(self):
|
||||
"""
|
||||
All job creation endpoints must extract user_email from AuthResult.
|
||||
|
||||
This is a regression guard for Issue 1 (user_email not being set).
|
||||
"""
|
||||
from backend.api.routes import file_upload, audio_search
|
||||
import inspect
|
||||
|
||||
endpoints_to_check = [
|
||||
(file_upload, 'upload_and_create_job'),
|
||||
(file_upload, 'create_job_from_url'),
|
||||
(audio_search, 'search_audio'),
|
||||
]
|
||||
|
||||
for module, func_name in endpoints_to_check:
|
||||
func = getattr(module, func_name)
|
||||
source = inspect.getsource(func)
|
||||
|
||||
assert 'auth_result.user_email' in source, \
|
||||
f"{func_name} must extract user_email from auth_result"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_from_url_endpoint_sets_user_email(self, mock_auth_result_with_email):
|
||||
"""
|
||||
/jobs/create-from-url endpoint must set user_email from authenticated user.
|
||||
"""
|
||||
from backend.api.routes.file_upload import create_job_from_url, CreateJobFromUrlRequest
|
||||
|
||||
mock_request = Mock()
|
||||
mock_request.headers = {}
|
||||
mock_request.client = Mock(host="127.0.0.1")
|
||||
mock_request.url = Mock(path="/api/jobs/create-from-url")
|
||||
|
||||
mock_background_tasks = Mock()
|
||||
body = CreateJobFromUrlRequest(
|
||||
url="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
|
||||
artist="Rick Astley",
|
||||
title="Never Gonna Give You Up",
|
||||
)
|
||||
|
||||
with patch('backend.api.routes.file_upload.job_manager') as mock_jm, \
|
||||
patch('backend.api.routes.file_upload.worker_service') as mock_worker, \
|
||||
patch('backend.api.routes.file_upload.get_credential_manager') as mock_cred:
|
||||
|
||||
mock_job = Mock()
|
||||
mock_job.job_id = "test-job-456"
|
||||
mock_jm.create_job.return_value = mock_job
|
||||
mock_cred.return_value.check_youtube_credentials.return_value = Mock(status=Mock(value="valid"))
|
||||
|
||||
await create_job_from_url(
|
||||
request=mock_request,
|
||||
background_tasks=mock_background_tasks,
|
||||
body=body,
|
||||
auth_result=mock_auth_result_with_email,
|
||||
)
|
||||
|
||||
mock_jm.create_job.assert_called_once()
|
||||
job_create_arg = mock_jm.create_job.call_args[0][0]
|
||||
assert job_create_arg.user_email == "testuser@example.com", \
|
||||
"create-from-url must set user_email from AuthResult"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_search_endpoint_sets_user_email(self, mock_auth_result_with_email):
|
||||
"""
|
||||
/audio-search/search endpoint must set user_email from authenticated user.
|
||||
"""
|
||||
from backend.api.routes.audio_search import search_audio, AudioSearchRequest
|
||||
|
||||
mock_request = Mock()
|
||||
mock_request.headers = {}
|
||||
mock_request.client = Mock(host="127.0.0.1")
|
||||
mock_request.url = Mock(path="/api/audio-search/search")
|
||||
|
||||
mock_background_tasks = Mock()
|
||||
body = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
with patch('backend.api.routes.audio_search.job_manager') as mock_jm, \
|
||||
patch('backend.api.routes.audio_search.get_audio_search_service') as mock_search, \
|
||||
patch('backend.api.routes.audio_search.get_credential_manager') as mock_cred:
|
||||
|
||||
mock_job = Mock()
|
||||
mock_job.job_id = "test-job-789"
|
||||
mock_job.state_data = {}
|
||||
mock_jm.create_job.return_value = mock_job
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_cred.return_value.check_youtube_credentials.return_value = Mock(status=Mock(value="valid"))
|
||||
mock_search.return_value.search.return_value = []
|
||||
|
||||
await search_audio(
|
||||
request=mock_request,
|
||||
background_tasks=mock_background_tasks,
|
||||
body=body,
|
||||
auth_result=mock_auth_result_with_email,
|
||||
)
|
||||
|
||||
mock_jm.create_job.assert_called_once()
|
||||
job_create_arg = mock_jm.create_job.call_args[0][0]
|
||||
assert job_create_arg.user_email == "testuser@example.com", \
|
||||
"audio-search must set user_email from AuthResult"
|
||||
|
||||
def test_effective_user_email_prefers_auth_over_form(self):
|
||||
"""
|
||||
When both auth_result.user_email and form user_email are provided,
|
||||
the authenticated user's email should take precedence.
|
||||
"""
|
||||
auth_email = "authenticated@example.com"
|
||||
form_email = "form@example.com"
|
||||
|
||||
# This is the logic used in the endpoints
|
||||
effective_user_email = auth_email or form_email
|
||||
|
||||
assert effective_user_email == auth_email, \
|
||||
"Authenticated user's email should take precedence over form parameter"
|
||||
|
||||
def test_effective_user_email_falls_back_to_form(self):
|
||||
"""
|
||||
When auth_result.user_email is None (e.g., admin token),
|
||||
form user_email should be used as fallback.
|
||||
"""
|
||||
auth_email = None
|
||||
form_email = "form@example.com"
|
||||
|
||||
effective_user_email = auth_email or form_email
|
||||
|
||||
assert effective_user_email == form_email, \
|
||||
"Form parameter should be used when auth has no email"
|
||||
|
||||
|
||||
class TestUrlJobWorkerSequencing:
|
||||
"""
|
||||
Issue 2: YouTube URL download race condition.
|
||||
|
||||
Root cause: Both audio and lyrics workers were triggered in parallel.
|
||||
For URL jobs, lyrics worker would timeout waiting for audio to download.
|
||||
|
||||
These tests verify that URL jobs trigger workers sequentially.
|
||||
"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_url_job_triggers_only_audio_worker_initially(self):
|
||||
"""
|
||||
create-from-url must only trigger audio worker, not lyrics worker.
|
||||
The audio worker will trigger lyrics after download completes.
|
||||
"""
|
||||
from backend.api.routes.file_upload import _trigger_audio_worker_only
|
||||
|
||||
with patch('backend.api.routes.file_upload.worker_service') as mock_ws:
|
||||
mock_ws.trigger_audio_worker = AsyncMock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock()
|
||||
|
||||
await _trigger_audio_worker_only("test-job-id")
|
||||
|
||||
mock_ws.trigger_audio_worker.assert_called_once_with("test-job-id")
|
||||
mock_ws.trigger_lyrics_worker.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_parallel_worker_triggers_both_workers(self):
|
||||
"""
|
||||
_trigger_workers_parallel should trigger both workers (for uploaded files).
|
||||
"""
|
||||
from backend.api.routes.file_upload import _trigger_workers_parallel
|
||||
|
||||
with patch('backend.api.routes.file_upload.worker_service') as mock_ws:
|
||||
mock_ws.trigger_audio_worker = AsyncMock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock()
|
||||
|
||||
await _trigger_workers_parallel("test-job-id")
|
||||
|
||||
mock_ws.trigger_audio_worker.assert_called_once_with("test-job-id")
|
||||
mock_ws.trigger_lyrics_worker.assert_called_once_with("test-job-id")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_worker_triggers_lyrics_after_url_download(self):
|
||||
"""
|
||||
Audio worker must trigger lyrics worker after successful URL download.
|
||||
"""
|
||||
from backend.workers.audio_worker import _trigger_lyrics_worker_after_url_download
|
||||
|
||||
# Mock at the source module where it's imported from
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_ws:
|
||||
mock_ws = Mock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock()
|
||||
mock_get_ws.return_value = mock_ws
|
||||
|
||||
await _trigger_lyrics_worker_after_url_download("test-job-id")
|
||||
|
||||
mock_ws.trigger_lyrics_worker.assert_called_once_with("test-job-id")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_worker_lyrics_trigger_handles_errors_gracefully(self):
|
||||
"""
|
||||
If lyrics worker trigger fails, audio processing should continue.
|
||||
"""
|
||||
from backend.workers.audio_worker import _trigger_lyrics_worker_after_url_download
|
||||
|
||||
# Mock at the source module where it's imported from
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_ws:
|
||||
mock_ws = Mock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock(side_effect=Exception("Network error"))
|
||||
mock_get_ws.return_value = mock_ws
|
||||
|
||||
# Should not raise exception
|
||||
await _trigger_lyrics_worker_after_url_download("test-job-id")
|
||||
|
||||
|
||||
class TestAudioSearchCacheIndependence:
|
||||
"""
|
||||
Issue 3: Audio search cache not persisting across Cloud Run instances.
|
||||
|
||||
Root cause: AudioSearchService used in-memory cache that doesn't persist
|
||||
across horizontally scaled instances.
|
||||
|
||||
These tests verify downloads work using state_data, not in-memory cache.
|
||||
"""
|
||||
|
||||
def test_search_results_stored_in_job_state_data(self):
|
||||
"""
|
||||
Search results must be stored in job.state_data for persistence.
|
||||
This tests the Job model's ability to store search results.
|
||||
"""
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
state_data={
|
||||
'audio_search_results': [
|
||||
{
|
||||
'provider': 'YouTube',
|
||||
'title': 'Test Song',
|
||||
'artist': 'Test Artist',
|
||||
'url': 'https://youtube.com/watch?v=abc123',
|
||||
'source_id': 'abc123',
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
assert 'audio_search_results' in job.state_data
|
||||
assert len(job.state_data['audio_search_results']) == 1
|
||||
assert job.state_data['audio_search_results'][0]['url'] == 'https://youtube.com/watch?v=abc123'
|
||||
|
||||
def test_audio_search_endpoint_stores_results_in_state_data(self):
|
||||
"""
|
||||
Verify that the search_audio code path stores results in job.state_data.
|
||||
|
||||
This tests that the audio_search route has the correct pattern for
|
||||
persisting search results. The actual integration is tested elsewhere.
|
||||
"""
|
||||
# Verify the import and module structure exists
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
|
||||
# Get the source code of search_audio
|
||||
source = inspect.getsource(audio_search.search_audio)
|
||||
|
||||
# Verify the endpoint stores results in state_data
|
||||
# The code uses job_manager.update_job with state_data dict
|
||||
assert 'audio_search_results' in source, \
|
||||
"search_audio must store results under 'audio_search_results' key"
|
||||
assert 'state_data' in source, \
|
||||
"search_audio must use state_data for persistence"
|
||||
|
||||
def test_download_code_has_youtube_direct_download_branch(self):
|
||||
"""
|
||||
Verify that _download_and_start_processing has the YouTube direct download path.
|
||||
|
||||
This checks that the code path exists for downloading YouTube audio directly
|
||||
using the URL from state_data, avoiding the in-memory cache dependency.
|
||||
"""
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
|
||||
# Get the source of _download_and_start_processing
|
||||
source = inspect.getsource(audio_search._download_and_start_processing)
|
||||
|
||||
# Verify the YouTube direct download branch exists
|
||||
assert "source_name == 'YouTube'" in source, \
|
||||
"_download_and_start_processing must check for YouTube source"
|
||||
assert 'download_url' in source, \
|
||||
"_download_and_start_processing must use download_url from state_data"
|
||||
assert 'download_from_url' in source, \
|
||||
"_download_and_start_processing must call download_from_url for YouTube"
|
||||
|
||||
def test_youtube_download_branch_avoids_cache_dependency(self):
|
||||
"""
|
||||
Verify that the YouTube download path uses URL from state_data,
|
||||
not the in-memory cache that doesn't persist across Cloud Run instances.
|
||||
"""
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
|
||||
source = inspect.getsource(audio_search._download_and_start_processing)
|
||||
|
||||
# The fix specifically uses selected.get('url') which comes from state_data
|
||||
# rather than audio_search_service cache
|
||||
assert "selected.get('url')" in source or "download_url" in source, \
|
||||
"YouTube download must use URL from selected result (state_data)"
|
||||
|
||||
# Verify it doesn't rely on cache for YouTube
|
||||
# The branch is: elif source_name == 'YouTube' and download_url:
|
||||
assert "YouTube" in source and "download_url" in source, \
|
||||
"YouTube branch must check for download_url availability"
|
||||
|
||||
|
||||
class TestTranscriptionTimeout:
|
||||
"""
|
||||
Issue 4: Jobs stuck in downloading state.
|
||||
|
||||
Root cause: Lyrics worker's AudioShake transcription could hang forever
|
||||
without proper timeout, leaving jobs stuck.
|
||||
|
||||
These tests verify timeout handling for transcription.
|
||||
"""
|
||||
|
||||
def test_transcription_timeout_constant_exists(self):
|
||||
"""
|
||||
Verify transcription timeout constant is defined in lyrics_worker.
|
||||
"""
|
||||
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS > 0
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS >= 300, "Timeout should be at least 5 minutes for long songs"
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS <= 1500, "Timeout shouldn't be more than 25 minutes"
|
||||
|
||||
def test_transcription_timeout_value(self):
|
||||
"""
|
||||
Verify the specific timeout value (20 minutes = 1200 seconds).
|
||||
|
||||
This accounts for:
|
||||
- Cloud Run cold start / worker initialization (1-5 min)
|
||||
- AudioShake transcription (1-2 min)
|
||||
- spaCy model loading for correction (2-3 min on cold start)
|
||||
- Agentic AI correction (1-3 min)
|
||||
"""
|
||||
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS == 1200, "Transcription timeout should be 20 minutes (1200 seconds)"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_asyncio_wait_for_raises_timeout_error(self):
|
||||
"""
|
||||
Verify asyncio.wait_for properly raises TimeoutError.
|
||||
This is a sanity check for the timeout mechanism we use.
|
||||
"""
|
||||
async def slow_operation():
|
||||
await asyncio.sleep(10)
|
||||
return "completed"
|
||||
|
||||
with pytest.raises(asyncio.TimeoutError):
|
||||
await asyncio.wait_for(slow_operation(), timeout=0.1)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lyrics_worker_timeout_converts_to_descriptive_error(self):
|
||||
"""
|
||||
Verify that timeout is converted to a descriptive exception message.
|
||||
Tests the actual error conversion logic pattern used in lyrics_worker.
|
||||
"""
|
||||
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
|
||||
# This simulates what happens in lyrics_worker when timeout occurs
|
||||
error_message = None
|
||||
try:
|
||||
# Simulate the timeout handling pattern from lyrics_worker
|
||||
raise asyncio.TimeoutError()
|
||||
except asyncio.TimeoutError:
|
||||
error_message = f"Transcription timed out after {TRANSCRIPTION_TIMEOUT_SECONDS} seconds"
|
||||
|
||||
assert error_message is not None
|
||||
assert "timed out" in error_message.lower()
|
||||
assert "1200" in error_message
|
||||
|
||||
def test_lyrics_worker_exception_marks_job_failed(self):
|
||||
"""
|
||||
Any exception in lyrics worker should mark job as failed.
|
||||
"""
|
||||
from backend.models.job import JobStatus
|
||||
|
||||
# Simulate the exception handling in lyrics_worker
|
||||
job_status = JobStatus.DOWNLOADING
|
||||
error_occurred = True
|
||||
|
||||
if error_occurred:
|
||||
# This is what happens in the except block
|
||||
job_status = JobStatus.FAILED
|
||||
error_message = "Lyrics transcription failed: Transcription timed out after 1200 seconds"
|
||||
|
||||
assert job_status == JobStatus.FAILED
|
||||
assert "timed out" in error_message.lower()
|
||||
|
||||
|
||||
class TestJobOwnershipFiltering:
|
||||
"""
|
||||
Additional tests for job ownership and filtering.
|
||||
|
||||
These verify that jobs are properly associated with users
|
||||
and can be filtered by user_email.
|
||||
"""
|
||||
|
||||
def test_job_model_has_user_email_field(self):
|
||||
"""Job model must have user_email field."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
artist="Test",
|
||||
title="Test",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert hasattr(job, 'user_email')
|
||||
assert job.user_email == "user@example.com"
|
||||
|
||||
def test_job_create_model_has_user_email_field(self):
|
||||
"""JobCreate model must have user_email field."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test",
|
||||
title="Test",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert hasattr(job_create, 'user_email')
|
||||
assert job_create.user_email == "user@example.com"
|
||||
|
||||
def test_job_create_user_email_is_optional(self):
|
||||
"""JobCreate user_email should be optional for backward compatibility."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test",
|
||||
title="Test",
|
||||
)
|
||||
|
||||
assert job_create.user_email is None
|
||||
|
||||
def test_jobs_can_be_filtered_by_user_email(self):
|
||||
"""Jobs should be filterable by user_email."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
jobs = [
|
||||
Job(
|
||||
job_id="job-1",
|
||||
artist="Test",
|
||||
title="Test 1",
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user1@example.com",
|
||||
),
|
||||
Job(
|
||||
job_id="job-2",
|
||||
artist="Test",
|
||||
title="Test 2",
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user2@example.com",
|
||||
),
|
||||
Job(
|
||||
job_id="job-3",
|
||||
artist="Test",
|
||||
title="Test 3",
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user1@example.com",
|
||||
),
|
||||
]
|
||||
|
||||
user1_jobs = [j for j in jobs if j.user_email == "user1@example.com"]
|
||||
|
||||
assert len(user1_jobs) == 2
|
||||
assert all(j.user_email == "user1@example.com" for j in user1_jobs)
|
||||
@@ -36,6 +36,29 @@ from karaoke_gen.audio_processor import AudioProcessor
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def _trigger_lyrics_worker_after_url_download(job_id: str) -> None:
|
||||
"""
|
||||
Trigger lyrics worker after URL audio download completes.
|
||||
|
||||
For URL jobs, we use sequential triggering:
|
||||
1. Audio worker downloads and uploads audio to GCS
|
||||
2. Audio worker triggers lyrics worker (this function)
|
||||
3. Both workers then proceed in parallel (audio separation + lyrics transcription)
|
||||
|
||||
This prevents the race condition where lyrics worker times out waiting for audio.
|
||||
"""
|
||||
from backend.services.worker_service import get_worker_service
|
||||
|
||||
try:
|
||||
worker_service = get_worker_service()
|
||||
await worker_service.trigger_lyrics_worker(job_id)
|
||||
logger.info(f"Job {job_id}: Triggered lyrics worker after URL download")
|
||||
except Exception as e:
|
||||
# Log but don't fail - audio processing can still continue
|
||||
# The job will eventually timeout if lyrics worker doesn't run
|
||||
logger.error(f"Job {job_id}: Failed to trigger lyrics worker: {e}")
|
||||
|
||||
|
||||
# Default model names - used by create_audio_processor and stored in state_data
|
||||
DEFAULT_CLEAN_MODEL = "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
DEFAULT_BACKING_MODELS = ["mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"]
|
||||
@@ -445,29 +468,34 @@ async def download_audio(
|
||||
# Case 3: Fresh URL that needs downloading
|
||||
if job.url:
|
||||
logger.info(f"Job {job_id}: Downloading from URL: {job.url}")
|
||||
|
||||
|
||||
# Use provided job_manager or create new one
|
||||
jm = job_manager_instance or JobManager()
|
||||
|
||||
|
||||
local_path = await download_from_url(
|
||||
job.url,
|
||||
temp_dir,
|
||||
job.artist,
|
||||
job.url,
|
||||
temp_dir,
|
||||
job.artist,
|
||||
job.title,
|
||||
job_manager=jm,
|
||||
job_id=job_id
|
||||
)
|
||||
|
||||
|
||||
if local_path and os.path.exists(local_path):
|
||||
# Upload to GCS and update job
|
||||
gcs_path = f"jobs/{job_id}/input/{os.path.basename(local_path)}"
|
||||
url = storage.upload_file(local_path, gcs_path)
|
||||
|
||||
|
||||
# Update job with GCS path for lyrics worker
|
||||
jm.update_job(job_id, {'input_media_gcs_path': gcs_path})
|
||||
jm.update_file_url(job_id, 'input', 'audio', url)
|
||||
|
||||
|
||||
logger.info(f"Job {job_id}: Downloaded and uploaded audio to GCS: {gcs_path}")
|
||||
|
||||
# For URL jobs, trigger lyrics worker now that audio is available
|
||||
# This is the sequential trigger pattern - audio first, then lyrics
|
||||
await _trigger_lyrics_worker_after_url_download(job_id)
|
||||
|
||||
return local_path
|
||||
else:
|
||||
logger.error(f"Job {job_id}: Failed to download from URL: {job.url}")
|
||||
|
||||
@@ -32,6 +32,7 @@ from backend.models.job import JobStatus
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.lyrics_cache_service import LyricsCacheService
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
from backend.workers.style_helper import load_style_config
|
||||
from backend.services.tracing import job_span, add_span_event, add_span_attribute
|
||||
@@ -44,6 +45,20 @@ from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Timeout for entire transcription process including agentic AI correction (20 minutes)
|
||||
# This needs to account for:
|
||||
# - Cloud Run cold start / worker initialization (1-5 min)
|
||||
# - AudioShake transcription (1-2 min)
|
||||
# - spaCy model loading for correction (2-3 min on cold start)
|
||||
# - Agentic AI correction (1-3 min)
|
||||
TRANSCRIPTION_TIMEOUT_SECONDS = 1200
|
||||
|
||||
# Default agentic correction timeout (3 minutes)
|
||||
# Configurable via AGENTIC_CORRECTION_TIMEOUT_SECONDS environment variable
|
||||
# If agentic AI takes longer than this, abort and use uncorrected transcription
|
||||
# Human review will correct any issues
|
||||
DEFAULT_AGENTIC_TIMEOUT_SECONDS = 180
|
||||
|
||||
|
||||
def _configure_agentic_ai():
|
||||
"""Configure environment variables for agentic AI correction.
|
||||
@@ -215,7 +230,40 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
raise Exception("Failed to download audio file")
|
||||
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
|
||||
download_span.set_attribute("audio_file", os.path.basename(audio_path))
|
||||
|
||||
|
||||
# Set up LyricsTranscriber cache directory and sync from GCS
|
||||
# This allows reusing cached AudioShake/lyrics API responses across Cloud Run instances
|
||||
cache_dir = os.path.join(temp_dir, "lyrics-cache")
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
os.environ["LYRICS_TRANSCRIBER_CACHE_DIR"] = cache_dir
|
||||
job_log.info(f"LyricsTranscriber cache dir: {cache_dir}")
|
||||
|
||||
# Sync cache from GCS (download any existing cached API responses)
|
||||
# Note: Cache sync is best-effort - failures should not fail the job
|
||||
cache_service = LyricsCacheService(storage)
|
||||
audio_hash = None
|
||||
lyrics_hash = None
|
||||
with job_span("sync-cache-from-gcs", job_id):
|
||||
try:
|
||||
# Compute cache keys
|
||||
audio_hash = cache_service.compute_audio_hash(audio_path)
|
||||
# Use lyrics_artist/lyrics_title for lyrics hash (these are what LyricsTranscriber uses)
|
||||
lyrics_search_artist = getattr(job, 'lyrics_artist', None) or job.artist
|
||||
lyrics_search_title = getattr(job, 'lyrics_title', None) or job.title
|
||||
lyrics_hash = cache_service.compute_lyrics_hash(lyrics_search_artist, lyrics_search_title)
|
||||
|
||||
job_log.info(f"Cache keys: audio_hash={audio_hash[:8]}..., lyrics_hash={lyrics_hash[:8]}...")
|
||||
add_span_attribute("audio_hash", audio_hash[:8])
|
||||
add_span_attribute("lyrics_hash", lyrics_hash[:8])
|
||||
|
||||
# Download relevant cache files from GCS
|
||||
cache_stats = cache_service.sync_cache_from_gcs(cache_dir, audio_hash, lyrics_hash)
|
||||
job_log.info(f"Cache sync from GCS: {cache_stats['downloaded']} downloaded, {cache_stats['not_found']} not found")
|
||||
add_span_attribute("cache_hits", cache_stats["downloaded"])
|
||||
except Exception as e:
|
||||
job_log.warning(f"Cache sync from GCS failed (non-fatal): {e}", exc_info=True)
|
||||
add_span_attribute("cache_sync_error", str(e)[:100])
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
# The status is managed at a higher level - workers just track their progress
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
@@ -285,26 +333,73 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
if lyrics_search_title != job.title:
|
||||
job_log.info(f" Lyrics search title override: {lyrics_search_title}")
|
||||
logger.info(f"[job:{job_id}] Calling lyrics_processor.transcribe_lyrics()")
|
||||
|
||||
# Run transcription + correction
|
||||
|
||||
# Run transcription + correction with timeout
|
||||
# AudioShake typically takes 1-2 minutes, but can hang.
|
||||
with job_span("audioshake-transcription", job_id) as trans_span:
|
||||
trans_start = time.time()
|
||||
add_span_event("transcription_started")
|
||||
|
||||
# Run transcription in thread pool to avoid blocking the event loop
|
||||
# (transcribe_lyrics is synchronous and takes 1-2 minutes)
|
||||
# Note: transcribe_lyrics internally calls AudioShake and lyrics APIs
|
||||
with metrics.time_external_api("audioshake", job_id):
|
||||
result = await asyncio.to_thread(
|
||||
lyrics_processor.transcribe_lyrics,
|
||||
input_audio_wav=audio_path,
|
||||
artist=job.artist, # Original artist for file naming
|
||||
title=job.title, # Original title for file naming
|
||||
track_output_dir=temp_dir,
|
||||
lyrics_artist=lyrics_search_artist, # Override for lyrics search
|
||||
lyrics_title=lyrics_search_title # Override for lyrics search
|
||||
|
||||
# Calculate deadline for agentic correction
|
||||
# If agentic AI is enabled, the correction loop will check this deadline
|
||||
# and abort early if exceeded, returning uncorrected transcription for human review
|
||||
settings = get_settings()
|
||||
agentic_deadline = None
|
||||
timeout_seconds = settings.agentic_correction_timeout_seconds
|
||||
if settings.use_agentic_ai:
|
||||
agentic_deadline = time.time() + timeout_seconds
|
||||
job_log.info(
|
||||
f"Agentic AI enabled with {timeout_seconds}s timeout"
|
||||
)
|
||||
|
||||
|
||||
# Run transcription in thread pool
|
||||
# When agentic AI is enabled, wrap with outer timeout as safety net
|
||||
# Inner deadline check in corrector.py will break out of gap loop gracefully
|
||||
# Outer timeout catches case where single LLM call hangs for too long
|
||||
transcription_coro = asyncio.to_thread(
|
||||
lyrics_processor.transcribe_lyrics,
|
||||
input_audio_wav=audio_path,
|
||||
artist=job.artist, # Original artist for file naming
|
||||
title=job.title, # Original title for file naming
|
||||
track_output_dir=temp_dir,
|
||||
lyrics_artist=lyrics_search_artist, # Override for lyrics search
|
||||
lyrics_title=lyrics_search_title, # Override for lyrics search
|
||||
agentic_deadline=agentic_deadline, # Deadline for agentic timeout
|
||||
)
|
||||
|
||||
with metrics.time_external_api("audioshake", job_id):
|
||||
if settings.use_agentic_ai:
|
||||
# Outer timeout must be generous to allow for AudioShake time
|
||||
# (can take 2-5+ minutes for long songs) PLUS agentic correction.
|
||||
# The inner deadline check in corrector.py handles the 3-minute
|
||||
# correction limit gracefully by returning partial results.
|
||||
# This outer timeout is a safety net for completely hung LLM calls.
|
||||
outer_timeout = TRANSCRIPTION_TIMEOUT_SECONDS # Same as non-agentic
|
||||
try:
|
||||
result = await asyncio.wait_for(
|
||||
transcription_coro,
|
||||
timeout=outer_timeout
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
# This should rarely trigger since inner deadline check breaks first
|
||||
# But provides hard stop if e.g. single LLM call hangs completely
|
||||
job_log.error(
|
||||
f"HARD TIMEOUT: Transcription exceeded {outer_timeout}s. "
|
||||
"This indicates the inner deadline check failed to trigger."
|
||||
)
|
||||
raise RuntimeError(
|
||||
f"Lyrics transcription timed out after {outer_timeout}s"
|
||||
) from None
|
||||
else:
|
||||
# Non-agentic mode: use general timeout (10 minutes)
|
||||
try:
|
||||
result = await asyncio.wait_for(
|
||||
transcription_coro,
|
||||
timeout=TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
raise Exception(f"Transcription timed out after {TRANSCRIPTION_TIMEOUT_SECONDS} seconds")
|
||||
|
||||
trans_duration = time.time() - trans_start
|
||||
trans_span.set_attribute("duration_seconds", trans_duration)
|
||||
add_span_event("transcription_completed", {"duration_seconds": trans_duration})
|
||||
@@ -319,7 +414,23 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
|
||||
job_log.info("All lyrics data uploaded successfully")
|
||||
logger.info(f"[job:{job_id}] All lyrics data uploaded successfully")
|
||||
|
||||
|
||||
# Sync new cache files to GCS (upload any newly created cache entries)
|
||||
# This persists cache across Cloud Run instances for future jobs
|
||||
# Note: Cache upload is best-effort - failures should not fail the job
|
||||
with job_span("sync-cache-to-gcs", job_id):
|
||||
try:
|
||||
if audio_hash and lyrics_hash:
|
||||
upload_stats = cache_service.sync_cache_to_gcs(cache_dir, audio_hash, lyrics_hash)
|
||||
job_log.info(f"Cache sync to GCS: {upload_stats['uploaded']} uploaded, {upload_stats['skipped']} already existed")
|
||||
add_span_attribute("cache_uploads", upload_stats["uploaded"])
|
||||
else:
|
||||
job_log.warning("Skipping cache upload: missing audio_hash or lyrics_hash")
|
||||
add_span_attribute("cache_upload_skipped", "missing_hashes")
|
||||
except Exception as e:
|
||||
job_log.warning(f"Cache upload to GCS failed (non-fatal): {e}")
|
||||
add_span_attribute("cache_upload_error", str(e)[:100])
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
'stage': 'lyrics_complete',
|
||||
|
||||
+216
-74
@@ -35,6 +35,7 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.rclone_service import get_rclone_service
|
||||
from backend.services.youtube_service import get_youtube_service
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.config import get_settings
|
||||
from backend.workers.style_helper import load_style_config
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
@@ -55,6 +56,122 @@ VIDEO_WORKER_LOGGERS = [
|
||||
]
|
||||
|
||||
|
||||
async def _encode_via_gce(
|
||||
job_id: str,
|
||||
job,
|
||||
job_manager: JobManager,
|
||||
storage: StorageService,
|
||||
temp_dir: str,
|
||||
base_name: str,
|
||||
job_log,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Encode videos using the high-performance GCE encoding worker.
|
||||
|
||||
This offloads FFmpeg encoding to a dedicated C4 GCE instance with
|
||||
Intel Granite Rapids 3.9 GHz CPU for 2-3x faster encoding.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
job: Job object
|
||||
job_manager: Job manager instance
|
||||
storage: Storage service instance
|
||||
temp_dir: Temporary directory for output files
|
||||
base_name: Base name for output files (e.g., "Artist - Title")
|
||||
job_log: Job-specific logger
|
||||
|
||||
Returns:
|
||||
Result dict with paths to encoded files
|
||||
"""
|
||||
settings = get_settings()
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
job_log.info("Using GCE encoding worker for high-performance encoding")
|
||||
logger.info(f"[job:{job_id}] Using GCE encoding worker")
|
||||
|
||||
# Construct GCS paths for the encoding worker
|
||||
bucket_name = settings.gcs_bucket_name
|
||||
input_gcs_path = f"gs://{bucket_name}/jobs/{job_id}/"
|
||||
output_gcs_path = f"gs://{bucket_name}/jobs/{job_id}/encoded/"
|
||||
|
||||
# Determine instrumental selection for encoding config
|
||||
instrumental_selection = job.state_data.get('instrumental_selection', 'clean')
|
||||
existing_instrumental = getattr(job, 'existing_instrumental_gcs_path', None)
|
||||
|
||||
encoding_config = {
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mp4_720p"],
|
||||
"base_name": base_name,
|
||||
"instrumental_selection": instrumental_selection,
|
||||
"existing_instrumental": existing_instrumental,
|
||||
"ffmpeg_threads": 8, # c4-standard-8 has 8 vCPUs
|
||||
}
|
||||
|
||||
job_log.info(f"Submitting encoding job to GCE worker")
|
||||
job_log.info(f" Input: {input_gcs_path}")
|
||||
job_log.info(f" Output: {output_gcs_path}")
|
||||
|
||||
def progress_callback(progress: int):
|
||||
# Update job progress (encoding is 75-95% of total)
|
||||
scaled_progress = 75 + int(progress * 0.2) # Map 0-100 to 75-95
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.ENCODING,
|
||||
progress=scaled_progress,
|
||||
message=f"Encoding videos ({progress}%)"
|
||||
)
|
||||
|
||||
try:
|
||||
# Submit and wait for encoding
|
||||
with job_span("gce-encoding", job_id) as encoding_span:
|
||||
encode_start = time.time()
|
||||
add_span_event("gce_encoding_started")
|
||||
|
||||
result = await encoding_service.encode_videos(
|
||||
job_id=job_id,
|
||||
input_gcs_path=input_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
encoding_config=encoding_config,
|
||||
progress_callback=progress_callback,
|
||||
)
|
||||
|
||||
encode_duration = time.time() - encode_start
|
||||
encoding_span.set_attribute("duration_seconds", encode_duration)
|
||||
add_span_event("gce_encoding_completed", {"duration_seconds": encode_duration})
|
||||
|
||||
job_log.info(f"GCE encoding complete in {encode_duration:.1f}s")
|
||||
logger.info(f"[job:{job_id}] GCE encoding complete in {encode_duration:.1f}s")
|
||||
|
||||
# Download encoded files from GCS to temp_dir
|
||||
output_files = result.get("output_files", [])
|
||||
local_files = {}
|
||||
|
||||
for gcs_path in output_files:
|
||||
filename = os.path.basename(gcs_path)
|
||||
local_path = os.path.join(temp_dir, filename)
|
||||
|
||||
job_log.info(f"Downloading encoded file: {filename}")
|
||||
storage.download_file(gcs_path, local_path)
|
||||
|
||||
# Map to result keys expected by _upload_results
|
||||
if "4K Lossless" in filename or "4k_lossless" in filename.lower():
|
||||
local_files["final_video"] = local_path
|
||||
elif "4K Lossy" in filename or "4k_lossy" in filename.lower():
|
||||
local_files["final_video_lossy"] = local_path
|
||||
elif "720p" in filename:
|
||||
local_files["final_video_720p"] = local_path
|
||||
elif filename.endswith(".mkv"):
|
||||
local_files["final_video_mkv"] = local_path
|
||||
|
||||
job_log.info(f"Downloaded {len(local_files)} encoded files")
|
||||
|
||||
return local_files
|
||||
|
||||
except Exception as e:
|
||||
job_log.error(f"GCE encoding failed: {e}")
|
||||
logger.error(f"[job:{job_id}] GCE encoding failed: {e}")
|
||||
raise
|
||||
|
||||
|
||||
async def generate_video(job_id: str) -> bool:
|
||||
"""
|
||||
Generate final karaoke videos using KaraokeFinalise.
|
||||
@@ -184,80 +301,105 @@ async def generate_video(job_id: str) -> bool:
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
job_log.info(f"Countdown padding detected: {countdown_padding_seconds}s - instrumental will be padded if needed")
|
||||
|
||||
# Log finalization parameters
|
||||
job_log.info("Creating KaraokeFinalise with parameters:")
|
||||
job_log.info(f" enable_cdg: {getattr(job, 'enable_cdg', False)}")
|
||||
job_log.info(f" enable_txt: {getattr(job, 'enable_txt', False)}")
|
||||
job_log.info(f" brand_prefix: {getattr(job, 'brand_prefix', None)}")
|
||||
job_log.info(f" discord_webhook: {'configured' if getattr(job, 'discord_webhook_url', None) else 'not configured'}")
|
||||
job_log.info(f" instrumental source: {instrumental_source}")
|
||||
job_log.info(f" countdown_padding_seconds: {countdown_padding_seconds}")
|
||||
if existing_instrumental_path:
|
||||
job_log.info(f" using user-provided instrumental (selection was: {instrumental_selection})")
|
||||
|
||||
# Create KaraokeFinalise with ALL the parameters from the job
|
||||
# This reuses all existing functionality!
|
||||
|
||||
# Set up YouTube description file if template is provided
|
||||
youtube_desc_path = None
|
||||
if youtube_credentials and getattr(job, 'youtube_description_template', None):
|
||||
youtube_desc_path = os.path.join(temp_dir, "youtube_description.txt")
|
||||
with open(youtube_desc_path, 'w') as f:
|
||||
f.write(job.youtube_description_template)
|
||||
job_log.info("YouTube description template written to temp file")
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
logger=logger,
|
||||
log_level=logging.INFO,
|
||||
dry_run=False,
|
||||
instrumental_format="flac",
|
||||
# CDG/TXT generation
|
||||
enable_cdg=getattr(job, 'enable_cdg', False),
|
||||
enable_txt=getattr(job, 'enable_txt', False),
|
||||
cdg_styles=cdg_styles,
|
||||
# Brand code and organization (server-side mode uses rclone)
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
organised_dir=None, # Not used in cloud - files stay in GCS
|
||||
organised_dir_rclone_root=getattr(job, 'organised_dir_rclone_root', None),
|
||||
public_share_dir=None, # Not used in cloud
|
||||
# Notifications
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
# YouTube upload (server-side with pre-loaded credentials)
|
||||
youtube_client_secrets_file=None, # Not used with pre-stored credentials
|
||||
youtube_description_file=youtube_desc_path,
|
||||
user_youtube_credentials=youtube_credentials, # Pre-loaded from Secret Manager
|
||||
rclone_destination=None,
|
||||
email_template_file=None,
|
||||
# Server-side optimizations
|
||||
non_interactive=True,
|
||||
server_side_mode=True,
|
||||
selected_instrumental_file=instrumental_file,
|
||||
# Audio synchronization - ensure instrumental matches vocal padding
|
||||
countdown_padding_seconds=countdown_padding_seconds,
|
||||
)
|
||||
|
||||
# Call process() - this does ALL the work:
|
||||
# - Encodes to 4 video formats
|
||||
# - Generates CDG/TXT packages if enabled
|
||||
# - Posts Discord notification if configured
|
||||
# - Handles brand code generation
|
||||
with job_span("karaoke-finalise", job_id) as finalise_span:
|
||||
finalise_start = time.time()
|
||||
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
|
||||
logger.info(f"[job:{job_id}] Starting KaraokeFinalise.process()")
|
||||
add_span_event("finalise_started")
|
||||
|
||||
result = finalise.process()
|
||||
|
||||
finalise_duration = time.time() - finalise_start
|
||||
finalise_span.set_attribute("duration_seconds", finalise_duration)
|
||||
add_span_event("finalise_completed", {"duration_seconds": finalise_duration})
|
||||
|
||||
job_log.info("KaraokeFinalise.process() complete!")
|
||||
if result.get('brand_code'):
|
||||
job_log.info(f"Brand code: {result.get('brand_code')}")
|
||||
logger.info(f"[job:{job_id}] KaraokeFinalise.process() complete in {finalise_duration:.1f}s")
|
||||
logger.info(f"[job:{job_id}] Brand code: {result.get('brand_code')}")
|
||||
# Check if GCE encoding is enabled for high-performance encoding
|
||||
encoding_service = get_encoding_service()
|
||||
use_gce_encoding = encoding_service.is_enabled
|
||||
|
||||
if use_gce_encoding:
|
||||
# ============ GCE ENCODING PATH ============
|
||||
# Use dedicated C4 GCE instance for 2-3x faster FFmpeg encoding
|
||||
job_log.info("GCE encoding enabled - using high-performance encoding worker")
|
||||
job_log.info(f" instrumental source: {instrumental_source}")
|
||||
|
||||
result = await _encode_via_gce(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
job_manager=job_manager,
|
||||
storage=storage,
|
||||
temp_dir=temp_dir,
|
||||
base_name=base_name,
|
||||
job_log=job_log,
|
||||
)
|
||||
|
||||
# GCE encoding doesn't generate brand_code, CDG/TXT, or Discord notifications
|
||||
# These will be handled by distribution if needed
|
||||
job_log.info("GCE encoding complete - encoded files downloaded")
|
||||
|
||||
else:
|
||||
# ============ STANDARD ENCODING PATH ============
|
||||
# Use KaraokeFinalise for encoding (runs on Cloud Run)
|
||||
|
||||
# Log finalization parameters
|
||||
job_log.info("Using KaraokeFinalise for encoding (standard path)")
|
||||
job_log.info(f" enable_cdg: {getattr(job, 'enable_cdg', False)}")
|
||||
job_log.info(f" enable_txt: {getattr(job, 'enable_txt', False)}")
|
||||
job_log.info(f" brand_prefix: {getattr(job, 'brand_prefix', None)}")
|
||||
job_log.info(f" discord_webhook: {'configured' if getattr(job, 'discord_webhook_url', None) else 'not configured'}")
|
||||
job_log.info(f" instrumental source: {instrumental_source}")
|
||||
job_log.info(f" countdown_padding_seconds: {countdown_padding_seconds}")
|
||||
if existing_instrumental_path:
|
||||
job_log.info(f" using user-provided instrumental (selection was: {instrumental_selection})")
|
||||
|
||||
# Set up YouTube description file if template is provided
|
||||
youtube_desc_path = None
|
||||
if youtube_credentials and getattr(job, 'youtube_description_template', None):
|
||||
youtube_desc_path = os.path.join(temp_dir, "youtube_description.txt")
|
||||
with open(youtube_desc_path, 'w') as f:
|
||||
f.write(job.youtube_description_template)
|
||||
job_log.info("YouTube description template written to temp file")
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
logger=logger,
|
||||
log_level=logging.INFO,
|
||||
dry_run=False,
|
||||
instrumental_format="flac",
|
||||
# CDG/TXT generation
|
||||
enable_cdg=getattr(job, 'enable_cdg', False),
|
||||
enable_txt=getattr(job, 'enable_txt', False),
|
||||
cdg_styles=cdg_styles,
|
||||
# Brand code and organization (server-side mode uses rclone)
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
organised_dir=None, # Not used in cloud - files stay in GCS
|
||||
organised_dir_rclone_root=getattr(job, 'organised_dir_rclone_root', None),
|
||||
public_share_dir=None, # Not used in cloud
|
||||
# Notifications
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
# YouTube upload (server-side with pre-loaded credentials)
|
||||
youtube_client_secrets_file=None, # Not used with pre-stored credentials
|
||||
youtube_description_file=youtube_desc_path,
|
||||
user_youtube_credentials=youtube_credentials, # Pre-loaded from Secret Manager
|
||||
rclone_destination=None,
|
||||
email_template_file=None,
|
||||
# Server-side optimizations
|
||||
non_interactive=True,
|
||||
server_side_mode=True,
|
||||
selected_instrumental_file=instrumental_file,
|
||||
# Audio synchronization - ensure instrumental matches vocal padding
|
||||
countdown_padding_seconds=countdown_padding_seconds,
|
||||
)
|
||||
|
||||
# Call process() - this does ALL the work:
|
||||
# - Encodes to 4 video formats
|
||||
# - Generates CDG/TXT packages if enabled
|
||||
# - Posts Discord notification if configured
|
||||
# - Handles brand code generation
|
||||
with job_span("karaoke-finalise", job_id) as finalise_span:
|
||||
finalise_start = time.time()
|
||||
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
|
||||
logger.info(f"[job:{job_id}] Starting KaraokeFinalise.process()")
|
||||
add_span_event("finalise_started")
|
||||
|
||||
result = finalise.process()
|
||||
|
||||
finalise_duration = time.time() - finalise_start
|
||||
finalise_span.set_attribute("duration_seconds", finalise_duration)
|
||||
add_span_event("finalise_completed", {"duration_seconds": finalise_duration})
|
||||
|
||||
job_log.info("KaraokeFinalise.process() complete!")
|
||||
if result.get('brand_code'):
|
||||
job_log.info(f"Brand code: {result.get('brand_code')}")
|
||||
logger.info(f"[job:{job_id}] KaraokeFinalise.process() complete in {finalise_duration:.1f}s")
|
||||
logger.info(f"[job:{job_id}] Brand code: {result.get('brand_code')}")
|
||||
|
||||
# Native API distribution uploads (used by remote CLI instead of rclone)
|
||||
with job_span("distribution", job_id):
|
||||
|
||||
+1
-1
@@ -75,7 +75,7 @@ steps:
|
||||
--concurrency 1 \
|
||||
--max-instances 50 \
|
||||
--min-instances 0 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
|
||||
|
||||
images:
|
||||
|
||||
+111
@@ -272,6 +272,117 @@ Content-Type: application/json
|
||||
|
||||
Bonus credit for detailed feedback (50+ chars).
|
||||
|
||||
## Admin Endpoints
|
||||
|
||||
All admin endpoints require an admin-role session token.
|
||||
|
||||
### Admin Dashboard Stats
|
||||
|
||||
```http
|
||||
GET /api/admin/stats/overview
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Returns platform statistics:
|
||||
```json
|
||||
{
|
||||
"total_users": 100,
|
||||
"active_users_7d": 25,
|
||||
"active_users_30d": 60,
|
||||
"total_jobs": 500,
|
||||
"jobs_last_7d": 50,
|
||||
"jobs_last_30d": 150,
|
||||
"jobs_by_status": {
|
||||
"pending": 5,
|
||||
"processing": 3,
|
||||
"awaiting_review": 10,
|
||||
"complete": 400,
|
||||
"failed": 20
|
||||
},
|
||||
"total_credits_issued_30d": 200,
|
||||
"total_beta_testers": 30
|
||||
}
|
||||
```
|
||||
|
||||
### List Users (Admin)
|
||||
|
||||
```http
|
||||
GET /api/users/admin/users
|
||||
GET /api/users/admin/users?search=user@example.com
|
||||
GET /api/users/admin/users?limit=20&offset=0&sort_by=created_at&sort_order=desc
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Returns paginated user list with total count.
|
||||
|
||||
### User Detail (Admin)
|
||||
|
||||
```http
|
||||
GET /api/users/admin/users/{email}/detail
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Returns full user profile including:
|
||||
- User info and stats
|
||||
- Recent credit transactions (last 20)
|
||||
- Recent jobs (last 10)
|
||||
- Active sessions count
|
||||
|
||||
### Add Credits (Admin)
|
||||
|
||||
```http
|
||||
POST /api/users/admin/credits
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"email": "user@example.com",
|
||||
"amount": 5,
|
||||
"reason": "Beta reward"
|
||||
}
|
||||
```
|
||||
|
||||
### Set User Role (Admin)
|
||||
|
||||
```http
|
||||
POST /api/users/admin/users/{email}/role
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
Content-Type: application/json
|
||||
|
||||
{"role": "admin"}
|
||||
```
|
||||
|
||||
### Enable/Disable User (Admin)
|
||||
|
||||
```http
|
||||
POST /api/users/admin/users/{email}/enable
|
||||
POST /api/users/admin/users/{email}/disable
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
### Beta Stats (Admin)
|
||||
|
||||
```http
|
||||
GET /api/users/admin/beta/stats
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
### Beta Feedback List (Admin)
|
||||
|
||||
```http
|
||||
GET /api/users/admin/beta/feedback?limit=50
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
### Delete Job (Admin)
|
||||
|
||||
```http
|
||||
DELETE /api/jobs/{job_id}?delete_files=true
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Admins can delete any job. Regular users can only delete their own jobs.
|
||||
|
||||
## Rate Limits
|
||||
|
||||
No rate limits currently implemented.
|
||||
|
||||
@@ -23,12 +23,15 @@
|
||||
│ • Video Worker - Final assembly and encoding │
|
||||
└───────────────────────────┬─────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────┼─────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────────┐
|
||||
│ Firestore│ │ GCS │ │Secret Manager│
|
||||
│ (jobs) │ │ (files) │ │ (API keys) │
|
||||
└──────────┘ └──────────┘ └──────────────┘
|
||||
┌─────────────┼─────────────┬───────────────┐
|
||||
▼ ▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ Firestore│ │ GCS │ │Secret Manager│ │GCE Encoding │
|
||||
│ (jobs) │ │ (files) │ │ (API keys) │ │ Worker (*) │
|
||||
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
|
||||
|
||||
(*) GCE Encoding Worker: C4-standard-8 VM with Intel Granite Rapids for
|
||||
high-performance FFmpeg encoding. Optional - falls back to Cloud Run.
|
||||
```
|
||||
|
||||
## Processing Pipeline
|
||||
|
||||
+11
-1
@@ -151,7 +151,11 @@ ADMIN_TOKENS=token1,token2
|
||||
|
||||
```bash
|
||||
NEXT_PUBLIC_API_URL=http://localhost:8000
|
||||
KARAOKE_ACCESS_TOKEN=... # For E2E tests
|
||||
|
||||
# For E2E tests (see frontend/.env.local.example)
|
||||
KARAOKE_ACCESS_TOKEN=... # Skip email enrollment
|
||||
TESTMAIL_API_KEY=... # testmail.app API key
|
||||
TESTMAIL_NAMESPACE=... # testmail.app namespace
|
||||
```
|
||||
|
||||
## Key Scripts
|
||||
@@ -162,6 +166,12 @@ KARAOKE_ACCESS_TOKEN=... # For E2E tests
|
||||
| `./scripts/run-backend-local.sh` | Start backend with emulators |
|
||||
| `./scripts/start-emulators.sh` | Start emulators only |
|
||||
| `./scripts/stop-emulators.sh` | Stop emulators |
|
||||
| `./scripts/debug-job.sh <job_id>` | Debug a cloud job (status, logs, files) |
|
||||
| `./scripts/get_job.py <job_id>` | Fetch job data from Firestore |
|
||||
| `./scripts/fetch_job_logs.py <job_id>` | Fetch cloud job logs to local file |
|
||||
| `./scripts/analyze_log_timing.py <log>` | Profile karaoke-gen performance from logs |
|
||||
| `./scripts/benchmark_ffmpeg.py` | Isolated FFmpeg encoding benchmark |
|
||||
| `./scripts/compare_local_vs_remote.py` | Compare local vs cloud karaoke-gen performance |
|
||||
|
||||
## Project Structure
|
||||
|
||||
|
||||
@@ -0,0 +1,239 @@
|
||||
# E2E Happy Path Test Specification
|
||||
|
||||
> **STATUS**: ✅ **PASSING** - Test completes all 12 steps in ~38 minutes. Last verified 2026-01-02.
|
||||
>
|
||||
> **IMPORTANT**: This document defines the requirements for a comprehensive end-to-end test that validates the entire karaoke generation flow with REAL user interactions only. NO API shortcuts allowed.
|
||||
|
||||
## Overview
|
||||
|
||||
**Goal**: A single Playwright test that validates the complete karaoke generation journey from first-time user signup through to distribution, using only browser interactions like a real user would.
|
||||
|
||||
**Environment**: Production (`gen.nomadkaraoke.com` / `api.nomadkaraoke.com`)
|
||||
|
||||
**Automation**: Runs daily via GitHub Actions and can be triggered on-demand.
|
||||
|
||||
## Test Requirements
|
||||
|
||||
### 1. New User Registration (testmail.app)
|
||||
|
||||
- [ ] Navigate to landing page (`gen.nomadkaraoke.com`)
|
||||
- [ ] Click "Sign In" or "Get Started" button
|
||||
- [ ] Generate a unique test email address (testmail.app namespace)
|
||||
- [ ] Enter the test email in the beta enrollment form
|
||||
- [ ] Submit the form
|
||||
- [ ] Wait for welcome/magic-link email via testmail.app API
|
||||
- [ ] Extract the magic link from the email
|
||||
- [ ] Click/navigate to the magic link to authenticate
|
||||
- [ ] Verify user is logged in and has credits
|
||||
|
||||
**Note**: When `use_test_token=true` workflow input is set, steps 1-9 above are skipped and the test uses a pre-authenticated token instead.
|
||||
|
||||
### 2. Create Karaoke Job (UI Only)
|
||||
|
||||
- [ ] Navigate to the karaoke creation page
|
||||
- [ ] Enter artist name: `piri` (test song with cached results)
|
||||
- [ ] Enter song title: `dog`
|
||||
- [ ] Click "Search" or equivalent button
|
||||
- [ ] Wait for search results to appear in the UI
|
||||
|
||||
### 3. Audio Selection (UI Only)
|
||||
|
||||
- [ ] Audio selection dialog/panel should appear
|
||||
- [ ] View available audio options in the UI
|
||||
- [ ] Click to select the first/best audio option
|
||||
- [ ] Confirm the selection via UI button
|
||||
- [ ] Wait for job to transition to processing state
|
||||
|
||||
### 4. Wait for Lyrics Transcription
|
||||
|
||||
- [ ] Monitor job status in the UI (not via API polling)
|
||||
- [ ] Wait for job to reach "awaiting lyrics review" state
|
||||
- [ ] This may take 10-15+ minutes for audio download + transcription
|
||||
|
||||
### 5. Lyrics Review (FULL UI Interaction)
|
||||
|
||||
> **CRITICAL**: This MUST be done through the UI, not via API calls
|
||||
|
||||
- [ ] Click to open the lyrics review interface
|
||||
- [ ] Verify lyrics are displayed in the editor
|
||||
- [ ] Interact with the lyrics editor:
|
||||
- [ ] View synced lyrics with timestamps
|
||||
- [ ] Optionally make a small edit to prove UI interaction works
|
||||
- [ ] Preview the video if preview functionality exists
|
||||
- [ ] Click "Approve" / "Submit" / "Complete Review" button in UI
|
||||
- [ ] Wait for job to transition to next state
|
||||
|
||||
### 6. Instrumental Selection (FULL UI Interaction)
|
||||
|
||||
> **CRITICAL**: This MUST be done through the UI, not via API calls
|
||||
|
||||
- [ ] Wait for instrumental options to become available
|
||||
- [ ] Instrumental selection dialog/panel should appear in UI
|
||||
- [ ] View available instrumental options (e.g., "clean", "with backing vocals")
|
||||
- [ ] Click to select the preferred instrumental option
|
||||
- [ ] Confirm selection via UI button
|
||||
- [ ] Wait for job to transition to video rendering state
|
||||
|
||||
### 7. Wait for Video Rendering & Encoding
|
||||
|
||||
- [ ] Monitor job progress in the UI
|
||||
- [ ] Wait for video rendering to complete
|
||||
- [ ] Wait for final encoding to complete
|
||||
- [ ] Job should reach "complete" status
|
||||
|
||||
### 8. Verify Completion & Downloads
|
||||
|
||||
- [ ] Verify job shows as "complete" in the UI
|
||||
- [ ] Verify download links are visible and clickable
|
||||
- [ ] Click at least one download link to verify it works
|
||||
- [ ] Verify the downloaded file is valid (non-zero size, correct format)
|
||||
|
||||
### 9. Distribution Verification & Cleanup
|
||||
|
||||
> **CRITICAL**: Test MUST clean up after itself to avoid polluting production distribution channels
|
||||
|
||||
- [ ] Verify job.state_data contains distribution results:
|
||||
- [ ] `brand_code` - The generated brand code (e.g., "NOMAD-1234")
|
||||
- [ ] `youtube_url` - YouTube video URL (if YouTube enabled)
|
||||
- [ ] `dropbox_link` - Dropbox shared link (if Dropbox enabled)
|
||||
- [ ] `gdrive_files` - Google Drive file IDs (if GDrive enabled)
|
||||
- [ ] **CLEANUP** - After verification, delete all distributed content:
|
||||
- [ ] Delete YouTube video via API (using stored youtube_url)
|
||||
- [ ] Delete Dropbox folder via API (using brand_code + dropbox_path)
|
||||
- [ ] Delete Google Drive files via API (using gdrive_files IDs)
|
||||
- [ ] Delete the test job from the system (GCS + Firestore)
|
||||
|
||||
## Technical Requirements
|
||||
|
||||
### Timeouts
|
||||
|
||||
| Stage | Timeout |
|
||||
|-------|---------|
|
||||
| Page navigation | 30 seconds |
|
||||
| Email arrival | 120 seconds |
|
||||
| Audio search | 60 seconds |
|
||||
| Lyrics transcription | 20 minutes |
|
||||
| Video rendering | 15 minutes |
|
||||
| Final encoding | 10 minutes |
|
||||
| **Total test** | **60 minutes** |
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
TESTMAIL_NAMESPACE=xxx # Required - for email testing (testmail.app)
|
||||
TESTMAIL_API_KEY=xxx # Required - for email testing (testmail.app)
|
||||
E2E_TEST_TOKEN=xxx # Optional - skip signup, use existing account
|
||||
# When use_test_token=true, skips email signup and uses pre-authenticated token
|
||||
```
|
||||
|
||||
### Test Song
|
||||
|
||||
Use `piri - dog` because:
|
||||
- Has cached flacfetch results (faster audio search)
|
||||
- Known to work reliably
|
||||
- Short song = faster processing
|
||||
|
||||
### Screenshots & Artifacts
|
||||
|
||||
Capture screenshots at each major milestone:
|
||||
1. Landing page loaded
|
||||
2. Beta form submitted
|
||||
3. Email received
|
||||
4. Logged in successfully
|
||||
5. Job created
|
||||
6. Audio selected
|
||||
7. Lyrics review opened
|
||||
8. Lyrics approved
|
||||
9. Instrumental selected
|
||||
10. Job completed
|
||||
11. Download verified
|
||||
|
||||
### Logging
|
||||
|
||||
Log detailed progress at each step with timestamps for debugging.
|
||||
|
||||
## GitHub Actions Workflow
|
||||
|
||||
**File**: `.github/workflows/e2e-happy-path.yml`
|
||||
|
||||
### Triggers
|
||||
|
||||
1. **Daily schedule**: Run at 6 AM UTC every day
|
||||
2. **Manual dispatch**: Can be triggered on-demand with optional parameters
|
||||
3. **Label trigger**: When `e2e-full` label added to PR
|
||||
|
||||
### Secrets Required
|
||||
|
||||
- `TESTMAIL_API_KEY` - For email testing (testmail.app)
|
||||
- `TESTMAIL_NAMESPACE` - testmail.app namespace for unique email addresses
|
||||
|
||||
### Workflow Steps
|
||||
|
||||
1. Checkout code
|
||||
2. Setup Node.js
|
||||
3. Install dependencies
|
||||
4. Install Playwright browsers
|
||||
5. Run the e2e happy path test
|
||||
6. Upload artifacts (screenshots, videos, logs)
|
||||
7. Report results
|
||||
|
||||
### Success Criteria
|
||||
|
||||
- All UI interactions complete successfully
|
||||
- Job reaches "complete" status
|
||||
- Download link works
|
||||
- No JavaScript errors in console
|
||||
- All screenshots captured
|
||||
|
||||
## What This Test Proves
|
||||
|
||||
When this test passes, it proves:
|
||||
|
||||
1. **User signup works** - New users can register via magic link
|
||||
2. **Authentication works** - Email verification and login flow functional
|
||||
3. **Job creation works** - Can search and create karaoke jobs
|
||||
4. **Audio pipeline works** - Audio search, selection, and download functional
|
||||
5. **Transcription works** - Lyrics transcription completes successfully
|
||||
6. **Lyrics review UI works** - Users can review and approve lyrics
|
||||
7. **Instrumental selection UI works** - Users can select instrumental versions
|
||||
8. **Video rendering works** - Full video generation pipeline functional
|
||||
9. **Downloads work** - Users can download their completed karaoke videos
|
||||
10. **End-to-end integration** - All systems work together correctly
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
1. **NO API calls to complete review** - Must click buttons in UI
|
||||
2. **NO API calls to select instrumental** - Must click buttons in UI
|
||||
3. **NO pre-authenticated tokens** - Must go through full signup
|
||||
4. **NO skipping steps** - Every user-facing interaction must happen
|
||||
5. **NO hardcoded waits** - Use proper Playwright waiting strategies
|
||||
|
||||
## File Location
|
||||
|
||||
**Test file**: `frontend/e2e/production/happy-path-real-user.spec.ts`
|
||||
|
||||
This is separate from the existing `full-user-journey.spec.ts` which uses API shortcuts.
|
||||
|
||||
## Iteration Plan
|
||||
|
||||
1. ~~Create initial test with all steps stubbed~~
|
||||
2. ~~Implement each section one at a time~~
|
||||
3. ~~Run via GHA workflow~~
|
||||
4. ~~Debug failures using artifacts~~
|
||||
5. ~~Iterate until fully passing~~ ✅ Completed 2026-01-02
|
||||
6. Add to required CI checks once stable
|
||||
|
||||
## Key Fixes Applied
|
||||
|
||||
During iteration, three issues were fixed:
|
||||
|
||||
1. **STEP 8 status detection**: "rendering" status comes BEFORE instrumental selection, not after. Changed to only skip on "encoding".
|
||||
2. **Instrumental page navigation**: Changed `waitUntil: 'networkidle'` to `'load'` because audio players prevent networkidle.
|
||||
3. **Download verification**: Changed HEAD to GET request with `maxRedirects: 0` since download endpoint doesn't support HEAD.
|
||||
|
||||
See [LESSONS-LEARNED.md](LESSONS-LEARNED.md) for details.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-01-02*
|
||||
*Author: Claude (via Andrew's request)*
|
||||
@@ -32,6 +32,31 @@ Audio separation and lyrics transcription run in parallel:
|
||||
- Progress tracked separately in `state_data`
|
||||
- Screens generation waits for both to complete
|
||||
|
||||
### Cloud Run Performance: Cold Starts and FFmpeg
|
||||
|
||||
**Problem**: Cloud karaoke generation was ~3x slower than local CLI (15 min vs 5 min for a typical song).
|
||||
|
||||
**Root causes identified**:
|
||||
|
||||
1. **Cold start penalty**: Cloud Run with `min-instances: 0` caused 77-second cold starts for FFmpeg-heavy operations (title screen generation). Warm instances take only 6 seconds.
|
||||
|
||||
2. **FFmpeg version**: Debian's package (5.1.8) is outdated and built with security-focused `--toolchain=hardened` which reduces performance. Static builds (7.0.2) from johnvansickle.com are ~20-40% faster.
|
||||
|
||||
3. **Generic x86_64 build**: Debian packages use generic CPU instructions, not optimized for Cloud Run's Cascade Lake/Ice Lake processors.
|
||||
|
||||
**Solutions applied**:
|
||||
- Set `min-instances: 4` to keep workers warm (eliminates cold start entirely)
|
||||
- Use John Van Sickle's static FFmpeg 7.0.2 instead of Debian package
|
||||
- Added `/api/health/detailed` endpoint to expose FFmpeg version for debugging
|
||||
|
||||
**Key insight**: For CPU-bound encoding workloads on Cloud Run:
|
||||
- GPU acceleration often doesn't help (filters like libass aren't GPU-accelerated)
|
||||
- Newer FFmpeg versions have significant encoding optimizations
|
||||
- Static builds with CPU-specific tuning outperform generic distribution packages
|
||||
- Cold starts hurt more than expected for initialization-heavy binaries like FFmpeg
|
||||
|
||||
**Debugging**: Use `scripts/compare_local_vs_remote.py` to benchmark local vs cloud performance.
|
||||
|
||||
## Common Gotchas
|
||||
|
||||
### Cross-Domain localStorage Isolation
|
||||
@@ -295,6 +320,77 @@ await expect(page.getByText('Hello,')).toBeVisible({ timeout: 5000 });
|
||||
|
||||
**Why it matters**: Playwright assertions have built-in retry logic and fail fast when conditions aren't met. Arbitrary timeouts either waste time (waiting when condition is already met) or cause flaky tests (not waiting long enough on slow CI).
|
||||
|
||||
### Playwright: Use 'load' Instead of 'networkidle' for Audio-Heavy Pages
|
||||
|
||||
**Problem**: Pages with audio players or ongoing audio network requests never reach `networkidle` state.
|
||||
|
||||
```typescript
|
||||
// BAD - hangs indefinitely on pages with audio players
|
||||
await page.goto(url, { waitUntil: 'networkidle' });
|
||||
```
|
||||
|
||||
**Symptom**: Test times out after 30-45 minutes waiting for navigation to complete, even though the page is fully rendered and usable.
|
||||
|
||||
**Solution**: Use `waitUntil: 'load'` plus a small fixed wait if needed:
|
||||
|
||||
```typescript
|
||||
// GOOD - works with audio-heavy pages
|
||||
await page.goto(url, { waitUntil: 'load' });
|
||||
await page.waitForTimeout(3000); // Allow JS to initialize
|
||||
```
|
||||
|
||||
**Root cause**: Audio players make continuous network requests for streaming/buffering, preventing the network from ever becoming "idle".
|
||||
|
||||
### Playwright: HEAD Requests May Not Be Supported
|
||||
|
||||
**Problem**: Using `page.request.head()` to verify download URLs returns 405 Method Not Allowed on some endpoints.
|
||||
|
||||
```typescript
|
||||
// BAD - many download endpoints don't support HEAD
|
||||
const response = await page.request.head(downloadUrl);
|
||||
// Returns 405 instead of 200
|
||||
```
|
||||
|
||||
**Solution**: Use GET with `maxRedirects: 0` to check accessibility without downloading:
|
||||
|
||||
```typescript
|
||||
// GOOD - works with all endpoints, doesn't follow redirects
|
||||
const response = await page.request.get(downloadUrl, { maxRedirects: 0 });
|
||||
expect(response.status() >= 200 && response.status() < 400).toBe(true);
|
||||
```
|
||||
|
||||
**Why maxRedirects: 0**: Download URLs often redirect to cloud storage (GCS, S3). A 302/307 redirect means the URL is valid. Following the redirect would download the entire file.
|
||||
|
||||
### E2E Tests: Understand Job State Machine Flow
|
||||
|
||||
**Problem**: E2E test skipped instrumental selection because it checked for "rendering" in status, but rendering happens BEFORE instrumental selection, not after.
|
||||
|
||||
```typescript
|
||||
// BAD - assumes rendering means instrumental is done
|
||||
if (status.includes('rendering') || status.includes('encoding')) {
|
||||
console.log('Past instrumental stage');
|
||||
break;
|
||||
}
|
||||
```
|
||||
|
||||
**Actual job flow**:
|
||||
```text
|
||||
downloading → transcribing → in_review → rendering_video →
|
||||
awaiting_instrumental_selection → encoding → complete
|
||||
```
|
||||
|
||||
**Solution**: Only skip on statuses that are definitively past the target stage:
|
||||
|
||||
```typescript
|
||||
// GOOD - encoding is definitely past instrumental
|
||||
if (status.includes('encoding') && !status.includes('awaiting')) {
|
||||
console.log('Past instrumental stage');
|
||||
break;
|
||||
}
|
||||
```
|
||||
|
||||
**Lesson**: Before writing status-checking logic, trace through the actual state machine to understand which states come before/after your target state.
|
||||
|
||||
## Deployment Notes
|
||||
|
||||
### Cloud Run Timeouts
|
||||
@@ -304,6 +400,62 @@ Default 5-minute timeout may not be enough for video encoding. Consider:
|
||||
- Breaking encoding into smaller chunks
|
||||
- Increasing timeout for specific operations
|
||||
|
||||
### Cloud Tasks dispatch_deadline (Separate from Cloud Run Timeout)
|
||||
|
||||
**Problem**: Cloud Tasks HTTP handlers have a separate `dispatch_deadline` (default 10 minutes, max 30 minutes) that controls how long Cloud Tasks waits for the HTTP response. This is DIFFERENT from Cloud Run's service timeout.
|
||||
|
||||
**Symptom**: Worker killed silently after 10 minutes even though Cloud Run timeout was set to 30 minutes. Job appears stuck with no error.
|
||||
|
||||
**Solution**: Set explicit `dispatch_deadline` when creating Cloud Tasks:
|
||||
```python
|
||||
from google.protobuf import duration_pb2
|
||||
|
||||
task = {
|
||||
"http_request": {...},
|
||||
"dispatch_deadline": duration_pb2.Duration(seconds=1800), # 30 min max
|
||||
}
|
||||
```
|
||||
|
||||
**Key insight**: For tasks that might take >30 minutes, Cloud Tasks HTTP targets won't work. Use Cloud Run Jobs instead (supports up to 24-hour execution).
|
||||
|
||||
| Timeout Type | Default | Max | What It Controls |
|
||||
|-------------|---------|-----|------------------|
|
||||
| Cloud Tasks `dispatch_deadline` | 10 min | 30 min | How long Cloud Tasks waits for HTTP response |
|
||||
| Cloud Run service timeout | 5 min | 60 min | How long a single HTTP request can run |
|
||||
| Cloud Run Jobs timeout | 10 min | 24 hr | How long a batch job execution can run |
|
||||
|
||||
See `docs/archive/2026-01-01-worker-timeout-fixes.md` for full details.
|
||||
|
||||
### Two-Layer Timeout Pattern for Long-Running Loops
|
||||
|
||||
**Problem**: A loop processing many items (e.g., 74 LLM calls at 15s each = 18+ minutes) can block indefinitely. Single outer timeout isn't enough because it may interrupt mid-operation and leave state inconsistent.
|
||||
|
||||
**Solution**: Use two layers of timeout protection:
|
||||
|
||||
```python
|
||||
# Layer 1: Inner deadline check (cooperative, clean exit)
|
||||
deadline = time.time() + timeout_seconds
|
||||
for item in items:
|
||||
if time.time() > deadline:
|
||||
logger.warning("Deadline exceeded, breaking early")
|
||||
break # Exit gracefully with partial/no results
|
||||
process(item)
|
||||
|
||||
# Layer 2: Outer asyncio.wait_for (safety net for hung operations)
|
||||
result = await asyncio.wait_for(
|
||||
asyncio.to_thread(run_loop_with_deadline),
|
||||
timeout=timeout_seconds + 60 # Buffer for cleanup
|
||||
)
|
||||
```
|
||||
|
||||
**Why both layers**:
|
||||
- Inner check: Handles normal case (many items). Exits cleanly between iterations.
|
||||
- Outer timeout: Catches edge case (single operation hangs). Hard stop if inner check never runs.
|
||||
|
||||
**Design choice**: Use `break` not `raise` for inner timeout. This returns partial/uncorrected results rather than failing the job, allowing downstream human review to fix issues.
|
||||
|
||||
See `docs/archive/2025-12-31-agentic-timeout-implementation.md` for implementation details.
|
||||
|
||||
### Secret Manager Access
|
||||
|
||||
Cloud Run service account needs `Secret Manager Secret Accessor` role. This is managed via Pulumi in `infrastructure/`.
|
||||
@@ -356,6 +508,47 @@ Code must handle both formats when invoking LangChain ChatVertexAI.
|
||||
|
||||
See `docs/archive/2025-12-30-gemini3-agentic-correction-fix.md` for full details.
|
||||
|
||||
### LangChain gRPC vs REST Providers
|
||||
|
||||
**Problem**: `langchain-google-vertexai` uses gRPC which can hang indefinitely during connection establishment with no timeout, causing cloud jobs to freeze silently.
|
||||
|
||||
**Symptoms**:
|
||||
- Job hangs at "Initializing model..." with no further logs
|
||||
- No timeout or error - just waits forever
|
||||
- Hard to diagnose because no exception is raised
|
||||
|
||||
**Solution**: Use `langchain-google-genai` (REST-based) instead. This package supports BOTH Vertex AI (service account auth) and Google AI Studio (API key auth):
|
||||
```python
|
||||
# Before (gRPC, can hang)
|
||||
from langchain_google_vertexai import ChatVertexAI
|
||||
model = ChatVertexAI(model=model_name, project=project)
|
||||
|
||||
# After (REST, reliable) - Using Vertex AI with service account/ADC
|
||||
from langchain_google_genai import ChatGoogleGenerativeAI
|
||||
model = ChatGoogleGenerativeAI(model=model_name, project=project_id) # Uses ADC
|
||||
|
||||
# Alternative - Using Google AI Studio with API key
|
||||
model = ChatGoogleGenerativeAI(model=model_name, google_api_key=api_key)
|
||||
```
|
||||
|
||||
**Key insight**: When `project` parameter is provided, `ChatGoogleGenerativeAI` automatically uses the Vertex AI backend with Application Default Credentials (ADC). On Cloud Run, this uses the attached service account - no API key needed.
|
||||
|
||||
**Additional protection**: Wrap model initialization in ThreadPoolExecutor timeout:
|
||||
```python
|
||||
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
|
||||
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(create_model)
|
||||
try:
|
||||
model = future.result(timeout=30) # Fail fast
|
||||
except FuturesTimeoutError:
|
||||
raise InitializationTimeoutError("Model init timed out")
|
||||
```
|
||||
|
||||
**Why ThreadPoolExecutor**: Works cross-platform (Windows, Linux, macOS). Python's `signal.alarm` only works on Unix.
|
||||
|
||||
See `docs/archive/2025-12-30-langchain-vertexai-to-genai-migration.md` for full details.
|
||||
|
||||
### CI/CD: GitHub Actions
|
||||
|
||||
Migrated from Cloud Build to GitHub Actions for:
|
||||
@@ -376,6 +569,51 @@ GitHub-hosted runners have ~14GB disk, which caused failures during Docker build
|
||||
|
||||
See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
|
||||
|
||||
### Docker Disk Space Management on Self-Hosted Runners
|
||||
|
||||
**Problem**: Even with 200GB SSDs, self-hosted runners can fill up with Docker images. Each CI build creates new dangling images (~15GB each), and without aggressive cleanup, disk fills in days.
|
||||
|
||||
**Symptoms**:
|
||||
- CI jobs fail with `No space left on device`
|
||||
- Jobs that require Docker (build, deploy, emulator tests) fail
|
||||
- Runners 11-20 at 100% while 1-10 at 30% (newer runners had less cleanup history)
|
||||
|
||||
**Root cause**: Original cleanup cron used `docker system prune -af --filter "until=168h"` which only removes images older than 7 days. With frequent CI builds, most dangling images are < 7 days old and never get cleaned.
|
||||
|
||||
**Solution** (two-layer approach):
|
||||
|
||||
1. **Hourly threshold-based cleanup** (on runners):
|
||||
```bash
|
||||
# /etc/cron.hourly/docker-cleanup
|
||||
THRESHOLD=70
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
if [ "$USAGE" -gt "$THRESHOLD" ]; then
|
||||
docker system prune -af # No age filter!
|
||||
docker builder prune -af
|
||||
fi
|
||||
```
|
||||
|
||||
2. **Pre-job disk check** (in CI workflow):
|
||||
```yaml
|
||||
- name: Check disk space
|
||||
run: |
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage exceeds 85%. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
**Key insights**:
|
||||
- Remove the age filter (`--filter "until=168h"`) - dangling images should always be cleaned
|
||||
- Threshold-based cleanup is better than time-based (don't clean if disk is fine)
|
||||
- Pre-job checks provide fail-fast behavior with clear error messages
|
||||
- `docker builder prune -af` catches buildkit cache that `system prune` misses
|
||||
|
||||
## Performance Observations
|
||||
|
||||
| Operation | Duration |
|
||||
@@ -508,6 +746,178 @@ rm /tmp/policy.yaml key.json # After copying key contents
|
||||
|
||||
**Lesson**: Service account keys are a security risk and should be avoided when possible. Use Workload Identity Federation instead. But for third-party services that require JSON keys (like LangFuse), this temporary policy override is the safest approach.
|
||||
|
||||
### Cloud Run Horizontal Scaling and In-Memory Cache
|
||||
|
||||
**Problem**: In-memory caches (singletons, module-level dicts) don't persist across Cloud Run instances. When scaling horizontally, request 1 may hit instance A and populate the cache, but request 2 may hit instance B which has an empty cache.
|
||||
|
||||
**Example**: `AudioSearchService` cached search results in `self._cached_results`. Search hit instance A, but download hit instance B:
|
||||
```
|
||||
Error: No cached result for index 0. Available indices: 0--1. Run search() first.
|
||||
```
|
||||
|
||||
**Solution**: Never rely on in-memory state across requests. Persist to Firestore/GCS:
|
||||
```python
|
||||
# BAD - cache lost if next request hits different instance
|
||||
self._cached_results = search_results
|
||||
result = self._cached_results[selection_index]
|
||||
|
||||
# GOOD - persist in job state_data
|
||||
job_manager.update_job(job_id, {'state_data': {'search_results': results}})
|
||||
# Later...
|
||||
job = job_manager.get_job(job_id)
|
||||
result = job.state_data['search_results'][selection_index]
|
||||
```
|
||||
|
||||
**Lesson**: Design for stateless request handling. Any state that must survive between requests must be persisted externally.
|
||||
|
||||
### GCS Downloads Preserve Directory Structure
|
||||
|
||||
**Problem**: When downloading files from GCS, the directory structure is preserved. Code that uses non-recursive glob patterns won't find files in subdirectories.
|
||||
|
||||
**Example**: GCE encoding worker downloaded from `gs://bucket/jobs/{job_id}/` but files were stored as:
|
||||
- `videos/with_vocals.mkv`
|
||||
- `stems/instrumental_clean.flac`
|
||||
|
||||
The original code used `work_dir.glob("*.mkv")` which only searched the root directory, causing "No video files found" errors.
|
||||
|
||||
**Solution**: Use recursive glob patterns and filter results:
|
||||
```python
|
||||
# BAD - only searches root directory
|
||||
input_files = list(work_dir.glob("*.mkv"))
|
||||
|
||||
# GOOD - searches all subdirectories
|
||||
input_files = list(work_dir.glob("**/*.mkv"))
|
||||
# Filter out outputs directory to avoid re-encoding
|
||||
input_files = [f for f in input_files if "outputs" not in str(f)]
|
||||
```
|
||||
|
||||
**Related gotcha**: Case sensitivity matters. GCS stores filenames as-is, so if files are uploaded as `instrumental_clean.flac` (lowercase), searching for `*Instrumental*.flac` (capital I) won't find them. Use case-insensitive patterns or search for both cases.
|
||||
|
||||
**Lesson**: When writing code that processes files downloaded from GCS, always use recursive glob patterns and verify the expected directory structure matches what's actually stored.
|
||||
|
||||
### GCS Blob Paths vs Full URIs
|
||||
|
||||
**Problem**: When uploading files to GCS and returning paths to other services, there's confusion between "blob paths" (just the path within a bucket) and "full GCS URIs" (gs://bucket/path).
|
||||
|
||||
**Symptoms**:
|
||||
- 404 errors when downloading files that were just uploaded
|
||||
- Paths look like: `bucket/gs://bucket/path/file.mp4` (doubled prefix)
|
||||
- Storage service prepends bucket name to what's already a full URI
|
||||
|
||||
**Example of the bug**:
|
||||
```python
|
||||
# GCE worker uploads to gs://bucket/jobs/id/encoded/
|
||||
# Then returns full URI in response: "gs://bucket/jobs/id/encoded/file.mp4"
|
||||
# Backend tries: storage.download_file("gs://bucket/jobs/id/encoded/file.mp4")
|
||||
# Storage service prepends bucket: "bucket/gs://bucket/jobs/id/encoded/file.mp4"
|
||||
# Result: 404 - No such object
|
||||
```
|
||||
|
||||
**Solution**: Return blob paths (path within bucket), not full GCS URIs:
|
||||
```python
|
||||
# output_gcs_path = "gs://bucket/jobs/id/encoded/"
|
||||
gcs_path = output_gcs_path.replace("gs://", "")
|
||||
parts = gcs_path.split("/", 1)
|
||||
bucket = parts[0]
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
|
||||
# Return blob paths like: "jobs/id/encoded/file.mp4"
|
||||
blob_path = f"{prefix}/{filename}" if prefix else filename
|
||||
```
|
||||
|
||||
**Related gotcha**: Double slashes in paths. When the prefix already ends with "/" and you concatenate with another "/":
|
||||
```python
|
||||
# BAD - creates "encoded//file.mp4"
|
||||
prefix = "jobs/id/encoded/" # Has trailing slash
|
||||
blob_name = f"{prefix}/{filename}" # Adds another slash
|
||||
|
||||
# GOOD - strip trailing slash first
|
||||
prefix = parts[1].rstrip("/")
|
||||
blob_name = f"{prefix}/{filename}" if prefix else filename
|
||||
```
|
||||
|
||||
**Lesson**: When building systems that pass GCS paths between services, establish a clear contract: either always use full URIs (`gs://bucket/path`) or always use blob paths (`path/within/bucket`). Document which format each API expects.
|
||||
|
||||
### Library Caches Need GCS Sync in Cloud Run
|
||||
|
||||
**Problem**: Libraries like LyricsTranscriber cache API responses to local directories (e.g., `~/lyrics-transcriber-cache`). In Cloud Run, each container is ephemeral - the cache is lost on every restart or scale-up, causing redundant API calls.
|
||||
|
||||
**Symptoms**:
|
||||
- Same song processed multiple times always hits external APIs
|
||||
- Higher API costs than expected (AudioShake charges per transcription)
|
||||
- Slower processing even for repeated jobs
|
||||
|
||||
**Solution**: Sync library cache to/from GCS before/after each job:
|
||||
|
||||
```python
|
||||
# Set cache dir via environment variable (library reads this)
|
||||
cache_dir = os.path.join(temp_dir, "lyrics-cache")
|
||||
os.environ["LYRICS_TRANSCRIBER_CACHE_DIR"] = cache_dir
|
||||
|
||||
# Download relevant cache files from GCS before processing
|
||||
cache_service.sync_cache_from_gcs(cache_dir, audio_hash, lyrics_hash)
|
||||
|
||||
# ... run processing ...
|
||||
|
||||
# Upload new cache files to GCS after processing
|
||||
cache_service.sync_cache_to_gcs(cache_dir, audio_hash, lyrics_hash)
|
||||
```
|
||||
|
||||
**Key design decisions**:
|
||||
1. Download only relevant files (by hash) - don't sync entire cache directory
|
||||
2. Skip files that already exist in GCS on upload (same input = same output)
|
||||
3. Use environment variable to configure library cache dir (avoids modifying library code)
|
||||
4. Cache indefinitely (no TTL) - API responses don't change for same inputs
|
||||
|
||||
**Lesson**: When using third-party libraries with local caching in Cloud Run, check if they support configurable cache directories. If so, sync to GCS for persistence across instances.
|
||||
|
||||
### Sequential vs Parallel Worker Triggering
|
||||
|
||||
**Problem**: For URL-based jobs (YouTube), triggering both audio and lyrics workers in parallel caused failures. Lyrics worker needs the audio file, but audio worker hadn't downloaded it yet.
|
||||
|
||||
```python
|
||||
# BAD - for URL jobs, audio isn't ready yet
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id) # Both start together
|
||||
# Lyrics worker times out waiting for audio
|
||||
```
|
||||
|
||||
**Solution**: For URL jobs, trigger workers sequentially:
|
||||
```python
|
||||
# GOOD - audio downloads first, then triggers lyrics
|
||||
if job.url and not job.input_media_gcs_path:
|
||||
# Only trigger audio worker; it will trigger lyrics after download
|
||||
background_tasks.add_task(worker_service.trigger_audio_worker, job_id)
|
||||
else:
|
||||
# For uploaded files, parallel is fine
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
```
|
||||
|
||||
**Lesson**: Understand data dependencies between workers. If worker B needs output from worker A, don't run them in parallel.
|
||||
|
||||
### Always Extract Auth Context Into Jobs
|
||||
|
||||
**Problem**: Job creation endpoints used `require_auth` but never extracted `auth_result.user_email` to store on the job. Jobs were created with `user_email=None`, so users couldn't see their own jobs.
|
||||
|
||||
```python
|
||||
# BAD - auth used for access control but email not captured
|
||||
@router.post("/jobs/upload")
|
||||
async def upload(auth_result = Depends(require_auth)):
|
||||
job = JobCreate(...) # user_email not set!
|
||||
```
|
||||
|
||||
**Solution**: Extract user identity from auth and set on job:
|
||||
```python
|
||||
# GOOD - capture authenticated user's identity
|
||||
@router.post("/jobs/upload")
|
||||
async def upload(auth_result: AuthResult = Depends(require_auth)):
|
||||
effective_user_email = auth_result.user_email or form_user_email
|
||||
job = JobCreate(user_email=effective_user_email, ...)
|
||||
```
|
||||
|
||||
**Lesson**: Auth provides identity, not just access control. If jobs need owner association, explicitly extract and store the user identifier.
|
||||
|
||||
See `docs/archive/2024-12-31-job-failure-investigation.md` for full details.
|
||||
|
||||
## What We'd Do Differently
|
||||
|
||||
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
|
||||
|
||||
+20
-2
@@ -7,6 +7,7 @@
|
||||
- **Web App**: https://gen.nomadkaraoke.com
|
||||
- `/` - Landing page with pricing and beta enrollment
|
||||
- `/app` - Main app (upload audio, review lyrics, download videos)
|
||||
- `/admin` - Admin dashboard (user management, job monitoring, beta program)
|
||||
- **Backend API**: https://api.nomadkaraoke.com - FastAPI on Cloud Run
|
||||
- **CLI Tools**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
|
||||
@@ -16,7 +17,7 @@
|
||||
|-----------|--------|
|
||||
| Audio upload & separation | Working |
|
||||
| Lyrics transcription | Working |
|
||||
| Agentic AI correction | Working (Gemini 3 Flash via Vertex AI) |
|
||||
| Agentic AI correction | Working (Gemini 2 Flash via Vertex AI) |
|
||||
| Human lyrics review | Working |
|
||||
| Preview video generation | Working |
|
||||
| Instrumental selection | Working |
|
||||
@@ -25,15 +26,32 @@
|
||||
| Magic link auth | Working |
|
||||
| Payment flow (Stripe) | Working |
|
||||
| Beta tester program | Working |
|
||||
| Admin dashboard | Working |
|
||||
| CI/CD self-hosted runner | Working (GCP) |
|
||||
| E2E happy path test | Working (38 min full pipeline) |
|
||||
|
||||
## Known Issues
|
||||
|
||||
- CDG format generation requires additional style configuration
|
||||
- Long audio files (>10 min) may timeout on some workers
|
||||
|
||||
## Recent Changes
|
||||
|
||||
- **Admin Dashboard** (2026-01-03): Added comprehensive admin dashboard at `/admin` with: stats overview (users, jobs, credits), user management (search, sort, add credits, toggle role, enable/disable), job browser (filter by status/user, view details, delete), and beta program monitoring. Query-param based routing for static export compatibility. See [API.md](API.md#admin-endpoints) for backend endpoints.
|
||||
|
||||
- **GCE Encoding Worker** (2026-01-03): Added optional high-performance encoding on dedicated GCE VM (C4-standard-8 with Intel Granite Rapids 3.9 GHz). Uses `USE_GCE_ENCODING=true` env var. Falls back to Cloud Run if unavailable. See [ARCHITECTURE.md](ARCHITECTURE.md).
|
||||
|
||||
- **Runner Disk Space Auto-Cleanup** (2026-01-03): Fixed CI failures caused by self-hosted runners filling up with Docker images. Changed cleanup from daily/7-day-old to hourly/threshold-based (70%). Added pre-job disk checks to all self-hosted CI jobs. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#docker-disk-space-management-on-self-hosted-runners)
|
||||
|
||||
- **E2E Happy Path Test** (2026-01-02): Added comprehensive E2E test that validates the complete karaoke generation journey using ONLY browser UI interactions (no API shortcuts). Test covers all 12 steps from landing page through cleanup. Runs daily via GitHub Actions and takes ~38 minutes. See [E2E-HAPPY-PATH-TEST-SPEC.md](E2E-HAPPY-PATH-TEST-SPEC.md)
|
||||
|
||||
- **LyricsTranscriber Cache Persistence** (2026-01-02): Added GCS-backed cache for LyricsTranscriber to avoid redundant API calls. AudioShake transcription and lyrics provider responses (Genius, Spotify, LRCLib, Musixmatch) are now cached to `gs://karaoke-gen-storage/lyrics-transcriber-cache/`. Cache is synced before/after each job, persisting across Cloud Run instances. Significantly reduces API costs and speeds up repeated processing of same songs.
|
||||
|
||||
- **Worker Timeout Fixes** (2026-01-01): Fixed 3 timeout issues blocking job completion: (1) Lyrics transcription timeout increased to 20 min (PR #153), (2) Cloud Tasks dispatch_deadline added for audio worker - default 10 min was killing Modal API calls (PR #154), (3) Enabled Cloud Run Jobs for video encoding - 1-hour timeout vs 30-min Cloud Run service limit (PR #155). Full E2E pipeline now verified working. See [archive/2026-01-01-worker-timeout-fixes.md](archive/2026-01-01-worker-timeout-fixes.md)
|
||||
|
||||
- **Agentic Correction Timeout** (2025-12-31): Added 3-minute configurable timeout for agentic AI lyrics correction. Prevents stuck jobs when songs have many gaps (74+ gaps could take 30+ minutes). On timeout, skips correction and proceeds to human review with raw transcription. See [archive/2025-12-31-agentic-timeout-implementation.md](archive/2025-12-31-agentic-timeout-implementation.md)
|
||||
- **Job Failure Fixes** (2025-12-31): Fixed 4 critical issues causing job failures: (1) user_email not set on jobs - users couldn't see their jobs, (2) YouTube URL race condition - lyrics worker started before audio downloaded, (3) Audio search cache not persisting across Cloud Run instances, (4) Jobs stuck in downloading state - added 10min transcription timeout. See [archive/2024-12-31-job-failure-investigation.md](archive/2024-12-31-job-failure-investigation.md)
|
||||
- **Vertex AI Auth Fix** (2025-12-31): Fixed `ChatGoogleGenerativeAI` to use Vertex AI backend with service account auth on Cloud Run. PR #145 broke cloud deployments by requiring `GOOGLE_API_KEY`. Solution: pass `project` parameter to trigger Vertex AI backend with ADC. See [archive/2025-12-31-vertex-ai-auth-fix.md](archive/2025-12-31-vertex-ai-auth-fix.md)
|
||||
- **LangChain Provider Migration** (2025-12-30): Migrated from `langchain-google-vertexai` (gRPC) to `langchain-google-genai` (REST) to fix silent hangs during model initialization. Added initialization and warm-up timeouts using ThreadPoolExecutor for fail-fast behavior. See [archive/2025-12-30-langchain-vertexai-to-genai-migration.md](archive/2025-12-30-langchain-vertexai-to-genai-migration.md)
|
||||
- **LangFuse Prompt Management** (2025-12-30): Replaced hardcoded LLM prompts with LangFuse-managed prompts for dynamic iteration without redeploying. Added migration script, LangFuse Vertex AI service account for GCP credits. See [archive/2025-12-30-langfuse-prompt-management.md](archive/2025-12-30-langfuse-prompt-management.md)
|
||||
- **Gemini 3 Flash Agentic Correction Fix** (2025-12-30): Fixed agentic correction hanging due to Gemini 3 requiring `global` location instead of regional endpoints, and multimodal response format handling. Added local test script for faster iteration. See [archive/2025-12-30-gemini3-agentic-correction-fix.md](archive/2025-12-30-gemini3-agentic-correction-fix.md)
|
||||
- **Cold Start UX** (2025-12-30): Frontend now shows friendly "Warming up the generator..." messages when backend is slow to respond (cold start after deployment or inactivity). Progressive messages appear after 5s and 15s to reassure users during 30s+ cold starts.
|
||||
|
||||
@@ -0,0 +1,291 @@
|
||||
# Job Failure Investigation Report
|
||||
|
||||
**Date**: 2024-12-31
|
||||
**Investigated Jobs**: 76296c31, 67bc8ca8, 02207418, cadd7153, 6a461e4e, 14035e2b, 643983e3, 9726a99b
|
||||
**Affected User**: [redacted]@gmail.com (plus E2E test jobs)
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Investigation revealed **4 distinct issues** causing job failures and the user complaint that "jobs don't show up in my account":
|
||||
|
||||
1. **CRITICAL: user_email not set on jobs** - Jobs aren't associated with users
|
||||
2. **YouTube URL download race condition** - Lyrics worker starts before audio is downloaded
|
||||
3. **Audio search cache not shared across instances** - Cloud Run scaling breaks download
|
||||
4. **Lyrics worker stalling silently** - Jobs stuck in "downloading" state forever
|
||||
|
||||
---
|
||||
|
||||
## Issue 1: user_email Not Being Set on Jobs (CRITICAL)
|
||||
|
||||
### Symptoms
|
||||
- User reports: "jobs polling is always returning a []"
|
||||
- User has active account with 2 credits, active session
|
||||
- User investigation shows `Jobs Created: 0` despite creating jobs
|
||||
- All 8 investigated jobs have `user_email: null`
|
||||
|
||||
### Root Cause
|
||||
The job creation endpoints use `require_auth` dependency which returns `AuthResult` containing `user_email`. However, the endpoints **do not extract this email and set it on the job**.
|
||||
|
||||
**In `backend/api/routes/file_upload.py`:**
|
||||
- `user_email` is an optional form field (line 397) that clients must explicitly pass
|
||||
- The `auth_result.user_email` from authentication is ignored
|
||||
- Jobs are created with `user_email=None` unless client explicitly sends it
|
||||
|
||||
**Same issue in:**
|
||||
- `/jobs/upload` endpoint
|
||||
- `/jobs/create-from-url` endpoint
|
||||
- `/jobs/create-with-upload-urls` endpoint
|
||||
- `/api/audio-search/search` endpoint
|
||||
|
||||
### Fix Required
|
||||
Modify all job creation endpoints to:
|
||||
1. Extract `user_email` from `AuthResult` when available
|
||||
2. Prefer authenticated user's email over client-provided value
|
||||
3. Set this on the `JobCreate` model
|
||||
|
||||
**Example fix for `create-from-url`:**
|
||||
```python
|
||||
@router.post("/jobs/create-from-url")
|
||||
async def create_job_from_url(
|
||||
request: Request,
|
||||
body: CreateJobFromUrlRequest,
|
||||
auth_result: AuthResult = Depends(require_auth) # Use AuthResult, not tuple
|
||||
):
|
||||
# Extract user email from authenticated session
|
||||
effective_user_email = auth_result.user_email or body.user_email
|
||||
|
||||
job_create = JobCreate(
|
||||
...
|
||||
user_email=effective_user_email, # Now properly set
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
### Priority
|
||||
**CRITICAL** - This is blocking all authenticated users from seeing their jobs.
|
||||
|
||||
---
|
||||
|
||||
## Issue 2: YouTube URL Download Race Condition
|
||||
|
||||
### Symptoms
|
||||
- Jobs cadd7153, 6a461e4e, 14035e2b failed with: `Lyrics transcription failed: Failed to download audio file`
|
||||
- All have `input_media_gcs_path: null`
|
||||
- All were created from YouTube URLs
|
||||
|
||||
### Root Cause
|
||||
In `file_upload.py` at `/jobs/create-from-url`:
|
||||
1. Job is created with `url` but no `input_media_gcs_path`
|
||||
2. Job immediately transitions to `DOWNLOADING` state
|
||||
3. **Both audio and lyrics workers are triggered in parallel** (`_trigger_workers_parallel`)
|
||||
|
||||
The problem:
|
||||
- Audio worker is responsible for downloading from YouTube and uploading to GCS
|
||||
- Lyrics worker needs the audio file from GCS to transcribe
|
||||
- Lyrics worker polls for `input_media_gcs_path` for up to 5 minutes (max_wait_seconds=300)
|
||||
- If YouTube download is slow or fails, lyrics worker times out
|
||||
|
||||
**Additional YouTube download issues:**
|
||||
- `YOUTUBE_COOKIES` env var may not be set, causing "Sign in to confirm you're not a bot" errors
|
||||
- YouTube rate limiting can cause downloads to fail
|
||||
|
||||
### Evidence in `lyrics_worker.py:401-427`
|
||||
```python
|
||||
# For URL jobs, wait for audio worker to download and upload to GCS
|
||||
if job.url:
|
||||
while time.time() - start_time < max_wait_seconds:
|
||||
# Poll for input_media_gcs_path
|
||||
if updated_job.input_media_gcs_path:
|
||||
return local_path
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
logger.error(f"Timed out waiting for audio download")
|
||||
return None
|
||||
```
|
||||
|
||||
### Fix Required
|
||||
Option A (Recommended): **Sequential triggering for URL jobs**
|
||||
```python
|
||||
# In create-from-url endpoint:
|
||||
if job.url and not job.input_media_gcs_path:
|
||||
# Only trigger audio worker first for URL jobs
|
||||
# Lyrics worker triggered after audio is downloaded
|
||||
background_tasks.add_task(worker_service.trigger_audio_worker, job_id)
|
||||
else:
|
||||
# For uploaded files, parallel is fine
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
```
|
||||
|
||||
Option B: **Audio worker triggers lyrics worker after download**
|
||||
- Have `audio_worker.download_audio()` trigger lyrics worker upon successful URL download
|
||||
|
||||
### Priority
|
||||
**HIGH** - YouTube/URL-based jobs are failing consistently.
|
||||
|
||||
---
|
||||
|
||||
## Issue 3: Audio Search Cache Not Persisting
|
||||
|
||||
### Symptoms
|
||||
- Job 76296c31 failed with: `Audio download failed: No cached result for index 0. Available indices: 0--1. Run search() first.`
|
||||
- Job was created from `audio_search` mode
|
||||
|
||||
### Root Cause
|
||||
`AudioSearchService` is a singleton with in-memory cache (`self._cached_results`). Cloud Run uses horizontal scaling with multiple instances, so:
|
||||
|
||||
1. Request 1 (search): Hits instance A, caches results in A's memory
|
||||
2. Request 2 (select/download): May hit instance B, which has empty cache
|
||||
3. Download fails because `len(self._cached_results) = 0`
|
||||
|
||||
**In `audio_search_service.py:279-284`:**
|
||||
```python
|
||||
if result_index < 0 or result_index >= len(self._cached_results):
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. " # Shows "0--1" when empty
|
||||
"Run search() first."
|
||||
)
|
||||
```
|
||||
|
||||
### Partial Fix Already in Place
|
||||
The code stores search results in `job.state_data['audio_search_results']` and attempts to use `download_by_id()` which doesn't require cache. However, for YouTube sources, it falls back to cache-based download.
|
||||
|
||||
### Fix Required
|
||||
**Never rely on in-memory cache across requests.** Always use job.state_data.
|
||||
|
||||
In `audio_search.py` `_download_and_start_processing()`:
|
||||
```python
|
||||
# Get the selected result from job state_data (not service cache)
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
selected = search_results[selection_index]
|
||||
|
||||
# For YouTube, construct a fake AudioSearchResult from state_data
|
||||
# and call download_by_id with video URL
|
||||
if selected['provider'] == 'YouTube':
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name='YouTube',
|
||||
source_id=selected.get('source_id'), # Video ID
|
||||
output_dir=temp_dir,
|
||||
download_url=selected.get('url'), # YouTube URL
|
||||
)
|
||||
```
|
||||
|
||||
### Priority
|
||||
**HIGH** - Audio search mode is broken in multi-instance environments.
|
||||
|
||||
---
|
||||
|
||||
## Issue 4: Jobs Stuck in Downloading State
|
||||
|
||||
### Symptoms
|
||||
- Jobs 643983e3, 9726a99b have status `downloading` indefinitely
|
||||
- `state_data.audio_complete: true` - audio worker finished
|
||||
- `state_data.lyrics_progress.stage: "transcribing"` - lyrics worker started
|
||||
- NO `lyrics_complete` flag - lyrics worker never finished
|
||||
- Status never transitions beyond `downloading`
|
||||
|
||||
### Root Cause
|
||||
The lyrics worker appears to crash or timeout during AudioShake transcription without properly failing the job. Possible causes:
|
||||
|
||||
1. **AudioShake API timeout** - transcription takes too long
|
||||
2. **Cloud Run instance killed** - worker running longer than timeout allows
|
||||
3. **Unhandled exception** - error during transcription not caught properly
|
||||
4. **Memory exhaustion** - large audio files causing OOM
|
||||
|
||||
The lyrics worker's exception handling (`lyrics_worker.py:375-383`) should mark job as failed:
|
||||
```python
|
||||
except Exception as e:
|
||||
job_manager.mark_job_failed(
|
||||
job_id=job_id,
|
||||
error_message=f"Lyrics transcription failed: {str(e)}",
|
||||
)
|
||||
```
|
||||
|
||||
But if the Cloud Run instance is killed (e.g., timeout, OOM), this code never runs.
|
||||
|
||||
### Fix Required
|
||||
1. **Add job-level timeouts with cleanup**
|
||||
```python
|
||||
# In job_manager.py - scheduled task to clean up stale jobs
|
||||
async def cleanup_stale_jobs():
|
||||
# Find jobs in 'downloading' state for more than X hours
|
||||
# Check if workers are actually running
|
||||
# Mark as failed if stuck
|
||||
```
|
||||
|
||||
2. **Add heartbeat mechanism**
|
||||
```python
|
||||
# Workers periodically update timestamp
|
||||
job_manager.update_state_data(job_id, 'worker_heartbeat', datetime.now())
|
||||
|
||||
# Background job detects stale heartbeats and fails job
|
||||
```
|
||||
|
||||
3. **Improve AudioShake timeout handling**
|
||||
```python
|
||||
# In lyrics_worker.py, add explicit timeout:
|
||||
async with asyncio.timeout(600): # 10 minute timeout
|
||||
result = await asyncio.to_thread(lyrics_processor.transcribe_lyrics, ...)
|
||||
```
|
||||
|
||||
### Priority
|
||||
**MEDIUM** - Jobs are stuck but not actively failing. Need monitoring/cleanup mechanism.
|
||||
|
||||
---
|
||||
|
||||
## Summary of Fixes
|
||||
|
||||
| Issue | Priority | Fix Complexity | Affected Code |
|
||||
|-------|----------|---------------|---------------|
|
||||
| 1. user_email not set | CRITICAL | Low | `file_upload.py`, `audio_search.py` |
|
||||
| 2. URL download race | HIGH | Medium | `file_upload.py`, `audio_worker.py` |
|
||||
| 3. Audio search cache | HIGH | Medium | `audio_search.py`, `audio_search_service.py` |
|
||||
| 4. Stuck jobs cleanup | MEDIUM | Medium | `job_manager.py`, workers |
|
||||
|
||||
---
|
||||
|
||||
## Implemented Fixes
|
||||
|
||||
### Issue 1: user_email - FIXED
|
||||
**Changes:**
|
||||
- Updated all job creation endpoints to use `AuthResult` type instead of tuple
|
||||
- Extract `auth_result.user_email` and set it on `JobCreate`
|
||||
- Prefer authenticated user's email over client-provided value
|
||||
|
||||
**Files modified:**
|
||||
- `backend/api/routes/file_upload.py` - All job creation endpoints
|
||||
- `backend/api/routes/audio_search.py` - Audio search endpoint
|
||||
|
||||
### Issue 2: URL Download Race Condition - FIXED
|
||||
**Changes:**
|
||||
- For URL jobs, only trigger audio worker initially (not lyrics)
|
||||
- Audio worker triggers lyrics worker after successful URL download
|
||||
- Added `_trigger_audio_worker_only()` function for URL jobs
|
||||
- Added `_trigger_lyrics_worker_after_url_download()` in audio_worker
|
||||
|
||||
**Files modified:**
|
||||
- `backend/api/routes/file_upload.py` - Changed create-from-url to use sequential triggering
|
||||
- `backend/workers/audio_worker.py` - Trigger lyrics worker after URL download
|
||||
|
||||
### Issue 3: Audio Search Cache - FIXED
|
||||
**Changes:**
|
||||
- For YouTube sources without remote flacfetch, download directly using URL from `state_data`
|
||||
- Avoids relying on in-memory cache which doesn't persist across Cloud Run instances
|
||||
- Added fallback warning for cache-based downloads
|
||||
|
||||
**Files modified:**
|
||||
- `backend/api/routes/audio_search.py` - Added direct YouTube download path
|
||||
|
||||
### Issue 4: Stuck Jobs Timeout - FIXED
|
||||
**Changes:**
|
||||
- Added 10-minute timeout around AudioShake transcription call
|
||||
- Uses `asyncio.wait_for` to ensure transcription doesn't hang forever
|
||||
- On timeout, raises exception which marks job as failed
|
||||
|
||||
**Files modified:**
|
||||
- `backend/workers/lyrics_worker.py` - Added timeout handling
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
All 812 backend tests pass after fixes.
|
||||
@@ -0,0 +1,178 @@
|
||||
# LangChain Vertex AI Migration Plan
|
||||
|
||||
**Date**: 2025-12-30
|
||||
**Status**: Planned
|
||||
**Priority**: Medium (deprecation warning, not yet breaking)
|
||||
|
||||
## Background
|
||||
|
||||
The `ChatVertexAI` class from `langchain-google-vertexai` was deprecated in version 3.2.0 and will be removed in 4.0.0. The recommended replacement is `ChatGoogleGenerativeAI` from the `langchain-google-genai` package.
|
||||
|
||||
Deprecation warning observed in logs:
|
||||
```
|
||||
LangChainDeprecationWarning: The class `ChatVertexAI` was deprecated in LangChain 3.2.0
|
||||
and will be removed in 4.0.0. An updated version of the class exists in the
|
||||
`langchain-google-genai` package and should be used instead.
|
||||
```
|
||||
|
||||
## Current Implementation
|
||||
|
||||
**File**: `lyrics_transcriber_temp/lyrics_transcriber/correction/agentic/providers/model_factory.py`
|
||||
|
||||
```python
|
||||
from langchain_google_vertexai import ChatVertexAI
|
||||
|
||||
model = ChatVertexAI(
|
||||
model=model_name,
|
||||
project=config.gcp_project_id,
|
||||
location=config.gcp_location,
|
||||
timeout=config.request_timeout_seconds,
|
||||
max_retries=config.max_retries,
|
||||
callbacks=callbacks,
|
||||
)
|
||||
```
|
||||
|
||||
## Target Implementation
|
||||
|
||||
```python
|
||||
from langchain_google_genai import ChatGoogleGenerativeAI
|
||||
|
||||
model = ChatGoogleGenerativeAI(
|
||||
model=model_name,
|
||||
vertexai=True, # Keep using Vertex AI authentication
|
||||
project=config.gcp_project_id,
|
||||
location=config.gcp_location,
|
||||
timeout=config.request_timeout_seconds,
|
||||
max_retries=config.max_retries,
|
||||
callbacks=callbacks,
|
||||
)
|
||||
```
|
||||
|
||||
## Migration Steps
|
||||
|
||||
### 1. Update Dependencies
|
||||
|
||||
In `pyproject.toml` (or wherever dependencies are managed):
|
||||
|
||||
```toml
|
||||
# Remove
|
||||
langchain-google-vertexai = "^x.x.x"
|
||||
|
||||
# Add
|
||||
langchain-google-genai = "^4.0.0"
|
||||
```
|
||||
|
||||
### 2. Update model_factory.py
|
||||
|
||||
**File**: `lyrics_transcriber_temp/lyrics_transcriber/correction/agentic/providers/model_factory.py`
|
||||
|
||||
```python
|
||||
# Change import
|
||||
# Old:
|
||||
from langchain_google_vertexai import ChatVertexAI
|
||||
|
||||
# New:
|
||||
from langchain_google_genai import ChatGoogleGenerativeAI
|
||||
```
|
||||
|
||||
Update the `_create_vertexai_model` method:
|
||||
|
||||
```python
|
||||
def _create_vertexai_model(
|
||||
self, model_name: str, callbacks: List[Any], config: ProviderConfig
|
||||
) -> Any:
|
||||
"""Create ChatGoogleGenerativeAI model for Google Gemini via Vertex AI.
|
||||
|
||||
Uses Application Default Credentials (ADC) for authentication.
|
||||
In Cloud Run, this uses the service account automatically.
|
||||
Locally, run: gcloud auth application-default login
|
||||
"""
|
||||
from langchain_google_genai import ChatGoogleGenerativeAI
|
||||
|
||||
model = ChatGoogleGenerativeAI(
|
||||
model=model_name,
|
||||
vertexai=True, # Use Vertex AI backend (not Google AI Studio)
|
||||
project=config.gcp_project_id,
|
||||
location=config.gcp_location,
|
||||
timeout=config.request_timeout_seconds,
|
||||
max_retries=config.max_retries,
|
||||
callbacks=callbacks,
|
||||
)
|
||||
logger.debug(f"🤖 Created Vertex AI model: {model_name} (project={config.gcp_project_id})")
|
||||
return model
|
||||
```
|
||||
|
||||
### 3. Alternative: Environment Variable Approach
|
||||
|
||||
Instead of passing `vertexai=True`, you can set environment variables:
|
||||
|
||||
```bash
|
||||
export GOOGLE_GENAI_USE_VERTEXAI=true
|
||||
export GOOGLE_CLOUD_PROJECT=your-project-id
|
||||
export GOOGLE_CLOUD_LOCATION=global # or us-central1, etc.
|
||||
```
|
||||
|
||||
Then the code simplifies to:
|
||||
|
||||
```python
|
||||
model = ChatGoogleGenerativeAI(
|
||||
model=model_name,
|
||||
timeout=config.request_timeout_seconds,
|
||||
max_retries=config.max_retries,
|
||||
callbacks=callbacks,
|
||||
)
|
||||
```
|
||||
|
||||
## Key Differences
|
||||
|
||||
| Aspect | ChatVertexAI (old) | ChatGoogleGenerativeAI (new) |
|
||||
|--------|-------------------|------------------------------|
|
||||
| Package | `langchain-google-vertexai` | `langchain-google-genai` |
|
||||
| Protocol | gRPC | REST |
|
||||
| Vertex AI Support | Native | Via `vertexai=True` param |
|
||||
| Auth Methods | ADC, Service Account | Same + API keys |
|
||||
| Multimodal | Yes | Yes |
|
||||
|
||||
## Potential Benefits
|
||||
|
||||
1. **REST instead of gRPC**: May resolve the `fork_posix.cc` gRPC warnings we observed:
|
||||
```
|
||||
I0000 00:00:1767135828.790289 30025640 fork_posix.cc:71] Other threads are
|
||||
currently calling into gRPC, skipping fork() handlers
|
||||
```
|
||||
|
||||
2. **Unified SDK**: The `google-genai` SDK is the consolidated approach for all Google AI services.
|
||||
|
||||
3. **Future-proof**: Avoids breaking changes when `langchain-google-vertexai` 4.0 removes `ChatVertexAI`.
|
||||
|
||||
## Potential Risks
|
||||
|
||||
1. **REST vs gRPC latency**: Some users reported slightly slower calls with REST. Monitor LangFuse traces after migration.
|
||||
|
||||
2. **Behavior differences**: Test thoroughly to ensure identical behavior for:
|
||||
- Multimodal response format handling
|
||||
- Timeout behavior
|
||||
- Retry logic
|
||||
- Error messages
|
||||
|
||||
## Testing Plan
|
||||
|
||||
1. Run existing agentic correction unit tests
|
||||
2. Run a full local CLI job with `USE_AGENTIC_AI=1`
|
||||
3. Compare LangFuse traces before/after for:
|
||||
- Response times
|
||||
- Success rates
|
||||
- Error types
|
||||
4. Verify Cloud Run deployment works with new package
|
||||
|
||||
## References
|
||||
|
||||
- [langchain-google-genai 4.0.0 Announcement](https://github.com/langchain-ai/langchain-google/discussions/1422)
|
||||
- [ChatGoogleGenerativeAI Docs](https://reference.langchain.com/python/integrations/langchain_google_genai/)
|
||||
- [LangChain Google Releases](https://github.com/langchain-ai/langchain-google/releases)
|
||||
|
||||
## Notes
|
||||
|
||||
- This migration was identified during investigation of 499 timeout errors in agentic correction (see PR #143)
|
||||
- The gRPC → REST change might actually help with the connection issues we observed
|
||||
- Authentication via Vertex AI (ADC/service accounts) is fully supported in the new package
|
||||
@@ -0,0 +1,95 @@
|
||||
# LangChain Vertex AI to Google GenAI Migration
|
||||
|
||||
**Date**: 2025-12-30
|
||||
**PR**: #145
|
||||
|
||||
## Problem
|
||||
|
||||
Cloud job `b1b2648c` hung silently during agentic correction. No logs appeared after "LangFuse prompt service initialized". Investigation revealed `ChatVertexAI` (langchain-google-vertexai) was hanging during gRPC connection establishment with no timeout.
|
||||
|
||||
## Solution
|
||||
|
||||
Migrated from `langchain-google-vertexai` to `langchain-google-genai`:
|
||||
|
||||
| Before | After |
|
||||
|--------|-------|
|
||||
| `ChatVertexAI` (gRPC-based) | `ChatGoogleGenerativeAI` (REST-based) |
|
||||
| Vertex AI service account auth | GOOGLE_API_KEY (AI Studio) |
|
||||
| No initialization timeout | 30s init + 15s warm-up timeouts |
|
||||
| DEBUG-level logs | INFO-level logs with timing |
|
||||
|
||||
## Key Changes
|
||||
|
||||
### Dependencies
|
||||
```toml
|
||||
# Before
|
||||
langchain-google-vertexai = ">=3.1.1"
|
||||
|
||||
# After
|
||||
langchain-google-genai = ">=2.0.0"
|
||||
```
|
||||
|
||||
### Model Creation
|
||||
```python
|
||||
# Before (gRPC, can hang)
|
||||
from langchain_google_vertexai import ChatVertexAI
|
||||
model = ChatVertexAI(model=model_name, project=project, location=location)
|
||||
|
||||
# After (REST, explicit API key)
|
||||
from langchain_google_genai import ChatGoogleGenerativeAI
|
||||
model = ChatGoogleGenerativeAI(
|
||||
model=model_name,
|
||||
google_api_key=api_key,
|
||||
convert_system_message_to_human=True, # Gemini doesn't support system messages
|
||||
)
|
||||
```
|
||||
|
||||
### Timeout Implementation
|
||||
Used `ThreadPoolExecutor` for cross-platform timeout (works on Windows, Linux, macOS):
|
||||
|
||||
```python
|
||||
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
|
||||
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(self._factory.create_chat_model, model, config)
|
||||
try:
|
||||
self._chat_model = future.result(timeout=30)
|
||||
except FuturesTimeoutError:
|
||||
raise InitializationTimeoutError("Model initialization timed out")
|
||||
```
|
||||
|
||||
### Warm-up Pattern
|
||||
Added lightweight "warm-up" request after model creation to establish connection before real work:
|
||||
|
||||
```python
|
||||
warm_up_prompt = 'Respond with exactly: {"status":"ready"}'
|
||||
response = model.invoke([HumanMessage(content=warm_up_prompt)])
|
||||
```
|
||||
|
||||
## Files Modified
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `pyproject.toml` | Replaced dependency, bumped version |
|
||||
| `lyrics_transcriber_temp/pyproject.toml` | Same |
|
||||
| `.../providers/config.py` | Added `initialization_timeout_seconds`, `warmup_timeout_seconds` |
|
||||
| `.../providers/model_factory.py` | Migrated to `ChatGoogleGenerativeAI`, added timing logs |
|
||||
| `.../providers/langchain_bridge.py` | Added ThreadPoolExecutor timeout wrapper, warm-up, INFO logs |
|
||||
| `.../agent.py` | Added timing logs for classify_gap |
|
||||
|
||||
## Why REST over gRPC
|
||||
|
||||
1. **Simpler debugging** - HTTP requests are easier to trace than gRPC streams
|
||||
2. **No connection pooling issues** - REST is stateless
|
||||
3. **Better timeout behavior** - HTTP timeouts are well-understood
|
||||
4. **Cloud-friendly** - Works better through proxies and load balancers
|
||||
|
||||
## Authentication
|
||||
|
||||
The new approach uses Google AI Studio API keys instead of Vertex AI service accounts:
|
||||
|
||||
- **Get key**: https://aistudio.google.com/app/apikey
|
||||
- **Set env var**: `GOOGLE_API_KEY=<your-key>`
|
||||
- **Benefits**: Simpler setup, no IAM configuration needed
|
||||
|
||||
Note: This uses Google AI Studio quotas, not GCP billing. For high-volume production use, consider keeping Vertex AI with proper timeout handling.
|
||||
@@ -0,0 +1,68 @@
|
||||
# Agentic AI Correction Timeout Implementation
|
||||
|
||||
**Date**: 2025-12-31
|
||||
**PR**: #149
|
||||
**Version**: 0.85.0
|
||||
|
||||
## Problem
|
||||
|
||||
Agentic AI lyrics correction was causing stuck jobs in production. Songs with many gaps (74+) could take 30+ minutes to process since each gap takes 10-30 seconds for LLM inference. This blocked the entire job pipeline.
|
||||
|
||||
Example from production logs:
|
||||
```
|
||||
Processing 22/74 gaps (~15s each) = ~18 minutes elapsed
|
||||
Estimated total: 74 gaps × 15s = 18+ minutes
|
||||
```
|
||||
|
||||
## Solution
|
||||
|
||||
Implemented a configurable 3-minute timeout (default) with two layers:
|
||||
|
||||
### Layer 1: Inner Deadline Check (Cooperative)
|
||||
In `corrector.py`, check deadline before processing each gap:
|
||||
```python
|
||||
if deadline and time.time() > deadline:
|
||||
self.logger.warning("AGENTIC TIMEOUT: Deadline exceeded...")
|
||||
break # Return uncorrected transcription for human review
|
||||
```
|
||||
|
||||
### Layer 2: Outer asyncio.wait_for (Safety Net)
|
||||
In `lyrics_worker.py`, wrap the entire transcription call:
|
||||
```python
|
||||
result = await asyncio.wait_for(
|
||||
asyncio.to_thread(lyrics_processor.transcribe_lyrics, ...),
|
||||
timeout=timeout_seconds + 60 # Extra buffer
|
||||
)
|
||||
```
|
||||
|
||||
## Behavior
|
||||
|
||||
| Scenario | What Happens |
|
||||
|----------|--------------|
|
||||
| Completes in <3 min | Normal flow, corrections applied |
|
||||
| Gap loop exceeds deadline | Inner check breaks, returns uncorrected |
|
||||
| LLM hangs indefinitely | Outer timeout fires (rare), job fails |
|
||||
| Any timeout | Job proceeds to human review with raw transcription |
|
||||
|
||||
## Files Changed
|
||||
|
||||
- `backend/config.py` - Added `agentic_correction_timeout_seconds` config
|
||||
- `backend/workers/lyrics_worker.py` - Deadline calculation, outer timeout wrapper
|
||||
- `karaoke_gen/lyrics_processor.py` - Pass deadline to controller
|
||||
- `lyrics_transcriber_temp/lyrics_transcriber/core/controller.py` - Accept deadline parameter
|
||||
- `lyrics_transcriber_temp/lyrics_transcriber/correction/corrector.py` - Deadline check in gap loop
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variable: `AGENTIC_CORRECTION_TIMEOUT_SECONDS` (default: 180)
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. **Break, don't raise**: Inner timeout uses `break` to exit the gap loop gracefully, returning whatever we have (uncorrected). This is better than raising an exception because:
|
||||
- No retry logic needed
|
||||
- Job continues to human review
|
||||
- User can manually correct any issues
|
||||
|
||||
2. **Two-layer protection**: The inner check handles normal cases (many gaps). The outer timeout catches edge cases (single LLM call hanging). Both are needed for reliability.
|
||||
|
||||
3. **Configurable timeout**: Default 3 minutes balances correction quality vs. user experience. It can be adjusted via environment variable per deployment.
|
||||
@@ -0,0 +1,72 @@
|
||||
# ChatGoogleGenerativeAI Vertex AI Auth Fix
|
||||
|
||||
**Date**: 2025-12-31
|
||||
**PR**: #147
|
||||
**Related**: PR #145 (LangChain provider migration)
|
||||
|
||||
## Background
|
||||
|
||||
PR #145 migrated from `ChatVertexAI` (gRPC) to `ChatGoogleGenerativeAI` (REST) to fix silent hangs during model initialization. However, the migration only configured API key authentication, breaking Cloud Run deployments that use service account credentials.
|
||||
|
||||
## Problem
|
||||
|
||||
After deploying PR #145, Cloud Run lyrics worker failed with:
|
||||
```
|
||||
🤖 Failed to initialize chat model: GOOGLE_API_KEY environment variable is required for Google/Gemini models
|
||||
```
|
||||
|
||||
The code required `GOOGLE_API_KEY` even though:
|
||||
- Cloud Run already has `GOOGLE_CLOUD_PROJECT` set
|
||||
- Service account credentials are available via ADC (Application Default Credentials)
|
||||
- Vertex AI IAM permissions were already configured
|
||||
|
||||
## Root Cause
|
||||
|
||||
`ChatGoogleGenerativeAI` supports TWO authentication backends:
|
||||
|
||||
| Backend | Trigger | Auth Method |
|
||||
|---------|---------|-------------|
|
||||
| Vertex AI | `project` param set | Service account / ADC |
|
||||
| Google AI Studio | Only `google_api_key` | API key |
|
||||
|
||||
PR #145 only passed `google_api_key`, forcing the Google AI Studio backend which doesn't support service accounts.
|
||||
|
||||
## Solution
|
||||
|
||||
Pass `project` parameter when `GOOGLE_CLOUD_PROJECT` is set to trigger Vertex AI backend:
|
||||
|
||||
```python
|
||||
# Before (broken)
|
||||
model = ChatGoogleGenerativeAI(model=model_name, google_api_key=api_key)
|
||||
|
||||
# After (works with service account)
|
||||
model_kwargs = {"model": model_name, ...}
|
||||
if project:
|
||||
model_kwargs["project"] = project # Triggers Vertex AI backend
|
||||
if api_key:
|
||||
model_kwargs["google_api_key"] = api_key # Optional fallback
|
||||
model = ChatGoogleGenerativeAI(**model_kwargs)
|
||||
```
|
||||
|
||||
## Key Insight
|
||||
|
||||
`ChatGoogleGenerativeAI` from `langchain-google-genai` is a unified client that supports both Vertex AI and Google AI Studio APIs. The `project` parameter is the key discriminator - when present, it uses Vertex AI with ADC auth.
|
||||
|
||||
## Testing
|
||||
|
||||
Verified that on Cloud Run with `GOOGLE_CLOUD_PROJECT=nomadkaraoke`:
|
||||
- No `GOOGLE_API_KEY` required
|
||||
- Uses attached service account via ADC
|
||||
- REST API (avoids gRPC hang issues from PR #145)
|
||||
|
||||
## Files Changed
|
||||
|
||||
- `lyrics_transcriber_temp/.../providers/model_factory.py` - Added project param logic
|
||||
- `docs/LESSONS-LEARNED.md` - Updated documentation
|
||||
|
||||
## Timing Analysis Context
|
||||
|
||||
This fix was discovered during a timing analysis comparing local vs remote job performance. Key findings:
|
||||
- Remote jobs had 38+ minute delays in lyrics worker
|
||||
- Root cause: gRPC hangs (fixed by PR #145) + auth misconfiguration (fixed by this PR)
|
||||
- With both fixes, remote processing should match local performance
|
||||
@@ -0,0 +1,78 @@
|
||||
# Worker Timeout Fixes
|
||||
|
||||
**Date**: 2026-01-01
|
||||
**PRs**: #153, #154, #155
|
||||
**Status**: Complete
|
||||
|
||||
## Problem
|
||||
|
||||
Jobs were failing at different stages due to worker timeouts:
|
||||
|
||||
1. **Lyrics worker** - Timeout after 600 seconds (10 minutes) during transcription
|
||||
2. **Audio worker** - Killed after 10 minutes by Cloud Tasks default `dispatch_deadline`
|
||||
3. **Video worker** - Timeout after 30 minutes (Cloud Run service timeout)
|
||||
|
||||
## Root Causes
|
||||
|
||||
### Lyrics Transcription Timeout (PR #153)
|
||||
- `TRANSCRIPTION_TIMEOUT_SECONDS` was 600 (10 minutes)
|
||||
- AudioShake transcription + agentic correction can take 15+ minutes for complex songs
|
||||
- **Fix**: Increased to 1200 seconds (20 minutes)
|
||||
|
||||
### Cloud Tasks dispatch_deadline (PR #154)
|
||||
- Cloud Tasks HTTP handlers have a default `dispatch_deadline` of 10 minutes
|
||||
- Audio separation via Modal API takes 15-20+ minutes
|
||||
- Worker was killed before Modal could complete, but no error surfaced
|
||||
- **Symptom**: Job stuck at `downloading` with `audio_complete: false`
|
||||
- **Fix**: Added explicit `dispatch_deadline` per worker type:
|
||||
```python
|
||||
WORKER_DISPATCH_DEADLINES = {
|
||||
"audio": 1800, # 30 min
|
||||
"lyrics": 1500, # 25 min
|
||||
"screens": 600, # 10 min
|
||||
"render-video": 1800, # 30 min
|
||||
"video": 1800, # 30 min
|
||||
}
|
||||
```
|
||||
|
||||
### Video Encoding Timeout (PR #155)
|
||||
- Cloud Run service timeout is 1800 seconds (30 minutes)
|
||||
- `KaraokeFinalise.process()` takes 30-40+ minutes for average songs
|
||||
- Video encoding includes: 4 format encodings, CDG generation, TXT packages
|
||||
- **Symptom**: Job stuck at `encoding` with `updated_at` frozen
|
||||
- **Fix**: Enabled `USE_CLOUD_RUN_JOBS_FOR_VIDEO=true` in `cloudbuild.yaml`
|
||||
- Cloud Run Jobs support up to 1-hour execution (configured in `infrastructure/__main__.py`)
|
||||
|
||||
## Key Insight: dispatch_deadline vs Cloud Run Timeout
|
||||
|
||||
These are different timeouts:
|
||||
|
||||
| Timeout | Default | Max | Controls |
|
||||
|---------|---------|-----|----------|
|
||||
| Cloud Tasks `dispatch_deadline` | 10 min | 30 min | How long Cloud Tasks waits for HTTP response |
|
||||
| Cloud Run service timeout | 5 min | 60 min | How long Cloud Run allows request to run |
|
||||
| Cloud Run Jobs timeout | 10 min | 24 hr | How long a batch job can run |
|
||||
|
||||
For worker tasks that take >30 minutes, Cloud Tasks HTTP targets can't work. Use Cloud Run Jobs instead.
|
||||
|
||||
## Verification
|
||||
|
||||
Job `eb725b2c` completed successfully through the full pipeline:
|
||||
- Audio separation: Complete
|
||||
- Lyrics transcription: Complete
|
||||
- Video encoding: Complete
|
||||
- All 4 video formats generated
|
||||
- All stems, lyrics, and packages available
|
||||
|
||||
## Files Changed
|
||||
|
||||
- `backend/services/worker_service.py` - Added `dispatch_deadline` to Cloud Tasks
|
||||
- `cloudbuild.yaml` - Enabled `USE_CLOUD_RUN_JOBS_FOR_VIDEO=true`
|
||||
- `pyproject.toml` - Version bumps (0.86.4, 0.86.5)
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Cloud Tasks dispatch_deadline is separate from Cloud Run timeout** - Both need to be configured for long-running workers
|
||||
2. **Default dispatch_deadline (10 min) is often insufficient** - Modal API, video encoding, and other heavy processing need more time
|
||||
3. **Cloud Run Jobs for truly long tasks** - For anything that might take >30 minutes, Cloud Tasks HTTP targets won't work
|
||||
4. **E2E tests reveal timeout issues** - Production E2E tests caught these issues that unit tests missed
|
||||
@@ -0,0 +1,19 @@
|
||||
# Environment variables for local development and testing
|
||||
# Copy this file to .env.local and fill in your values
|
||||
|
||||
# =============================================================================
|
||||
# E2E Testing - Email (for beta enrollment flow)
|
||||
# =============================================================================
|
||||
|
||||
# testmail.app credentials for email testing
|
||||
# Get these from https://testmail.app dashboard
|
||||
TESTMAIL_API_KEY=your-api-key-here
|
||||
TESTMAIL_NAMESPACE=your-namespace-here
|
||||
|
||||
# =============================================================================
|
||||
# E2E Testing - Authentication (optional)
|
||||
# =============================================================================
|
||||
|
||||
# Skip email enrollment by providing an existing access token
|
||||
# Useful for faster local testing when you already have credits
|
||||
# KARAOKE_ACCESS_TOKEN=your-token-here
|
||||
@@ -0,0 +1,286 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState } from "react"
|
||||
import { adminApi, AdminBetaStats, AdminBetaFeedback } from "@/lib/api"
|
||||
import { StatsCard, StatsGrid } from "@/components/admin/stats-card"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import { Badge } from "@/components/ui/badge"
|
||||
import {
|
||||
Table,
|
||||
TableBody,
|
||||
TableCell,
|
||||
TableHead,
|
||||
TableHeader,
|
||||
TableRow,
|
||||
} from "@/components/ui/table"
|
||||
import {
|
||||
Users,
|
||||
MessageSquare,
|
||||
Star,
|
||||
RefreshCw,
|
||||
Loader2,
|
||||
ThumbsUp,
|
||||
ThumbsDown,
|
||||
} from "lucide-react"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { useToast } from "@/hooks/use-toast"
|
||||
|
||||
export default function AdminBetaPage() {
|
||||
const { toast } = useToast()
|
||||
const [stats, setStats] = useState<AdminBetaStats | null>(null)
|
||||
const [feedback, setFeedback] = useState<AdminBetaFeedback[]>([])
|
||||
const [loading, setLoading] = useState(true)
|
||||
|
||||
const loadData = async () => {
|
||||
try {
|
||||
setLoading(true)
|
||||
const [statsData, feedbackData] = await Promise.all([
|
||||
adminApi.getBetaStats(),
|
||||
adminApi.getBetaFeedback(50),
|
||||
])
|
||||
setStats(statsData)
|
||||
setFeedback(feedbackData.feedback || [])
|
||||
} catch (err: any) {
|
||||
console.error("Failed to load beta data:", err)
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to load beta program data",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
loadData()
|
||||
}, [])
|
||||
|
||||
const renderStars = (rating?: number) => {
|
||||
if (!rating) return "—"
|
||||
return (
|
||||
<div className="flex gap-0.5">
|
||||
{[1, 2, 3, 4, 5].map((star) => (
|
||||
<Star
|
||||
key={star}
|
||||
className={`w-4 h-4 ${
|
||||
star <= rating ? "fill-yellow-400 text-yellow-400" : "text-muted"
|
||||
}`}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
const formatDate = (dateStr?: string) => {
|
||||
if (!dateStr) return "—"
|
||||
const date = new Date(dateStr)
|
||||
if (isNaN(date.getTime())) return "—"
|
||||
return date.toLocaleDateString()
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
{/* Header */}
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<h1 className="text-2xl font-bold tracking-tight">Beta Program</h1>
|
||||
<p className="text-muted-foreground">
|
||||
Manage beta testers and view feedback
|
||||
</p>
|
||||
</div>
|
||||
<Button variant="outline" size="sm" onClick={loadData} disabled={loading}>
|
||||
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
|
||||
Refresh
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{/* Stats Grid */}
|
||||
<StatsGrid>
|
||||
<StatsCard
|
||||
title="Total Beta Testers"
|
||||
value={stats?.total_beta_testers ?? 0}
|
||||
icon={Users}
|
||||
loading={loading}
|
||||
/>
|
||||
<StatsCard
|
||||
title="Active Testers"
|
||||
value={stats?.active_testers ?? 0}
|
||||
icon={Users}
|
||||
loading={loading}
|
||||
/>
|
||||
<StatsCard
|
||||
title="Pending Feedback"
|
||||
value={stats?.pending_feedback ?? 0}
|
||||
icon={MessageSquare}
|
||||
loading={loading}
|
||||
/>
|
||||
<StatsCard
|
||||
title="Total Feedback"
|
||||
value={stats?.total_feedback_submissions ?? 0}
|
||||
icon={MessageSquare}
|
||||
loading={loading}
|
||||
/>
|
||||
</StatsGrid>
|
||||
|
||||
{/* Average Ratings */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle className="text-base">Average Ratings</CardTitle>
|
||||
<CardDescription>Overall feedback scores from beta testers</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
{loading ? (
|
||||
<div className="flex items-center justify-center py-8">
|
||||
<Loader2 className="w-6 h-6 animate-spin text-muted-foreground" />
|
||||
</div>
|
||||
) : stats?.average_ratings ? (
|
||||
<div className="grid gap-4 md:grid-cols-4">
|
||||
<div className="rounded-lg border p-4">
|
||||
<p className="text-sm text-muted-foreground">Overall</p>
|
||||
<div className="flex items-center gap-2 mt-1">
|
||||
<span className="text-2xl font-bold">
|
||||
{stats.average_ratings.overall?.toFixed(1) || "—"}
|
||||
</span>
|
||||
<span className="text-muted-foreground">/ 5</span>
|
||||
</div>
|
||||
</div>
|
||||
<div className="rounded-lg border p-4">
|
||||
<p className="text-sm text-muted-foreground">Ease of Use</p>
|
||||
<div className="flex items-center gap-2 mt-1">
|
||||
<span className="text-2xl font-bold">
|
||||
{stats.average_ratings.ease_of_use?.toFixed(1) || "—"}
|
||||
</span>
|
||||
<span className="text-muted-foreground">/ 5</span>
|
||||
</div>
|
||||
</div>
|
||||
<div className="rounded-lg border p-4">
|
||||
<p className="text-sm text-muted-foreground">Lyrics Accuracy</p>
|
||||
<div className="flex items-center gap-2 mt-1">
|
||||
<span className="text-2xl font-bold">
|
||||
{stats.average_ratings.lyrics_accuracy?.toFixed(1) || "—"}
|
||||
</span>
|
||||
<span className="text-muted-foreground">/ 5</span>
|
||||
</div>
|
||||
</div>
|
||||
<div className="rounded-lg border p-4">
|
||||
<p className="text-sm text-muted-foreground">Correction Experience</p>
|
||||
<div className="flex items-center gap-2 mt-1">
|
||||
<span className="text-2xl font-bold">
|
||||
{stats.average_ratings.correction_experience?.toFixed(1) || "—"}
|
||||
</span>
|
||||
<span className="text-muted-foreground">/ 5</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<p className="text-muted-foreground">No ratings available yet</p>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Feedback List */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle className="text-base">Recent Feedback</CardTitle>
|
||||
<CardDescription>Feedback submissions from beta testers</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
{loading ? (
|
||||
<div className="flex items-center justify-center py-8">
|
||||
<Loader2 className="w-6 h-6 animate-spin text-muted-foreground" />
|
||||
</div>
|
||||
) : feedback.length === 0 ? (
|
||||
<p className="text-muted-foreground text-center py-8">No feedback submissions yet</p>
|
||||
) : (
|
||||
<div className="space-y-4">
|
||||
{feedback.map((item) => (
|
||||
<div key={item.id} className="border rounded-lg p-4">
|
||||
<div className="flex items-start justify-between gap-4">
|
||||
<div className="flex-1">
|
||||
<div className="flex items-center gap-2 mb-2">
|
||||
<span className="font-medium text-sm">{item.user_email}</span>
|
||||
{item.job_id && (
|
||||
<Badge variant="outline" className="text-xs">
|
||||
Job: {item.job_id}
|
||||
</Badge>
|
||||
)}
|
||||
<span className="text-xs text-muted-foreground">
|
||||
{formatDate(item.created_at)}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{/* Ratings */}
|
||||
<div className="flex flex-wrap gap-4 mb-3 text-sm">
|
||||
<div className="flex items-center gap-1">
|
||||
<span className="text-muted-foreground">Overall:</span>
|
||||
{renderStars(item.overall_rating)}
|
||||
</div>
|
||||
{item.ease_of_use_rating && (
|
||||
<div className="flex items-center gap-1">
|
||||
<span className="text-muted-foreground">Ease:</span>
|
||||
{renderStars(item.ease_of_use_rating)}
|
||||
</div>
|
||||
)}
|
||||
{item.lyrics_accuracy_rating && (
|
||||
<div className="flex items-center gap-1">
|
||||
<span className="text-muted-foreground">Accuracy:</span>
|
||||
{renderStars(item.lyrics_accuracy_rating)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Comments */}
|
||||
{item.what_went_well && (
|
||||
<div className="mb-2">
|
||||
<p className="text-xs text-muted-foreground">What went well:</p>
|
||||
<p className="text-sm">{item.what_went_well}</p>
|
||||
</div>
|
||||
)}
|
||||
{item.what_could_improve && (
|
||||
<div className="mb-2">
|
||||
<p className="text-xs text-muted-foreground">What could improve:</p>
|
||||
<p className="text-sm">{item.what_could_improve}</p>
|
||||
</div>
|
||||
)}
|
||||
{item.additional_comments && (
|
||||
<div>
|
||||
<p className="text-xs text-muted-foreground">Additional comments:</p>
|
||||
<p className="text-sm">{item.additional_comments}</p>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Recommend/Use Again */}
|
||||
<div className="flex flex-col gap-1 text-sm">
|
||||
{item.would_recommend !== undefined && (
|
||||
<div className="flex items-center gap-1">
|
||||
{item.would_recommend ? (
|
||||
<ThumbsUp className="w-4 h-4 text-green-500" />
|
||||
) : (
|
||||
<ThumbsDown className="w-4 h-4 text-red-500" />
|
||||
)}
|
||||
<span className="text-xs text-muted-foreground">Recommend</span>
|
||||
</div>
|
||||
)}
|
||||
{item.would_use_again !== undefined && (
|
||||
<div className="flex items-center gap-1">
|
||||
{item.would_use_again ? (
|
||||
<ThumbsUp className="w-4 h-4 text-green-500" />
|
||||
) : (
|
||||
<ThumbsDown className="w-4 h-4 text-red-500" />
|
||||
)}
|
||||
<span className="text-xs text-muted-foreground">Use Again</span>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,428 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState } from "react"
|
||||
import { useSearchParams, useRouter } from "next/navigation"
|
||||
import { api, adminApi, Job } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Badge } from "@/components/ui/badge"
|
||||
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import {
|
||||
Accordion,
|
||||
AccordionContent,
|
||||
AccordionItem,
|
||||
AccordionTrigger,
|
||||
} from "@/components/ui/accordion"
|
||||
import {
|
||||
AlertDialog,
|
||||
AlertDialogAction,
|
||||
AlertDialogCancel,
|
||||
AlertDialogContent,
|
||||
AlertDialogDescription,
|
||||
AlertDialogFooter,
|
||||
AlertDialogHeader,
|
||||
AlertDialogTitle,
|
||||
} from "@/components/ui/alert-dialog"
|
||||
import {
|
||||
ArrowLeft,
|
||||
RefreshCw,
|
||||
Loader2,
|
||||
Trash2,
|
||||
User,
|
||||
Clock,
|
||||
FileText,
|
||||
XCircle,
|
||||
} from "lucide-react"
|
||||
import { useToast } from "@/hooks/use-toast"
|
||||
|
||||
export default function AdminJobDetailPage() {
|
||||
const searchParams = useSearchParams()
|
||||
const router = useRouter()
|
||||
const { toast } = useToast()
|
||||
const jobId = searchParams.get("id") || ""
|
||||
|
||||
const [job, setJob] = useState<Job | null>(null)
|
||||
const [logs, setLogs] = useState<any[]>([])
|
||||
const [loading, setLoading] = useState(true)
|
||||
const [logsLoading, setLogsLoading] = useState(false)
|
||||
|
||||
// Delete dialog
|
||||
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false)
|
||||
const [deleting, setDeleting] = useState(false)
|
||||
|
||||
const loadJob = async () => {
|
||||
if (!jobId) {
|
||||
setLoading(false)
|
||||
return
|
||||
}
|
||||
try {
|
||||
setLoading(true)
|
||||
const data = await api.getJob(jobId)
|
||||
setJob(data)
|
||||
} catch (err: any) {
|
||||
console.error("Failed to load job:", err)
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to load job",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
const loadLogs = async () => {
|
||||
if (!jobId) return
|
||||
try {
|
||||
setLogsLoading(true)
|
||||
const data = await api.getJobLogs(jobId, 200)
|
||||
setLogs(data)
|
||||
} catch (err: any) {
|
||||
console.error("Failed to load logs:", err)
|
||||
} finally {
|
||||
setLogsLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
loadJob()
|
||||
loadLogs()
|
||||
}, [jobId])
|
||||
|
||||
const handleDelete = async () => {
|
||||
try {
|
||||
setDeleting(true)
|
||||
await adminApi.deleteJob(jobId)
|
||||
toast({
|
||||
title: "Job Deleted",
|
||||
description: `Job ${jobId} has been deleted`,
|
||||
})
|
||||
router.push("/admin/jobs")
|
||||
} catch (err: any) {
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to delete job",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setDeleting(false)
|
||||
}
|
||||
}
|
||||
|
||||
const getStatusVariant = (status: string) => {
|
||||
if (status === "complete" || status === "prep_complete") return "default"
|
||||
if (status === "failed") return "destructive"
|
||||
if (status === "cancelled") return "outline"
|
||||
if (status.includes("awaiting")) return "secondary"
|
||||
return "secondary"
|
||||
}
|
||||
|
||||
const getLogLevelColor = (level: string) => {
|
||||
switch (level?.toUpperCase()) {
|
||||
case "ERROR":
|
||||
return "text-red-500"
|
||||
case "WARNING":
|
||||
return "text-yellow-500"
|
||||
case "INFO":
|
||||
return "text-blue-500"
|
||||
case "DEBUG":
|
||||
return "text-gray-500"
|
||||
default:
|
||||
return "text-foreground"
|
||||
}
|
||||
}
|
||||
|
||||
const formatDate = (dateStr?: string) => {
|
||||
if (!dateStr) return "—"
|
||||
return new Date(dateStr).toLocaleString()
|
||||
}
|
||||
|
||||
if (!jobId) {
|
||||
return (
|
||||
<div className="text-center py-12">
|
||||
<p className="text-muted-foreground">No job ID provided</p>
|
||||
<Button variant="outline" className="mt-4" onClick={() => router.push("/admin/jobs")}>
|
||||
<ArrowLeft className="w-4 h-4 mr-2" />
|
||||
Back to Jobs
|
||||
</Button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
if (loading) {
|
||||
return (
|
||||
<div className="flex items-center justify-center py-12">
|
||||
<Loader2 className="w-8 h-8 animate-spin text-muted-foreground" />
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
if (!job) {
|
||||
return (
|
||||
<div className="text-center py-12">
|
||||
<p className="text-muted-foreground">Job not found</p>
|
||||
<Button variant="outline" className="mt-4" onClick={() => router.push("/admin/jobs")}>
|
||||
<ArrowLeft className="w-4 h-4 mr-2" />
|
||||
Back to Jobs
|
||||
</Button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
{/* Header */}
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-4">
|
||||
<Button variant="ghost" size="icon" onClick={() => router.push("/admin/jobs")}>
|
||||
<ArrowLeft className="w-4 h-4" />
|
||||
</Button>
|
||||
<div>
|
||||
<h1 className="text-2xl font-bold tracking-tight flex items-center gap-2">
|
||||
Job {job.job_id}
|
||||
<Badge variant={getStatusVariant(job.status)}>
|
||||
{job.status.replace(/_/g, " ")}
|
||||
</Badge>
|
||||
</h1>
|
||||
{job.artist && job.title && (
|
||||
<p className="text-muted-foreground">{job.artist} - {job.title}</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
<div className="flex gap-2">
|
||||
<Button variant="outline" size="sm" onClick={() => { loadJob(); loadLogs(); }} disabled={loading}>
|
||||
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
|
||||
Refresh
|
||||
</Button>
|
||||
<Button
|
||||
variant="destructive"
|
||||
size="sm"
|
||||
onClick={() => setDeleteDialogOpen(true)}
|
||||
>
|
||||
<Trash2 className="w-4 h-4 mr-2" />
|
||||
Delete
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Overview Cards */}
|
||||
<div className="grid gap-4 md:grid-cols-4">
|
||||
<Card>
|
||||
<CardHeader className="pb-2">
|
||||
<CardTitle className="text-sm font-medium flex items-center gap-2">
|
||||
<User className="w-4 h-4" />
|
||||
User
|
||||
</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<p className="text-sm truncate">
|
||||
{job.user_email || "Unknown"}
|
||||
</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
<Card>
|
||||
<CardHeader className="pb-2">
|
||||
<CardTitle className="text-sm font-medium flex items-center gap-2">
|
||||
<Clock className="w-4 h-4" />
|
||||
Created
|
||||
</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<p className="text-sm">{formatDate(job.created_at)}</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
<Card>
|
||||
<CardHeader className="pb-2">
|
||||
<CardTitle className="text-sm font-medium flex items-center gap-2">
|
||||
<Clock className="w-4 h-4" />
|
||||
Updated
|
||||
</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<p className="text-sm">{formatDate(job.updated_at)}</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
<Card>
|
||||
<CardHeader className="pb-2">
|
||||
<CardTitle className="text-sm font-medium flex items-center gap-2">
|
||||
Progress
|
||||
</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="flex items-center gap-2">
|
||||
<div className="flex-1 h-2 bg-muted rounded-full overflow-hidden">
|
||||
<div
|
||||
className="h-full bg-primary transition-all"
|
||||
style={{ width: `${job.progress}%` }}
|
||||
/>
|
||||
</div>
|
||||
<span className="text-sm font-medium">{job.progress}%</span>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
{/* Error Message */}
|
||||
{job.error_message && (
|
||||
<Card className="border-destructive">
|
||||
<CardHeader>
|
||||
<CardTitle className="text-destructive flex items-center gap-2">
|
||||
<XCircle className="w-5 h-5" />
|
||||
Error
|
||||
</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<p className="text-sm">{job.error_message}</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
)}
|
||||
|
||||
{/* Accordion Sections */}
|
||||
<Accordion type="multiple" className="w-full" defaultValue={["timeline", "logs"]}>
|
||||
{/* Timeline */}
|
||||
<AccordionItem value="timeline">
|
||||
<AccordionTrigger>
|
||||
<div className="flex items-center gap-2">
|
||||
<Clock className="w-4 h-4" />
|
||||
Timeline ({job.timeline?.length || 0} events)
|
||||
</div>
|
||||
</AccordionTrigger>
|
||||
<AccordionContent>
|
||||
<div className="space-y-2 max-h-[400px] overflow-y-auto">
|
||||
{job.timeline && job.timeline.length > 0 ? (
|
||||
job.timeline.map((event, i) => (
|
||||
<div
|
||||
key={i}
|
||||
className="flex items-start gap-3 text-sm border-l-2 border-muted pl-4 py-1"
|
||||
>
|
||||
<div className="flex-1">
|
||||
<div className="flex items-center gap-2">
|
||||
<Badge variant="outline" className="text-xs">
|
||||
{event.status}
|
||||
</Badge>
|
||||
{event.progress !== undefined && (
|
||||
<span className="text-muted-foreground">
|
||||
{event.progress}%
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
{event.message && (
|
||||
<p className="text-muted-foreground mt-1">{event.message}</p>
|
||||
)}
|
||||
</div>
|
||||
<span className="text-xs text-muted-foreground whitespace-nowrap">
|
||||
{formatDate(event.timestamp)}
|
||||
</span>
|
||||
</div>
|
||||
))
|
||||
) : (
|
||||
<p className="text-muted-foreground text-sm">No timeline events</p>
|
||||
)}
|
||||
</div>
|
||||
</AccordionContent>
|
||||
</AccordionItem>
|
||||
|
||||
{/* Worker Logs */}
|
||||
<AccordionItem value="logs">
|
||||
<AccordionTrigger>
|
||||
<div className="flex items-center gap-2">
|
||||
<FileText className="w-4 h-4" />
|
||||
Worker Logs ({logs.length} entries)
|
||||
{logsLoading && <Loader2 className="w-4 h-4 animate-spin ml-2" />}
|
||||
</div>
|
||||
</AccordionTrigger>
|
||||
<AccordionContent>
|
||||
<div className="space-y-1 max-h-[500px] overflow-y-auto font-mono text-xs">
|
||||
{logs.length > 0 ? (
|
||||
logs.map((log, i) => (
|
||||
<div key={i} className="flex gap-2 py-0.5 border-b border-muted/50">
|
||||
<span className="text-muted-foreground whitespace-nowrap">
|
||||
{log.timestamp ? new Date(log.timestamp).toLocaleTimeString() : "—"}
|
||||
</span>
|
||||
<span className={`w-16 ${getLogLevelColor(log.level)}`}>
|
||||
[{log.level}]
|
||||
</span>
|
||||
<span className="text-muted-foreground w-20">
|
||||
{log.worker || "—"}
|
||||
</span>
|
||||
<span className="flex-1 break-all">{log.message}</span>
|
||||
</div>
|
||||
))
|
||||
) : (
|
||||
<p className="text-muted-foreground">No logs available</p>
|
||||
)}
|
||||
</div>
|
||||
</AccordionContent>
|
||||
</AccordionItem>
|
||||
|
||||
{/* State Data */}
|
||||
<AccordionItem value="state">
|
||||
<AccordionTrigger>
|
||||
<div className="flex items-center gap-2">
|
||||
<FileText className="w-4 h-4" />
|
||||
State Data
|
||||
</div>
|
||||
</AccordionTrigger>
|
||||
<AccordionContent>
|
||||
<pre className="text-xs bg-muted p-4 rounded-md overflow-x-auto max-h-[400px]">
|
||||
{JSON.stringify(job.state_data || {}, null, 2)}
|
||||
</pre>
|
||||
</AccordionContent>
|
||||
</AccordionItem>
|
||||
|
||||
{/* File URLs */}
|
||||
<AccordionItem value="files">
|
||||
<AccordionTrigger>
|
||||
<div className="flex items-center gap-2">
|
||||
<FileText className="w-4 h-4" />
|
||||
File URLs
|
||||
</div>
|
||||
</AccordionTrigger>
|
||||
<AccordionContent>
|
||||
<pre className="text-xs bg-muted p-4 rounded-md overflow-x-auto max-h-[400px]">
|
||||
{JSON.stringify(job.file_urls || {}, null, 2)}
|
||||
</pre>
|
||||
</AccordionContent>
|
||||
</AccordionItem>
|
||||
|
||||
{/* Raw Job Data */}
|
||||
<AccordionItem value="raw">
|
||||
<AccordionTrigger>
|
||||
<div className="flex items-center gap-2">
|
||||
<FileText className="w-4 h-4" />
|
||||
Raw Job Data
|
||||
</div>
|
||||
</AccordionTrigger>
|
||||
<AccordionContent>
|
||||
<pre className="text-xs bg-muted p-4 rounded-md overflow-x-auto max-h-[500px]">
|
||||
{JSON.stringify(job, null, 2)}
|
||||
</pre>
|
||||
</AccordionContent>
|
||||
</AccordionItem>
|
||||
</Accordion>
|
||||
|
||||
{/* Delete Confirmation Dialog */}
|
||||
<AlertDialog open={deleteDialogOpen} onOpenChange={setDeleteDialogOpen}>
|
||||
<AlertDialogContent>
|
||||
<AlertDialogHeader>
|
||||
<AlertDialogTitle>Delete Job</AlertDialogTitle>
|
||||
<AlertDialogDescription>
|
||||
Are you sure you want to delete job {jobId}? This action cannot be
|
||||
undone and will also delete all associated files.
|
||||
</AlertDialogDescription>
|
||||
</AlertDialogHeader>
|
||||
<AlertDialogFooter>
|
||||
<AlertDialogCancel>Cancel</AlertDialogCancel>
|
||||
<AlertDialogAction
|
||||
onClick={handleDelete}
|
||||
disabled={deleting}
|
||||
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
|
||||
>
|
||||
{deleting && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
|
||||
Delete
|
||||
</AlertDialogAction>
|
||||
</AlertDialogFooter>
|
||||
</AlertDialogContent>
|
||||
</AlertDialog>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,327 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState, useCallback } from "react"
|
||||
import { useRouter } from "next/navigation"
|
||||
import { adminApi, Job } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Input } from "@/components/ui/input"
|
||||
import { Badge } from "@/components/ui/badge"
|
||||
import {
|
||||
Table,
|
||||
TableBody,
|
||||
TableCell,
|
||||
TableHead,
|
||||
TableHeader,
|
||||
TableRow,
|
||||
} from "@/components/ui/table"
|
||||
import {
|
||||
Select,
|
||||
SelectContent,
|
||||
SelectItem,
|
||||
SelectTrigger,
|
||||
SelectValue,
|
||||
} from "@/components/ui/select"
|
||||
import {
|
||||
AlertDialog,
|
||||
AlertDialogAction,
|
||||
AlertDialogCancel,
|
||||
AlertDialogContent,
|
||||
AlertDialogDescription,
|
||||
AlertDialogFooter,
|
||||
AlertDialogHeader,
|
||||
AlertDialogTitle,
|
||||
} from "@/components/ui/alert-dialog"
|
||||
import {
|
||||
Search,
|
||||
RefreshCw,
|
||||
Loader2,
|
||||
Trash2,
|
||||
ExternalLink,
|
||||
} from "lucide-react"
|
||||
import { useToast } from "@/hooks/use-toast"
|
||||
|
||||
const statusOptions = [
|
||||
{ value: "all", label: "All Statuses" },
|
||||
{ value: "pending", label: "Pending" },
|
||||
{ value: "downloading", label: "Downloading" },
|
||||
{ value: "separating_stage1", label: "Separating (Stage 1)" },
|
||||
{ value: "separating_stage2", label: "Separating (Stage 2)" },
|
||||
{ value: "transcribing", label: "Transcribing" },
|
||||
{ value: "correcting", label: "Correcting" },
|
||||
{ value: "awaiting_review", label: "Awaiting Review" },
|
||||
{ value: "in_review", label: "In Review" },
|
||||
{ value: "rendering_video", label: "Rendering Video" },
|
||||
{ value: "awaiting_instrumental_selection", label: "Awaiting Instrumental" },
|
||||
{ value: "generating_video", label: "Generating Video" },
|
||||
{ value: "encoding", label: "Encoding" },
|
||||
{ value: "complete", label: "Complete" },
|
||||
{ value: "failed", label: "Failed" },
|
||||
{ value: "cancelled", label: "Cancelled" },
|
||||
]
|
||||
|
||||
export default function AdminJobsPage() {
|
||||
const router = useRouter()
|
||||
const { toast } = useToast()
|
||||
const [jobs, setJobs] = useState<Job[]>([])
|
||||
const [loading, setLoading] = useState(true)
|
||||
const [statusFilter, setStatusFilter] = useState("all")
|
||||
const [userEmailFilter, setUserEmailFilter] = useState("")
|
||||
const [searchInput, setSearchInput] = useState("")
|
||||
|
||||
// Delete dialog
|
||||
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false)
|
||||
const [jobToDelete, setJobToDelete] = useState<Job | null>(null)
|
||||
const [deleting, setDeleting] = useState(false)
|
||||
|
||||
const loadJobs = useCallback(async () => {
|
||||
try {
|
||||
setLoading(true)
|
||||
const data = await adminApi.listAllJobs({
|
||||
status: statusFilter !== "all" ? statusFilter : undefined,
|
||||
user_email: userEmailFilter || undefined,
|
||||
limit: 100,
|
||||
})
|
||||
setJobs(data)
|
||||
} catch (err: any) {
|
||||
console.error("Failed to load jobs:", err)
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to load jobs",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}, [statusFilter, userEmailFilter, toast])
|
||||
|
||||
useEffect(() => {
|
||||
loadJobs()
|
||||
}, [loadJobs])
|
||||
|
||||
const handleSearch = () => {
|
||||
setUserEmailFilter(searchInput)
|
||||
}
|
||||
|
||||
const handleKeyPress = (e: React.KeyboardEvent) => {
|
||||
if (e.key === "Enter") {
|
||||
handleSearch()
|
||||
}
|
||||
}
|
||||
|
||||
const handleDelete = async () => {
|
||||
if (!jobToDelete) return
|
||||
|
||||
try {
|
||||
setDeleting(true)
|
||||
await adminApi.deleteJob(jobToDelete.job_id)
|
||||
toast({
|
||||
title: "Job Deleted",
|
||||
description: `Job ${jobToDelete.job_id} has been deleted`,
|
||||
})
|
||||
setDeleteDialogOpen(false)
|
||||
setJobToDelete(null)
|
||||
loadJobs()
|
||||
} catch (err: any) {
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to delete job",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setDeleting(false)
|
||||
}
|
||||
}
|
||||
|
||||
const getStatusVariant = (status: string) => {
|
||||
if (status === "complete" || status === "prep_complete") return "default"
|
||||
if (status === "failed") return "destructive"
|
||||
if (status === "cancelled") return "outline"
|
||||
if (status.includes("awaiting")) return "secondary"
|
||||
return "secondary"
|
||||
}
|
||||
|
||||
const formatDate = (dateStr?: string) => {
|
||||
if (!dateStr) return "—"
|
||||
return new Date(dateStr).toLocaleString()
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
{/* Header */}
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<h1 className="text-2xl font-bold tracking-tight">Jobs</h1>
|
||||
<p className="text-muted-foreground">
|
||||
View and manage all jobs in the system
|
||||
</p>
|
||||
</div>
|
||||
<Button variant="outline" size="sm" onClick={loadJobs} disabled={loading}>
|
||||
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
|
||||
Refresh
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{/* Filters */}
|
||||
<div className="flex flex-col sm:flex-row gap-4">
|
||||
<div className="flex-1 flex gap-2">
|
||||
<Input
|
||||
placeholder="Filter by user email..."
|
||||
value={searchInput}
|
||||
onChange={(e) => setSearchInput(e.target.value)}
|
||||
onKeyDown={handleKeyPress}
|
||||
className="max-w-sm"
|
||||
/>
|
||||
<Button variant="secondary" onClick={handleSearch}>
|
||||
<Search className="w-4 h-4" />
|
||||
</Button>
|
||||
</div>
|
||||
<Select value={statusFilter} onValueChange={setStatusFilter}>
|
||||
<SelectTrigger className="w-[200px]">
|
||||
<SelectValue placeholder="Filter by status" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
{statusOptions.map((option) => (
|
||||
<SelectItem key={option.value} value={option.value}>
|
||||
{option.label}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</div>
|
||||
|
||||
{/* Table */}
|
||||
<div className="rounded-md border">
|
||||
<Table>
|
||||
<TableHeader>
|
||||
<TableRow>
|
||||
<TableHead>Job ID</TableHead>
|
||||
<TableHead>User</TableHead>
|
||||
<TableHead>Artist / Title</TableHead>
|
||||
<TableHead>Status</TableHead>
|
||||
<TableHead>Progress</TableHead>
|
||||
<TableHead>Created</TableHead>
|
||||
<TableHead className="text-right">Actions</TableHead>
|
||||
</TableRow>
|
||||
</TableHeader>
|
||||
<TableBody>
|
||||
{loading ? (
|
||||
<TableRow>
|
||||
<TableCell colSpan={7} className="text-center py-8">
|
||||
<Loader2 className="w-6 h-6 animate-spin mx-auto text-muted-foreground" />
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
) : jobs.length === 0 ? (
|
||||
<TableRow>
|
||||
<TableCell colSpan={7} className="text-center py-8 text-muted-foreground">
|
||||
No jobs found
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
) : (
|
||||
jobs.map((job) => (
|
||||
<TableRow
|
||||
key={job.job_id}
|
||||
className="cursor-pointer hover:bg-muted/50"
|
||||
onClick={() => router.push(`/admin/jobs/${job.job_id}`)}
|
||||
>
|
||||
<TableCell className="font-mono text-sm">{job.job_id}</TableCell>
|
||||
<TableCell className="text-sm text-muted-foreground max-w-[150px] truncate">
|
||||
{job.user_email || "—"}
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
{job.artist && job.title ? (
|
||||
<span className="max-w-[200px] truncate block">
|
||||
{job.artist} - {job.title}
|
||||
</span>
|
||||
) : job.filename ? (
|
||||
<span className="max-w-[200px] truncate block text-muted-foreground">
|
||||
{job.filename}
|
||||
</span>
|
||||
) : (
|
||||
"—"
|
||||
)}
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
<Badge variant={getStatusVariant(job.status)}>
|
||||
{job.status.replace(/_/g, " ")}
|
||||
</Badge>
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
<div className="flex items-center gap-2">
|
||||
<div className="w-16 h-2 bg-muted rounded-full overflow-hidden">
|
||||
<div
|
||||
className="h-full bg-primary transition-all"
|
||||
style={{ width: `${job.progress}%` }}
|
||||
/>
|
||||
</div>
|
||||
<span className="text-xs text-muted-foreground">
|
||||
{job.progress}%
|
||||
</span>
|
||||
</div>
|
||||
</TableCell>
|
||||
<TableCell className="text-sm">
|
||||
{formatDate(job.created_at)}
|
||||
</TableCell>
|
||||
<TableCell className="text-right">
|
||||
<div className="flex justify-end gap-1" onClick={(e) => e.stopPropagation()}>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
onClick={() => router.push(`/admin/jobs/${job.job_id}`)}
|
||||
title="View details"
|
||||
>
|
||||
<ExternalLink className="w-4 h-4" />
|
||||
</Button>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
onClick={() => {
|
||||
setJobToDelete(job)
|
||||
setDeleteDialogOpen(true)
|
||||
}}
|
||||
title="Delete job"
|
||||
className="text-destructive hover:text-destructive"
|
||||
>
|
||||
<Trash2 className="w-4 h-4" />
|
||||
</Button>
|
||||
</div>
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
))
|
||||
)}
|
||||
</TableBody>
|
||||
</Table>
|
||||
</div>
|
||||
|
||||
{/* Summary */}
|
||||
<p className="text-sm text-muted-foreground">
|
||||
Showing {jobs.length} jobs
|
||||
{statusFilter !== "all" && ` with status "${statusFilter}"`}
|
||||
{userEmailFilter && ` for user "${userEmailFilter}"`}
|
||||
</p>
|
||||
|
||||
{/* Delete Confirmation Dialog */}
|
||||
<AlertDialog open={deleteDialogOpen} onOpenChange={setDeleteDialogOpen}>
|
||||
<AlertDialogContent>
|
||||
<AlertDialogHeader>
|
||||
<AlertDialogTitle>Delete Job</AlertDialogTitle>
|
||||
<AlertDialogDescription>
|
||||
Are you sure you want to delete job {jobToDelete?.job_id}? This action
|
||||
cannot be undone and will also delete all associated files.
|
||||
</AlertDialogDescription>
|
||||
</AlertDialogHeader>
|
||||
<AlertDialogFooter>
|
||||
<AlertDialogCancel>Cancel</AlertDialogCancel>
|
||||
<AlertDialogAction
|
||||
onClick={handleDelete}
|
||||
disabled={deleting}
|
||||
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
|
||||
>
|
||||
{deleting && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
|
||||
Delete
|
||||
</AlertDialogAction>
|
||||
</AlertDialogFooter>
|
||||
</AlertDialogContent>
|
||||
</AlertDialog>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
"use client"
|
||||
|
||||
import React, { useEffect, useState } from "react"
|
||||
import { useRouter } from "next/navigation"
|
||||
import { getAccessToken } from "@/lib/api"
|
||||
import { useAuth } from "@/lib/auth"
|
||||
import { Loader2 } from "lucide-react"
|
||||
import { SidebarProvider, SidebarInset, SidebarTrigger } from "@/components/ui/sidebar"
|
||||
import { AdminSidebar } from "@/components/admin/admin-sidebar"
|
||||
import { Separator } from "@/components/ui/separator"
|
||||
import {
|
||||
Breadcrumb,
|
||||
BreadcrumbItem,
|
||||
BreadcrumbLink,
|
||||
BreadcrumbList,
|
||||
BreadcrumbPage,
|
||||
BreadcrumbSeparator,
|
||||
} from "@/components/ui/breadcrumb"
|
||||
import { usePathname } from "next/navigation"
|
||||
|
||||
function AdminBreadcrumb() {
|
||||
const pathname = usePathname()
|
||||
|
||||
const getBreadcrumbs = () => {
|
||||
const segments = pathname.split("/").filter(Boolean)
|
||||
const breadcrumbs: { label: string; href?: string }[] = []
|
||||
|
||||
// Always start with Admin
|
||||
if (segments[0] === "admin") {
|
||||
if (segments.length === 1) {
|
||||
breadcrumbs.push({ label: "Dashboard" })
|
||||
} else {
|
||||
breadcrumbs.push({ label: "Admin", href: "/admin" })
|
||||
|
||||
if (segments[1] === "users") {
|
||||
if (segments.length === 2) {
|
||||
breadcrumbs.push({ label: "Users" })
|
||||
} else {
|
||||
breadcrumbs.push({ label: "Users", href: "/admin/users" })
|
||||
breadcrumbs.push({ label: decodeURIComponent(segments[2]) })
|
||||
}
|
||||
} else if (segments[1] === "jobs") {
|
||||
if (segments.length === 2) {
|
||||
breadcrumbs.push({ label: "Jobs" })
|
||||
} else if (segments[2]) {
|
||||
breadcrumbs.push({ label: "Jobs", href: "/admin/jobs" })
|
||||
breadcrumbs.push({ label: decodeURIComponent(segments[2]) })
|
||||
}
|
||||
} else if (segments[1] === "beta") {
|
||||
breadcrumbs.push({ label: "Beta Program" })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return breadcrumbs
|
||||
}
|
||||
|
||||
const breadcrumbs = getBreadcrumbs()
|
||||
|
||||
return (
|
||||
<Breadcrumb>
|
||||
<BreadcrumbList>
|
||||
{breadcrumbs.map((crumb, index) => (
|
||||
<React.Fragment key={`${crumb.href || ''}-${crumb.label}`}>
|
||||
{index > 0 && <BreadcrumbSeparator />}
|
||||
<BreadcrumbItem>
|
||||
{crumb.href ? (
|
||||
<BreadcrumbLink href={crumb.href}>{crumb.label}</BreadcrumbLink>
|
||||
) : (
|
||||
<BreadcrumbPage>{crumb.label}</BreadcrumbPage>
|
||||
)}
|
||||
</BreadcrumbItem>
|
||||
</React.Fragment>
|
||||
))}
|
||||
</BreadcrumbList>
|
||||
</Breadcrumb>
|
||||
)
|
||||
}
|
||||
|
||||
export default function AdminLayout({
|
||||
children,
|
||||
}: {
|
||||
children: React.ReactNode
|
||||
}) {
|
||||
const router = useRouter()
|
||||
const { user } = useAuth()
|
||||
const [isChecking, setIsChecking] = useState(true)
|
||||
|
||||
useEffect(() => {
|
||||
const token = getAccessToken()
|
||||
if (!token) {
|
||||
router.replace("/")
|
||||
return
|
||||
}
|
||||
|
||||
// Check if user is admin (either by role or @nomadkaraoke.com email)
|
||||
if (user) {
|
||||
const isAdmin = user.role === "admin" || user.email?.endsWith("@nomadkaraoke.com")
|
||||
if (!isAdmin) {
|
||||
router.replace("/app")
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// If we have a token but no user yet, wait for auth to load
|
||||
if (!user) {
|
||||
return
|
||||
}
|
||||
|
||||
setIsChecking(false)
|
||||
}, [router, user])
|
||||
|
||||
// Show loading while checking auth
|
||||
if (isChecking || !user) {
|
||||
return (
|
||||
<div
|
||||
className="min-h-screen flex items-center justify-center"
|
||||
style={{ background: "var(--bg)" }}
|
||||
>
|
||||
<Loader2 className="w-8 h-8 animate-spin" style={{ color: "var(--text-muted)" }} />
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<SidebarProvider>
|
||||
<AdminSidebar />
|
||||
<SidebarInset>
|
||||
<header className="flex h-16 shrink-0 items-center gap-2 border-b px-4">
|
||||
<SidebarTrigger className="-ml-1" />
|
||||
<Separator orientation="vertical" className="mr-2 h-4" />
|
||||
<AdminBreadcrumb />
|
||||
</header>
|
||||
<main className="flex-1 p-4 md:p-6">{children}</main>
|
||||
</SidebarInset>
|
||||
</SidebarProvider>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState } from "react"
|
||||
import { adminApi, AdminStatsOverview } from "@/lib/api"
|
||||
import { StatsCard, StatsGrid } from "@/components/admin/stats-card"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import { Badge } from "@/components/ui/badge"
|
||||
import {
|
||||
Users,
|
||||
Briefcase,
|
||||
CreditCard,
|
||||
TestTube2,
|
||||
Clock,
|
||||
CheckCircle,
|
||||
XCircle,
|
||||
AlertCircle,
|
||||
Loader2,
|
||||
RefreshCw,
|
||||
} from "lucide-react"
|
||||
import { Button } from "@/components/ui/button"
|
||||
|
||||
export default function AdminDashboardPage() {
|
||||
const [stats, setStats] = useState<AdminStatsOverview | null>(null)
|
||||
const [loading, setLoading] = useState(true)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
|
||||
const loadStats = async () => {
|
||||
try {
|
||||
setLoading(true)
|
||||
setError(null)
|
||||
const data = await adminApi.getStats()
|
||||
setStats(data)
|
||||
} catch (err: unknown) {
|
||||
const message = err instanceof Error ? err.message : "Failed to load statistics"
|
||||
setError(message)
|
||||
console.error("Failed to load admin stats:", err)
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
loadStats()
|
||||
}, [])
|
||||
|
||||
if (error) {
|
||||
return (
|
||||
<div className="flex flex-col items-center justify-center py-12">
|
||||
<XCircle className="w-12 h-12 text-destructive mb-4" />
|
||||
<h2 className="text-lg font-semibold mb-2">Failed to load statistics</h2>
|
||||
<p className="text-muted-foreground mb-4">{error}</p>
|
||||
<Button onClick={loadStats}>
|
||||
<RefreshCw className="w-4 h-4 mr-2" />
|
||||
Retry
|
||||
</Button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
{/* Header */}
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<h1 className="text-2xl font-bold tracking-tight">Dashboard</h1>
|
||||
<p className="text-muted-foreground">
|
||||
Overview of your karaoke platform
|
||||
</p>
|
||||
</div>
|
||||
<Button variant="outline" size="sm" onClick={loadStats} disabled={loading}>
|
||||
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
|
||||
Refresh
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{/* Main Stats Grid */}
|
||||
<StatsGrid>
|
||||
<StatsCard
|
||||
title="Total Users"
|
||||
value={stats?.total_users ?? 0}
|
||||
description={`${stats?.active_users_7d ?? 0} active in last 7 days`}
|
||||
icon={Users}
|
||||
loading={loading}
|
||||
/>
|
||||
<StatsCard
|
||||
title="Total Jobs"
|
||||
value={stats?.total_jobs ?? 0}
|
||||
description={`${stats?.jobs_last_7d ?? 0} in last 7 days`}
|
||||
icon={Briefcase}
|
||||
loading={loading}
|
||||
/>
|
||||
<StatsCard
|
||||
title="Credits Issued (30d)"
|
||||
value={stats?.total_credits_issued_30d ?? 0}
|
||||
description="Credits added to accounts"
|
||||
icon={CreditCard}
|
||||
loading={loading}
|
||||
/>
|
||||
<StatsCard
|
||||
title="Beta Testers"
|
||||
value={stats?.total_beta_testers ?? 0}
|
||||
description="Active beta participants"
|
||||
icon={TestTube2}
|
||||
loading={loading}
|
||||
/>
|
||||
</StatsGrid>
|
||||
|
||||
{/* Secondary Stats */}
|
||||
<div className="grid gap-4 md:grid-cols-2">
|
||||
{/* Job Status Breakdown */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle className="text-base">Job Status Breakdown</CardTitle>
|
||||
<CardDescription>Current status of all jobs in the system</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
{loading ? (
|
||||
<div className="flex items-center justify-center py-8">
|
||||
<Loader2 className="w-6 h-6 animate-spin text-muted-foreground" />
|
||||
</div>
|
||||
) : (
|
||||
<div className="space-y-3">
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<Clock className="w-4 h-4 text-yellow-500" />
|
||||
<span className="text-sm">Pending</span>
|
||||
</div>
|
||||
<Badge variant="secondary">{stats?.jobs_by_status?.pending ?? 0}</Badge>
|
||||
</div>
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<Loader2 className="w-4 h-4 text-blue-500" />
|
||||
<span className="text-sm">Processing</span>
|
||||
</div>
|
||||
<Badge variant="secondary">{stats?.jobs_by_status?.processing ?? 0}</Badge>
|
||||
</div>
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<AlertCircle className="w-4 h-4 text-orange-500" />
|
||||
<span className="text-sm">Awaiting Review</span>
|
||||
</div>
|
||||
<Badge variant="secondary">{stats?.jobs_by_status?.awaiting_review ?? 0}</Badge>
|
||||
</div>
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<AlertCircle className="w-4 h-4 text-purple-500" />
|
||||
<span className="text-sm">Awaiting Instrumental</span>
|
||||
</div>
|
||||
<Badge variant="secondary">{stats?.jobs_by_status?.awaiting_instrumental ?? 0}</Badge>
|
||||
</div>
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<CheckCircle className="w-4 h-4 text-green-500" />
|
||||
<span className="text-sm">Complete</span>
|
||||
</div>
|
||||
<Badge variant="secondary">{stats?.jobs_by_status?.complete ?? 0}</Badge>
|
||||
</div>
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<XCircle className="w-4 h-4 text-red-500" />
|
||||
<span className="text-sm">Failed</span>
|
||||
</div>
|
||||
<Badge variant="destructive">{stats?.jobs_by_status?.failed ?? 0}</Badge>
|
||||
</div>
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-2">
|
||||
<XCircle className="w-4 h-4 text-gray-500" />
|
||||
<span className="text-sm">Cancelled</span>
|
||||
</div>
|
||||
<Badge variant="outline">{stats?.jobs_by_status?.cancelled ?? 0}</Badge>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Activity Summary */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle className="text-base">Activity Summary</CardTitle>
|
||||
<CardDescription>User and job activity over time</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
{loading ? (
|
||||
<div className="flex items-center justify-center py-8">
|
||||
<Loader2 className="w-6 h-6 animate-spin text-muted-foreground" />
|
||||
</div>
|
||||
) : (
|
||||
<div className="space-y-4">
|
||||
<div>
|
||||
<h4 className="text-sm font-medium mb-2">Users</h4>
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
<div className="rounded-lg border p-3">
|
||||
<p className="text-2xl font-bold">{stats?.active_users_7d ?? 0}</p>
|
||||
<p className="text-xs text-muted-foreground">Active (7d)</p>
|
||||
</div>
|
||||
<div className="rounded-lg border p-3">
|
||||
<p className="text-2xl font-bold">{stats?.active_users_30d ?? 0}</p>
|
||||
<p className="text-xs text-muted-foreground">Active (30d)</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<h4 className="text-sm font-medium mb-2">Jobs</h4>
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
<div className="rounded-lg border p-3">
|
||||
<p className="text-2xl font-bold">{stats?.jobs_last_7d ?? 0}</p>
|
||||
<p className="text-xs text-muted-foreground">Created (7d)</p>
|
||||
</div>
|
||||
<div className="rounded-lg border p-3">
|
||||
<p className="text-2xl font-bold">{stats?.jobs_last_30d ?? 0}</p>
|
||||
<p className="text-xs text-muted-foreground">Created (30d)</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,485 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState } from "react"
|
||||
import { useSearchParams, useRouter } from "next/navigation"
|
||||
import { adminApi, AdminUserDetail } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Badge } from "@/components/ui/badge"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import {
|
||||
Table,
|
||||
TableBody,
|
||||
TableCell,
|
||||
TableHead,
|
||||
TableHeader,
|
||||
TableRow,
|
||||
} from "@/components/ui/table"
|
||||
import {
|
||||
Dialog,
|
||||
DialogContent,
|
||||
DialogDescription,
|
||||
DialogFooter,
|
||||
DialogHeader,
|
||||
DialogTitle,
|
||||
} from "@/components/ui/dialog"
|
||||
import { Input } from "@/components/ui/input"
|
||||
import { Label } from "@/components/ui/label"
|
||||
import {
|
||||
ArrowLeft,
|
||||
RefreshCw,
|
||||
Loader2,
|
||||
Plus,
|
||||
Shield,
|
||||
ShieldOff,
|
||||
UserX,
|
||||
UserCheck,
|
||||
CreditCard,
|
||||
Briefcase,
|
||||
Clock,
|
||||
} from "lucide-react"
|
||||
import { useToast } from "@/hooks/use-toast"
|
||||
|
||||
export default function AdminUserDetailPage() {
|
||||
const searchParams = useSearchParams()
|
||||
const router = useRouter()
|
||||
const { toast } = useToast()
|
||||
const email = searchParams.get("email") || ""
|
||||
|
||||
const [user, setUser] = useState<AdminUserDetail | null>(null)
|
||||
const [loading, setLoading] = useState(true)
|
||||
const [actionLoading, setActionLoading] = useState(false)
|
||||
|
||||
// Credit dialog
|
||||
const [creditDialogOpen, setCreditDialogOpen] = useState(false)
|
||||
const [creditAmount, setCreditAmount] = useState("")
|
||||
const [creditReason, setCreditReason] = useState("")
|
||||
|
||||
const loadUser = async () => {
|
||||
if (!email) {
|
||||
setLoading(false)
|
||||
return
|
||||
}
|
||||
try {
|
||||
setLoading(true)
|
||||
const data = await adminApi.getUserDetail(email)
|
||||
setUser(data)
|
||||
} catch (err: any) {
|
||||
console.error("Failed to load user:", err)
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to load user",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
loadUser()
|
||||
}, [email])
|
||||
|
||||
const handleAddCredits = async () => {
|
||||
if (!creditAmount || !creditReason) return
|
||||
|
||||
const amount = parseInt(creditAmount.trim(), 10)
|
||||
if (!Number.isInteger(amount) || amount < 1 || amount > 100000) {
|
||||
toast({
|
||||
title: "Invalid Amount",
|
||||
description: "Please enter a valid amount between 1 and 100,000",
|
||||
variant: "destructive",
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
try {
|
||||
setActionLoading(true)
|
||||
await adminApi.addCredits(email, amount, creditReason)
|
||||
toast({
|
||||
title: "Credits Added",
|
||||
description: `Added ${creditAmount} credits`,
|
||||
})
|
||||
setCreditDialogOpen(false)
|
||||
setCreditAmount("")
|
||||
setCreditReason("")
|
||||
loadUser()
|
||||
} catch (err: any) {
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to add credits",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setActionLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
const handleToggleRole = async () => {
|
||||
if (!user) return
|
||||
const newRole = user.role === "admin" ? "user" : "admin"
|
||||
|
||||
try {
|
||||
setActionLoading(true)
|
||||
await adminApi.setUserRole(email, newRole)
|
||||
toast({
|
||||
title: "Role Updated",
|
||||
description: `User is now ${newRole}`,
|
||||
})
|
||||
loadUser()
|
||||
} catch (err: any) {
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to update role",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setActionLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
const handleToggleStatus = async () => {
|
||||
if (!user) return
|
||||
|
||||
try {
|
||||
setActionLoading(true)
|
||||
if (user.is_active) {
|
||||
await adminApi.disableUser(email)
|
||||
toast({
|
||||
title: "User Disabled",
|
||||
description: "User has been disabled",
|
||||
})
|
||||
} else {
|
||||
await adminApi.enableUser(email)
|
||||
toast({
|
||||
title: "User Enabled",
|
||||
description: "User has been enabled",
|
||||
})
|
||||
}
|
||||
loadUser()
|
||||
} catch (err: any) {
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to update user status",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setActionLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
const formatDate = (dateStr?: string) => {
|
||||
if (!dateStr) return "Never"
|
||||
return new Date(dateStr).toLocaleString()
|
||||
}
|
||||
|
||||
if (!email) {
|
||||
return (
|
||||
<div className="text-center py-12">
|
||||
<p className="text-muted-foreground">No user email provided</p>
|
||||
<Button variant="outline" className="mt-4" onClick={() => router.push("/admin/users")}>
|
||||
<ArrowLeft className="w-4 h-4 mr-2" />
|
||||
Back to Users
|
||||
</Button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
if (loading) {
|
||||
return (
|
||||
<div className="flex items-center justify-center py-12">
|
||||
<Loader2 className="w-8 h-8 animate-spin text-muted-foreground" />
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
if (!user) {
|
||||
return (
|
||||
<div className="text-center py-12">
|
||||
<p className="text-muted-foreground">User not found</p>
|
||||
<Button variant="outline" className="mt-4" onClick={() => router.push("/admin/users")}>
|
||||
<ArrowLeft className="w-4 h-4 mr-2" />
|
||||
Back to Users
|
||||
</Button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
{/* Header */}
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-4">
|
||||
<Button variant="ghost" size="icon" onClick={() => router.push("/admin/users")}>
|
||||
<ArrowLeft className="w-4 h-4" />
|
||||
</Button>
|
||||
<div>
|
||||
<h1 className="text-2xl font-bold tracking-tight flex items-center gap-2">
|
||||
{user.display_name || user.email}
|
||||
<Badge variant={user.role === "admin" ? "default" : "secondary"}>
|
||||
{user.role}
|
||||
</Badge>
|
||||
{!user.is_active && <Badge variant="destructive">Disabled</Badge>}
|
||||
</h1>
|
||||
<p className="text-muted-foreground">{user.email}</p>
|
||||
</div>
|
||||
</div>
|
||||
<div className="flex gap-2">
|
||||
<Button variant="outline" size="sm" onClick={loadUser} disabled={loading}>
|
||||
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
|
||||
Refresh
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Actions */}
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button onClick={() => setCreditDialogOpen(true)} disabled={actionLoading}>
|
||||
<Plus className="w-4 h-4 mr-2" />
|
||||
Add Credits
|
||||
</Button>
|
||||
<Button
|
||||
variant="outline"
|
||||
onClick={handleToggleRole}
|
||||
disabled={actionLoading}
|
||||
>
|
||||
{user.role === "admin" ? (
|
||||
<>
|
||||
<ShieldOff className="w-4 h-4 mr-2" />
|
||||
Remove Admin
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<Shield className="w-4 h-4 mr-2" />
|
||||
Make Admin
|
||||
</>
|
||||
)}
|
||||
</Button>
|
||||
<Button
|
||||
variant={user.is_active ? "destructive" : "outline"}
|
||||
onClick={handleToggleStatus}
|
||||
disabled={actionLoading}
|
||||
>
|
||||
{user.is_active ? (
|
||||
<>
|
||||
<UserX className="w-4 h-4 mr-2" />
|
||||
Disable User
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<UserCheck className="w-4 h-4 mr-2" />
|
||||
Enable User
|
||||
</>
|
||||
)}
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{/* Stats Grid */}
|
||||
<div className="grid gap-4 md:grid-cols-4">
|
||||
<Card>
|
||||
<CardHeader className="pb-2">
|
||||
<CardTitle className="text-sm font-medium flex items-center gap-2">
|
||||
<CreditCard className="w-4 h-4" />
|
||||
Credits
|
||||
</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<p className="text-2xl font-bold">{user.credits}</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
<Card>
|
||||
<CardHeader className="pb-2">
|
||||
<CardTitle className="text-sm font-medium flex items-center gap-2">
|
||||
<Briefcase className="w-4 h-4" />
|
||||
Jobs Created
|
||||
</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<p className="text-2xl font-bold">{user.total_jobs_created}</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
<Card>
|
||||
<CardHeader className="pb-2">
|
||||
<CardTitle className="text-sm font-medium flex items-center gap-2">
|
||||
<Briefcase className="w-4 h-4" />
|
||||
Jobs Completed
|
||||
</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<p className="text-2xl font-bold">{user.total_jobs_completed}</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
<Card>
|
||||
<CardHeader className="pb-2">
|
||||
<CardTitle className="text-sm font-medium flex items-center gap-2">
|
||||
<Clock className="w-4 h-4" />
|
||||
Active Sessions
|
||||
</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<p className="text-2xl font-bold">{user.active_sessions_count}</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
<div className="grid gap-6 md:grid-cols-2">
|
||||
{/* Account Info */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Account Information</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent className="space-y-3">
|
||||
<div className="flex justify-between">
|
||||
<span className="text-muted-foreground">Email Verified</span>
|
||||
<Badge variant={user.email_verified ? "default" : "secondary"}>
|
||||
{user.email_verified ? "Yes" : "No"}
|
||||
</Badge>
|
||||
</div>
|
||||
<div className="flex justify-between">
|
||||
<span className="text-muted-foreground">Beta Tester</span>
|
||||
<Badge variant={user.is_beta_tester ? "default" : "secondary"}>
|
||||
{user.is_beta_tester ? user.beta_tester_status || "Yes" : "No"}
|
||||
</Badge>
|
||||
</div>
|
||||
<div className="flex justify-between">
|
||||
<span className="text-muted-foreground">Created</span>
|
||||
<span className="text-sm">{formatDate(user.created_at)}</span>
|
||||
</div>
|
||||
<div className="flex justify-between">
|
||||
<span className="text-muted-foreground">Last Login</span>
|
||||
<span className="text-sm">{formatDate(user.last_login_at)}</span>
|
||||
</div>
|
||||
<div className="flex justify-between">
|
||||
<span className="text-muted-foreground">Last Updated</span>
|
||||
<span className="text-sm">{formatDate(user.updated_at)}</span>
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Credit Transactions */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Recent Credit Transactions</CardTitle>
|
||||
<CardDescription>Last 20 transactions</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
{user.credit_transactions.length === 0 ? (
|
||||
<p className="text-muted-foreground text-sm">No transactions</p>
|
||||
) : (
|
||||
<div className="space-y-2 max-h-[300px] overflow-y-auto">
|
||||
{user.credit_transactions.slice().reverse().map((txn, i) => (
|
||||
<div key={txn.id || i} className="flex justify-between items-center text-sm border-b pb-2">
|
||||
<div>
|
||||
<span className={txn.amount > 0 ? "text-green-500" : "text-red-500"}>
|
||||
{txn.amount > 0 ? "+" : ""}{txn.amount}
|
||||
</span>
|
||||
<span className="text-muted-foreground ml-2">{txn.reason}</span>
|
||||
</div>
|
||||
<span className="text-xs text-muted-foreground">
|
||||
{new Date(txn.created_at).toLocaleDateString()}
|
||||
</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
{/* Recent Jobs */}
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle>Recent Jobs</CardTitle>
|
||||
<CardDescription>Last 10 jobs by this user</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
{user.recent_jobs.length === 0 ? (
|
||||
<p className="text-muted-foreground text-sm">No jobs</p>
|
||||
) : (
|
||||
<Table>
|
||||
<TableHeader>
|
||||
<TableRow>
|
||||
<TableHead>Job ID</TableHead>
|
||||
<TableHead>Artist / Title</TableHead>
|
||||
<TableHead>Status</TableHead>
|
||||
<TableHead>Created</TableHead>
|
||||
</TableRow>
|
||||
</TableHeader>
|
||||
<TableBody>
|
||||
{user.recent_jobs.map((job) => (
|
||||
<TableRow
|
||||
key={job.job_id}
|
||||
className="cursor-pointer hover:bg-muted/50"
|
||||
onClick={() => router.push(`/admin/jobs/detail?id=${job.job_id}`)}
|
||||
>
|
||||
<TableCell className="font-mono text-sm">{job.job_id}</TableCell>
|
||||
<TableCell>
|
||||
{job.artist && job.title ? `${job.artist} - ${job.title}` : "—"}
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
<Badge variant={
|
||||
job.status === "complete" ? "default" :
|
||||
job.status === "failed" ? "destructive" :
|
||||
"secondary"
|
||||
}>
|
||||
{job.status}
|
||||
</Badge>
|
||||
</TableCell>
|
||||
<TableCell className="text-sm">
|
||||
{job.created_at ? new Date(job.created_at).toLocaleString() : "—"}
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
))}
|
||||
</TableBody>
|
||||
</Table>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
{/* Add Credits Dialog */}
|
||||
<Dialog open={creditDialogOpen} onOpenChange={setCreditDialogOpen}>
|
||||
<DialogContent>
|
||||
<DialogHeader>
|
||||
<DialogTitle>Add Credits</DialogTitle>
|
||||
<DialogDescription>
|
||||
Add credits to {user.email}
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
<div className="space-y-4 py-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="amount">Amount</Label>
|
||||
<Input
|
||||
id="amount"
|
||||
type="number"
|
||||
min="1"
|
||||
placeholder="Enter amount"
|
||||
value={creditAmount}
|
||||
onChange={(e) => setCreditAmount(e.target.value)}
|
||||
/>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="reason">Reason</Label>
|
||||
<Input
|
||||
id="reason"
|
||||
placeholder="e.g., Beta reward, Support credit"
|
||||
value={creditReason}
|
||||
onChange={(e) => setCreditReason(e.target.value)}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<DialogFooter>
|
||||
<Button variant="outline" onClick={() => setCreditDialogOpen(false)}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
onClick={handleAddCredits}
|
||||
disabled={!creditAmount || !creditReason || actionLoading}
|
||||
>
|
||||
{actionLoading && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
|
||||
Add Credits
|
||||
</Button>
|
||||
</DialogFooter>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,375 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState, useCallback } from "react"
|
||||
import { useRouter } from "next/navigation"
|
||||
import { adminApi, AdminUser, AdminUserListResponse } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Input } from "@/components/ui/input"
|
||||
import { Badge } from "@/components/ui/badge"
|
||||
import {
|
||||
Table,
|
||||
TableBody,
|
||||
TableCell,
|
||||
TableHead,
|
||||
TableHeader,
|
||||
TableRow,
|
||||
} from "@/components/ui/table"
|
||||
import {
|
||||
Select,
|
||||
SelectContent,
|
||||
SelectItem,
|
||||
SelectTrigger,
|
||||
SelectValue,
|
||||
} from "@/components/ui/select"
|
||||
import {
|
||||
Dialog,
|
||||
DialogContent,
|
||||
DialogDescription,
|
||||
DialogFooter,
|
||||
DialogHeader,
|
||||
DialogTitle,
|
||||
DialogTrigger,
|
||||
} from "@/components/ui/dialog"
|
||||
import { Label } from "@/components/ui/label"
|
||||
import {
|
||||
Search,
|
||||
RefreshCw,
|
||||
ChevronLeft,
|
||||
ChevronRight,
|
||||
Plus,
|
||||
Loader2,
|
||||
CreditCard,
|
||||
Shield,
|
||||
ShieldOff,
|
||||
UserX,
|
||||
UserCheck,
|
||||
} from "lucide-react"
|
||||
import { useToast } from "@/hooks/use-toast"
|
||||
|
||||
export default function AdminUsersPage() {
|
||||
const router = useRouter()
|
||||
const { toast } = useToast()
|
||||
const [users, setUsers] = useState<AdminUser[]>([])
|
||||
const [loading, setLoading] = useState(true)
|
||||
const [total, setTotal] = useState(0)
|
||||
const [offset, setOffset] = useState(0)
|
||||
const [limit] = useState(20)
|
||||
const [hasMore, setHasMore] = useState(false)
|
||||
const [search, setSearch] = useState("")
|
||||
const [searchInput, setSearchInput] = useState("")
|
||||
const [sortBy, setSortBy] = useState<"created_at" | "last_login_at" | "credits" | "email">("created_at")
|
||||
const [sortOrder, setSortOrder] = useState<"asc" | "desc">("desc")
|
||||
|
||||
// Credit dialog state
|
||||
const [creditDialogOpen, setCreditDialogOpen] = useState(false)
|
||||
const [selectedUser, setSelectedUser] = useState<AdminUser | null>(null)
|
||||
const [creditAmount, setCreditAmount] = useState("")
|
||||
const [creditReason, setCreditReason] = useState("")
|
||||
const [addingCredits, setAddingCredits] = useState(false)
|
||||
|
||||
const loadUsers = useCallback(async () => {
|
||||
try {
|
||||
setLoading(true)
|
||||
const data = await adminApi.listUsers({
|
||||
limit,
|
||||
offset,
|
||||
search: search || undefined,
|
||||
sort_by: sortBy,
|
||||
sort_order: sortOrder,
|
||||
})
|
||||
setUsers(data.users)
|
||||
setTotal(data.total)
|
||||
setHasMore(data.has_more)
|
||||
} catch (err: any) {
|
||||
console.error("Failed to load users:", err)
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to load users",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}, [limit, offset, search, sortBy, sortOrder, toast])
|
||||
|
||||
useEffect(() => {
|
||||
loadUsers()
|
||||
}, [loadUsers])
|
||||
|
||||
const handleSearch = () => {
|
||||
setOffset(0)
|
||||
setSearch(searchInput)
|
||||
}
|
||||
|
||||
const handleKeyPress = (e: React.KeyboardEvent) => {
|
||||
if (e.key === "Enter") {
|
||||
handleSearch()
|
||||
}
|
||||
}
|
||||
|
||||
const handleAddCredits = async () => {
|
||||
if (!selectedUser || !creditAmount || !creditReason) return
|
||||
|
||||
const amount = parseInt(creditAmount.trim(), 10)
|
||||
if (!Number.isInteger(amount) || amount < 1 || amount > 100000) {
|
||||
toast({
|
||||
title: "Invalid Amount",
|
||||
description: "Please enter a valid amount between 1 and 100,000",
|
||||
variant: "destructive",
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
try {
|
||||
setAddingCredits(true)
|
||||
await adminApi.addCredits(selectedUser.email, amount, creditReason)
|
||||
toast({
|
||||
title: "Credits Added",
|
||||
description: `Added ${creditAmount} credits to ${selectedUser.email}`,
|
||||
})
|
||||
setCreditDialogOpen(false)
|
||||
setCreditAmount("")
|
||||
setCreditReason("")
|
||||
setSelectedUser(null)
|
||||
loadUsers()
|
||||
} catch (err: any) {
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to add credits",
|
||||
variant: "destructive",
|
||||
})
|
||||
} finally {
|
||||
setAddingCredits(false)
|
||||
}
|
||||
}
|
||||
|
||||
const handleToggleRole = async (user: AdminUser) => {
|
||||
const newRole = user.role === "admin" ? "user" : "admin"
|
||||
try {
|
||||
await adminApi.setUserRole(user.email, newRole)
|
||||
toast({
|
||||
title: "Role Updated",
|
||||
description: `${user.email} is now ${newRole}`,
|
||||
})
|
||||
loadUsers()
|
||||
} catch (err: any) {
|
||||
toast({
|
||||
title: "Error",
|
||||
description: err.message || "Failed to update role",
|
||||
variant: "destructive",
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
{/* Header */}
|
||||
<div className="flex items-center justify-between">
|
||||
<div>
|
||||
<h1 className="text-2xl font-bold tracking-tight">Users</h1>
|
||||
<p className="text-muted-foreground">
|
||||
Manage user accounts and credits
|
||||
</p>
|
||||
</div>
|
||||
<Button variant="outline" size="sm" onClick={loadUsers} disabled={loading}>
|
||||
<RefreshCw className={`w-4 h-4 mr-2 ${loading ? "animate-spin" : ""}`} />
|
||||
Refresh
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{/* Filters */}
|
||||
<div className="flex flex-col sm:flex-row gap-4">
|
||||
<div className="flex-1 flex gap-2">
|
||||
<Input
|
||||
placeholder="Search by email..."
|
||||
value={searchInput}
|
||||
onChange={(e) => setSearchInput(e.target.value)}
|
||||
onKeyPress={handleKeyPress}
|
||||
className="max-w-sm"
|
||||
/>
|
||||
<Button variant="secondary" onClick={handleSearch}>
|
||||
<Search className="w-4 h-4" />
|
||||
</Button>
|
||||
</div>
|
||||
<div className="flex gap-2">
|
||||
<Select value={sortBy} onValueChange={(v) => setSortBy(v as any)}>
|
||||
<SelectTrigger className="w-[140px]">
|
||||
<SelectValue placeholder="Sort by" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
<SelectItem value="created_at">Created</SelectItem>
|
||||
<SelectItem value="last_login_at">Last Login</SelectItem>
|
||||
<SelectItem value="credits">Credits</SelectItem>
|
||||
<SelectItem value="email">Email</SelectItem>
|
||||
</SelectContent>
|
||||
</Select>
|
||||
<Select value={sortOrder} onValueChange={(v) => setSortOrder(v as any)}>
|
||||
<SelectTrigger className="w-[100px]">
|
||||
<SelectValue placeholder="Order" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
<SelectItem value="desc">Desc</SelectItem>
|
||||
<SelectItem value="asc">Asc</SelectItem>
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Table */}
|
||||
<div className="rounded-md border">
|
||||
<Table>
|
||||
<TableHeader>
|
||||
<TableRow>
|
||||
<TableHead>Email</TableHead>
|
||||
<TableHead>Role</TableHead>
|
||||
<TableHead className="text-right">Credits</TableHead>
|
||||
<TableHead className="text-right">Jobs</TableHead>
|
||||
<TableHead className="text-right">Actions</TableHead>
|
||||
</TableRow>
|
||||
</TableHeader>
|
||||
<TableBody>
|
||||
{loading ? (
|
||||
<TableRow>
|
||||
<TableCell colSpan={5} className="text-center py-8">
|
||||
<Loader2 className="w-6 h-6 animate-spin mx-auto text-muted-foreground" />
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
) : users.length === 0 ? (
|
||||
<TableRow>
|
||||
<TableCell colSpan={5} className="text-center py-8 text-muted-foreground">
|
||||
No users found
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
) : (
|
||||
users.map((user) => (
|
||||
<TableRow
|
||||
key={user.email}
|
||||
className="cursor-pointer hover:bg-muted/50"
|
||||
onClick={() => router.push(`/admin/users/detail?email=${encodeURIComponent(user.email)}`)}
|
||||
>
|
||||
<TableCell className="font-medium">
|
||||
{user.display_name || user.email}
|
||||
{user.display_name && (
|
||||
<span className="text-muted-foreground text-sm block">
|
||||
{user.email}
|
||||
</span>
|
||||
)}
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
<Badge variant={user.role === "admin" ? "default" : "secondary"}>
|
||||
{user.role}
|
||||
</Badge>
|
||||
</TableCell>
|
||||
<TableCell className="text-right">{user.credits}</TableCell>
|
||||
<TableCell className="text-right">
|
||||
{user.total_jobs_created ?? 0} / {user.total_jobs_completed ?? 0}
|
||||
</TableCell>
|
||||
<TableCell className="text-right">
|
||||
<div className="flex justify-end gap-1" onClick={(e) => e.stopPropagation()}>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
onClick={() => {
|
||||
setSelectedUser(user)
|
||||
setCreditDialogOpen(true)
|
||||
}}
|
||||
title="Add credits"
|
||||
>
|
||||
<Plus className="w-4 h-4" />
|
||||
</Button>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
onClick={() => handleToggleRole(user)}
|
||||
title={user.role === "admin" ? "Remove admin" : "Make admin"}
|
||||
>
|
||||
{user.role === "admin" ? (
|
||||
<ShieldOff className="w-4 h-4" />
|
||||
) : (
|
||||
<Shield className="w-4 h-4" />
|
||||
)}
|
||||
</Button>
|
||||
</div>
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
))
|
||||
)}
|
||||
</TableBody>
|
||||
</Table>
|
||||
</div>
|
||||
|
||||
{/* Pagination */}
|
||||
<div className="flex items-center justify-between">
|
||||
<p className="text-sm text-muted-foreground">
|
||||
{total === 0 ? "No users found" : `Showing ${offset + 1}-${Math.min(offset + users.length, total)} of ${total} users`}
|
||||
</p>
|
||||
<div className="flex gap-2">
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
onClick={() => setOffset(Math.max(0, offset - limit))}
|
||||
disabled={offset === 0}
|
||||
>
|
||||
<ChevronLeft className="w-4 h-4" />
|
||||
Previous
|
||||
</Button>
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
onClick={() => setOffset(offset + limit)}
|
||||
disabled={!hasMore}
|
||||
>
|
||||
Next
|
||||
<ChevronRight className="w-4 h-4" />
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Add Credits Dialog */}
|
||||
<Dialog open={creditDialogOpen} onOpenChange={setCreditDialogOpen}>
|
||||
<DialogContent>
|
||||
<DialogHeader>
|
||||
<DialogTitle>Add Credits</DialogTitle>
|
||||
<DialogDescription>
|
||||
Add credits to {selectedUser?.email}
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
<div className="space-y-4 py-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="amount">Amount</Label>
|
||||
<Input
|
||||
id="amount"
|
||||
type="number"
|
||||
min="1"
|
||||
placeholder="Enter amount"
|
||||
value={creditAmount}
|
||||
onChange={(e) => setCreditAmount(e.target.value)}
|
||||
/>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="reason">Reason</Label>
|
||||
<Input
|
||||
id="reason"
|
||||
placeholder="e.g., Beta reward, Support credit"
|
||||
value={creditReason}
|
||||
onChange={(e) => setCreditReason(e.target.value)}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<DialogFooter>
|
||||
<Button variant="outline" onClick={() => setCreditDialogOpen(false)}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
onClick={handleAddCredits}
|
||||
disabled={!creditAmount || !creditReason || addingCredits}
|
||||
>
|
||||
{addingCredits && <Loader2 className="w-4 h-4 mr-2 animate-spin" />}
|
||||
Add Credits
|
||||
</Button>
|
||||
</DialogFooter>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -3,11 +3,10 @@
|
||||
import { useState, useEffect, useCallback } from "react"
|
||||
import { useRouter } from "next/navigation"
|
||||
import { api, Job, getAccessToken } from "@/lib/api"
|
||||
import { useAutoMode, getAutoModeFromUrl } from "@/lib/auto-mode"
|
||||
import { useJobNotifications, useVisibilityRefresh } from "@/hooks/use-notifications"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import { Music2, RefreshCw, Loader2, Zap, ZapOff, Moon, Sun } from "lucide-react"
|
||||
import { Music2, RefreshCw, Loader2, Moon, Sun } from "lucide-react"
|
||||
import { WarmingUpLoader } from "@/components/WarmingUpLoader"
|
||||
import { JobCard } from "@/components/job"
|
||||
import { JobSubmission } from "@/components/job/JobSubmission"
|
||||
@@ -28,7 +27,6 @@ export default function AppPage() {
|
||||
const [isLoadingJobs, setIsLoadingJobs] = useState(true)
|
||||
const [isInitialLoad, setIsInitialLoad] = useState(true)
|
||||
const [isAuthenticated, setIsAuthenticated] = useState<boolean | null>(null)
|
||||
const { enabled: autoModeEnabled, setEnabled: setAutoMode, toggle: toggleAutoMode } = useAutoMode()
|
||||
const { isDarkMode, toggleTheme, mounted } = useTheme()
|
||||
|
||||
// Memoize loadJobs for use with visibility refresh
|
||||
@@ -69,14 +67,6 @@ export default function AppPage() {
|
||||
setIsAuthenticated(true)
|
||||
}, [router])
|
||||
|
||||
// Initialize auto-mode from URL parameter on mount
|
||||
useEffect(() => {
|
||||
const urlAutoMode = getAutoModeFromUrl()
|
||||
if (urlAutoMode) {
|
||||
setAutoMode(true)
|
||||
}
|
||||
}, [setAutoMode])
|
||||
|
||||
// Load jobs on mount (only if authenticated)
|
||||
useEffect(() => {
|
||||
if (isAuthenticated !== true) {
|
||||
@@ -109,7 +99,7 @@ export default function AppPage() {
|
||||
|
||||
return (
|
||||
<div className="min-h-screen" style={{ background: 'var(--bg)' }}>
|
||||
{/* AutoProcessor - handles non-interactive mode */}
|
||||
{/* AutoProcessor - handles non-interactive mode for jobs with that flag */}
|
||||
<AutoProcessor jobs={jobs} onJobsChanged={loadJobs} />
|
||||
|
||||
{/* Header */}
|
||||
@@ -122,36 +112,6 @@ export default function AppPage() {
|
||||
</div>
|
||||
</div>
|
||||
<div className="flex items-center gap-1 sm:gap-2 shrink-0">
|
||||
<TooltipProvider>
|
||||
<Tooltip>
|
||||
<TooltipTrigger asChild>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={toggleAutoMode}
|
||||
className={`min-h-[40px] px-2 sm:px-3 ${autoModeEnabled ? "text-amber-400 hover:text-amber-300 bg-amber-500/10" : ""}`}
|
||||
style={!autoModeEnabled ? { color: 'var(--text-muted)' } : undefined}
|
||||
>
|
||||
{autoModeEnabled ? (
|
||||
<Zap className="w-4 h-4 sm:mr-2" />
|
||||
) : (
|
||||
<ZapOff className="w-4 h-4 sm:mr-2" />
|
||||
)}
|
||||
<span className="hidden sm:inline">Auto</span>
|
||||
</Button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent side="bottom" className="max-w-xs">
|
||||
<p className="font-medium">
|
||||
{autoModeEnabled ? "Auto Mode Enabled" : "Auto Mode Disabled"}
|
||||
</p>
|
||||
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
|
||||
{autoModeEnabled
|
||||
? "Jobs will auto-complete review and select clean instrumental (like -y flag)"
|
||||
: "Click to enable non-interactive mode"}
|
||||
</p>
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
</TooltipProvider>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
@@ -193,27 +153,6 @@ export default function AppPage() {
|
||||
</header>
|
||||
|
||||
<main className="px-4 py-8 space-y-6">
|
||||
{/* Auto Mode Banner */}
|
||||
{autoModeEnabled && (
|
||||
<div className="bg-amber-500/10 border border-amber-500/30 rounded-lg p-4 flex items-center gap-3">
|
||||
<Zap className="w-5 h-5 text-amber-400 flex-shrink-0" />
|
||||
<div className="flex-1">
|
||||
<p className="text-sm text-amber-200 font-medium">Non-Interactive Mode Active</p>
|
||||
<p className="text-xs text-amber-200/70 mt-0.5">
|
||||
Jobs will automatically accept lyrics and select clean instrumental (equivalent to -y flag)
|
||||
</p>
|
||||
</div>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={toggleAutoMode}
|
||||
className="text-amber-200 hover:text-amber-100 hover:bg-amber-500/20"
|
||||
>
|
||||
Disable
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="grid lg:grid-cols-2 gap-8">
|
||||
{/* Submit Job Card */}
|
||||
<Card className="backdrop-blur" style={{ borderColor: 'var(--card-border)', backgroundColor: 'var(--card)' }}>
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
import { useEffect, useCallback, useRef } from "react"
|
||||
import { api, Job } from "@/lib/api"
|
||||
import { useAutoMode } from "@/lib/auto-mode"
|
||||
|
||||
interface AutoProcessorProps {
|
||||
/** List of jobs to monitor */
|
||||
@@ -13,27 +12,27 @@ interface AutoProcessorProps {
|
||||
|
||||
/**
|
||||
* AutoProcessor component that automatically handles interactive states
|
||||
* when auto-mode is enabled.
|
||||
* for jobs that have non_interactive mode enabled.
|
||||
*
|
||||
* This is equivalent to the -y flag in karaoke-gen-remote CLI:
|
||||
* - Auto-completes lyrics review (accepts auto-corrected lyrics)
|
||||
* - Auto-selects "clean" instrumental
|
||||
* - Auto-selects instrumental based on backing vocals analyzer recommendation
|
||||
* - Auto-selects first/best audio search result
|
||||
*/
|
||||
export function AutoProcessor({ jobs, onJobsChanged }: AutoProcessorProps) {
|
||||
const { enabled, markProcessing, unmarkProcessing, isProcessing } = useAutoMode()
|
||||
const processedRef = useRef<Set<string>>(new Set())
|
||||
const processingRef = useRef<Set<string>>(new Set())
|
||||
|
||||
const processJob = useCallback(async (job: Job, action: string) => {
|
||||
const processingKey = `${job.job_id}-${action}`
|
||||
|
||||
// Skip if already being processed
|
||||
if (isProcessing(processingKey) || processedRef.current.has(processingKey)) {
|
||||
// Skip if already being processed or already processed
|
||||
if (processingRef.current.has(processingKey) || processedRef.current.has(processingKey)) {
|
||||
return
|
||||
}
|
||||
|
||||
// Mark as processing
|
||||
markProcessing(processingKey)
|
||||
processingRef.current.add(processingKey)
|
||||
processedRef.current.add(processingKey)
|
||||
|
||||
console.log(`[AutoProcessor] Auto-processing ${action} for job ${job.job_id}`)
|
||||
@@ -46,11 +45,16 @@ export function AutoProcessor({ jobs, onJobsChanged }: AutoProcessorProps) {
|
||||
console.log(`[AutoProcessor] Auto-completed review for job ${job.job_id}`)
|
||||
break
|
||||
|
||||
case 'instrumental':
|
||||
// Auto-select "clean" instrumental
|
||||
await api.selectInstrumental(job.job_id, 'clean')
|
||||
console.log(`[AutoProcessor] Auto-selected clean instrumental for job ${job.job_id}`)
|
||||
case 'instrumental': {
|
||||
// Use backing vocals analyzer recommendation (default to clean if review needed or missing)
|
||||
const analysis = job.state_data?.backing_vocals_analysis
|
||||
const recommendation = analysis?.recommended_selection
|
||||
// Only use 'with_backing' if explicitly recommended by analyzer, otherwise clean (safe default)
|
||||
const selection = recommendation === 'with_backing' ? 'with_backing' : 'clean'
|
||||
await api.selectInstrumental(job.job_id, selection)
|
||||
console.log(`[AutoProcessor] Auto-selected ${selection} instrumental for job ${job.job_id} (analyzer recommendation: ${recommendation || 'none'})`)
|
||||
break
|
||||
}
|
||||
|
||||
case 'audio':
|
||||
// Auto-select first audio search result (index 0)
|
||||
@@ -66,15 +70,18 @@ export function AutoProcessor({ jobs, onJobsChanged }: AutoProcessorProps) {
|
||||
// Remove from processed so it can be retried
|
||||
processedRef.current.delete(processingKey)
|
||||
} finally {
|
||||
unmarkProcessing(processingKey)
|
||||
processingRef.current.delete(processingKey)
|
||||
}
|
||||
}, [isProcessing, markProcessing, unmarkProcessing, onJobsChanged])
|
||||
}, [onJobsChanged])
|
||||
|
||||
useEffect(() => {
|
||||
if (!enabled) return
|
||||
|
||||
// Find jobs that need auto-processing
|
||||
// Find jobs that need auto-processing (only jobs with non_interactive flag)
|
||||
for (const job of jobs) {
|
||||
// Only process jobs with non_interactive mode enabled
|
||||
if (!job.non_interactive) {
|
||||
continue
|
||||
}
|
||||
|
||||
switch (job.status) {
|
||||
case 'awaiting_review':
|
||||
processJob(job, 'review')
|
||||
@@ -87,7 +94,7 @@ export function AutoProcessor({ jobs, onJobsChanged }: AutoProcessorProps) {
|
||||
break
|
||||
}
|
||||
}
|
||||
}, [enabled, jobs, processJob])
|
||||
}, [jobs, processJob])
|
||||
|
||||
// This component doesn't render anything - it's purely for side effects
|
||||
return null
|
||||
|
||||
@@ -7,14 +7,6 @@
|
||||
import { render, screen } from '@testing-library/react'
|
||||
import { JobCard } from '../job/JobCard'
|
||||
import { Job } from '@/lib/api'
|
||||
import { useAutoMode } from '@/lib/auto-mode'
|
||||
|
||||
// Mock the auto-mode hook
|
||||
jest.mock('@/lib/auto-mode', () => ({
|
||||
useAutoMode: jest.fn()
|
||||
}))
|
||||
|
||||
const mockUseAutoMode = useAutoMode as jest.Mock
|
||||
|
||||
// Mock the child components
|
||||
jest.mock('../job/JobActions', () => ({
|
||||
@@ -48,11 +40,6 @@ describe('JobCard', () => {
|
||||
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks()
|
||||
// Default mock: auto-mode disabled
|
||||
mockUseAutoMode.mockReturnValue({
|
||||
enabled: false,
|
||||
isProcessing: jest.fn().mockReturnValue(false)
|
||||
})
|
||||
})
|
||||
|
||||
it('renders job information', () => {
|
||||
@@ -104,14 +91,7 @@ describe('JobCard', () => {
|
||||
expect(card).toHaveClass('border-red-500/30')
|
||||
})
|
||||
|
||||
describe('when auto-mode is disabled', () => {
|
||||
beforeEach(() => {
|
||||
mockUseAutoMode.mockReturnValue({
|
||||
enabled: false,
|
||||
isProcessing: jest.fn().mockReturnValue(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('regular jobs (non_interactive: false)', () => {
|
||||
it('shows lyrics review link for awaiting_review status', () => {
|
||||
const reviewJob = { ...mockJob, status: 'awaiting_review', review_token: 'test-token-123' }
|
||||
render(<JobCard job={reviewJob} onRefresh={mockOnRefresh} />)
|
||||
@@ -146,18 +126,24 @@ describe('JobCard', () => {
|
||||
expect(href).toContain('baseApiUrl=')
|
||||
expect(href).toContain('instrumentalToken=test-instrumental-token')
|
||||
})
|
||||
|
||||
it('does not show Auto badge for regular jobs', () => {
|
||||
render(<JobCard job={mockJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.queryByText('Auto')).not.toBeInTheDocument()
|
||||
})
|
||||
})
|
||||
|
||||
describe('when auto-mode is enabled', () => {
|
||||
beforeEach(() => {
|
||||
mockUseAutoMode.mockReturnValue({
|
||||
enabled: true,
|
||||
isProcessing: jest.fn().mockReturnValue(false)
|
||||
})
|
||||
describe('non-interactive jobs (non_interactive: true)', () => {
|
||||
it('shows Auto badge for non-interactive jobs', () => {
|
||||
const nonInteractiveJob = { ...mockJob, non_interactive: true }
|
||||
render(<JobCard job={nonInteractiveJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText('Auto')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('shows auto-accept message for awaiting_review status', () => {
|
||||
const reviewJob = { ...mockJob, status: 'awaiting_review' }
|
||||
const reviewJob = { ...mockJob, status: 'awaiting_review', non_interactive: true }
|
||||
render(<JobCard job={reviewJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText(/Will auto-accept/)).toBeInTheDocument()
|
||||
@@ -165,32 +151,20 @@ describe('JobCard', () => {
|
||||
})
|
||||
|
||||
it('shows auto-select message for awaiting_instrumental_selection status', () => {
|
||||
const instrumentalJob = { ...mockJob, status: 'awaiting_instrumental_selection' }
|
||||
const instrumentalJob = { ...mockJob, status: 'awaiting_instrumental_selection', non_interactive: true }
|
||||
render(<JobCard job={instrumentalJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText(/Will auto-select clean/)).toBeInTheDocument()
|
||||
expect(screen.getByText(/Will auto-select/)).toBeInTheDocument()
|
||||
expect(screen.queryByText('Select Instrumental')).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('shows auto-select message for awaiting_audio_selection status', () => {
|
||||
const audioJob = { ...mockJob, status: 'awaiting_audio_selection' }
|
||||
it('shows auto-select first message for awaiting_audio_selection status', () => {
|
||||
const audioJob = { ...mockJob, status: 'awaiting_audio_selection', non_interactive: true }
|
||||
render(<JobCard job={audioJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText(/Will auto-select first/)).toBeInTheDocument()
|
||||
expect(screen.queryByRole('button', { name: 'Select Audio' })).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('shows auto-processing indicator when processing', () => {
|
||||
mockUseAutoMode.mockReturnValue({
|
||||
enabled: true,
|
||||
isProcessing: jest.fn().mockReturnValue(true)
|
||||
})
|
||||
|
||||
const reviewJob = { ...mockJob, status: 'awaiting_review' }
|
||||
render(<JobCard job={reviewJob} onRefresh={mockOnRefresh} />)
|
||||
|
||||
expect(screen.getByText(/Auto-processing.../)).toBeInTheDocument()
|
||||
})
|
||||
})
|
||||
|
||||
it('shows output links for complete jobs', () => {
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
"use client"
|
||||
|
||||
import Link from "next/link"
|
||||
import { usePathname } from "next/navigation"
|
||||
import {
|
||||
LayoutDashboard,
|
||||
Users,
|
||||
Briefcase,
|
||||
TestTube2,
|
||||
ArrowLeft,
|
||||
Settings,
|
||||
} from "lucide-react"
|
||||
|
||||
import {
|
||||
Sidebar,
|
||||
SidebarContent,
|
||||
SidebarFooter,
|
||||
SidebarGroup,
|
||||
SidebarGroupContent,
|
||||
SidebarGroupLabel,
|
||||
SidebarHeader,
|
||||
SidebarMenu,
|
||||
SidebarMenuButton,
|
||||
SidebarMenuItem,
|
||||
SidebarSeparator,
|
||||
} from "@/components/ui/sidebar"
|
||||
|
||||
const navItems = [
|
||||
{
|
||||
title: "Dashboard",
|
||||
href: "/admin",
|
||||
icon: LayoutDashboard,
|
||||
},
|
||||
{
|
||||
title: "Users",
|
||||
href: "/admin/users",
|
||||
icon: Users,
|
||||
},
|
||||
{
|
||||
title: "Jobs",
|
||||
href: "/admin/jobs",
|
||||
icon: Briefcase,
|
||||
},
|
||||
{
|
||||
title: "Beta Program",
|
||||
href: "/admin/beta",
|
||||
icon: TestTube2,
|
||||
},
|
||||
]
|
||||
|
||||
export function AdminSidebar() {
|
||||
const pathname = usePathname()
|
||||
|
||||
const isActive = (href: string) => {
|
||||
if (href === "/admin") {
|
||||
return pathname === "/admin"
|
||||
}
|
||||
return pathname.startsWith(href)
|
||||
}
|
||||
|
||||
return (
|
||||
<Sidebar>
|
||||
<SidebarHeader className="border-b border-sidebar-border">
|
||||
<div className="flex items-center gap-2 px-2 py-2">
|
||||
<Settings className="w-6 h-6 text-primary" />
|
||||
<span className="font-semibold text-lg">Admin Panel</span>
|
||||
</div>
|
||||
</SidebarHeader>
|
||||
|
||||
<SidebarContent>
|
||||
<SidebarGroup>
|
||||
<SidebarGroupLabel>Navigation</SidebarGroupLabel>
|
||||
<SidebarGroupContent>
|
||||
<SidebarMenu>
|
||||
{navItems.map((item) => (
|
||||
<SidebarMenuItem key={item.href}>
|
||||
<SidebarMenuButton asChild isActive={isActive(item.href)}>
|
||||
<Link href={item.href}>
|
||||
<item.icon className="w-4 h-4" />
|
||||
<span>{item.title}</span>
|
||||
</Link>
|
||||
</SidebarMenuButton>
|
||||
</SidebarMenuItem>
|
||||
))}
|
||||
</SidebarMenu>
|
||||
</SidebarGroupContent>
|
||||
</SidebarGroup>
|
||||
</SidebarContent>
|
||||
|
||||
<SidebarSeparator />
|
||||
|
||||
<SidebarFooter>
|
||||
<SidebarMenu>
|
||||
<SidebarMenuItem>
|
||||
<SidebarMenuButton asChild>
|
||||
<Link href="/app">
|
||||
<ArrowLeft className="w-4 h-4" />
|
||||
<span>Back to App</span>
|
||||
</Link>
|
||||
</SidebarMenuButton>
|
||||
</SidebarMenuItem>
|
||||
</SidebarMenu>
|
||||
</SidebarFooter>
|
||||
</Sidebar>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
export { AdminSidebar } from "./admin-sidebar"
|
||||
export { StatsCard, StatsGrid } from "./stats-card"
|
||||
@@ -0,0 +1,87 @@
|
||||
"use client"
|
||||
|
||||
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
import { Skeleton } from "@/components/ui/skeleton"
|
||||
import { cn } from "@/lib/utils"
|
||||
import { LucideIcon } from "lucide-react"
|
||||
|
||||
interface StatsCardProps {
|
||||
title: string
|
||||
value: string | number
|
||||
description?: string
|
||||
icon?: LucideIcon
|
||||
trend?: {
|
||||
value: number
|
||||
label: string
|
||||
positive?: boolean
|
||||
}
|
||||
loading?: boolean
|
||||
className?: string
|
||||
}
|
||||
|
||||
export function StatsCard({
|
||||
title,
|
||||
value,
|
||||
description,
|
||||
icon: Icon,
|
||||
trend,
|
||||
loading,
|
||||
className,
|
||||
}: StatsCardProps) {
|
||||
if (loading) {
|
||||
return (
|
||||
<Card className={cn("", className)}>
|
||||
<CardHeader className="flex flex-row items-center justify-between space-y-0 pb-2">
|
||||
<CardTitle className="text-sm font-medium">
|
||||
<Skeleton className="h-4 w-24" />
|
||||
</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<Skeleton className="h-8 w-16" />
|
||||
<Skeleton className="h-3 w-32 mt-2" />
|
||||
</CardContent>
|
||||
</Card>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<Card className={cn("", className)}>
|
||||
<CardHeader className="flex flex-row items-center justify-between space-y-0 pb-2">
|
||||
<CardTitle className="text-sm font-medium text-muted-foreground">
|
||||
{title}
|
||||
</CardTitle>
|
||||
{Icon && <Icon className="h-4 w-4 text-muted-foreground" />}
|
||||
</CardHeader>
|
||||
<CardContent>
|
||||
<div className="text-2xl font-bold">{value}</div>
|
||||
{description && (
|
||||
<p className="text-xs text-muted-foreground mt-1">{description}</p>
|
||||
)}
|
||||
{trend && (
|
||||
<p className={cn(
|
||||
"text-xs mt-1",
|
||||
trend.positive === true ? "text-green-500" : trend.positive === false ? "text-red-500" : "text-muted-foreground"
|
||||
)}>
|
||||
{trend.positive === true ? "+" : ""}{trend.value}% {trend.label}
|
||||
</p>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
)
|
||||
}
|
||||
|
||||
interface StatsGridProps {
|
||||
children: React.ReactNode
|
||||
className?: string
|
||||
}
|
||||
|
||||
export function StatsGrid({ children, className }: StatsGridProps) {
|
||||
return (
|
||||
<div className={cn(
|
||||
"grid gap-4 md:grid-cols-2 lg:grid-cols-4",
|
||||
className
|
||||
)}>
|
||||
{children}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -2,9 +2,8 @@
|
||||
|
||||
import { useState } from "react"
|
||||
import { Job } from "@/lib/api"
|
||||
import { useAutoMode } from "@/lib/auto-mode"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Loader2, ExternalLink, Zap } from "lucide-react"
|
||||
import { ExternalLink, Zap } from "lucide-react"
|
||||
import { JobActions } from "./JobActions"
|
||||
import { JobLogs } from "./JobLogs"
|
||||
import { OutputLinks } from "./OutputLinks"
|
||||
@@ -77,7 +76,6 @@ interface JobCardProps {
|
||||
export function JobCard({ job, onRefresh }: JobCardProps) {
|
||||
const [showLogs, setShowLogs] = useState(false)
|
||||
const [showAudioSearch, setShowAudioSearch] = useState(false)
|
||||
const { enabled: autoModeEnabled, isProcessing: isAutoProcessing } = useAutoMode()
|
||||
|
||||
const createdAt = new Date(job.created_at).toLocaleString()
|
||||
const isInteractive = job.status === "awaiting_review" ||
|
||||
@@ -86,14 +84,6 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
|
||||
job.status === "awaiting_audio_selection"
|
||||
const isComplete = job.status === "complete"
|
||||
const isFailed = job.status === "failed"
|
||||
const isProcessing = !isComplete && !isFailed && job.status !== "cancelled"
|
||||
|
||||
// Check if this job is being auto-processed
|
||||
const isBeingAutoProcessed = autoModeEnabled && isInteractive && (
|
||||
isAutoProcessing(`${job.job_id}-review`) ||
|
||||
isAutoProcessing(`${job.job_id}-instrumental`) ||
|
||||
isAutoProcessing(`${job.job_id}-audio`)
|
||||
)
|
||||
|
||||
// Build review URL for awaiting_review status
|
||||
const getReviewUrl = () => {
|
||||
@@ -125,22 +115,14 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
|
||||
|
||||
// Render primary action button based on status
|
||||
const renderPrimaryAction = () => {
|
||||
if (autoModeEnabled && isInteractive) {
|
||||
// For non-interactive jobs in interactive states, show auto-processing message
|
||||
if (job.non_interactive && isInteractive) {
|
||||
return (
|
||||
<div className="flex items-center gap-1 text-xs text-amber-400">
|
||||
{isBeingAutoProcessed ? (
|
||||
<>
|
||||
<Loader2 className="w-3 h-3 animate-spin" />
|
||||
Auto-processing...
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<Zap className="w-3 h-3" />
|
||||
Will auto-{job.status === "awaiting_review" || job.status === "in_review" ? "accept" :
|
||||
job.status === "awaiting_instrumental_selection" ? "select clean" :
|
||||
"select first"}
|
||||
</>
|
||||
)}
|
||||
<Zap className="w-3 h-3" />
|
||||
Will auto-{job.status === "awaiting_review" || job.status === "in_review" ? "accept" :
|
||||
job.status === "awaiting_instrumental_selection" ? "select" :
|
||||
"select first"}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -198,10 +180,18 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
|
||||
backgroundColor: 'var(--card)',
|
||||
}}
|
||||
>
|
||||
{/* Header row with title */}
|
||||
<p className="font-medium truncate" style={{ color: 'var(--text)' }}>
|
||||
{job.artist || "Unknown"} - {job.title || "Unknown"}
|
||||
</p>
|
||||
{/* Header row with title and non-interactive badge */}
|
||||
<div className="flex items-start justify-between gap-2">
|
||||
<p className="font-medium truncate" style={{ color: 'var(--text)' }}>
|
||||
{job.artist || "Unknown"} - {job.title || "Unknown"}
|
||||
</p>
|
||||
{job.non_interactive && (
|
||||
<span className="shrink-0 inline-flex items-center gap-1 text-[10px] px-1.5 py-0.5 rounded bg-amber-500/20 text-amber-400 font-medium">
|
||||
<Zap className="w-2.5 h-2.5" />
|
||||
Auto
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Meta row: ID, date, status */}
|
||||
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
|
||||
@@ -254,4 +244,3 @@ export function JobCard({ job, onRefresh }: JobCardProps) {
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -39,6 +39,9 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
const [selectedTheme, setSelectedTheme] = useState<string | undefined>()
|
||||
const [colorOverrides, setColorOverrides] = useState<ColorOverrides>({})
|
||||
|
||||
// Non-interactive mode (shared across all tabs)
|
||||
const [nonInteractive, setNonInteractive] = useState(false)
|
||||
|
||||
async function handleUploadSubmit(e: React.FormEvent) {
|
||||
e.preventDefault()
|
||||
setError("")
|
||||
@@ -57,6 +60,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
await api.uploadJob(uploadFile, uploadArtist.trim(), uploadTitle.trim(), {
|
||||
theme_id: selectedTheme,
|
||||
color_overrides: cleanColorOverrides(colorOverrides),
|
||||
non_interactive: nonInteractive,
|
||||
})
|
||||
setUploadFile(null)
|
||||
setUploadArtist("")
|
||||
@@ -91,6 +95,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
{
|
||||
theme_id: selectedTheme,
|
||||
color_overrides: cleanColorOverrides(colorOverrides),
|
||||
non_interactive: nonInteractive,
|
||||
}
|
||||
)
|
||||
setYoutubeUrl("")
|
||||
@@ -122,6 +127,7 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
await api.searchAudio(searchArtist.trim(), searchTitle.trim(), false, {
|
||||
theme_id: selectedTheme,
|
||||
color_overrides: cleanColorOverrides(colorOverrides),
|
||||
non_interactive: nonInteractive,
|
||||
})
|
||||
setSearchArtist("")
|
||||
setSearchTitle("")
|
||||
@@ -250,6 +256,29 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
/>
|
||||
</div>
|
||||
|
||||
{/* Non-interactive mode */}
|
||||
<div className="space-y-2 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
|
||||
<div className="flex items-start gap-3">
|
||||
<input
|
||||
type="checkbox"
|
||||
id="non-interactive-upload"
|
||||
checked={nonInteractive}
|
||||
onChange={(e) => setNonInteractive(e.target.checked)}
|
||||
disabled={isSubmitting}
|
||||
className="mt-1 w-4 h-4 rounded border-gray-600 bg-gray-700 text-amber-500 focus:ring-amber-500 focus:ring-offset-gray-800"
|
||||
/>
|
||||
<div>
|
||||
<Label htmlFor="non-interactive-upload" className="cursor-pointer" style={{ color: 'var(--text)' }}>
|
||||
Skip lyrics review (non-interactive)
|
||||
</Label>
|
||||
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
|
||||
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn't perfect.
|
||||
Skipping review will likely result in incorrect words in your karaoke video.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{error && activeTab === "upload" && (
|
||||
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
|
||||
)}
|
||||
@@ -332,6 +361,29 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
/>
|
||||
</div>
|
||||
|
||||
{/* Non-interactive mode */}
|
||||
<div className="space-y-2 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
|
||||
<div className="flex items-start gap-3">
|
||||
<input
|
||||
type="checkbox"
|
||||
id="non-interactive-url"
|
||||
checked={nonInteractive}
|
||||
onChange={(e) => setNonInteractive(e.target.checked)}
|
||||
disabled={isSubmitting}
|
||||
className="mt-1 w-4 h-4 rounded border-gray-600 bg-gray-700 text-amber-500 focus:ring-amber-500 focus:ring-offset-gray-800"
|
||||
/>
|
||||
<div>
|
||||
<Label htmlFor="non-interactive-url" className="cursor-pointer" style={{ color: 'var(--text)' }}>
|
||||
Skip lyrics review (non-interactive)
|
||||
</Label>
|
||||
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
|
||||
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn't perfect.
|
||||
Skipping review will likely result in incorrect words in your karaoke video.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{error && activeTab === "url" && (
|
||||
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
|
||||
)}
|
||||
@@ -397,6 +449,29 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
/>
|
||||
</div>
|
||||
|
||||
{/* Non-interactive mode */}
|
||||
<div className="space-y-2 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
|
||||
<div className="flex items-start gap-3">
|
||||
<input
|
||||
type="checkbox"
|
||||
id="non-interactive-search"
|
||||
checked={nonInteractive}
|
||||
onChange={(e) => setNonInteractive(e.target.checked)}
|
||||
disabled={isSubmitting}
|
||||
className="mt-1 w-4 h-4 rounded border-gray-600 bg-gray-700 text-amber-500 focus:ring-amber-500 focus:ring-offset-gray-800"
|
||||
/>
|
||||
<div>
|
||||
<Label htmlFor="non-interactive-search" className="cursor-pointer" style={{ color: 'var(--text)' }}>
|
||||
Skip lyrics review (non-interactive)
|
||||
</Label>
|
||||
<p className="text-xs mt-1" style={{ color: 'var(--text-muted)' }}>
|
||||
<span className="text-amber-500 font-medium">Warning:</span> auto lyrics correction isn't perfect.
|
||||
Skipping review will likely result in incorrect words in your karaoke video.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{error && activeTab === "search" && (
|
||||
<p className="text-sm text-red-400 bg-red-500/10 rounded p-2">{error}</p>
|
||||
)}
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState } from "react"
|
||||
import { useEffect, useState, useRef, useCallback } from "react"
|
||||
import { api } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Download, Loader2, ExternalLink, FolderOpen } from "lucide-react"
|
||||
import { Download, Loader2, ExternalLink, FolderOpen, Copy } from "lucide-react"
|
||||
|
||||
interface OutputLinksProps {
|
||||
jobId: string
|
||||
@@ -14,17 +14,28 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
|
||||
const [youtubeUrl, setYoutubeUrl] = useState<string | null>(null)
|
||||
const [dropboxUrl, setDropboxUrl] = useState<string | null>(null)
|
||||
const [isLoading, setIsLoading] = useState(false)
|
||||
const [copiedUrl, setCopiedUrl] = useState<string | null>(null)
|
||||
const copyTimeoutRef = useRef<NodeJS.Timeout | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
loadOutputLinks()
|
||||
}, [jobId])
|
||||
|
||||
// Cleanup timeout on unmount
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
if (copyTimeoutRef.current) {
|
||||
clearTimeout(copyTimeoutRef.current)
|
||||
}
|
||||
}
|
||||
}, [])
|
||||
|
||||
async function loadOutputLinks() {
|
||||
setIsLoading(true)
|
||||
try {
|
||||
const result = await api.getDownloadUrls(jobId)
|
||||
setDownloadUrls(result.download_urls)
|
||||
|
||||
|
||||
// Extract YouTube and Dropbox URLs from state_data if available
|
||||
const job = await api.getJob(jobId)
|
||||
if (job.state_data?.youtube_url) {
|
||||
@@ -40,6 +51,31 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
|
||||
}
|
||||
}
|
||||
|
||||
const copyToClipboard = useCallback(async (url: string) => {
|
||||
// Check if clipboard API is available
|
||||
if (!navigator?.clipboard?.writeText) {
|
||||
console.error("Clipboard API not available")
|
||||
return
|
||||
}
|
||||
|
||||
try {
|
||||
await navigator.clipboard.writeText(url)
|
||||
setCopiedUrl(url)
|
||||
|
||||
// Clear any existing timeout
|
||||
if (copyTimeoutRef.current) {
|
||||
clearTimeout(copyTimeoutRef.current)
|
||||
}
|
||||
|
||||
// Set new timeout
|
||||
copyTimeoutRef.current = setTimeout(() => {
|
||||
setCopiedUrl(null)
|
||||
}, 2000)
|
||||
} catch (err) {
|
||||
console.error("Failed to copy to clipboard:", err)
|
||||
}
|
||||
}, [])
|
||||
|
||||
if (isLoading) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 text-xs" style={{ color: 'var(--text-muted)' }}>
|
||||
@@ -60,33 +96,55 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
|
||||
<div className="flex flex-wrap gap-2">
|
||||
{/* Distribution Links */}
|
||||
{youtubeUrl && (
|
||||
<a
|
||||
href={youtubeUrl}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-red-600 hover:bg-red-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<ExternalLink className="w-3 h-3" />
|
||||
View on YouTube
|
||||
</a>
|
||||
<div className="flex gap-1">
|
||||
<a
|
||||
href={youtubeUrl}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded-l bg-red-600 hover:bg-red-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<ExternalLink className="w-3 h-3" />
|
||||
View on YouTube
|
||||
</a>
|
||||
<button
|
||||
type="button"
|
||||
onClick={(e) => { e.stopPropagation(); copyToClipboard(youtubeUrl) }}
|
||||
className="inline-flex items-center text-xs px-2 py-1.5 rounded-r bg-red-700 hover:bg-red-600 text-white"
|
||||
title="Copy YouTube URL"
|
||||
aria-label={copiedUrl === youtubeUrl ? "YouTube URL copied" : "Copy YouTube URL"}
|
||||
>
|
||||
{copiedUrl === youtubeUrl ? <span className="text-[10px]">Copied!</span> : <Copy className="w-3 h-3" />}
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
{dropboxUrl && (
|
||||
<a
|
||||
href={dropboxUrl}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-blue-600 hover:bg-blue-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<FolderOpen className="w-3 h-3" />
|
||||
Dropbox Folder
|
||||
</a>
|
||||
<div className="flex gap-1">
|
||||
<a
|
||||
href={dropboxUrl}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded-l bg-blue-600 hover:bg-blue-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<FolderOpen className="w-3 h-3" />
|
||||
Dropbox Folder
|
||||
</a>
|
||||
<button
|
||||
type="button"
|
||||
onClick={(e) => { e.stopPropagation(); copyToClipboard(dropboxUrl) }}
|
||||
className="inline-flex items-center text-xs px-2 py-1.5 rounded-r bg-blue-700 hover:bg-blue-600 text-white"
|
||||
title="Copy Dropbox URL"
|
||||
aria-label={copiedUrl === dropboxUrl ? "Dropbox URL copied" : "Copy Dropbox URL"}
|
||||
>
|
||||
{copiedUrl === dropboxUrl ? <span className="text-[10px]">Copied!</span> : <Copy className="w-3 h-3" />}
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
{/* Download Links */}
|
||||
{downloadUrls?.finals?.lossy_720p && (
|
||||
{downloadUrls?.finals?.lossy_720p_mp4 && (
|
||||
<a
|
||||
href={api.getDownloadUrl(jobId, "finals", "lossy_720p")}
|
||||
href={api.getDownloadUrl(jobId, "finals", "lossy_720p_mp4")}
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-green-600 hover:bg-green-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
@@ -104,6 +162,16 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
|
||||
4K Video
|
||||
</a>
|
||||
)}
|
||||
{downloadUrls?.videos?.with_vocals && (
|
||||
<a
|
||||
href={api.getDownloadUrl(jobId, "videos", "with_vocals")}
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-purple-600 hover:bg-purple-500 text-white"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<Download className="w-3 h-3" />
|
||||
With Vocals
|
||||
</a>
|
||||
)}
|
||||
{downloadUrls?.packages?.cdg_zip && (
|
||||
<a
|
||||
href={api.getDownloadUrl(jobId, "packages", "cdg_zip")}
|
||||
@@ -134,4 +202,3 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
/**
|
||||
* Email Testing Utilities using MailSlurp
|
||||
* Email Testing Utilities using testmail.app
|
||||
*
|
||||
* MailSlurp provides disposable email inboxes for e2e testing.
|
||||
* Free tier: 50 emails/month - perfect for CI/CD testing.
|
||||
* testmail.app provides namespace-based email addresses for e2e testing.
|
||||
* Emails arrive at: {namespace}.{tag}@inbox.testmail.app
|
||||
*
|
||||
* Setup:
|
||||
* 1. Create account at https://app.mailslurp.com
|
||||
* 2. Get API key from dashboard
|
||||
* 3. Set MAILSLURP_API_KEY environment variable
|
||||
* 1. Create account at https://testmail.app
|
||||
* 2. Get API key and namespace from dashboard
|
||||
* 3. Set TESTMAIL_API_KEY and TESTMAIL_NAMESPACE environment variables
|
||||
*
|
||||
* Usage in tests:
|
||||
* const emailHelper = await createEmailHelper();
|
||||
@@ -17,9 +17,27 @@
|
||||
* const magicLink = emailHelper.extractMagicLink(email);
|
||||
*/
|
||||
|
||||
import { MailSlurp, InboxDto, Email } from 'mailslurp-client';
|
||||
const TESTMAIL_API_KEY = process.env.TESTMAIL_API_KEY;
|
||||
const TESTMAIL_NAMESPACE = process.env.TESTMAIL_NAMESPACE;
|
||||
const TESTMAIL_API_URL = 'https://api.testmail.app/api/json';
|
||||
|
||||
const MAILSLURP_API_KEY = process.env.MAILSLURP_API_KEY;
|
||||
/**
|
||||
* Inbox object compatible with existing test code
|
||||
*/
|
||||
export interface InboxDto {
|
||||
id: string;
|
||||
emailAddress: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Email object compatible with existing test code
|
||||
*/
|
||||
export interface Email {
|
||||
subject?: string;
|
||||
body?: string;
|
||||
from?: string;
|
||||
timestamp?: number;
|
||||
}
|
||||
|
||||
export interface EmailHelper {
|
||||
isAvailable: boolean;
|
||||
@@ -30,40 +48,122 @@ export interface EmailHelper {
|
||||
deleteInbox: (inboxId: string) => Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* testmail.app API response structure
|
||||
*/
|
||||
interface TestmailResponse {
|
||||
result: string;
|
||||
message?: string;
|
||||
count: number;
|
||||
limit: number;
|
||||
offset: number;
|
||||
emails: TestmailEmail[];
|
||||
}
|
||||
|
||||
interface TestmailEmail {
|
||||
from: string;
|
||||
subject: string;
|
||||
html: string;
|
||||
text: string;
|
||||
timestamp: number;
|
||||
tag: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate a unique tag for this test run
|
||||
*/
|
||||
function generateUniqueTag(): string {
|
||||
const timestamp = Date.now();
|
||||
const random = Math.random().toString(36).substring(2, 8);
|
||||
return `test-${timestamp}-${random}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates an email testing helper.
|
||||
* Returns a helper with isAvailable=false if MAILSLURP_API_KEY is not set.
|
||||
* Returns a helper with isAvailable=false if testmail.app env vars are not set.
|
||||
*/
|
||||
export async function createEmailHelper(): Promise<EmailHelper> {
|
||||
if (!MAILSLURP_API_KEY) {
|
||||
console.warn('MAILSLURP_API_KEY not set - email testing disabled');
|
||||
if (!TESTMAIL_API_KEY || !TESTMAIL_NAMESPACE) {
|
||||
console.warn('TESTMAIL_API_KEY or TESTMAIL_NAMESPACE not set - email testing disabled');
|
||||
return createDisabledHelper();
|
||||
}
|
||||
|
||||
const mailslurp = new MailSlurp({ apiKey: MAILSLURP_API_KEY });
|
||||
|
||||
return {
|
||||
isAvailable: true,
|
||||
|
||||
/**
|
||||
* Create a new disposable inbox for testing
|
||||
* Create a new "inbox" for testing.
|
||||
* In testmail.app, this just generates a unique tag - no API call needed.
|
||||
*/
|
||||
async createInbox(): Promise<InboxDto> {
|
||||
const inbox = await mailslurp.createInbox();
|
||||
console.log(`Created test inbox: ${inbox.emailAddress}`);
|
||||
return inbox;
|
||||
const tag = generateUniqueTag();
|
||||
const emailAddress = `${TESTMAIL_NAMESPACE}.${tag}@inbox.testmail.app`;
|
||||
console.log(`Created test inbox: ${emailAddress}`);
|
||||
return { id: tag, emailAddress };
|
||||
},
|
||||
|
||||
/**
|
||||
* Wait for an email to arrive in the inbox
|
||||
* @param inboxId - The inbox ID to check
|
||||
* @param inboxId - The tag to check (returned from createInbox as id)
|
||||
* @param timeout - Timeout in ms (default: 60000 = 1 minute)
|
||||
*/
|
||||
async waitForEmail(inboxId: string, timeout = 60000): Promise<Email> {
|
||||
console.log(`Waiting for email in inbox ${inboxId}...`);
|
||||
const email = await mailslurp.waitForLatestEmail(inboxId, timeout);
|
||||
console.log(`Received email: ${email.subject}`);
|
||||
return email;
|
||||
console.log(`Waiting for email with tag ${inboxId}...`);
|
||||
|
||||
// testmail.app livequery timeout is in seconds, max ~60s per request
|
||||
// For longer timeouts, we poll multiple times
|
||||
const pollInterval = Math.min(timeout, 30000); // 30s max per poll
|
||||
const startTime = Date.now();
|
||||
|
||||
while (Date.now() - startTime < timeout) {
|
||||
const remainingTime = timeout - (Date.now() - startTime);
|
||||
const waitTime = Math.min(pollInterval, remainingTime);
|
||||
const waitSeconds = Math.ceil(waitTime / 1000);
|
||||
|
||||
const url = new URL(TESTMAIL_API_URL);
|
||||
url.searchParams.set('apikey', TESTMAIL_API_KEY!);
|
||||
url.searchParams.set('namespace', TESTMAIL_NAMESPACE!);
|
||||
url.searchParams.set('tag', inboxId);
|
||||
url.searchParams.set('livequery', 'true');
|
||||
url.searchParams.set('wait', waitSeconds.toString());
|
||||
|
||||
try {
|
||||
const response = await fetch(url.toString());
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`testmail.app API error: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const data: TestmailResponse = await response.json();
|
||||
|
||||
if (data.result !== 'success') {
|
||||
throw new Error(`testmail.app API error: ${data.message || 'Unknown error'}`);
|
||||
}
|
||||
|
||||
if (data.emails && data.emails.length > 0) {
|
||||
// Return the most recent email
|
||||
const latestEmail = data.emails.sort((a, b) => b.timestamp - a.timestamp)[0];
|
||||
console.log(`Received email: ${latestEmail.subject}`);
|
||||
|
||||
// Convert to our Email format (use html as body, fallback to text)
|
||||
return {
|
||||
subject: latestEmail.subject,
|
||||
body: latestEmail.html || latestEmail.text,
|
||||
from: latestEmail.from,
|
||||
timestamp: latestEmail.timestamp,
|
||||
};
|
||||
}
|
||||
} catch (error) {
|
||||
// If it's a timeout or network error, continue polling
|
||||
if (Date.now() - startTime < timeout) {
|
||||
console.log(`Polling for email... (${Math.round((Date.now() - startTime) / 1000)}s elapsed)`);
|
||||
continue;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error(`Timed out waiting for email after ${timeout}ms`);
|
||||
},
|
||||
|
||||
/**
|
||||
@@ -135,21 +235,22 @@ export async function createEmailHelper(): Promise<EmailHelper> {
|
||||
},
|
||||
|
||||
/**
|
||||
* Delete an inbox after testing
|
||||
* Delete an inbox after testing.
|
||||
* In testmail.app, emails auto-expire - this is a no-op.
|
||||
*/
|
||||
async deleteInbox(inboxId: string): Promise<void> {
|
||||
await mailslurp.deleteInbox(inboxId);
|
||||
console.log(`Deleted inbox ${inboxId}`);
|
||||
// testmail.app emails auto-expire, no cleanup needed
|
||||
console.log(`Inbox ${inboxId} cleanup skipped (testmail.app auto-expires)`);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a disabled helper for when MailSlurp is not configured
|
||||
* Creates a disabled helper for when testmail.app is not configured
|
||||
*/
|
||||
function createDisabledHelper(): EmailHelper {
|
||||
const notAvailable = () => {
|
||||
throw new Error('Email testing not available - set MAILSLURP_API_KEY');
|
||||
throw new Error('Email testing not available - set TESTMAIL_API_KEY and TESTMAIL_NAMESPACE');
|
||||
};
|
||||
|
||||
return {
|
||||
@@ -166,5 +267,5 @@ function createDisabledHelper(): EmailHelper {
|
||||
* Check if email testing is available
|
||||
*/
|
||||
export function isEmailTestingAvailable(): boolean {
|
||||
return !!MAILSLURP_API_KEY;
|
||||
return !!(TESTMAIL_API_KEY && TESTMAIL_NAMESPACE);
|
||||
}
|
||||
@@ -0,0 +1,219 @@
|
||||
import { test, expect, Page } from "@playwright/test";
|
||||
|
||||
/**
|
||||
* Admin Dashboard Production Tests
|
||||
*
|
||||
* Tests the admin dashboard functionality on live production.
|
||||
* Requires KARAOKE_ADMIN_TOKEN environment variable with admin privileges.
|
||||
* Admin tokens must belong to a user with @nomadkaraoke.com email domain.
|
||||
*
|
||||
* Run with:
|
||||
* KARAOKE_ADMIN_TOKEN=xxx npx playwright test e2e/production/admin-dashboard.spec.ts --reporter=list
|
||||
*/
|
||||
|
||||
const PROD_URL = "https://gen.nomadkaraoke.com";
|
||||
const API_URL = "https://api.nomadkaraoke.com";
|
||||
|
||||
interface UserResponse {
|
||||
user: {
|
||||
email: string;
|
||||
role: string;
|
||||
};
|
||||
}
|
||||
|
||||
async function validateAdminToken(
|
||||
token: string
|
||||
): Promise<{ isAdmin: boolean; email?: string; error?: string }> {
|
||||
try {
|
||||
const response = await fetch(`${API_URL}/api/users/me`, {
|
||||
headers: { Authorization: `Bearer ${token}` },
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
return { isAdmin: false, error: `API returned ${response.status}` };
|
||||
}
|
||||
|
||||
const data = (await response.json()) as UserResponse;
|
||||
const isAdmin =
|
||||
data.user?.role === "admin" ||
|
||||
data.user?.email?.endsWith("@nomadkaraoke.com");
|
||||
|
||||
return {
|
||||
isAdmin,
|
||||
email: data.user?.email,
|
||||
error: isAdmin ? undefined : `User ${data.user?.email} has role "${data.user?.role}", not admin`,
|
||||
};
|
||||
} catch (err) {
|
||||
return { isAdmin: false, error: `Failed to validate token: ${err}` };
|
||||
}
|
||||
}
|
||||
|
||||
function getAdminToken(): string | null {
|
||||
// Prefer KARAOKE_ADMIN_TOKEN, fall back to KARAOKE_ACCESS_TOKEN
|
||||
return process.env.KARAOKE_ADMIN_TOKEN || process.env.KARAOKE_ACCESS_TOKEN || null;
|
||||
}
|
||||
|
||||
async function authenticatePage(page: Page, token: string): Promise<void> {
|
||||
await page.addInitScript((t) => {
|
||||
localStorage.setItem("karaoke_access_token", t);
|
||||
}, token);
|
||||
}
|
||||
|
||||
// Check if we have a valid admin token before running tests
|
||||
let adminToken: string | null = null;
|
||||
let tokenValidation: { isAdmin: boolean; email?: string; error?: string } | null = null;
|
||||
|
||||
test.beforeAll(async () => {
|
||||
adminToken = getAdminToken();
|
||||
if (adminToken) {
|
||||
tokenValidation = await validateAdminToken(adminToken);
|
||||
if (!tokenValidation.isAdmin) {
|
||||
console.log(`⚠️ Token validation failed: ${tokenValidation.error}`);
|
||||
console.log(" Admin tests require a token for a @nomadkaraoke.com user");
|
||||
console.log(" Set KARAOKE_ADMIN_TOKEN environment variable with an admin token");
|
||||
} else {
|
||||
console.log(`✓ Admin token validated for ${tokenValidation.email}`);
|
||||
}
|
||||
} else {
|
||||
console.log("⚠️ No admin token found (KARAOKE_ADMIN_TOKEN or KARAOKE_ACCESS_TOKEN)");
|
||||
}
|
||||
});
|
||||
|
||||
test.describe("Admin Dashboard - Production", () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
test.skip(!adminToken || !tokenValidation?.isAdmin, "Skipping: No valid admin token available");
|
||||
await authenticatePage(page, adminToken!);
|
||||
});
|
||||
|
||||
test("admin dashboard loads and shows stats", async ({ page }) => {
|
||||
await page.goto(`${PROD_URL}/admin`);
|
||||
|
||||
// Wait for page to load
|
||||
await expect(page.locator("h1")).toContainText("Admin Dashboard", {
|
||||
timeout: 30000,
|
||||
});
|
||||
|
||||
// Check stats cards are present
|
||||
await expect(page.getByText("Total Users")).toBeVisible();
|
||||
await expect(page.getByText("Total Jobs")).toBeVisible();
|
||||
await expect(page.getByText("Active Users")).toBeVisible();
|
||||
|
||||
// Check job status breakdown card is present
|
||||
await expect(page.getByText("Jobs by Status")).toBeVisible();
|
||||
});
|
||||
|
||||
test("admin sidebar navigation works", async ({ page }) => {
|
||||
await page.goto(`${PROD_URL}/admin`);
|
||||
|
||||
// Wait for dashboard to load
|
||||
await expect(page.locator("h1")).toContainText("Admin Dashboard", {
|
||||
timeout: 30000,
|
||||
});
|
||||
|
||||
// Click Users in sidebar
|
||||
await page.getByRole("link", { name: /Users/i }).first().click();
|
||||
await expect(page).toHaveURL(/\/admin\/users/);
|
||||
await expect(page.locator("h1")).toContainText("Users");
|
||||
|
||||
// Click Jobs in sidebar
|
||||
await page.getByRole("link", { name: /Jobs/i }).first().click();
|
||||
await expect(page).toHaveURL(/\/admin\/jobs/);
|
||||
await expect(page.locator("h1")).toContainText("Jobs");
|
||||
|
||||
// Click Beta Program in sidebar
|
||||
await page.getByRole("link", { name: /Beta/i }).first().click();
|
||||
await expect(page).toHaveURL(/\/admin\/beta/);
|
||||
await expect(page.locator("h1")).toContainText("Beta Program");
|
||||
|
||||
// Navigate back to dashboard
|
||||
await page.getByRole("link", { name: /Dashboard/i }).first().click();
|
||||
await expect(page).toHaveURL(/\/admin$/);
|
||||
});
|
||||
|
||||
test("users page loads with user list", async ({ page }) => {
|
||||
await page.goto(`${PROD_URL}/admin/users`);
|
||||
|
||||
// Wait for page to load
|
||||
await expect(page.locator("h1")).toContainText("Users", { timeout: 30000 });
|
||||
|
||||
// Check table headers are present
|
||||
await expect(page.getByRole("columnheader", { name: "Email" })).toBeVisible();
|
||||
await expect(page.getByRole("columnheader", { name: "Role" })).toBeVisible();
|
||||
await expect(page.getByRole("columnheader", { name: "Credits" })).toBeVisible();
|
||||
|
||||
// Check search input is present
|
||||
await expect(page.getByPlaceholder(/Search/i)).toBeVisible();
|
||||
|
||||
// Check pagination controls
|
||||
await expect(page.getByText(/Showing/)).toBeVisible();
|
||||
});
|
||||
|
||||
test("jobs page loads with job list", async ({ page }) => {
|
||||
await page.goto(`${PROD_URL}/admin/jobs`);
|
||||
|
||||
// Wait for page to load
|
||||
await expect(page.locator("h1")).toContainText("Jobs", { timeout: 30000 });
|
||||
|
||||
// Check table headers are present
|
||||
await expect(page.getByRole("columnheader", { name: "Job ID" })).toBeVisible();
|
||||
await expect(page.getByRole("columnheader", { name: "Status" })).toBeVisible();
|
||||
|
||||
// Check filter controls
|
||||
await expect(page.getByPlaceholder(/Filter by user/i)).toBeVisible();
|
||||
});
|
||||
|
||||
test("beta page loads with stats", async ({ page }) => {
|
||||
await page.goto(`${PROD_URL}/admin/beta`);
|
||||
|
||||
// Wait for page to load
|
||||
await expect(page.locator("h1")).toContainText("Beta Program", {
|
||||
timeout: 30000,
|
||||
});
|
||||
|
||||
// Check stats cards
|
||||
await expect(page.getByText("Total Beta Testers")).toBeVisible();
|
||||
await expect(page.getByText("Feedback Submissions")).toBeVisible();
|
||||
});
|
||||
});
|
||||
|
||||
test.describe("Admin API - Production", () => {
|
||||
test.beforeEach(async () => {
|
||||
test.skip(!adminToken || !tokenValidation?.isAdmin, "Skipping: No valid admin token available");
|
||||
});
|
||||
|
||||
test("API stats endpoint returns data", async ({ request }) => {
|
||||
const response = await request.get(`${API_URL}/api/admin/stats/overview`, {
|
||||
headers: {
|
||||
Authorization: `Bearer ${adminToken}`,
|
||||
},
|
||||
});
|
||||
|
||||
expect(response.status()).toBe(200);
|
||||
|
||||
const data = await response.json();
|
||||
expect(data).toHaveProperty("total_users");
|
||||
expect(data).toHaveProperty("total_jobs");
|
||||
expect(data).toHaveProperty("jobs_by_status");
|
||||
expect(typeof data.total_users).toBe("number");
|
||||
expect(typeof data.total_jobs).toBe("number");
|
||||
|
||||
console.log(`Stats: ${data.total_users} users, ${data.total_jobs} jobs`);
|
||||
});
|
||||
|
||||
test("API users list endpoint returns data", async ({ request }) => {
|
||||
const response = await request.get(`${API_URL}/api/users/admin/users?limit=5`, {
|
||||
headers: {
|
||||
Authorization: `Bearer ${adminToken}`,
|
||||
},
|
||||
});
|
||||
|
||||
expect(response.status()).toBe(200);
|
||||
|
||||
const data = await response.json();
|
||||
expect(data).toHaveProperty("users");
|
||||
expect(data).toHaveProperty("total");
|
||||
expect(Array.isArray(data.users)).toBe(true);
|
||||
|
||||
console.log(`Users: ${data.total} total, fetched ${data.users.length}`);
|
||||
});
|
||||
});
|
||||
@@ -25,7 +25,8 @@ import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-tes
|
||||
* 10. Verify Outputs - Check download URLs
|
||||
*
|
||||
* Environment Variables:
|
||||
* - MAILSLURP_API_KEY: For email testing (required for full flow)
|
||||
* - TESTMAIL_API_KEY: For email testing (required for full flow)
|
||||
* - TESTMAIL_NAMESPACE: Your testmail.app namespace
|
||||
* - KARAOKE_ACCESS_TOKEN: Skip enrollment and use existing token
|
||||
*/
|
||||
|
||||
@@ -47,8 +48,9 @@ const TIMEOUTS = {
|
||||
action: 30_000, // 30s for UI actions
|
||||
expect: 60_000, // 60s for assertions
|
||||
apiCall: 120_000, // 2min for API calls
|
||||
download: 900_000, // 15min for remote flacfetch downloads (torrents can be slow)
|
||||
jobProcessing: 600_000, // 10min for job processing
|
||||
fullTest: 900_000, // 15min for full test
|
||||
fullTest: 2400_000, // 40min for full test (download + processing + video generation)
|
||||
} as const;
|
||||
|
||||
// =============================================================================
|
||||
@@ -188,11 +190,11 @@ test.describe('Production E2E - Full User Journey', () => {
|
||||
});
|
||||
console.log(' Authenticated successfully with existing token');
|
||||
} else if (isEmailTestingAvailable()) {
|
||||
console.log(' Starting beta enrollment with MailSlurp');
|
||||
console.log(' Starting beta enrollment with testmail.app');
|
||||
const emailHelper = await createEmailHelper();
|
||||
|
||||
if (!emailHelper.isAvailable) {
|
||||
throw new Error('MailSlurp not available');
|
||||
throw new Error('testmail.app not available');
|
||||
}
|
||||
|
||||
// Create test inbox
|
||||
@@ -248,7 +250,7 @@ test.describe('Production E2E - Full User Journey', () => {
|
||||
}
|
||||
} else {
|
||||
throw new Error(
|
||||
'No KARAOKE_ACCESS_TOKEN and no MAILSLURP_API_KEY - cannot authenticate'
|
||||
'No KARAOKE_ACCESS_TOKEN and no TESTMAIL_API_KEY/TESTMAIL_NAMESPACE - cannot authenticate'
|
||||
);
|
||||
}
|
||||
|
||||
@@ -388,73 +390,48 @@ test.describe('Production E2E - Full User Journey', () => {
|
||||
console.log('STEP 5 COMPLETE: Audio selection handled');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 6: Wait for Processing (Lyrics)
|
||||
// STEP 6: Wait for Processing (Download + Audio Separation + Lyrics)
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 6: Wait for Processing');
|
||||
console.log('========================================');
|
||||
|
||||
console.log(' Polling job status...');
|
||||
console.log(' Polling job status (this may take 15-20 minutes for remote downloads)...');
|
||||
const { status: statusAfterProcessing, job: jobAfterProcessing } = await pollJobStatus(
|
||||
request,
|
||||
jobId,
|
||||
['in_review', 'awaiting_instrumental_selection', 'completed'],
|
||||
accessToken!
|
||||
['awaiting_review', 'in_review', 'awaiting_instrumental_selection', 'complete'],
|
||||
accessToken!,
|
||||
TIMEOUTS.download // Use longer timeout for downloads
|
||||
);
|
||||
|
||||
console.log(`STEP 6 COMPLETE: Job reached ${statusAfterProcessing}`);
|
||||
|
||||
// =========================================================================
|
||||
// STEP 7: Lyrics Review
|
||||
// STEP 7: Complete Lyrics Review (via API)
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 7: Lyrics Review');
|
||||
console.log('========================================');
|
||||
|
||||
if (statusAfterProcessing === 'in_review') {
|
||||
await page.reload();
|
||||
await page.waitForLoadState('networkidle');
|
||||
if (statusAfterProcessing === 'awaiting_review' || statusAfterProcessing === 'in_review') {
|
||||
console.log(' Completing lyrics review via API (accepting AI-generated lyrics)...');
|
||||
|
||||
const reviewBtn = page.getByRole('button', { name: /review.*lyrics/i }).first();
|
||||
|
||||
if (await reviewBtn.isVisible({ timeout: 10000 }).catch(() => false)) {
|
||||
console.log(' Opening lyrics review...');
|
||||
|
||||
// This opens a new tab
|
||||
const [popup] = await Promise.all([
|
||||
page.context().waitForEvent('page', { timeout: 15000 }).catch(() => null),
|
||||
reviewBtn.click(),
|
||||
]);
|
||||
|
||||
if (popup) {
|
||||
await popup.waitForLoadState('networkidle');
|
||||
await popup.waitForTimeout(5000);
|
||||
await popup.screenshot({
|
||||
path: 'test-results/07a-review-ui.png',
|
||||
fullPage: true,
|
||||
});
|
||||
|
||||
// Click Preview Video
|
||||
const previewBtn = popup.getByRole('button', { name: /preview.*video/i }).first();
|
||||
if (await previewBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
console.log(' Clicking Preview Video...');
|
||||
await previewBtn.click();
|
||||
await popup.waitForTimeout(30000); // Preview generation
|
||||
|
||||
// Click Complete Review
|
||||
const completeBtn = popup.getByRole('button', { name: /complete.*review/i }).first();
|
||||
if (await completeBtn.isVisible({ timeout: 60000 }).catch(() => false)) {
|
||||
console.log(' Clicking Complete Review...');
|
||||
await completeBtn.click();
|
||||
await popup.waitForTimeout(3000);
|
||||
console.log(' Review completed');
|
||||
}
|
||||
}
|
||||
|
||||
if (!popup.isClosed()) {
|
||||
await popup.close();
|
||||
}
|
||||
// Use the API to complete the review - this accepts the current lyrics as-is
|
||||
const completeReviewResponse = await request.post(
|
||||
`${API_URL}/api/jobs/${jobId}/complete-review`,
|
||||
{
|
||||
headers: { Authorization: `Bearer ${accessToken}` },
|
||||
timeout: TIMEOUTS.apiCall,
|
||||
}
|
||||
);
|
||||
|
||||
if (completeReviewResponse.ok()) {
|
||||
const result = await completeReviewResponse.json();
|
||||
console.log(` Review completed: ${result.message}`);
|
||||
} else {
|
||||
const errorText = await completeReviewResponse.text();
|
||||
throw new Error(`Failed to complete review: ${completeReviewResponse.status()} - ${errorText}`);
|
||||
}
|
||||
} else {
|
||||
console.log(` Skipping - job status is ${statusAfterProcessing}`);
|
||||
@@ -463,43 +440,44 @@ test.describe('Production E2E - Full User Journey', () => {
|
||||
console.log('STEP 7 COMPLETE: Lyrics review handled');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 8: Instrumental Selection
|
||||
// STEP 8: Wait for Video Rendering and Instrumental Selection (via API)
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 8: Instrumental Selection');
|
||||
console.log('========================================');
|
||||
|
||||
// Wait for job to reach instrumental selection
|
||||
// Wait for job to reach instrumental selection (after video rendering)
|
||||
console.log(' Waiting for video rendering to complete...');
|
||||
const { status: statusBeforeInstrumental } = await pollJobStatus(
|
||||
request,
|
||||
jobId,
|
||||
['awaiting_instrumental_selection', 'completed'],
|
||||
['awaiting_instrumental_selection', 'complete'],
|
||||
accessToken!,
|
||||
120000 // 2 minutes
|
||||
TIMEOUTS.jobProcessing // Video rendering can take several minutes
|
||||
).catch(() => ({ status: 'unknown', job: null }));
|
||||
|
||||
if (statusBeforeInstrumental === 'awaiting_instrumental_selection') {
|
||||
await page.reload();
|
||||
await page.waitForLoadState('networkidle');
|
||||
console.log(' Selecting clean instrumental via API...');
|
||||
|
||||
const instrumentalBtn = page.getByRole('button', { name: /select.*instrumental/i }).first();
|
||||
if (await instrumentalBtn.isVisible({ timeout: 10000 }).catch(() => false)) {
|
||||
console.log(' Opening instrumental selection...');
|
||||
await instrumentalBtn.click();
|
||||
await page.waitForTimeout(2000);
|
||||
|
||||
const dialog = page.locator('[role="dialog"]');
|
||||
if (await dialog.isVisible()) {
|
||||
await page.screenshot({ path: 'test-results/08-instrumental-dialog.png' });
|
||||
|
||||
// Select first instrumental (usually "Clean")
|
||||
const selectBtn = dialog.getByRole('button', { name: /select/i }).first();
|
||||
if (await selectBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
await selectBtn.click();
|
||||
console.log(' Instrumental selected');
|
||||
await page.waitForTimeout(3000);
|
||||
}
|
||||
// Use the API to select the clean instrumental
|
||||
const selectInstrumentalResponse = await request.post(
|
||||
`${API_URL}/api/jobs/${jobId}/select-instrumental`,
|
||||
{
|
||||
headers: {
|
||||
Authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
data: { selection: 'clean' },
|
||||
timeout: TIMEOUTS.apiCall,
|
||||
}
|
||||
);
|
||||
|
||||
if (selectInstrumentalResponse.ok()) {
|
||||
const result = await selectInstrumentalResponse.json();
|
||||
console.log(` Instrumental selected: ${result.message}`);
|
||||
} else {
|
||||
const errorText = await selectInstrumentalResponse.text();
|
||||
throw new Error(`Failed to select instrumental: ${selectInstrumentalResponse.status()} - ${errorText}`);
|
||||
}
|
||||
} else {
|
||||
console.log(` Skipping - job status is ${statusBeforeInstrumental}`);
|
||||
@@ -508,17 +486,17 @@ test.describe('Production E2E - Full User Journey', () => {
|
||||
console.log('STEP 8 COMPLETE: Instrumental selection handled');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 9: Wait for Completion
|
||||
// STEP 9: Wait for Final Video Generation and Completion
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 9: Wait for Completion');
|
||||
console.log('========================================');
|
||||
|
||||
console.log(' Waiting for job to complete...');
|
||||
console.log(' Waiting for final video generation (encoding 4 formats)...');
|
||||
const { status: finalStatus, job: finalJob } = await pollJobStatus(
|
||||
request,
|
||||
jobId,
|
||||
['completed'],
|
||||
['complete'], // Note: status is 'complete' not 'completed'
|
||||
accessToken!,
|
||||
TIMEOUTS.jobProcessing
|
||||
);
|
||||
@@ -526,29 +504,57 @@ test.describe('Production E2E - Full User Journey', () => {
|
||||
console.log('STEP 9 COMPLETE: Job completed');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 10: Verify Outputs
|
||||
// STEP 10: Verify Outputs and Download URLs
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 10: Verify Outputs');
|
||||
console.log('========================================');
|
||||
|
||||
console.log(` Job ID: ${finalJob.job_id}`);
|
||||
console.log(` Artist: ${finalJob.artist}`);
|
||||
console.log(` Title: ${finalJob.title}`);
|
||||
console.log(` Theme ID: ${finalJob.theme_id || 'N/A'}`);
|
||||
console.log(` YouTube URL: ${finalJob.youtube_url || 'N/A'}`);
|
||||
console.log(` Dropbox URL: ${finalJob.dropbox_url || 'N/A'}`);
|
||||
console.log(` Google Drive URL: ${finalJob.gdrive_url || 'N/A'}`);
|
||||
|
||||
// Verify required outputs
|
||||
expect(finalJob.theme_id).toBeTruthy();
|
||||
// Verify required fields
|
||||
expect(finalJob.status).toBe('complete');
|
||||
expect(finalJob.artist).toBeTruthy();
|
||||
expect(finalJob.title).toBeTruthy();
|
||||
|
||||
// Check for download URLs
|
||||
// Check for download URLs - this is the key verification
|
||||
const downloadResponse = await request.get(`${API_URL}/api/jobs/${jobId}/download-urls`, {
|
||||
headers: { Authorization: `Bearer ${accessToken}` },
|
||||
timeout: TIMEOUTS.apiCall,
|
||||
}).catch(() => null);
|
||||
});
|
||||
|
||||
if (downloadResponse?.ok()) {
|
||||
if (downloadResponse.ok()) {
|
||||
const downloadUrls = await downloadResponse.json();
|
||||
console.log(` Download URLs available: ${Object.keys(downloadUrls).length}`);
|
||||
const urlKeys = Object.keys(downloadUrls);
|
||||
console.log(` Download formats available: ${urlKeys.length}`);
|
||||
|
||||
// Should have multiple video formats
|
||||
for (const key of urlKeys) {
|
||||
console.log(` - ${key}: ${downloadUrls[key] ? 'Available' : 'Missing'}`);
|
||||
}
|
||||
|
||||
// Verify we have at least one download URL
|
||||
expect(urlKeys.length).toBeGreaterThan(0);
|
||||
|
||||
// Verify at least one URL is actually present
|
||||
const hasUrl = urlKeys.some(key => downloadUrls[key]);
|
||||
expect(hasUrl).toBe(true);
|
||||
} else {
|
||||
console.log(` WARNING: Could not fetch download URLs (status ${downloadResponse.status()})`);
|
||||
}
|
||||
|
||||
// Get final job state for reference
|
||||
const finalJobResponse = await request.get(`${API_URL}/api/jobs/${jobId}`, {
|
||||
headers: { Authorization: `Bearer ${accessToken}` },
|
||||
timeout: TIMEOUTS.apiCall,
|
||||
});
|
||||
|
||||
if (finalJobResponse.ok()) {
|
||||
const fullJob = await finalJobResponse.json();
|
||||
console.log(` File URLs available: ${Object.keys(fullJob.file_urls || {}).length} categories`);
|
||||
}
|
||||
|
||||
await page.reload();
|
||||
|
||||
@@ -0,0 +1,935 @@
|
||||
import { test, expect, Page, BrowserContext } from '@playwright/test';
|
||||
import { createEmailHelper, isEmailTestingAvailable } from '../helpers/email-testing';
|
||||
|
||||
/**
|
||||
* E2E Happy Path Test - Real User Journey with Full UI Interactions
|
||||
*
|
||||
* This test validates the COMPLETE karaoke generation flow using ONLY
|
||||
* browser interactions - no API shortcuts allowed.
|
||||
*
|
||||
* See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements.
|
||||
*
|
||||
* Test Flow:
|
||||
* 1. Landing Page - Navigate and verify
|
||||
* 2. New User Signup - Create testmail.app inbox, submit beta form, verify email
|
||||
* 3. Magic Link Auth - Click magic link to authenticate
|
||||
* 4. Create Job - Search for test song via UI
|
||||
* 5. Audio Selection - Select audio source via UI dialog
|
||||
* 6. Wait for Processing - Monitor via UI status updates
|
||||
* 7. Lyrics Review - Open review UI, interact, submit via UI
|
||||
* 8. Instrumental Selection - Open selection UI, interact, submit via UI
|
||||
* 9. Wait for Completion - Monitor via UI
|
||||
* 10. Verify Downloads - Check download links work
|
||||
* 11. Verify Distribution - Check YouTube/Dropbox/GDrive indicators
|
||||
* 12. Cleanup - Delete distributed content and job
|
||||
*
|
||||
* Environment Variables:
|
||||
* - TESTMAIL_API_KEY + TESTMAIL_NAMESPACE: Required for email testing (can be skipped with E2E_TEST_TOKEN)
|
||||
* - E2E_TEST_TOKEN: Pre-configured user token to skip signup flow (for iterations)
|
||||
* - E2E_ADMIN_TOKEN: Required for cleanup endpoint (optional - cleanup skipped if not set)
|
||||
*/
|
||||
|
||||
/**
|
||||
* Check if we should use a pre-configured test token instead of creating a new user
|
||||
*/
|
||||
function hasPreConfiguredToken(): boolean {
|
||||
return !!process.env.E2E_TEST_TOKEN;
|
||||
}
|
||||
|
||||
function getPreConfiguredToken(): string {
|
||||
return process.env.E2E_TEST_TOKEN || '';
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// CONSTANTS
|
||||
// =============================================================================
|
||||
|
||||
const PROD_URL = 'https://gen.nomadkaraoke.com';
|
||||
const API_URL = 'https://api.nomadkaraoke.com';
|
||||
|
||||
// Test song - uses cached flacfetch results for speed
|
||||
const TEST_SONG = {
|
||||
artist: 'piri',
|
||||
title: 'dog',
|
||||
} as const;
|
||||
|
||||
// Timeouts
|
||||
const TIMEOUTS = {
|
||||
action: 30_000, // 30s for UI actions
|
||||
expect: 60_000, // 60s for assertions
|
||||
emailArrival: 120_000, // 2min for email to arrive
|
||||
audioSearch: 120_000, // 2min for audio search
|
||||
lyricsProcessing: 1500_000, // 25min for lyrics transcription (matches Cloud Tasks deadline)
|
||||
videoRendering: 900_000, // 15min for video rendering
|
||||
finalEncoding: 600_000, // 10min for final encoding
|
||||
fullTest: 3600_000, // 60min for full test
|
||||
} as const;
|
||||
|
||||
// =============================================================================
|
||||
// HELPERS
|
||||
// =============================================================================
|
||||
|
||||
/**
|
||||
* Wait for job status to appear in the UI
|
||||
*/
|
||||
async function waitForJobStatus(
|
||||
page: Page,
|
||||
jobCard: any,
|
||||
targetStatuses: string[],
|
||||
timeoutMs: number
|
||||
): Promise<string> {
|
||||
const startTime = Date.now();
|
||||
|
||||
while (Date.now() - startTime < timeoutMs) {
|
||||
// Check the status indicator text in the job card
|
||||
const statusText = await jobCard.locator('[class*="text-"]').first().textContent() || '';
|
||||
|
||||
for (const target of targetStatuses) {
|
||||
if (statusText.toLowerCase().includes(target.toLowerCase())) {
|
||||
console.log(` Found status: ${statusText}`);
|
||||
return target;
|
||||
}
|
||||
}
|
||||
|
||||
// Also check for "Action needed" indicator
|
||||
const actionNeeded = await jobCard.getByText('Action needed').isVisible().catch(() => false);
|
||||
if (actionNeeded) {
|
||||
console.log(` Action needed - checking for target status`);
|
||||
for (const target of targetStatuses) {
|
||||
if (statusText.toLowerCase().includes(target.replace('awaiting_', '').replace('_', ' '))) {
|
||||
return target;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Wait before next check
|
||||
await page.waitForTimeout(5000);
|
||||
|
||||
// Refresh to get latest status
|
||||
const refreshBtn = page.getByRole('button', { name: /refresh/i });
|
||||
if (await refreshBtn.isVisible().catch(() => false)) {
|
||||
await refreshBtn.click();
|
||||
await page.waitForTimeout(2000);
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error(`Timeout waiting for status: ${targetStatuses.join(' or ')}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the auth token from localStorage
|
||||
*/
|
||||
async function getAuthToken(page: Page): Promise<string | null> {
|
||||
return await page.evaluate(() => localStorage.getItem('karaoke_access_token'));
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// TEST
|
||||
// =============================================================================
|
||||
|
||||
test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
|
||||
// IMPORTANT: Disable retries for this test - each retry creates a new karaoke job
|
||||
// which wastes 15-20 minutes of processing time
|
||||
test.describe.configure({ retries: 0 });
|
||||
|
||||
test('Complete flow: New user signup -> Karaoke generation -> Distribution -> Cleanup', async ({
|
||||
page,
|
||||
context,
|
||||
}) => {
|
||||
test.setTimeout(TIMEOUTS.fullTest);
|
||||
|
||||
// Check if we have a pre-configured token (for iterations without using testmail.app)
|
||||
const usePreConfiguredToken = hasPreConfiguredToken();
|
||||
|
||||
// Skip if neither pre-configured token nor testmail.app is available
|
||||
if (!usePreConfiguredToken && !isEmailTestingAvailable()) {
|
||||
test.skip(true, 'Neither E2E_TEST_TOKEN nor TESTMAIL_API_KEY/TESTMAIL_NAMESPACE set - cannot run test');
|
||||
return;
|
||||
}
|
||||
|
||||
let accessToken: string | null = null;
|
||||
let jobId: string | null = null;
|
||||
let inboxId: string | null = null;
|
||||
|
||||
// Only create email helper if we're doing the full signup flow
|
||||
const emailHelper = usePreConfiguredToken ? null : await createEmailHelper();
|
||||
|
||||
try {
|
||||
// =========================================================================
|
||||
// STEP 1: Landing Page
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 1: Landing Page');
|
||||
console.log('========================================');
|
||||
|
||||
await page.goto(PROD_URL);
|
||||
await page.waitForLoadState('networkidle');
|
||||
|
||||
// Verify hero section
|
||||
await expect(page.locator('h1')).toContainText('Karaoke Video', { timeout: TIMEOUTS.expect });
|
||||
console.log(' Hero section visible');
|
||||
|
||||
// Verify pricing section
|
||||
await expect(page.locator('#pricing')).toBeVisible({ timeout: TIMEOUTS.expect });
|
||||
console.log(' Pricing section visible');
|
||||
|
||||
await page.screenshot({ path: 'test-results/01-landing-page.png', fullPage: true });
|
||||
console.log('STEP 1 COMPLETE: Landing page loads correctly');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 2 & 3: Authentication (either via pre-configured token or beta signup)
|
||||
// =========================================================================
|
||||
|
||||
if (usePreConfiguredToken) {
|
||||
// ----- FAST PATH: Use pre-configured token -----
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 2: Using Pre-Configured Token (skipping signup)');
|
||||
console.log('========================================');
|
||||
|
||||
accessToken = getPreConfiguredToken();
|
||||
console.log(` Using token: ${accessToken.substring(0, 8)}...`);
|
||||
|
||||
// Navigate to app and inject token into localStorage
|
||||
await page.goto(`${PROD_URL}/app`);
|
||||
await page.evaluate((token) => {
|
||||
localStorage.setItem('karaoke_access_token', token);
|
||||
}, accessToken);
|
||||
|
||||
// Reload to apply the token
|
||||
await page.reload();
|
||||
await page.waitForLoadState('networkidle');
|
||||
|
||||
await page.screenshot({ path: 'test-results/02-token-injected.png' });
|
||||
console.log('STEP 2 COMPLETE: Token injected, skipping beta enrollment');
|
||||
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 3: Verify Authentication');
|
||||
console.log('========================================');
|
||||
|
||||
// Verify we're authenticated by checking for the app UI
|
||||
await expect(page.locator('body')).not.toContainText('Sign in', { timeout: TIMEOUTS.action });
|
||||
console.log(' User authenticated');
|
||||
|
||||
// Try to find credit indicator
|
||||
const creditText = await page.getByText(/credit/i).first().textContent().catch(() => 'N/A');
|
||||
console.log(` Credits visible: ${creditText}`);
|
||||
|
||||
console.log('STEP 3 COMPLETE: User authenticated with pre-configured token');
|
||||
|
||||
} else {
|
||||
// ----- FULL PATH: Beta signup with testmail.app -----
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 2: New User Beta Enrollment');
|
||||
console.log('========================================');
|
||||
|
||||
// Create test inbox
|
||||
const inbox = await emailHelper!.createInbox();
|
||||
inboxId = inbox.id!;
|
||||
console.log(` Test email: ${inbox.emailAddress}`);
|
||||
|
||||
// Scroll to beta section first to ensure it's in view
|
||||
const betaSection = page.locator('text=Beta Tester Program').first();
|
||||
await betaSection.scrollIntoViewIfNeeded();
|
||||
await page.waitForTimeout(500);
|
||||
|
||||
// Open beta form - click the Join Beta Program button
|
||||
const betaButton = page.getByRole('button', { name: /join beta program/i });
|
||||
await expect(betaButton).toBeVisible({ timeout: TIMEOUTS.action });
|
||||
|
||||
// Take screenshot before clicking
|
||||
await page.screenshot({ path: 'test-results/02-before-beta-click.png' });
|
||||
console.log(' Clicking Join Beta Program button...');
|
||||
|
||||
// Click and wait for the form to appear
|
||||
await betaButton.click();
|
||||
|
||||
// Wait for beta form to appear - the button should be replaced by the form
|
||||
const betaEmailInput = page.locator('#beta-email');
|
||||
|
||||
// Poll for form visibility with retries
|
||||
let formVisible = false;
|
||||
for (let attempt = 0; attempt < 5; attempt++) {
|
||||
await page.waitForTimeout(1000);
|
||||
formVisible = await betaEmailInput.isVisible().catch(() => false);
|
||||
if (formVisible) break;
|
||||
console.log(` Form not visible yet, attempt ${attempt + 1}/5`);
|
||||
// Try clicking again if form didn't appear
|
||||
const buttonStillVisible = await betaButton.isVisible().catch(() => false);
|
||||
if (buttonStillVisible) {
|
||||
console.log(' Button still visible, clicking again...');
|
||||
await betaButton.click();
|
||||
}
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/02-after-beta-click.png' });
|
||||
|
||||
if (!formVisible) {
|
||||
// Debug: log the page state
|
||||
const pageContent = await page.content();
|
||||
console.log(' Page HTML length:', pageContent.length);
|
||||
console.log(' Looking for beta-email in HTML:', pageContent.includes('beta-email'));
|
||||
throw new Error('Beta form did not appear after clicking Join Beta Program button');
|
||||
}
|
||||
|
||||
await expect(betaEmailInput).toBeVisible({ timeout: TIMEOUTS.action });
|
||||
console.log(' Beta form visible');
|
||||
|
||||
// Fill beta form
|
||||
await betaEmailInput.fill(inbox.emailAddress!);
|
||||
|
||||
// Fill the promise/reason field
|
||||
const promiseField = page.locator('#beta-promise, textarea');
|
||||
await promiseField.first().fill('piri - dog (E2E test: testing complete karaoke generation flow)');
|
||||
|
||||
// Check the acceptance checkbox
|
||||
const acceptCheckbox = page.locator('#beta-accept, input[type="checkbox"]');
|
||||
await acceptCheckbox.first().check();
|
||||
|
||||
await page.screenshot({ path: 'test-results/02a-beta-form-filled.png' });
|
||||
console.log(' Beta form filled');
|
||||
|
||||
// Submit form - look for the submit button
|
||||
const submitButton = page.getByRole('button', { name: /get.*free.*credit|submit|enroll/i });
|
||||
await expect(submitButton).toBeEnabled({ timeout: TIMEOUTS.action });
|
||||
await submitButton.click();
|
||||
|
||||
// Wait for success and redirect
|
||||
await expect(
|
||||
page.getByText(/welcome to the beta|redirecting to the app/i)
|
||||
).toBeVisible({ timeout: 15000 });
|
||||
|
||||
await page.waitForURL(/\/app/, { timeout: 10000 });
|
||||
console.log(' Redirected to /app');
|
||||
|
||||
// Get the token from localStorage
|
||||
accessToken = await getAuthToken(page);
|
||||
if (!accessToken) {
|
||||
throw new Error('No access token received after beta enrollment');
|
||||
}
|
||||
console.log(' Session token received');
|
||||
|
||||
await page.screenshot({ path: 'test-results/02b-app-after-enrollment.png' });
|
||||
console.log('STEP 2 COMPLETE: Beta enrollment successful');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 3: Verify Credits (via UI)
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 3: Verify Credits');
|
||||
console.log('========================================');
|
||||
|
||||
// Look for credit indicator in the UI
|
||||
const creditText = await page.getByText(/credit/i).first().textContent();
|
||||
console.log(` Credits visible: ${creditText}`);
|
||||
|
||||
console.log('STEP 3 COMPLETE: User authenticated with credits');
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// STEP 4: Create Karaoke Job (via UI)
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 4: Create Karaoke Job');
|
||||
console.log('========================================');
|
||||
|
||||
// Navigate to Search tab
|
||||
await page.getByRole('tab', { name: /search/i }).click();
|
||||
await page.waitForTimeout(1000);
|
||||
|
||||
// Select a theme if required
|
||||
const themeSelect = page.locator('button[role="combobox"]').first();
|
||||
if (await themeSelect.isVisible({ timeout: 3000 }).catch(() => false)) {
|
||||
const currentTheme = await themeSelect.textContent();
|
||||
if (currentTheme?.toLowerCase().includes('select')) {
|
||||
await themeSelect.click();
|
||||
await page.waitForTimeout(500);
|
||||
await page.locator('[role="option"]').first().click();
|
||||
await page.waitForTimeout(500);
|
||||
console.log(' Selected first theme');
|
||||
}
|
||||
}
|
||||
|
||||
// Fill artist and title
|
||||
await page.getByLabel('Artist').fill(TEST_SONG.artist);
|
||||
await page.getByLabel('Title').fill(TEST_SONG.title);
|
||||
console.log(` Searching for: ${TEST_SONG.artist} - ${TEST_SONG.title}`);
|
||||
|
||||
await page.screenshot({ path: 'test-results/04a-search-form.png' });
|
||||
|
||||
// Submit search
|
||||
await page.getByRole('button', { name: /search.*create/i }).click();
|
||||
|
||||
// Wait for job card to appear
|
||||
console.log(' Waiting for job card...');
|
||||
const jobCard = page.locator('[class*="rounded-lg"][class*="border"]').filter({
|
||||
hasText: `${TEST_SONG.artist} - ${TEST_SONG.title}`
|
||||
}).first();
|
||||
|
||||
await expect(jobCard).toBeVisible({ timeout: TIMEOUTS.audioSearch });
|
||||
console.log(' Job card visible');
|
||||
|
||||
// Extract job ID from the card's full text content
|
||||
// The ID appears as "ID: xxxxxxxx" in the card text (e.g., "piri - dogID: af10c81f")
|
||||
const cardFullText = await jobCard.textContent() || '';
|
||||
const idMatch = cardFullText.match(/ID:\s*([a-f0-9]{8,})/i);
|
||||
if (idMatch) {
|
||||
jobId = idMatch[1];
|
||||
console.log(` Job ID: ${jobId}`);
|
||||
} else {
|
||||
console.log(` WARNING: Could not extract job ID from card text`);
|
||||
console.log(` Card text sample: ${cardFullText.substring(0, 100)}...`);
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/04b-job-created.png' });
|
||||
console.log('STEP 4 COMPLETE: Job created');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 5: Audio Selection (via UI)
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 5: Audio Selection');
|
||||
console.log('========================================');
|
||||
|
||||
// Wait for job to need audio selection or skip if not needed
|
||||
await page.waitForTimeout(5000);
|
||||
|
||||
// Check if Select Audio button appears
|
||||
const selectAudioBtn = jobCard.getByRole('button', { name: /select audio/i });
|
||||
if (await selectAudioBtn.isVisible({ timeout: 30000 }).catch(() => false)) {
|
||||
console.log(' Opening audio selection dialog...');
|
||||
await selectAudioBtn.click();
|
||||
await page.waitForTimeout(2000);
|
||||
|
||||
const dialog = page.locator('[role="dialog"]');
|
||||
if (await dialog.isVisible({ timeout: 5000 })) {
|
||||
await page.screenshot({ path: 'test-results/05a-audio-dialog-loading.png' });
|
||||
|
||||
// Wait for the loading spinner to disappear
|
||||
// The AudioSearchDialog shows "Loading..." while fetching results
|
||||
const loadingIndicator = dialog.getByText(/loading/i);
|
||||
try {
|
||||
await expect(loadingIndicator).not.toBeVisible({ timeout: 60000 });
|
||||
console.log(' Audio results loaded');
|
||||
} catch {
|
||||
console.log(' WARNING: Loading indicator timeout');
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/05b-audio-dialog-loaded.png' });
|
||||
|
||||
// Check for "No audio sources found" message
|
||||
const noResultsMsg = dialog.getByText(/no audio sources found/i);
|
||||
if (await noResultsMsg.isVisible({ timeout: 2000 }).catch(() => false)) {
|
||||
console.log(' WARNING: No audio sources found - this may cause the test to fail');
|
||||
await page.screenshot({ path: 'test-results/05b-no-audio-sources.png' });
|
||||
}
|
||||
|
||||
// Find and click first Select button
|
||||
const selectButtons = dialog.getByRole('button', { name: /^select$/i });
|
||||
const count = await selectButtons.count();
|
||||
console.log(` Found ${count} audio options`);
|
||||
|
||||
if (count > 0) {
|
||||
await selectButtons.first().click();
|
||||
console.log(' Audio selected via UI');
|
||||
|
||||
// Wait for dialog to close
|
||||
await dialog.waitFor({ state: 'hidden', timeout: TIMEOUTS.audioSearch }).catch(() => {
|
||||
console.log(' Dialog still open - pressing Escape');
|
||||
return page.keyboard.press('Escape');
|
||||
});
|
||||
} else {
|
||||
console.log(' ERROR: No Select buttons found - closing dialog');
|
||||
await page.keyboard.press('Escape');
|
||||
// This is a critical error - the job won't progress without audio selection
|
||||
throw new Error('No audio sources available for selection. The job cannot proceed.');
|
||||
}
|
||||
}
|
||||
} else {
|
||||
console.log(' No audio selection needed - job auto-selected or cached');
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/05c-after-audio.png' });
|
||||
console.log('STEP 5 COMPLETE: Audio selection handled');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 6: Wait for Lyrics Processing
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 6: Wait for Lyrics Processing');
|
||||
console.log('========================================');
|
||||
|
||||
console.log(' Waiting for lyrics transcription (this may take 10-20 minutes)...');
|
||||
|
||||
// Poll the UI for status changes
|
||||
let foundReviewStatus = false;
|
||||
|
||||
const startTime = Date.now();
|
||||
let pollCount = 0;
|
||||
while (Date.now() - startTime < TIMEOUTS.lyricsProcessing) {
|
||||
pollCount++;
|
||||
const elapsedMin = Math.floor((Date.now() - startTime) / 60000);
|
||||
const elapsedSec = Math.floor(((Date.now() - startTime) % 60000) / 1000);
|
||||
|
||||
// Refresh the page/jobs
|
||||
const refreshBtn = page.getByRole('button', { name: /refresh/i });
|
||||
if (await refreshBtn.isVisible().catch(() => false)) {
|
||||
await refreshBtn.click();
|
||||
await page.waitForTimeout(2000);
|
||||
}
|
||||
|
||||
// Check job card status text
|
||||
const statusText = await jobCard.textContent() || '';
|
||||
|
||||
// Check if the "Review Lyrics" link is visible (most reliable indicator)
|
||||
const reviewLink = jobCard.getByRole('link', { name: /review lyrics/i });
|
||||
const reviewLinkVisible = await reviewLink.isVisible().catch(() => false);
|
||||
|
||||
console.log(` [${elapsedMin}m ${elapsedSec}s] Poll #${pollCount}: reviewLink=${reviewLinkVisible}, status="${statusText.substring(0, 80)}..."`);
|
||||
|
||||
// Primary check: Is the Review Lyrics link visible?
|
||||
if (reviewLinkVisible) {
|
||||
foundReviewStatus = true;
|
||||
console.log(' Job ready for lyrics review (found Review Lyrics link)');
|
||||
break;
|
||||
}
|
||||
|
||||
// Secondary check: Text-based status detection
|
||||
if (statusText.toLowerCase().includes('review lyrics') ||
|
||||
(statusText.toLowerCase().includes('action needed') && statusText.toLowerCase().includes('review'))) {
|
||||
foundReviewStatus = true;
|
||||
console.log(' Job ready for lyrics review (found status text)');
|
||||
break;
|
||||
}
|
||||
|
||||
// Check for failure
|
||||
if (statusText.toLowerCase().includes('failed')) {
|
||||
await page.screenshot({ path: 'test-results/06-job-failed.png' });
|
||||
throw new Error('Job failed during lyrics processing');
|
||||
}
|
||||
|
||||
await page.waitForTimeout(10000); // Check every 10 seconds
|
||||
}
|
||||
|
||||
if (!foundReviewStatus) {
|
||||
const elapsedMin = Math.floor((Date.now() - startTime) / 60000);
|
||||
console.log(` TIMEOUT after ${elapsedMin} minutes (${pollCount} polls)`);
|
||||
await page.screenshot({ path: 'test-results/06-timeout.png' });
|
||||
throw new Error(`Timeout waiting for lyrics review stage after ${elapsedMin} minutes`);
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/06-ready-for-review.png' });
|
||||
console.log('STEP 6 COMPLETE: Lyrics processing finished');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 7: Lyrics Review (via UI - REAL USER FLOW)
|
||||
// The minimal path: scroll to bottom -> click "Preview Video" -> wait for video -> click "Complete Review"
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 7: Lyrics Review (UI Interaction)');
|
||||
console.log('========================================');
|
||||
|
||||
// Click the Review Lyrics button/link
|
||||
const reviewLink = jobCard.getByRole('link', { name: /review lyrics/i });
|
||||
await expect(reviewLink).toBeVisible({ timeout: TIMEOUTS.action });
|
||||
|
||||
// Get the review URL and open in new page
|
||||
const reviewUrl = await reviewLink.getAttribute('href');
|
||||
console.log(` Review URL: ${reviewUrl}`);
|
||||
|
||||
// Open review UI in new page
|
||||
const reviewPage = await context.newPage();
|
||||
await reviewPage.goto(reviewUrl!, { waitUntil: 'networkidle' });
|
||||
await reviewPage.waitForTimeout(3000);
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/07a-lyrics-review-opened.png', fullPage: true });
|
||||
console.log(' Lyrics review UI opened');
|
||||
|
||||
// Wait for the page content to load
|
||||
await expect(reviewPage.locator('body')).not.toBeEmpty({ timeout: TIMEOUTS.action });
|
||||
|
||||
// Wait for the main content to render - give MUI time to initialize
|
||||
await reviewPage.waitForTimeout(2000);
|
||||
|
||||
// Scroll to bottom of the page to see the "Preview Video" button
|
||||
console.log(' Scrolling to bottom of page...');
|
||||
await reviewPage.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
|
||||
await reviewPage.waitForTimeout(1000);
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/07b-scrolled-to-bottom.png', fullPage: true });
|
||||
|
||||
// Click "Preview Video" button (at bottom of the lyrics review page)
|
||||
// This button has text "Preview Video" and an OndemandVideo icon
|
||||
const previewVideoBtn = reviewPage.getByRole('button', { name: /preview video/i });
|
||||
await expect(previewVideoBtn).toBeVisible({ timeout: TIMEOUTS.action });
|
||||
console.log(' Found "Preview Video" button');
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/07c-before-preview-click.png', fullPage: true });
|
||||
await previewVideoBtn.click();
|
||||
console.log(' Clicked "Preview Video" button');
|
||||
|
||||
// Wait for the modal dialog to appear
|
||||
// The modal has title "Preview Video (With Vocals)"
|
||||
const previewModal = reviewPage.getByRole('dialog');
|
||||
await expect(previewModal).toBeVisible({ timeout: TIMEOUTS.action });
|
||||
console.log(' Preview modal opened');
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/07d-preview-modal.png', fullPage: true });
|
||||
|
||||
// Wait for video to load in the modal
|
||||
// The PreviewVideoSection component first shows "Generating preview video..."
|
||||
// then renders the <video> element once the preview is ready
|
||||
console.log(' Waiting for preview video generation...');
|
||||
|
||||
// First wait for the loading indicator to disappear (or video to appear)
|
||||
// The loading state shows "Generating preview video..."
|
||||
const loadingText = reviewPage.getByText(/generating preview video/i);
|
||||
try {
|
||||
// Wait up to 2 minutes for preview generation (it can be slow)
|
||||
await expect(loadingText).not.toBeVisible({ timeout: 120000 });
|
||||
console.log(' Preview generation complete');
|
||||
} catch {
|
||||
console.log(' WARNING: Loading indicator timeout - checking for video anyway');
|
||||
}
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/07e-after-generation.png', fullPage: true });
|
||||
|
||||
// Now check for the video element or an error message
|
||||
const videoElement = reviewPage.locator('video');
|
||||
const errorAlert = reviewPage.locator('[role="alert"]');
|
||||
|
||||
// Check if there's an error
|
||||
if (await errorAlert.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
const errorText = await errorAlert.textContent();
|
||||
console.log(` WARNING: Preview error: ${errorText}`);
|
||||
await reviewPage.screenshot({ path: 'test-results/07e-preview-error.png', fullPage: true });
|
||||
// Continue anyway - we can still click Complete Review even if preview failed
|
||||
} else if (await videoElement.isVisible({ timeout: 10000 }).catch(() => false)) {
|
||||
console.log(' Video element visible in modal');
|
||||
// Give the video a moment to buffer/load
|
||||
await reviewPage.waitForTimeout(3000);
|
||||
} else {
|
||||
console.log(' WARNING: No video element found, but continuing anyway');
|
||||
}
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/07f-video-state.png', fullPage: true });
|
||||
|
||||
// Click "Complete Review" button in the modal
|
||||
// This button has variant="contained", color="success", and endIcon={<CloudUpload />}
|
||||
const completeReviewBtn = reviewPage.getByRole('button', { name: /complete review/i });
|
||||
await expect(completeReviewBtn).toBeVisible({ timeout: TIMEOUTS.action });
|
||||
console.log(' Found "Complete Review" button');
|
||||
|
||||
await completeReviewBtn.click();
|
||||
console.log(' Clicked "Complete Review" button');
|
||||
|
||||
// Wait for the submission to process
|
||||
// The handleSubmitToServer function calls apiClient.submitCorrections() then window.close()
|
||||
await reviewPage.waitForTimeout(5000);
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/07g-after-complete-review.png', fullPage: true });
|
||||
|
||||
// The review page may close automatically after submission (window.close())
|
||||
// If it's still open, close it manually
|
||||
if (!reviewPage.isClosed()) {
|
||||
await reviewPage.close();
|
||||
console.log(' Closed review UI');
|
||||
} else {
|
||||
console.log(' Review UI closed automatically after submission');
|
||||
}
|
||||
|
||||
// Refresh main app to see updated status
|
||||
await page.bringToFront();
|
||||
const refreshBtn = page.getByRole('button', { name: /refresh/i });
|
||||
if (await refreshBtn.isVisible().catch(() => false)) {
|
||||
await refreshBtn.click();
|
||||
await page.waitForTimeout(3000);
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/07h-after-review-refresh.png' });
|
||||
console.log('STEP 7 COMPLETE: Lyrics review submitted via UI');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 8: Wait for Video Rendering & Instrumental Selection (via UI - REAL USER FLOW)
|
||||
// The instrumental UI has: selection-option elements, submit button with id="submit-btn"
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 8: Wait for Video Rendering & Instrumental Selection');
|
||||
console.log('========================================');
|
||||
|
||||
// Wait for instrumental selection to become available
|
||||
console.log(' Waiting for video rendering and instrumental selection...');
|
||||
|
||||
let foundInstrumentalStatus = false;
|
||||
const renderStartTime = Date.now();
|
||||
|
||||
while (Date.now() - renderStartTime < TIMEOUTS.videoRendering) {
|
||||
// Refresh
|
||||
if (await refreshBtn.isVisible().catch(() => false)) {
|
||||
await refreshBtn.click();
|
||||
await page.waitForTimeout(2000);
|
||||
}
|
||||
|
||||
// Check job card status
|
||||
const statusText = await jobCard.textContent() || '';
|
||||
console.log(` Status: ${statusText.substring(0, 80)}...`);
|
||||
|
||||
if (statusText.toLowerCase().includes('select instrumental') ||
|
||||
statusText.toLowerCase().includes('awaiting_instrumental') ||
|
||||
(statusText.toLowerCase().includes('action needed') && statusText.toLowerCase().includes('instrumental'))) {
|
||||
foundInstrumentalStatus = true;
|
||||
console.log(' Job ready for instrumental selection');
|
||||
break;
|
||||
}
|
||||
|
||||
// Check if already complete (auto-mode might have run)
|
||||
if (statusText.toLowerCase().includes('complete') && !statusText.toLowerCase().includes('prep')) {
|
||||
console.log(' Job already complete - skipping instrumental selection');
|
||||
break;
|
||||
}
|
||||
|
||||
// Check if encoding is in progress (past instrumental stage)
|
||||
// NOTE: "rendering" status is BEFORE instrumental selection, not after!
|
||||
// The flow is: rendering_video -> awaiting_instrumental_selection -> encoding -> complete
|
||||
// So only skip if job is at encoding (not rendering)
|
||||
if (statusText.toLowerCase().includes('encoding') && !statusText.toLowerCase().includes('awaiting')) {
|
||||
console.log(' Job is encoding - instrumental selection was handled');
|
||||
break;
|
||||
}
|
||||
|
||||
if (statusText.toLowerCase().includes('failed')) {
|
||||
throw new Error('Job failed during video rendering');
|
||||
}
|
||||
|
||||
await page.waitForTimeout(10000);
|
||||
}
|
||||
|
||||
if (foundInstrumentalStatus) {
|
||||
// Click the Select Instrumental button/link
|
||||
const instrumentalLink = jobCard.getByRole('link', { name: /select instrumental/i });
|
||||
await expect(instrumentalLink).toBeVisible({ timeout: TIMEOUTS.action });
|
||||
|
||||
const instrumentalUrl = await instrumentalLink.getAttribute('href');
|
||||
console.log(` Instrumental URL: ${instrumentalUrl}`);
|
||||
|
||||
// Open instrumental UI in new page
|
||||
const instrumentalPage = await context.newPage();
|
||||
// Handle both absolute and relative URLs
|
||||
const fullUrl = instrumentalUrl?.startsWith('http') ? instrumentalUrl : `${PROD_URL}${instrumentalUrl}`;
|
||||
// Use 'load' instead of 'networkidle' - instrumental page has ongoing audio requests
|
||||
// that prevent networkidle from ever being reached
|
||||
await instrumentalPage.goto(fullUrl, { waitUntil: 'load' });
|
||||
await instrumentalPage.waitForTimeout(3000);
|
||||
|
||||
await instrumentalPage.screenshot({ path: 'test-results/08a-instrumental-opened.png', fullPage: true });
|
||||
console.log(' Instrumental selection UI opened');
|
||||
|
||||
// Wait for the page to fully load - the instrumental UI renders selection options
|
||||
await instrumentalPage.waitForSelector('.selection-option, .selection-panel', { timeout: 30000 });
|
||||
|
||||
// Select "Clean" instrumental option
|
||||
// The options have class "selection-option" and contain labels with "Clean" or "With Backing"
|
||||
const cleanOption = instrumentalPage.locator('.selection-option:has-text("Clean")').first();
|
||||
if (await cleanOption.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
await cleanOption.click();
|
||||
console.log(' Selected "Clean" instrumental option');
|
||||
} else {
|
||||
// If we can't find "Clean" specifically, the first option might already be selected
|
||||
console.log(' Clean option not found - using default selection');
|
||||
}
|
||||
|
||||
await instrumentalPage.screenshot({ path: 'test-results/08b-clean-selected.png', fullPage: true });
|
||||
|
||||
// Click the submit button: "✓ Confirm & Continue" (id="submit-btn")
|
||||
const submitBtn = instrumentalPage.locator('#submit-btn');
|
||||
await expect(submitBtn).toBeVisible({ timeout: TIMEOUTS.action });
|
||||
console.log(' Found submit button');
|
||||
|
||||
await submitBtn.click();
|
||||
console.log(' Clicked "Confirm & Continue" button');
|
||||
|
||||
// Wait for submission to complete - the page shows a success screen then auto-closes
|
||||
await instrumentalPage.waitForTimeout(5000);
|
||||
|
||||
await instrumentalPage.screenshot({ path: 'test-results/08c-instrumental-submitted.png', fullPage: true });
|
||||
|
||||
// The page may auto-close after submission
|
||||
if (!instrumentalPage.isClosed()) {
|
||||
await instrumentalPage.close();
|
||||
console.log(' Closed instrumental UI');
|
||||
} else {
|
||||
console.log(' Instrumental UI closed automatically after submission');
|
||||
}
|
||||
}
|
||||
|
||||
// Refresh main app
|
||||
await page.bringToFront();
|
||||
if (await refreshBtn.isVisible().catch(() => false)) {
|
||||
await refreshBtn.click();
|
||||
await page.waitForTimeout(3000);
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/08d-after-instrumental.png' });
|
||||
console.log('STEP 8 COMPLETE: Instrumental selection handled via UI');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 9: Wait for Final Encoding and Completion
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 9: Wait for Completion');
|
||||
console.log('========================================');
|
||||
|
||||
console.log(' Waiting for video rendering and encoding (this may take 15-25 minutes)...');
|
||||
|
||||
let isComplete = false;
|
||||
const encodeStartTime = Date.now();
|
||||
// After instrumental selection, job needs: video rendering (~15 min) + encoding (~10 min)
|
||||
// Add extra buffer for variable processing times
|
||||
const completionTimeout = TIMEOUTS.videoRendering + TIMEOUTS.finalEncoding + 600_000; // 35 minutes total
|
||||
|
||||
while (Date.now() - encodeStartTime < completionTimeout) {
|
||||
if (await refreshBtn.isVisible().catch(() => false)) {
|
||||
await refreshBtn.click();
|
||||
await page.waitForTimeout(2000);
|
||||
}
|
||||
|
||||
const statusText = await jobCard.textContent() || '';
|
||||
|
||||
if (statusText.toLowerCase().includes('complete') && !statusText.toLowerCase().includes('prep')) {
|
||||
isComplete = true;
|
||||
console.log(' Job complete!');
|
||||
break;
|
||||
}
|
||||
|
||||
if (statusText.toLowerCase().includes('failed')) {
|
||||
throw new Error('Job failed during final encoding');
|
||||
}
|
||||
|
||||
console.log(` Status: ${statusText.substring(0, 80)}...`);
|
||||
await page.waitForTimeout(10000);
|
||||
}
|
||||
|
||||
if (!isComplete) {
|
||||
throw new Error('Timeout waiting for job completion');
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/09-job-complete.png' });
|
||||
console.log('STEP 9 COMPLETE: Job finished');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 10: Verify Downloads
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 10: Verify Downloads');
|
||||
console.log('========================================');
|
||||
|
||||
// Look for download links in the job card
|
||||
const downloadLinks = jobCard.locator('a[href*="download"], a[href*="storage.googleapis.com"]');
|
||||
const downloadCount = await downloadLinks.count();
|
||||
console.log(` Found ${downloadCount} download links`);
|
||||
|
||||
if (downloadCount > 0) {
|
||||
// Click first download link to verify it works
|
||||
const firstDownloadUrl = await downloadLinks.first().getAttribute('href');
|
||||
console.log(` First download URL: ${firstDownloadUrl?.substring(0, 80)}...`);
|
||||
|
||||
// Verify the download URL is accessible
|
||||
// Use GET instead of HEAD since the download endpoint may not support HEAD
|
||||
// The response might redirect to GCS, so accept 2xx/3xx status codes
|
||||
const response = await page.request.get(firstDownloadUrl!, {
|
||||
maxRedirects: 0, // Don't follow redirects, just check the initial response
|
||||
});
|
||||
const status = response.status();
|
||||
// Accept 200 (success) or 302/307 (redirect to actual download)
|
||||
expect(status >= 200 && status < 400).toBe(true);
|
||||
console.log(` Download URL verified accessible (status: ${status})`);
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/10-downloads-verified.png' });
|
||||
console.log('STEP 10 COMPLETE: Downloads verified');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 11: Verify Distribution (if enabled)
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 11: Verify Distribution');
|
||||
console.log('========================================');
|
||||
|
||||
// Check for distribution indicators in job card or state
|
||||
const cardText = await jobCard.textContent() || '';
|
||||
|
||||
if (cardText.toLowerCase().includes('youtube') ||
|
||||
cardText.toLowerCase().includes('dropbox') ||
|
||||
cardText.toLowerCase().includes('drive')) {
|
||||
console.log(' Distribution indicators found in UI');
|
||||
} else {
|
||||
console.log(' No distribution indicators visible in UI (may be in backend state)');
|
||||
}
|
||||
|
||||
console.log('STEP 11 COMPLETE: Distribution check done');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 12: Cleanup
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 12: Cleanup');
|
||||
console.log('========================================');
|
||||
|
||||
const adminToken = process.env.E2E_ADMIN_TOKEN;
|
||||
|
||||
if (adminToken && jobId) {
|
||||
console.log(' Cleaning up distribution and job...');
|
||||
|
||||
try {
|
||||
const cleanupResponse = await page.request.post(
|
||||
`${API_URL}/api/jobs/${jobId}/cleanup-distribution`,
|
||||
{
|
||||
headers: {
|
||||
'Authorization': `Bearer ${adminToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
data: { delete_job: true },
|
||||
}
|
||||
);
|
||||
|
||||
if (cleanupResponse.ok()) {
|
||||
const result = await cleanupResponse.json();
|
||||
console.log(' Cleanup results:');
|
||||
console.log(` YouTube: ${result.youtube?.status}`);
|
||||
console.log(` Dropbox: ${result.dropbox?.status}`);
|
||||
console.log(` GDrive: ${result.gdrive?.status}`);
|
||||
console.log(` Job deleted: ${result.job_deleted}`);
|
||||
} else {
|
||||
console.log(` WARNING: Cleanup failed with status ${cleanupResponse.status()}`);
|
||||
}
|
||||
} catch (e) {
|
||||
console.log(` WARNING: Cleanup error: ${e}`);
|
||||
}
|
||||
} else {
|
||||
console.log(' Skipping cleanup - E2E_ADMIN_TOKEN not set or no job ID');
|
||||
console.log(` Job ID for manual cleanup: ${jobId}`);
|
||||
}
|
||||
|
||||
console.log('STEP 12 COMPLETE: Cleanup done');
|
||||
|
||||
// =========================================================================
|
||||
// SUCCESS
|
||||
// =========================================================================
|
||||
console.log('\n========================================');
|
||||
console.log('TEST COMPLETE: Full user journey successful!');
|
||||
console.log('========================================');
|
||||
console.log(`Job ID: ${jobId}`);
|
||||
console.log('All UI interactions completed successfully.');
|
||||
|
||||
} finally {
|
||||
// Always clean up the email inbox (if we created one)
|
||||
if (inboxId && emailHelper) {
|
||||
try {
|
||||
await emailHelper.deleteInbox(inboxId);
|
||||
console.log(' Cleaned up test email inbox');
|
||||
} catch (e) {
|
||||
console.log(` Warning: Failed to delete inbox: ${e}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,130 @@
|
||||
import { test, expect } from '@playwright/test';
|
||||
|
||||
/**
|
||||
* FOCUSED TEST: Instrumental Selection UI Only
|
||||
*
|
||||
* This test is for rapid iteration on the instrumental selection step.
|
||||
* It requires an existing job that's already in "awaiting_instrumental_selection" state.
|
||||
*
|
||||
* Usage:
|
||||
* E2E_JOB_ID=xxx E2E_INSTRUMENTAL_TOKEN=yyy npx playwright test instrumental-selection-only.spec.ts \
|
||||
* --config=playwright.production.config.ts --headed
|
||||
*/
|
||||
|
||||
const PROD_URL = 'https://gen.nomadkaraoke.com';
|
||||
const API_URL = 'https://api.nomadkaraoke.com';
|
||||
|
||||
function getJobId(): string {
|
||||
const jobId = process.env.E2E_JOB_ID;
|
||||
if (!jobId) {
|
||||
throw new Error('E2E_JOB_ID environment variable is required');
|
||||
}
|
||||
return jobId;
|
||||
}
|
||||
|
||||
function getInstrumentalToken(): string {
|
||||
const token = process.env.E2E_INSTRUMENTAL_TOKEN;
|
||||
if (!token) {
|
||||
throw new Error('E2E_INSTRUMENTAL_TOKEN environment variable is required');
|
||||
}
|
||||
return token;
|
||||
}
|
||||
|
||||
test.describe('Instrumental Selection Only - Focused Test', () => {
|
||||
test.describe.configure({ retries: 0 });
|
||||
|
||||
test('Complete instrumental selection via UI', async ({ page }) => {
|
||||
test.setTimeout(120_000); // 2 minutes
|
||||
|
||||
const jobId = getJobId();
|
||||
const token = getInstrumentalToken();
|
||||
|
||||
console.log('========================================');
|
||||
console.log('FOCUSED TEST: Instrumental Selection Only');
|
||||
console.log('========================================');
|
||||
console.log(`Job ID: ${jobId}`);
|
||||
console.log(`Instrumental Token: ${token.substring(0, 8)}...`);
|
||||
console.log('');
|
||||
|
||||
// Build the instrumental selection URL
|
||||
const baseApiUrl = encodeURIComponent(`${API_URL}/api/jobs/${jobId}`);
|
||||
const instrumentalUrl = `${PROD_URL}/instrumental/?baseApiUrl=${baseApiUrl}&instrumentalToken=${token}`;
|
||||
|
||||
console.log(`Instrumental URL: ${instrumentalUrl}`);
|
||||
console.log('');
|
||||
|
||||
// Navigate to the instrumental selection page
|
||||
console.log('1. Opening instrumental selection UI...');
|
||||
await page.goto(instrumentalUrl, { waitUntil: 'networkidle' });
|
||||
await page.waitForTimeout(3000);
|
||||
|
||||
await page.screenshot({ path: 'test-results/instrumental-01-opened.png', fullPage: true });
|
||||
console.log(' Instrumental selection UI opened');
|
||||
|
||||
// Wait for content to load
|
||||
await expect(page.locator('body')).not.toBeEmpty({ timeout: 30_000 });
|
||||
|
||||
// Log page content for debugging
|
||||
const bodyText = await page.locator('body').textContent();
|
||||
console.log(` Page content preview: ${bodyText?.substring(0, 300)}...`);
|
||||
|
||||
// Check for error messages
|
||||
const errorMessages = await page.locator('[class*="error"], [class*="Error"]').allTextContents();
|
||||
if (errorMessages.length > 0) {
|
||||
console.log(` WARNING: Error messages found: ${errorMessages.join(', ')}`);
|
||||
}
|
||||
|
||||
// Look for selection options
|
||||
console.log('2. Looking for instrumental options...');
|
||||
|
||||
// The instrumental selection page has radio buttons or selection cards
|
||||
const selectButtons = page.locator('button:has-text("Select"), input[type="submit"], input[type="radio"], .selection-option');
|
||||
const optionCount = await selectButtons.count();
|
||||
console.log(` Found ${optionCount} selection elements`);
|
||||
|
||||
await page.screenshot({ path: 'test-results/instrumental-02-options.png', fullPage: true });
|
||||
|
||||
if (optionCount === 0) {
|
||||
// Try looking for any clickable elements
|
||||
const allButtons = await page.getByRole('button').allTextContents();
|
||||
console.log(` All buttons on page: ${allButtons.join(', ')}`);
|
||||
|
||||
// Maybe it's a simple form with a submit button
|
||||
const submitBtn = page.locator('#submit-btn, input[type="submit"], button[type="submit"]');
|
||||
if (await submitBtn.isVisible().catch(() => false)) {
|
||||
console.log(' Found submit button - clicking...');
|
||||
await submitBtn.click();
|
||||
await page.waitForTimeout(5000);
|
||||
await page.screenshot({ path: 'test-results/instrumental-03-after-submit.png', fullPage: true });
|
||||
console.log(' Selection submitted!');
|
||||
} else {
|
||||
throw new Error('No instrumental selection options found');
|
||||
}
|
||||
} else {
|
||||
// Click the first option
|
||||
console.log('3. Selecting first instrumental option...');
|
||||
await selectButtons.first().click();
|
||||
await page.waitForTimeout(2000);
|
||||
|
||||
await page.screenshot({ path: 'test-results/instrumental-03-selected.png', fullPage: true });
|
||||
|
||||
// Look for a submit/confirm button
|
||||
const submitBtn = page.locator('#submit-btn, input[type="submit"], button[type="submit"], button:has-text("Confirm"), button:has-text("Submit")');
|
||||
if (await submitBtn.isVisible().catch(() => false)) {
|
||||
console.log('4. Clicking submit button...');
|
||||
await submitBtn.click();
|
||||
await page.waitForTimeout(5000);
|
||||
console.log(' Selection submitted!');
|
||||
} else {
|
||||
console.log(' No submit button found - selection may have been automatic');
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/instrumental-04-after-submit.png', fullPage: true });
|
||||
}
|
||||
|
||||
console.log('');
|
||||
console.log('========================================');
|
||||
console.log('FOCUSED TEST COMPLETE');
|
||||
console.log('========================================');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,231 @@
|
||||
import { test, expect } from '@playwright/test';
|
||||
|
||||
/**
|
||||
* FOCUSED TEST: Lyrics Review UI Only
|
||||
*
|
||||
* This test is for rapid iteration on the lyrics review step.
|
||||
* It requires an existing job that's already in the "in_review" state.
|
||||
*
|
||||
* Usage:
|
||||
* E2E_JOB_ID=xxx E2E_REVIEW_TOKEN=yyy npx playwright test lyrics-review-only.spec.ts \
|
||||
* --config=playwright.production.config.ts
|
||||
*
|
||||
* To get a job in the right state:
|
||||
* 1. Run the happy-path test and let it reach STEP 6 (or manually create a job)
|
||||
* 2. Get the job ID and review token from the logs/UI
|
||||
* 3. Run this focused test with those values
|
||||
*/
|
||||
|
||||
const PROD_URL = 'https://gen.nomadkaraoke.com';
|
||||
const API_URL = 'https://api.nomadkaraoke.com';
|
||||
|
||||
// Get test parameters from environment
|
||||
function getJobId(): string {
|
||||
const jobId = process.env.E2E_JOB_ID;
|
||||
if (!jobId) {
|
||||
throw new Error('E2E_JOB_ID environment variable is required');
|
||||
}
|
||||
return jobId;
|
||||
}
|
||||
|
||||
function getReviewToken(): string | null {
|
||||
return process.env.E2E_REVIEW_TOKEN || null;
|
||||
}
|
||||
|
||||
function getAccessToken(): string | null {
|
||||
return process.env.E2E_TEST_TOKEN || null;
|
||||
}
|
||||
|
||||
test.describe('Lyrics Review Only - Focused Test', () => {
|
||||
// Disable retries for this focused test
|
||||
test.describe.configure({ retries: 0 });
|
||||
|
||||
test('Complete lyrics review via UI', async ({ page, context }) => {
|
||||
test.setTimeout(900_000); // 15 minutes - preview generation can take up to 10 min
|
||||
|
||||
const jobId = getJobId();
|
||||
const reviewToken = getReviewToken();
|
||||
const accessToken = getAccessToken();
|
||||
|
||||
console.log('========================================');
|
||||
console.log('FOCUSED TEST: Lyrics Review Only');
|
||||
console.log('========================================');
|
||||
console.log(`Job ID: ${jobId}`);
|
||||
console.log(`Review Token: ${reviewToken ? reviewToken.substring(0, 8) + '...' : 'NOT SET'}`);
|
||||
console.log(`Access Token: ${accessToken ? accessToken.substring(0, 8) + '...' : 'NOT SET'}`);
|
||||
console.log('');
|
||||
|
||||
// Build the review URL
|
||||
let reviewUrl: string;
|
||||
if (reviewToken) {
|
||||
// Direct review URL with token
|
||||
reviewUrl = `${PROD_URL}/lyrics//?baseApiUrl=${encodeURIComponent(API_URL + '/api/review/' + jobId)}&reviewToken=${reviewToken}`;
|
||||
} else if (accessToken) {
|
||||
// Use the main app to get to the review
|
||||
reviewUrl = `${PROD_URL}/lyrics//?baseApiUrl=${encodeURIComponent(API_URL + '/api/review/' + jobId)}`;
|
||||
} else {
|
||||
throw new Error('Either E2E_REVIEW_TOKEN or E2E_TEST_TOKEN must be set');
|
||||
}
|
||||
|
||||
console.log(`Review URL: ${reviewUrl}`);
|
||||
console.log('');
|
||||
|
||||
// Navigate to the review page
|
||||
console.log('1. Opening lyrics review UI...');
|
||||
await page.goto(reviewUrl, { waitUntil: 'networkidle' });
|
||||
await page.waitForTimeout(3000);
|
||||
|
||||
await page.screenshot({ path: 'test-results/focused-01-review-opened.png', fullPage: true });
|
||||
console.log(' Review UI opened');
|
||||
|
||||
// Wait for the page content to load
|
||||
await expect(page.locator('body')).not.toBeEmpty({ timeout: 30_000 });
|
||||
|
||||
// Wait for MUI to initialize
|
||||
await page.waitForTimeout(2000);
|
||||
|
||||
// Log page content for debugging
|
||||
const bodyText = await page.locator('body').textContent();
|
||||
console.log(` Page content preview: ${bodyText?.substring(0, 200)}...`);
|
||||
|
||||
// Look for any error messages
|
||||
const errorMessages = await page.locator('[class*="error"], [class*="Error"], .MuiAlert-standardError').allTextContents();
|
||||
if (errorMessages.length > 0) {
|
||||
console.log(` WARNING: Error messages found: ${errorMessages.join(', ')}`);
|
||||
}
|
||||
|
||||
// Scroll to bottom to find the "Preview Video" button
|
||||
console.log('2. Scrolling to bottom of page...');
|
||||
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
|
||||
await page.waitForTimeout(1000);
|
||||
|
||||
await page.screenshot({ path: 'test-results/focused-02-scrolled.png', fullPage: true });
|
||||
|
||||
// List all visible buttons for debugging
|
||||
const allButtons = await page.getByRole('button').allTextContents();
|
||||
console.log(` Visible buttons: ${allButtons.join(', ')}`);
|
||||
|
||||
// Click "Preview Video" button
|
||||
console.log('3. Looking for "Preview Video" button...');
|
||||
const previewVideoBtn = page.getByRole('button', { name: /preview video/i });
|
||||
|
||||
if (await previewVideoBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
console.log(' Found "Preview Video" button - clicking...');
|
||||
await page.screenshot({ path: 'test-results/focused-03-before-preview.png', fullPage: true });
|
||||
await previewVideoBtn.click();
|
||||
console.log(' Clicked "Preview Video" button');
|
||||
} else {
|
||||
// Maybe the button has different text or is not visible yet
|
||||
console.log(' Preview Video button not found with that name');
|
||||
console.log(' Taking screenshot for debugging...');
|
||||
await page.screenshot({ path: 'test-results/focused-03-no-preview-btn.png', fullPage: true });
|
||||
|
||||
// Try alternative selectors
|
||||
const finishBtn = page.getByRole('button', { name: /finish|complete|submit|done/i });
|
||||
if (await finishBtn.isVisible().catch(() => false)) {
|
||||
const btnText = await finishBtn.textContent();
|
||||
console.log(` Found alternative button: "${btnText}"`);
|
||||
}
|
||||
|
||||
throw new Error('Could not find Preview Video button');
|
||||
}
|
||||
|
||||
// Wait for the modal dialog to appear
|
||||
console.log('4. Waiting for preview modal...');
|
||||
const previewModal = page.getByRole('dialog');
|
||||
await expect(previewModal).toBeVisible({ timeout: 30_000 });
|
||||
console.log(' Preview modal opened');
|
||||
|
||||
await page.screenshot({ path: 'test-results/focused-04-modal-opened.png', fullPage: true });
|
||||
|
||||
// List what's in the modal for debugging
|
||||
const modalContent = await previewModal.textContent();
|
||||
console.log(` Modal content preview: ${modalContent?.substring(0, 200)}...`);
|
||||
|
||||
// Wait for video preview to generate
|
||||
// The PreviewVideoSection shows "Generating preview video..." while loading
|
||||
console.log('5. Waiting for preview video generation...');
|
||||
const loadingText = page.getByText(/generating preview video/i);
|
||||
|
||||
if (await loadingText.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
console.log(' Preview is generating... waiting up to 10 minutes');
|
||||
try {
|
||||
await expect(loadingText).not.toBeVisible({ timeout: 600_000 });
|
||||
console.log(' Preview generation complete');
|
||||
} catch {
|
||||
console.log(' WARNING: Preview generation timed out after 10 minutes');
|
||||
}
|
||||
} else {
|
||||
console.log(' No loading indicator found - preview may already be ready or errored');
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/focused-05-after-generation.png', fullPage: true });
|
||||
|
||||
// Check for video element or error
|
||||
const videoElement = page.locator('video');
|
||||
const errorAlert = page.locator('[role="alert"]');
|
||||
|
||||
if (await errorAlert.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
const errorText = await errorAlert.textContent();
|
||||
console.log(` WARNING: Preview error: ${errorText}`);
|
||||
await page.screenshot({ path: 'test-results/focused-05-preview-error.png', fullPage: true });
|
||||
// Continue anyway
|
||||
} else if (await videoElement.isVisible({ timeout: 10000 }).catch(() => false)) {
|
||||
console.log(' Video element visible');
|
||||
// Check video state
|
||||
const videoState = await videoElement.first().evaluate((el) => {
|
||||
const video = el as HTMLVideoElement;
|
||||
return {
|
||||
src: video.src,
|
||||
readyState: video.readyState,
|
||||
networkState: video.networkState,
|
||||
error: video.error?.message,
|
||||
};
|
||||
});
|
||||
console.log(` Video state: ${JSON.stringify(videoState)}`);
|
||||
await page.waitForTimeout(3000);
|
||||
} else {
|
||||
console.log(' WARNING: No video element found, but continuing anyway');
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/focused-06-video-state.png', fullPage: true });
|
||||
|
||||
// Click "Complete Review" button in the modal
|
||||
console.log('6. Looking for "Complete Review" button...');
|
||||
|
||||
// List all buttons in the modal
|
||||
const modalButtons = await previewModal.getByRole('button').allTextContents();
|
||||
console.log(` Modal buttons: ${modalButtons.join(', ')}`);
|
||||
|
||||
const completeReviewBtn = page.getByRole('button', { name: /complete review/i });
|
||||
|
||||
if (await completeReviewBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
console.log(' Found "Complete Review" button - clicking...');
|
||||
await completeReviewBtn.click();
|
||||
console.log(' Clicked "Complete Review" button');
|
||||
} else {
|
||||
// Try to find any submit-like button
|
||||
const submitBtn = previewModal.getByRole('button', { name: /submit|confirm|done|save/i });
|
||||
if (await submitBtn.isVisible().catch(() => false)) {
|
||||
const btnText = await submitBtn.textContent();
|
||||
console.log(` Found alternative submit button: "${btnText}" - clicking...`);
|
||||
await submitBtn.click();
|
||||
} else {
|
||||
console.log(' Could not find Complete Review or similar button');
|
||||
await page.screenshot({ path: 'test-results/focused-07-no-complete-btn.png', fullPage: true });
|
||||
throw new Error('Could not find Complete Review button');
|
||||
}
|
||||
}
|
||||
|
||||
// Wait for the submission to process
|
||||
console.log('7. Waiting for submission to complete...');
|
||||
await page.waitForTimeout(5000);
|
||||
|
||||
await page.screenshot({ path: 'test-results/focused-08-after-submit.png', fullPage: true });
|
||||
|
||||
console.log('');
|
||||
console.log('========================================');
|
||||
console.log('FOCUSED TEST COMPLETE');
|
||||
console.log('========================================');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,401 @@
|
||||
import { test, expect } from '@playwright/test';
|
||||
|
||||
/**
|
||||
* TEST EXISTING JOB - Test specific stages against existing jobs
|
||||
*
|
||||
* This test allows you to validate specific stages of the E2E flow
|
||||
* against jobs that are already in the right state.
|
||||
*
|
||||
* Usage:
|
||||
* E2E_TEST_TOKEN=xxx E2E_JOB_ID=yyy E2E_TEST_STAGE=lyrics_review \
|
||||
* npx playwright test test-existing-job.spec.ts --config=playwright.production.config.ts
|
||||
*
|
||||
* Required:
|
||||
* - E2E_TEST_TOKEN: Your user access token
|
||||
* - E2E_JOB_ID: The job ID to test against
|
||||
* - E2E_TEST_STAGE: The stage to test (lyrics_review, instrumental_selection, or completion)
|
||||
*
|
||||
* Job states required for each stage:
|
||||
* - lyrics_review: Job must be in "in_review" state
|
||||
* - instrumental_selection: Job must be in "awaiting_instrumental_selection" state
|
||||
* - completion: Job must be in a late-stage processing state
|
||||
*/
|
||||
|
||||
const PROD_URL = 'https://gen.nomadkaraoke.com';
|
||||
|
||||
function getAccessToken(): string {
|
||||
const token = process.env.E2E_TEST_TOKEN;
|
||||
if (!token) {
|
||||
throw new Error('E2E_TEST_TOKEN environment variable is required');
|
||||
}
|
||||
return token;
|
||||
}
|
||||
|
||||
function getJobId(): string {
|
||||
const jobId = process.env.E2E_JOB_ID;
|
||||
if (!jobId) {
|
||||
throw new Error('E2E_JOB_ID environment variable is required');
|
||||
}
|
||||
return jobId;
|
||||
}
|
||||
|
||||
function getTestStage(): string {
|
||||
const stage = process.env.E2E_TEST_STAGE || 'lyrics_review';
|
||||
const validStages = ['lyrics_review', 'instrumental_selection', 'completion'];
|
||||
if (!validStages.includes(stage)) {
|
||||
throw new Error(`Invalid E2E_TEST_STAGE: ${stage}. Valid options: ${validStages.join(', ')}`);
|
||||
}
|
||||
return stage;
|
||||
}
|
||||
|
||||
test.describe('Test Existing Job - Focused Stage Testing', () => {
|
||||
test.describe.configure({ retries: 0 });
|
||||
|
||||
test('Test specific stage against existing job', async ({ page, context }) => {
|
||||
test.setTimeout(300_000); // 5 minutes
|
||||
|
||||
const accessToken = getAccessToken();
|
||||
const jobId = getJobId();
|
||||
const testStage = getTestStage();
|
||||
|
||||
console.log('========================================');
|
||||
console.log('TEST EXISTING JOB');
|
||||
console.log('========================================');
|
||||
console.log(`Job ID: ${jobId}`);
|
||||
console.log(`Test Stage: ${testStage}`);
|
||||
console.log(`Access Token: ${accessToken.substring(0, 8)}...`);
|
||||
console.log('');
|
||||
|
||||
// STEP 1: Set up authentication
|
||||
console.log('1. Setting up authentication...');
|
||||
await page.goto(PROD_URL, { waitUntil: 'networkidle' });
|
||||
|
||||
// Set the access token in localStorage
|
||||
await page.evaluate((token) => {
|
||||
localStorage.setItem('karaoke_access_token', token);
|
||||
}, accessToken);
|
||||
|
||||
console.log(' Access token set in localStorage');
|
||||
|
||||
// STEP 2: Navigate to app page (where jobs are shown) and find the job
|
||||
console.log('2. Navigating to app page...');
|
||||
await page.goto(`${PROD_URL}/app`, { waitUntil: 'networkidle' });
|
||||
await page.waitForTimeout(3000);
|
||||
|
||||
await page.screenshot({ path: 'test-results/existing-01-jobs-page.png', fullPage: true });
|
||||
|
||||
// Find the job card by ID
|
||||
console.log(`3. Looking for job ${jobId}...`);
|
||||
|
||||
// The job ID appears in the job card - look for it
|
||||
const jobCard = page.locator(`[data-job-id="${jobId}"], :has-text("${jobId}")`).first();
|
||||
|
||||
// If we can't find by ID directly, look for any job card and check
|
||||
let foundJobCard = await jobCard.isVisible({ timeout: 5000 }).catch(() => false);
|
||||
|
||||
if (!foundJobCard) {
|
||||
console.log(' Job card not found by ID, looking for job in list...');
|
||||
// Get all job cards and look for ours
|
||||
const allCards = page.locator('[class*="card"], [class*="Card"], [class*="job"]');
|
||||
const cardCount = await allCards.count();
|
||||
console.log(` Found ${cardCount} potential job cards`);
|
||||
|
||||
// Check if the job ID appears on the page
|
||||
const pageText = await page.textContent('body');
|
||||
if (pageText?.includes(jobId)) {
|
||||
console.log(` Job ID ${jobId} found in page content`);
|
||||
} else {
|
||||
console.log(` WARNING: Job ID ${jobId} not found on page`);
|
||||
await page.screenshot({ path: 'test-results/existing-02-job-not-found.png', fullPage: true });
|
||||
throw new Error(`Job ${jobId} not found on jobs page`);
|
||||
}
|
||||
}
|
||||
|
||||
// Log the current job status
|
||||
const statusText = await page.textContent('body') || '';
|
||||
console.log(` Page content preview: ${statusText.substring(0, 300)}...`);
|
||||
|
||||
// STEP 3: Execute the appropriate test stage
|
||||
if (testStage === 'lyrics_review') {
|
||||
await testLyricsReview(page, context, jobId);
|
||||
} else if (testStage === 'instrumental_selection') {
|
||||
await testInstrumentalSelection(page, context, jobId);
|
||||
} else if (testStage === 'completion') {
|
||||
await testCompletion(page, jobId);
|
||||
}
|
||||
|
||||
console.log('');
|
||||
console.log('========================================');
|
||||
console.log('TEST COMPLETE');
|
||||
console.log('========================================');
|
||||
});
|
||||
});
|
||||
|
||||
async function testLyricsReview(page: any, context: any, jobId: string) {
|
||||
console.log('\n========================================');
|
||||
console.log('TESTING: Lyrics Review Stage');
|
||||
console.log('========================================');
|
||||
|
||||
// Look for the Review Lyrics link
|
||||
console.log('4. Looking for "Review Lyrics" link...');
|
||||
const reviewLink = page.getByRole('link', { name: /review lyrics/i });
|
||||
|
||||
if (!(await reviewLink.isVisible({ timeout: 10000 }).catch(() => false))) {
|
||||
// Maybe we need to expand the job card or the job isn't in the right state
|
||||
console.log(' Review Lyrics link not visible');
|
||||
|
||||
// Check for Action needed indicator
|
||||
const actionNeeded = page.getByText(/action needed/i);
|
||||
if (await actionNeeded.isVisible().catch(() => false)) {
|
||||
console.log(' Found "Action needed" indicator');
|
||||
}
|
||||
|
||||
// List all links on the page for debugging
|
||||
const allLinks = await page.getByRole('link').allTextContents();
|
||||
console.log(` Available links: ${allLinks.slice(0, 20).join(', ')}`);
|
||||
|
||||
await page.screenshot({ path: 'test-results/existing-03-no-review-link.png', fullPage: true });
|
||||
throw new Error('Job does not appear to be in the lyrics review state');
|
||||
}
|
||||
|
||||
// Get the review URL and open in new page
|
||||
const reviewUrl = await reviewLink.getAttribute('href');
|
||||
console.log(` Review URL: ${reviewUrl}`);
|
||||
|
||||
// Open review UI in new page
|
||||
console.log('5. Opening lyrics review UI...');
|
||||
const reviewPage = await context.newPage();
|
||||
await reviewPage.goto(reviewUrl!, { waitUntil: 'networkidle' });
|
||||
await reviewPage.waitForTimeout(3000);
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/existing-04-review-opened.png', fullPage: true });
|
||||
console.log(' Lyrics review UI opened');
|
||||
|
||||
// Wait for content to load
|
||||
await expect(reviewPage.locator('body')).not.toBeEmpty({ timeout: 30000 });
|
||||
await reviewPage.waitForTimeout(2000);
|
||||
|
||||
// Log page content for debugging
|
||||
const bodyText = await reviewPage.locator('body').textContent();
|
||||
console.log(` Page content: ${bodyText?.substring(0, 200)}...`);
|
||||
|
||||
// Look for any error messages
|
||||
const errorMessages = await reviewPage.locator('[class*="error"], .MuiAlert-standardError').allTextContents();
|
||||
if (errorMessages.length > 0) {
|
||||
console.log(` WARNING: Error messages: ${errorMessages.join(', ')}`);
|
||||
}
|
||||
|
||||
// Scroll to bottom to find the "Preview Video" button
|
||||
console.log('6. Scrolling to bottom of page...');
|
||||
await reviewPage.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
|
||||
await reviewPage.waitForTimeout(1000);
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/existing-05-scrolled.png', fullPage: true });
|
||||
|
||||
// List all visible buttons
|
||||
const allButtons = await reviewPage.getByRole('button').allTextContents();
|
||||
console.log(` Visible buttons: ${allButtons.join(', ')}`);
|
||||
|
||||
// Click "Preview Video" button
|
||||
console.log('7. Clicking "Preview Video" button...');
|
||||
const previewVideoBtn = reviewPage.getByRole('button', { name: /preview video/i });
|
||||
|
||||
if (!(await previewVideoBtn.isVisible({ timeout: 5000 }).catch(() => false))) {
|
||||
await reviewPage.screenshot({ path: 'test-results/existing-06-no-preview-btn.png', fullPage: true });
|
||||
throw new Error('Could not find Preview Video button');
|
||||
}
|
||||
|
||||
await previewVideoBtn.click();
|
||||
console.log(' Clicked Preview Video button');
|
||||
|
||||
// Wait for the modal dialog to appear
|
||||
console.log('8. Waiting for preview modal...');
|
||||
const previewModal = reviewPage.getByRole('dialog');
|
||||
await expect(previewModal).toBeVisible({ timeout: 30000 });
|
||||
console.log(' Preview modal opened');
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/existing-07-modal-opened.png', fullPage: true });
|
||||
|
||||
// Wait for video preview to generate
|
||||
console.log('9. Waiting for preview video generation...');
|
||||
const loadingText = reviewPage.getByText(/generating preview video/i);
|
||||
|
||||
if (await loadingText.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
console.log(' Preview is generating... waiting up to 2 minutes');
|
||||
try {
|
||||
await expect(loadingText).not.toBeVisible({ timeout: 120000 });
|
||||
console.log(' Preview generation complete');
|
||||
} catch {
|
||||
console.log(' WARNING: Preview generation timed out');
|
||||
}
|
||||
} else {
|
||||
console.log(' No loading indicator found - preview may already be ready');
|
||||
}
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/existing-08-after-generation.png', fullPage: true });
|
||||
|
||||
// Check for video element or error
|
||||
const videoElement = reviewPage.locator('video');
|
||||
const errorAlert = reviewPage.locator('[role="alert"]');
|
||||
|
||||
if (await errorAlert.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
const errorText = await errorAlert.textContent();
|
||||
console.log(` WARNING: Preview error: ${errorText}`);
|
||||
} else if (await videoElement.isVisible({ timeout: 10000 }).catch(() => false)) {
|
||||
console.log(' Video element visible');
|
||||
const videoState = await videoElement.first().evaluate((el) => {
|
||||
const video = el as HTMLVideoElement;
|
||||
return {
|
||||
src: video.src?.substring(0, 50),
|
||||
readyState: video.readyState,
|
||||
duration: video.duration,
|
||||
};
|
||||
});
|
||||
console.log(` Video state: ${JSON.stringify(videoState)}`);
|
||||
} else {
|
||||
console.log(' WARNING: No video element found');
|
||||
}
|
||||
|
||||
// Click "Complete Review" button
|
||||
console.log('10. Clicking "Complete Review" button...');
|
||||
const modalButtons = await previewModal.getByRole('button').allTextContents();
|
||||
console.log(` Modal buttons: ${modalButtons.join(', ')}`);
|
||||
|
||||
const completeReviewBtn = reviewPage.getByRole('button', { name: /complete review/i });
|
||||
|
||||
if (await completeReviewBtn.isVisible({ timeout: 5000 }).catch(() => false)) {
|
||||
await completeReviewBtn.click();
|
||||
console.log(' Clicked Complete Review button');
|
||||
} else {
|
||||
await reviewPage.screenshot({ path: 'test-results/existing-09-no-complete-btn.png', fullPage: true });
|
||||
throw new Error('Could not find Complete Review button');
|
||||
}
|
||||
|
||||
// Wait for submission
|
||||
console.log('11. Waiting for submission...');
|
||||
await reviewPage.waitForTimeout(5000);
|
||||
|
||||
await reviewPage.screenshot({ path: 'test-results/existing-10-after-submit.png', fullPage: true });
|
||||
console.log(' Lyrics review completed successfully!');
|
||||
|
||||
await reviewPage.close();
|
||||
}
|
||||
|
||||
async function testInstrumentalSelection(page: any, context: any, jobId: string) {
|
||||
console.log('\n========================================');
|
||||
console.log('TESTING: Instrumental Selection Stage');
|
||||
console.log('========================================');
|
||||
|
||||
// Look for the Select Instrumental link
|
||||
console.log('4. Looking for "Select Instrumental" link...');
|
||||
const selectLink = page.getByRole('link', { name: /select instrumental/i });
|
||||
|
||||
if (!(await selectLink.isVisible({ timeout: 10000 }).catch(() => false))) {
|
||||
// List all links for debugging
|
||||
const allLinks = await page.getByRole('link').allTextContents();
|
||||
console.log(` Available links: ${allLinks.slice(0, 20).join(', ')}`);
|
||||
|
||||
await page.screenshot({ path: 'test-results/existing-03-no-select-link.png', fullPage: true });
|
||||
throw new Error('Job does not appear to be in the instrumental selection state');
|
||||
}
|
||||
|
||||
// Get the selection URL and open in new page
|
||||
const selectUrl = await selectLink.getAttribute('href');
|
||||
console.log(` Selection URL: ${selectUrl}`);
|
||||
|
||||
// Open selection UI in new page
|
||||
console.log('5. Opening instrumental selection UI...');
|
||||
const selectPage = await context.newPage();
|
||||
await selectPage.goto(selectUrl!, { waitUntil: 'networkidle' });
|
||||
await selectPage.waitForTimeout(3000);
|
||||
|
||||
await selectPage.screenshot({ path: 'test-results/existing-04-selection-opened.png', fullPage: true });
|
||||
console.log(' Instrumental selection UI opened');
|
||||
|
||||
// Wait for content
|
||||
await expect(selectPage.locator('body')).not.toBeEmpty({ timeout: 30000 });
|
||||
|
||||
// Log page content
|
||||
const bodyText = await selectPage.locator('body').textContent();
|
||||
console.log(` Page content: ${bodyText?.substring(0, 200)}...`);
|
||||
|
||||
// Find and click a Select button (the first one should be fine)
|
||||
console.log('6. Looking for instrumental options...');
|
||||
const selectButtons = selectPage.locator('button:has-text("Select"), input[type="submit"][value*="Select"]');
|
||||
const buttonCount = await selectButtons.count();
|
||||
console.log(` Found ${buttonCount} Select buttons`);
|
||||
|
||||
if (buttonCount === 0) {
|
||||
await selectPage.screenshot({ path: 'test-results/existing-05-no-select-buttons.png', fullPage: true });
|
||||
throw new Error('No instrumental selection buttons found');
|
||||
}
|
||||
|
||||
// Click the first one
|
||||
console.log('7. Clicking first instrumental option...');
|
||||
await selectButtons.first().click();
|
||||
console.log(' Clicked Select button');
|
||||
|
||||
// Wait for the selection to process
|
||||
console.log('8. Waiting for selection to complete...');
|
||||
await selectPage.waitForTimeout(5000);
|
||||
|
||||
await selectPage.screenshot({ path: 'test-results/existing-06-after-selection.png', fullPage: true });
|
||||
|
||||
// Check for success message or redirect
|
||||
const currentUrl = selectPage.url();
|
||||
console.log(` Current URL: ${currentUrl}`);
|
||||
|
||||
console.log(' Instrumental selection completed successfully!');
|
||||
await selectPage.close();
|
||||
}
|
||||
|
||||
async function testCompletion(page: any, jobId: string) {
|
||||
console.log('\n========================================');
|
||||
console.log('TESTING: Completion Stage');
|
||||
console.log('========================================');
|
||||
|
||||
// This stage just monitors the job until completion
|
||||
console.log('4. Monitoring job for completion...');
|
||||
|
||||
const maxWaitTime = 30 * 60 * 1000; // 30 minutes
|
||||
const startTime = Date.now();
|
||||
let completed = false;
|
||||
|
||||
while (Date.now() - startTime < maxWaitTime) {
|
||||
// Refresh the page
|
||||
const refreshBtn = page.getByRole('button', { name: /refresh/i });
|
||||
if (await refreshBtn.isVisible().catch(() => false)) {
|
||||
await refreshBtn.click();
|
||||
await page.waitForTimeout(2000);
|
||||
} else {
|
||||
await page.reload({ waitUntil: 'networkidle' });
|
||||
await page.waitForTimeout(2000);
|
||||
}
|
||||
|
||||
// Check job status
|
||||
const statusText = await page.textContent('body') || '';
|
||||
console.log(` Status: ${statusText.substring(0, 100)}...`);
|
||||
|
||||
// Check for completion indicators
|
||||
if (statusText.toLowerCase().includes('complete') &&
|
||||
!statusText.toLowerCase().includes('processing')) {
|
||||
completed = true;
|
||||
console.log(' Job completed!');
|
||||
break;
|
||||
}
|
||||
|
||||
// Check for failure
|
||||
if (statusText.toLowerCase().includes('failed')) {
|
||||
throw new Error('Job failed during processing');
|
||||
}
|
||||
|
||||
await page.waitForTimeout(30000); // Check every 30 seconds
|
||||
}
|
||||
|
||||
if (!completed) {
|
||||
throw new Error('Job did not complete within timeout');
|
||||
}
|
||||
|
||||
await page.screenshot({ path: 'test-results/existing-completion.png', fullPage: true });
|
||||
console.log(' Completion test passed!');
|
||||
}
|
||||
@@ -1,171 +0,0 @@
|
||||
/**
|
||||
* Tests for auto-mode state management
|
||||
*/
|
||||
|
||||
import { renderHook, act } from "@testing-library/react"
|
||||
import { useAutoMode, getAutoModeFromUrl } from "../auto-mode"
|
||||
|
||||
// Mock localStorage
|
||||
const localStorageMock = (() => {
|
||||
let store: Record<string, string> = {}
|
||||
return {
|
||||
getItem: jest.fn((key: string) => store[key] || null),
|
||||
setItem: jest.fn((key: string, value: string) => {
|
||||
store[key] = value
|
||||
}),
|
||||
removeItem: jest.fn((key: string) => {
|
||||
delete store[key]
|
||||
}),
|
||||
clear: jest.fn(() => {
|
||||
store = {}
|
||||
}),
|
||||
}
|
||||
})()
|
||||
|
||||
Object.defineProperty(window, "localStorage", { value: localStorageMock })
|
||||
|
||||
describe("Auto Mode", () => {
|
||||
beforeEach(() => {
|
||||
localStorageMock.clear()
|
||||
jest.clearAllMocks()
|
||||
})
|
||||
|
||||
describe("useAutoMode hook", () => {
|
||||
it("should initialize with enabled=false", () => {
|
||||
const { result } = renderHook(() => useAutoMode())
|
||||
expect(result.current.enabled).toBe(false)
|
||||
})
|
||||
|
||||
it("should enable auto-mode", () => {
|
||||
const { result } = renderHook(() => useAutoMode())
|
||||
|
||||
act(() => {
|
||||
result.current.enable()
|
||||
})
|
||||
|
||||
expect(result.current.enabled).toBe(true)
|
||||
})
|
||||
|
||||
it("should disable auto-mode", () => {
|
||||
const { result } = renderHook(() => useAutoMode())
|
||||
|
||||
act(() => {
|
||||
result.current.enable()
|
||||
result.current.disable()
|
||||
})
|
||||
|
||||
expect(result.current.enabled).toBe(false)
|
||||
})
|
||||
|
||||
it("should toggle auto-mode", () => {
|
||||
const { result } = renderHook(() => useAutoMode())
|
||||
|
||||
act(() => {
|
||||
result.current.toggle()
|
||||
})
|
||||
expect(result.current.enabled).toBe(true)
|
||||
|
||||
act(() => {
|
||||
result.current.toggle()
|
||||
})
|
||||
expect(result.current.enabled).toBe(false)
|
||||
})
|
||||
|
||||
it("should set enabled directly", () => {
|
||||
const { result } = renderHook(() => useAutoMode())
|
||||
|
||||
act(() => {
|
||||
result.current.setEnabled(true)
|
||||
})
|
||||
expect(result.current.enabled).toBe(true)
|
||||
|
||||
act(() => {
|
||||
result.current.setEnabled(false)
|
||||
})
|
||||
expect(result.current.enabled).toBe(false)
|
||||
})
|
||||
|
||||
it("should track processing jobs", () => {
|
||||
const { result } = renderHook(() => useAutoMode())
|
||||
|
||||
expect(result.current.isProcessing("job-123-review")).toBe(false)
|
||||
|
||||
act(() => {
|
||||
result.current.markProcessing("job-123-review")
|
||||
})
|
||||
|
||||
expect(result.current.isProcessing("job-123-review")).toBe(true)
|
||||
expect(result.current.isProcessing("job-456-review")).toBe(false)
|
||||
|
||||
act(() => {
|
||||
result.current.unmarkProcessing("job-123-review")
|
||||
})
|
||||
|
||||
expect(result.current.isProcessing("job-123-review")).toBe(false)
|
||||
})
|
||||
|
||||
it("should handle multiple processing jobs", () => {
|
||||
const { result } = renderHook(() => useAutoMode())
|
||||
|
||||
act(() => {
|
||||
result.current.markProcessing("job-1-review")
|
||||
result.current.markProcessing("job-2-instrumental")
|
||||
})
|
||||
|
||||
expect(result.current.isProcessing("job-1-review")).toBe(true)
|
||||
expect(result.current.isProcessing("job-2-instrumental")).toBe(true)
|
||||
|
||||
act(() => {
|
||||
result.current.unmarkProcessing("job-1-review")
|
||||
})
|
||||
|
||||
expect(result.current.isProcessing("job-1-review")).toBe(false)
|
||||
expect(result.current.isProcessing("job-2-instrumental")).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe("getAutoModeFromUrl", () => {
|
||||
const originalLocation = window.location
|
||||
|
||||
beforeEach(() => {
|
||||
// @ts-ignore - mocking location
|
||||
delete window.location
|
||||
// @ts-ignore - mocking location
|
||||
window.location = { search: "" }
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
window.location = originalLocation
|
||||
})
|
||||
|
||||
it("should return false when no auto parameter", () => {
|
||||
// @ts-ignore
|
||||
window.location.search = ""
|
||||
expect(getAutoModeFromUrl()).toBe(false)
|
||||
})
|
||||
|
||||
it('should return true when auto=true', () => {
|
||||
// @ts-ignore
|
||||
window.location.search = "?auto=true"
|
||||
expect(getAutoModeFromUrl()).toBe(true)
|
||||
})
|
||||
|
||||
it('should return true when auto=1', () => {
|
||||
// @ts-ignore
|
||||
window.location.search = "?auto=1"
|
||||
expect(getAutoModeFromUrl()).toBe(true)
|
||||
})
|
||||
|
||||
it('should return false when auto=false', () => {
|
||||
// @ts-ignore
|
||||
window.location.search = "?auto=false"
|
||||
expect(getAutoModeFromUrl()).toBe(false)
|
||||
})
|
||||
|
||||
it("should handle multiple query params", () => {
|
||||
// @ts-ignore
|
||||
window.location.search = "?foo=bar&auto=true&baz=qux"
|
||||
expect(getAutoModeFromUrl()).toBe(true)
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -65,6 +65,8 @@ export interface Job {
|
||||
review_token?: string;
|
||||
instrumental_token?: string;
|
||||
audio_hash?: string;
|
||||
non_interactive?: boolean;
|
||||
user_email?: string;
|
||||
}
|
||||
|
||||
export interface UploadJobResponse {
|
||||
@@ -225,6 +227,7 @@ export const api = {
|
||||
enable_youtube_upload?: boolean;
|
||||
theme_id?: string;
|
||||
color_overrides?: ColorOverrides;
|
||||
non_interactive?: boolean;
|
||||
}
|
||||
): Promise<UploadJobResponse> {
|
||||
const formData = new FormData();
|
||||
@@ -250,6 +253,9 @@ export const api = {
|
||||
if (options?.color_overrides) {
|
||||
formData.append('color_overrides', JSON.stringify(options.color_overrides));
|
||||
}
|
||||
if (options?.non_interactive !== undefined) {
|
||||
formData.append('non_interactive', String(options.non_interactive));
|
||||
}
|
||||
|
||||
const response = await fetch(`${API_BASE_URL}/api/jobs/upload`, {
|
||||
method: 'POST',
|
||||
@@ -274,6 +280,7 @@ export const api = {
|
||||
enable_youtube_upload?: boolean;
|
||||
theme_id?: string;
|
||||
color_overrides?: ColorOverrides;
|
||||
non_interactive?: boolean;
|
||||
}
|
||||
): Promise<{ status: string; job_id: string; message: string }> {
|
||||
const body: Record<string, any> = { url };
|
||||
@@ -285,6 +292,7 @@ export const api = {
|
||||
if (options?.enable_youtube_upload !== undefined) body.enable_youtube_upload = options.enable_youtube_upload;
|
||||
if (options?.theme_id) body.theme_id = options.theme_id;
|
||||
if (options?.color_overrides) body.color_overrides = options.color_overrides;
|
||||
if (options?.non_interactive !== undefined) body.non_interactive = options.non_interactive;
|
||||
|
||||
const response = await fetch(`${API_BASE_URL}/api/jobs/create-from-url`, {
|
||||
method: 'POST',
|
||||
@@ -409,6 +417,7 @@ export const api = {
|
||||
enable_youtube_upload?: boolean;
|
||||
theme_id?: string;
|
||||
color_overrides?: ColorOverrides;
|
||||
non_interactive?: boolean;
|
||||
}
|
||||
): Promise<AudioSearchResponse> {
|
||||
const body: Record<string, any> = { artist, title, auto_download: autoDownload };
|
||||
@@ -418,6 +427,7 @@ export const api = {
|
||||
if (options?.enable_youtube_upload !== undefined) body.enable_youtube_upload = options.enable_youtube_upload;
|
||||
if (options?.theme_id) body.theme_id = options.theme_id;
|
||||
if (options?.color_overrides) body.color_overrides = options.color_overrides;
|
||||
if (options?.non_interactive !== undefined) body.non_interactive = options.non_interactive;
|
||||
|
||||
const response = await fetch(`${API_BASE_URL}/api/audio-search/search`, {
|
||||
method: 'POST',
|
||||
@@ -660,5 +670,270 @@ export interface BetaEnrollResponse {
|
||||
session_token: string | null;
|
||||
}
|
||||
|
||||
// ==========================================================================
|
||||
// Admin Types
|
||||
// ==========================================================================
|
||||
|
||||
export interface AdminStatsOverview {
|
||||
total_users: number;
|
||||
active_users_7d: number;
|
||||
active_users_30d: number;
|
||||
total_jobs: number;
|
||||
jobs_last_7d: number;
|
||||
jobs_last_30d: number;
|
||||
jobs_by_status: {
|
||||
pending: number;
|
||||
processing: number;
|
||||
awaiting_review: number;
|
||||
awaiting_instrumental: number;
|
||||
complete: number;
|
||||
failed: number;
|
||||
cancelled: number;
|
||||
};
|
||||
total_credits_issued_30d: number;
|
||||
total_beta_testers: number;
|
||||
}
|
||||
|
||||
export interface AdminUser {
|
||||
email: string;
|
||||
role: 'user' | 'admin';
|
||||
credits: number;
|
||||
display_name?: string;
|
||||
total_jobs_created?: number;
|
||||
total_jobs_completed?: number;
|
||||
}
|
||||
|
||||
export interface AdminUserListResponse {
|
||||
users: AdminUser[];
|
||||
total: number;
|
||||
offset: number;
|
||||
limit: number;
|
||||
has_more: boolean;
|
||||
}
|
||||
|
||||
export interface AdminUserDetail {
|
||||
email: string;
|
||||
role: 'user' | 'admin';
|
||||
credits: number;
|
||||
display_name?: string;
|
||||
is_active: boolean;
|
||||
email_verified: boolean;
|
||||
created_at?: string;
|
||||
updated_at?: string;
|
||||
last_login_at?: string;
|
||||
total_jobs_created: number;
|
||||
total_jobs_completed: number;
|
||||
is_beta_tester: boolean;
|
||||
beta_tester_status?: string;
|
||||
credit_transactions: Array<{
|
||||
id: string;
|
||||
amount: number;
|
||||
reason: string;
|
||||
created_at: string;
|
||||
job_id?: string;
|
||||
created_by?: string;
|
||||
}>;
|
||||
recent_jobs: Array<{
|
||||
job_id: string;
|
||||
status: string;
|
||||
artist?: string;
|
||||
title?: string;
|
||||
created_at?: string;
|
||||
}>;
|
||||
active_sessions_count: number;
|
||||
}
|
||||
|
||||
export interface AdminBetaStats {
|
||||
total_beta_testers: number;
|
||||
active_testers: number;
|
||||
pending_feedback: number;
|
||||
completed_feedback: number;
|
||||
total_feedback_submissions: number;
|
||||
average_ratings: {
|
||||
overall: number;
|
||||
ease_of_use: number;
|
||||
lyrics_accuracy: number;
|
||||
correction_experience: number;
|
||||
};
|
||||
}
|
||||
|
||||
export interface AdminBetaFeedback {
|
||||
id: string;
|
||||
user_email: string;
|
||||
job_id?: string;
|
||||
overall_rating: number;
|
||||
ease_of_use_rating?: number;
|
||||
lyrics_accuracy_rating?: number;
|
||||
correction_experience_rating?: number;
|
||||
what_went_well?: string;
|
||||
what_could_improve?: string;
|
||||
additional_comments?: string;
|
||||
would_recommend?: boolean;
|
||||
would_use_again?: boolean;
|
||||
created_at: string;
|
||||
}
|
||||
|
||||
export interface AdminJobListParams {
|
||||
status?: string;
|
||||
user_email?: string;
|
||||
environment?: string;
|
||||
created_after?: string;
|
||||
created_before?: string;
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
// Admin API namespace
|
||||
export const adminApi = {
|
||||
/**
|
||||
* Get admin dashboard statistics
|
||||
*/
|
||||
async getStats(): Promise<AdminStatsOverview> {
|
||||
const response = await fetch(`${API_BASE_URL}/api/admin/stats/overview`, {
|
||||
headers: getAuthHeaders()
|
||||
});
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* List all users with pagination and search
|
||||
*/
|
||||
async listUsers(params?: {
|
||||
limit?: number;
|
||||
offset?: number;
|
||||
search?: string;
|
||||
sort_by?: 'created_at' | 'last_login_at' | 'credits' | 'email';
|
||||
sort_order?: 'asc' | 'desc';
|
||||
include_inactive?: boolean;
|
||||
}): Promise<AdminUserListResponse> {
|
||||
const searchParams = new URLSearchParams();
|
||||
if (params?.limit) searchParams.set('limit', String(params.limit));
|
||||
if (params?.offset) searchParams.set('offset', String(params.offset));
|
||||
if (params?.search) searchParams.set('search', params.search);
|
||||
if (params?.sort_by) searchParams.set('sort_by', params.sort_by);
|
||||
if (params?.sort_order) searchParams.set('sort_order', params.sort_order);
|
||||
if (params?.include_inactive) searchParams.set('include_inactive', 'true');
|
||||
|
||||
const url = `${API_BASE_URL}/api/users/admin/users${searchParams.toString() ? '?' + searchParams.toString() : ''}`;
|
||||
const response = await fetch(url, {
|
||||
headers: getAuthHeaders()
|
||||
});
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Get detailed user information
|
||||
*/
|
||||
async getUserDetail(email: string): Promise<AdminUserDetail> {
|
||||
const response = await fetch(`${API_BASE_URL}/api/users/admin/users/${encodeURIComponent(email)}/detail`, {
|
||||
headers: getAuthHeaders()
|
||||
});
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Add credits to a user
|
||||
*/
|
||||
async addCredits(email: string, amount: number, reason: string): Promise<{
|
||||
status: string;
|
||||
email: string;
|
||||
credits_added: number;
|
||||
new_balance: number;
|
||||
message: string;
|
||||
}> {
|
||||
const response = await fetch(`${API_BASE_URL}/api/users/admin/credits`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
...getAuthHeaders()
|
||||
},
|
||||
body: JSON.stringify({ email, amount, reason }),
|
||||
});
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Set user role
|
||||
*/
|
||||
async setUserRole(email: string, role: 'user' | 'admin'): Promise<{ status: string; message: string }> {
|
||||
const response = await fetch(`${API_BASE_URL}/api/users/admin/users/${encodeURIComponent(email)}/role?role=${role}`, {
|
||||
method: 'POST',
|
||||
headers: getAuthHeaders()
|
||||
});
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Enable a user account
|
||||
*/
|
||||
async enableUser(email: string): Promise<{ status: string; message: string }> {
|
||||
const response = await fetch(`${API_BASE_URL}/api/users/admin/users/${encodeURIComponent(email)}/enable`, {
|
||||
method: 'POST',
|
||||
headers: getAuthHeaders()
|
||||
});
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Disable a user account
|
||||
*/
|
||||
async disableUser(email: string): Promise<{ status: string; message: string }> {
|
||||
const response = await fetch(`${API_BASE_URL}/api/users/admin/users/${encodeURIComponent(email)}/disable`, {
|
||||
method: 'POST',
|
||||
headers: getAuthHeaders()
|
||||
});
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* List all jobs (admin view)
|
||||
*/
|
||||
async listAllJobs(params?: AdminJobListParams): Promise<Job[]> {
|
||||
const searchParams = new URLSearchParams();
|
||||
if (params?.status) searchParams.set('status', params.status);
|
||||
if (params?.user_email) searchParams.set('user_email', params.user_email);
|
||||
if (params?.environment) searchParams.set('environment', params.environment);
|
||||
if (params?.created_after) searchParams.set('created_after', params.created_after);
|
||||
if (params?.created_before) searchParams.set('created_before', params.created_before);
|
||||
if (params?.limit) searchParams.set('limit', String(params.limit));
|
||||
|
||||
const url = `${API_BASE_URL}/api/jobs${searchParams.toString() ? '?' + searchParams.toString() : ''}`;
|
||||
const response = await fetch(url, {
|
||||
headers: getAuthHeaders()
|
||||
});
|
||||
return handleResponse<Job[]>(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Get beta program statistics
|
||||
*/
|
||||
async getBetaStats(): Promise<AdminBetaStats> {
|
||||
const response = await fetch(`${API_BASE_URL}/api/users/admin/beta/stats`, {
|
||||
headers: getAuthHeaders()
|
||||
});
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Get beta program feedback list
|
||||
*/
|
||||
async getBetaFeedback(limit: number = 50): Promise<{ feedback: AdminBetaFeedback[]; total: number }> {
|
||||
const response = await fetch(`${API_BASE_URL}/api/users/admin/beta/feedback?limit=${limit}`, {
|
||||
headers: getAuthHeaders()
|
||||
});
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Delete a job (admin)
|
||||
*/
|
||||
async deleteJob(jobId: string, deleteFiles: boolean = true): Promise<{ status: string; message: string }> {
|
||||
const response = await fetch(
|
||||
`${API_BASE_URL}/api/jobs/${jobId}?delete_files=${deleteFiles}`,
|
||||
{ method: 'DELETE', headers: getAuthHeaders() }
|
||||
);
|
||||
return handleResponse(response);
|
||||
},
|
||||
};
|
||||
|
||||
export { ApiError };
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
/**
|
||||
* Auto-mode state management for non-interactive processing.
|
||||
*
|
||||
* When auto-mode is enabled (via ?auto=true URL parameter or UI toggle),
|
||||
* the frontend automatically:
|
||||
* - Completes lyrics review without human intervention
|
||||
* - Selects "clean" instrumental automatically
|
||||
* - Selects the first/best audio search result automatically
|
||||
*
|
||||
* This is equivalent to the -y flag in karaoke-gen-remote CLI.
|
||||
*/
|
||||
|
||||
import { create } from 'zustand'
|
||||
import { persist } from 'zustand/middleware'
|
||||
|
||||
interface AutoModeState {
|
||||
/** Whether auto-mode is enabled */
|
||||
enabled: boolean
|
||||
/** Jobs currently being auto-processed (to prevent duplicate processing) */
|
||||
processingJobs: Set<string>
|
||||
/** Enable auto-mode */
|
||||
enable: () => void
|
||||
/** Disable auto-mode */
|
||||
disable: () => void
|
||||
/** Toggle auto-mode */
|
||||
toggle: () => void
|
||||
/** Set auto-mode state directly */
|
||||
setEnabled: (enabled: boolean) => void
|
||||
/** Mark a job as being processed */
|
||||
markProcessing: (jobId: string) => void
|
||||
/** Unmark a job as being processed */
|
||||
unmarkProcessing: (jobId: string) => void
|
||||
/** Check if a job is currently being processed */
|
||||
isProcessing: (jobId: string) => boolean
|
||||
}
|
||||
|
||||
export const useAutoMode = create<AutoModeState>()(
|
||||
persist(
|
||||
(set, get) => ({
|
||||
enabled: false,
|
||||
processingJobs: new Set<string>(),
|
||||
|
||||
enable: () => set({ enabled: true }),
|
||||
disable: () => set({ enabled: false }),
|
||||
toggle: () => set((state) => ({ enabled: !state.enabled })),
|
||||
setEnabled: (enabled: boolean) => set({ enabled }),
|
||||
|
||||
markProcessing: (jobId: string) => set((state) => {
|
||||
const newSet = new Set(state.processingJobs)
|
||||
newSet.add(jobId)
|
||||
return { processingJobs: newSet }
|
||||
}),
|
||||
|
||||
unmarkProcessing: (jobId: string) => set((state) => {
|
||||
const newSet = new Set(state.processingJobs)
|
||||
newSet.delete(jobId)
|
||||
return { processingJobs: newSet }
|
||||
}),
|
||||
|
||||
isProcessing: (jobId: string) => get().processingJobs.has(jobId),
|
||||
}),
|
||||
{
|
||||
name: 'karaoke-auto-mode',
|
||||
// Only persist the enabled state, not the processing jobs
|
||||
partialize: (state) => ({ enabled: state.enabled }),
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
/**
|
||||
* Check URL for auto-mode parameter.
|
||||
* Returns true if ?auto=true or ?auto=1 is in the URL.
|
||||
*/
|
||||
export function getAutoModeFromUrl(): boolean {
|
||||
if (typeof window === 'undefined') return false
|
||||
|
||||
const params = new URLSearchParams(window.location.search)
|
||||
const autoParam = params.get('auto')
|
||||
return autoParam === 'true' || autoParam === '1'
|
||||
}
|
||||
Generated
+2
-69
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "karaoke-gen-frontend",
|
||||
"version": "0.77.0",
|
||||
"version": "0.79.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "karaoke-gen-frontend",
|
||||
"version": "0.77.0",
|
||||
"version": "0.79.0",
|
||||
"dependencies": {
|
||||
"@hookform/resolvers": "^3.10.0",
|
||||
"@radix-ui/react-accordion": "1.2.2",
|
||||
@@ -75,7 +75,6 @@
|
||||
"concurrently": "^9.2.1",
|
||||
"jest": "^29.7.0",
|
||||
"jest-environment-jsdom": "^29.7.0",
|
||||
"mailslurp-client": "^16.0.0",
|
||||
"postcss": "^8.5",
|
||||
"tailwindcss": "^4.1.9",
|
||||
"ts-node": "^10.9.2",
|
||||
@@ -4504,16 +4503,6 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/cross-fetch": {
|
||||
"version": "4.1.0",
|
||||
"resolved": "https://registry.npmjs.org/cross-fetch/-/cross-fetch-4.1.0.tgz",
|
||||
"integrity": "sha512-uKm5PU+MHTootlWEY+mZ4vvXoCn4fLQxT9dSc1sXVMSFkINTJVN8cAQROpwcKm8bJ/c7rgZVIBWzH5T78sNZZw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"node-fetch": "^2.7.0"
|
||||
}
|
||||
},
|
||||
"node_modules/cross-spawn": {
|
||||
"version": "7.0.6",
|
||||
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
|
||||
@@ -7103,16 +7092,6 @@
|
||||
"@jridgewell/sourcemap-codec": "^1.5.5"
|
||||
}
|
||||
},
|
||||
"node_modules/mailslurp-client": {
|
||||
"version": "16.0.0",
|
||||
"resolved": "https://registry.npmjs.org/mailslurp-client/-/mailslurp-client-16.0.0.tgz",
|
||||
"integrity": "sha512-4g3Zauq5vcHZ2HAanfLpdcXcsZhZj6ToL/hUY10YS1vHK2DGt9EEO43ms5wzLFHCMFOCsx6So1JMIER/n8rIXw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"cross-fetch": "^4.1.0"
|
||||
}
|
||||
},
|
||||
"node_modules/make-dir": {
|
||||
"version": "4.0.0",
|
||||
"resolved": "https://registry.npmjs.org/make-dir/-/make-dir-4.0.0.tgz",
|
||||
@@ -7355,52 +7334,6 @@
|
||||
"node": "^10 || ^12 || >=14"
|
||||
}
|
||||
},
|
||||
"node_modules/node-fetch": {
|
||||
"version": "2.7.0",
|
||||
"resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz",
|
||||
"integrity": "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"whatwg-url": "^5.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": "4.x || >=6.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"encoding": "^0.1.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"encoding": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/node-fetch/node_modules/tr46": {
|
||||
"version": "0.0.3",
|
||||
"resolved": "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz",
|
||||
"integrity": "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/node-fetch/node_modules/webidl-conversions": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz",
|
||||
"integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==",
|
||||
"dev": true,
|
||||
"license": "BSD-2-Clause"
|
||||
},
|
||||
"node_modules/node-fetch/node_modules/whatwg-url": {
|
||||
"version": "5.0.0",
|
||||
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz",
|
||||
"integrity": "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"tr46": "~0.0.3",
|
||||
"webidl-conversions": "^3.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/node-int64": {
|
||||
"version": "0.4.0",
|
||||
"resolved": "https://registry.npmjs.org/node-int64/-/node-int64-0.4.0.tgz",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "karaoke-gen-frontend",
|
||||
"version": "0.78.0",
|
||||
"version": "0.79.0",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"build": "next build",
|
||||
@@ -93,7 +93,6 @@
|
||||
"concurrently": "^9.2.1",
|
||||
"jest": "^29.7.0",
|
||||
"jest-environment-jsdom": "^29.7.0",
|
||||
"mailslurp-client": "^16.0.0",
|
||||
"postcss": "^8.5",
|
||||
"tailwindcss": "^4.1.9",
|
||||
"ts-node": "^10.9.2",
|
||||
|
||||
@@ -74,7 +74,8 @@ export default defineConfig({
|
||||
},
|
||||
|
||||
// Much longer timeout for production tests (karaoke generation takes time!)
|
||||
timeout: 900000, // 15 minutes per test
|
||||
// Remote downloads can take 15+ minutes, plus processing and video generation
|
||||
timeout: 2400000, // 40 minutes per test
|
||||
|
||||
// Expect timeout for assertions
|
||||
expect: {
|
||||
|
||||
@@ -2,7 +2,7 @@ import { defineConfig, devices } from '@playwright/test';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
|
||||
// Load .env.local file if it exists (for KARAOKE_ACCESS_TOKEN, MAILSLURP_API_KEY)
|
||||
// Load .env.local file if it exists (for KARAOKE_ACCESS_TOKEN, TESTMAIL_API_KEY, TESTMAIL_NAMESPACE)
|
||||
const envLocalPath = path.join(__dirname, '.env.local');
|
||||
if (fs.existsSync(envLocalPath)) {
|
||||
const envContent = fs.readFileSync(envLocalPath, 'utf-8');
|
||||
|
||||
+605
-5
@@ -92,6 +92,58 @@ firestore_index_magic_links = firestore.Index(
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Gen Users: query by is_active, order by created_at (for admin user list)
|
||||
firestore_index_gen_users_active_created = firestore.Index(
|
||||
"firestore-index-gen-users-active-created",
|
||||
project=project_id,
|
||||
database=firestore_db.name,
|
||||
collection="gen_users",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Gen Users: query by is_active, order by last_login_at (for admin user list sorting)
|
||||
firestore_index_gen_users_active_login = firestore.Index(
|
||||
"firestore-index-gen-users-active-login",
|
||||
project=project_id,
|
||||
database=firestore_db.name,
|
||||
collection="gen_users",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="last_login_at", order="DESCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Gen Users: query by is_active, order by credits (for admin user list sorting)
|
||||
firestore_index_gen_users_active_credits = firestore.Index(
|
||||
"firestore-index-gen-users-active-credits",
|
||||
project=project_id,
|
||||
database=firestore_db.name,
|
||||
collection="gen_users",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="credits", order="DESCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Gen Users: query by is_active, order by email (for admin user list sorting)
|
||||
firestore_index_gen_users_active_email = firestore.Index(
|
||||
"firestore-index-gen-users-active-email",
|
||||
project=project_id,
|
||||
database=firestore_db.name,
|
||||
collection="gen_users",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="email", order="ASCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Create Cloud Storage Bucket
|
||||
bucket = storage.Bucket(
|
||||
"karaoke-storage",
|
||||
@@ -314,7 +366,7 @@ workload_identity_provider = gcp.iam.WorkloadIdentityPoolProvider(
|
||||
)
|
||||
|
||||
# Allow GitHub Actions to impersonate the service account
|
||||
# Both karaoke-gen and flacfetch repos need access
|
||||
# All nomadkaraoke repos that deploy to GCP need access
|
||||
github_actions_wif_binding = gcp.serviceaccount.IAMBinding(
|
||||
"github-actions-wif-binding",
|
||||
service_account_id=github_actions_sa.name,
|
||||
@@ -326,6 +378,9 @@ github_actions_wif_binding = gcp.serviceaccount.IAMBinding(
|
||||
workload_identity_pool.name.apply(
|
||||
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/flacfetch"
|
||||
),
|
||||
workload_identity_pool.name.apply(
|
||||
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/karaoke-decide"
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
@@ -752,6 +807,10 @@ pulumi.export("firestore_index_jobs_user_email", firestore_index_jobs_user_email
|
||||
pulumi.export("firestore_index_jobs_status", firestore_index_jobs_status.name)
|
||||
pulumi.export("firestore_index_sessions_active", firestore_index_sessions_active.name)
|
||||
pulumi.export("firestore_index_magic_links", firestore_index_magic_links.name)
|
||||
pulumi.export("firestore_index_gen_users_active_created", firestore_index_gen_users_active_created.name)
|
||||
pulumi.export("firestore_index_gen_users_active_login", firestore_index_gen_users_active_login.name)
|
||||
pulumi.export("firestore_index_gen_users_active_credits", firestore_index_gen_users_active_credits.name)
|
||||
pulumi.export("firestore_index_gen_users_active_email", firestore_index_gen_users_active_email.name)
|
||||
pulumi.export("service_account_email", service_account.email)
|
||||
pulumi.export("artifact_repo_url", artifact_repo.name.apply(
|
||||
lambda name: f"us-central1-docker.pkg.dev/{project_id}/karaoke-repo"
|
||||
@@ -1033,6 +1092,516 @@ pulumi.export("flacfetch_service_url", flacfetch_ip.address.apply(
|
||||
))
|
||||
pulumi.export("flacfetch_service_account", flacfetch_sa.email)
|
||||
|
||||
# ==================== High-Performance Encoding Worker ====================
|
||||
# A dedicated C4-standard-96 VM with Intel Granite Rapids CPU for fast video encoding.
|
||||
# This offloads CPU-intensive FFmpeg libx264 encoding from Cloud Run to achieve
|
||||
# 2-3x faster encoding times with dedicated high-frequency cores.
|
||||
# Cloud Run dispatches encoding jobs to this worker via HTTP.
|
||||
|
||||
# Service account for encoding worker VM
|
||||
encoding_worker_sa = serviceaccount.Account(
|
||||
"encoding-worker-sa",
|
||||
account_id="encoding-worker",
|
||||
display_name="Encoding Worker Service Account",
|
||||
description="Service account for high-performance video encoding VM",
|
||||
)
|
||||
|
||||
# Grant encoding worker service account GCS read/write permissions
|
||||
encoding_worker_storage_iam = gcp.storage.BucketIAMMember(
|
||||
"encoding-worker-storage-admin",
|
||||
bucket=bucket.name,
|
||||
role="roles/storage.objectAdmin",
|
||||
member=encoding_worker_sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant encoding worker service account Secret Manager access
|
||||
encoding_worker_secrets_iam = gcp.projects.IAMMember(
|
||||
"encoding-worker-secrets-access",
|
||||
project=project_id,
|
||||
role="roles/secretmanager.secretAccessor",
|
||||
member=encoding_worker_sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant encoding worker logging permissions
|
||||
encoding_worker_logging_iam = gcp.projects.IAMMember(
|
||||
"encoding-worker-logging-access",
|
||||
project=project_id,
|
||||
role="roles/logging.logWriter",
|
||||
member=encoding_worker_sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Static external IP for encoding worker (Cloud Run doesn't have VPC access by default)
|
||||
encoding_worker_ip = gcp.compute.Address(
|
||||
"encoding-worker-ip",
|
||||
name="encoding-worker-static-ip",
|
||||
region="us-central1",
|
||||
address_type="EXTERNAL",
|
||||
description="Static external IP for encoding worker service",
|
||||
)
|
||||
|
||||
# Startup script for the encoding worker VM
|
||||
ENCODING_WORKER_STARTUP_SCRIPT = """#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Log to a file for debugging
|
||||
exec > >(tee /var/log/encoding-worker-startup.log) 2>&1
|
||||
echo "Starting encoding worker setup at $(date)"
|
||||
|
||||
# Install dependencies
|
||||
apt-get update
|
||||
apt-get install -y python3-pip python3-venv docker.io curl git xz-utils
|
||||
|
||||
# Enable and start Docker
|
||||
systemctl enable docker
|
||||
systemctl start docker
|
||||
|
||||
# Install optimized static FFmpeg build (John Van Sickle)
|
||||
echo "Installing optimized FFmpeg..."
|
||||
curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz
|
||||
tar -xf /tmp/ffmpeg.tar.xz -C /tmp
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/
|
||||
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe
|
||||
rm -rf /tmp/ffmpeg*
|
||||
ffmpeg -version
|
||||
|
||||
# Create working directory
|
||||
mkdir -p /opt/encoding-worker
|
||||
cd /opt/encoding-worker
|
||||
|
||||
# Create Python virtual environment
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
|
||||
# Install Python dependencies
|
||||
pip install --upgrade pip
|
||||
pip install fastapi uvicorn google-cloud-storage aiofiles aiohttp
|
||||
|
||||
# Get API key from Secret Manager
|
||||
echo "Fetching API key from Secret Manager..."
|
||||
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
|
||||
|
||||
if [ -z "$ENCODING_API_KEY" ]; then
|
||||
echo "WARNING: No API key found in Secret Manager, generating temporary key"
|
||||
ENCODING_API_KEY=$(openssl rand -hex 32)
|
||||
echo "Temporary API key: $ENCODING_API_KEY"
|
||||
fi
|
||||
|
||||
# Create the encoding worker service
|
||||
cat > /opt/encoding-worker/main.py << 'PYTHON_CODE'
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
import uuid
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
|
||||
from google.cloud import storage
|
||||
from pydantic import BaseModel
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
app = FastAPI(title="Encoding Worker", version="1.0.0")
|
||||
|
||||
# API key authentication
|
||||
API_KEY = os.environ.get("ENCODING_API_KEY", "")
|
||||
|
||||
|
||||
async def verify_api_key(x_api_key: str = Header(None)):
|
||||
# Verify API key for authentication
|
||||
if not API_KEY:
|
||||
logger.warning("No API key configured - authentication disabled")
|
||||
return True
|
||||
if x_api_key != API_KEY:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
return True
|
||||
|
||||
# Job tracking
|
||||
jobs: dict[str, dict] = {}
|
||||
executor = ThreadPoolExecutor(max_workers=4) # 4 parallel encoding jobs
|
||||
|
||||
# GCS client
|
||||
storage_client = storage.Client()
|
||||
|
||||
class EncodeRequest(BaseModel):
|
||||
job_id: str
|
||||
input_gcs_path: str # gs://bucket/path/to/inputs/
|
||||
output_gcs_path: str # gs://bucket/path/to/outputs/
|
||||
encoding_config: dict # Video formats to generate
|
||||
|
||||
class JobStatus(BaseModel):
|
||||
job_id: str
|
||||
status: str # pending, running, complete, failed
|
||||
progress: int # 0-100
|
||||
error: Optional[str] = None
|
||||
output_files: Optional[list[str]] = None
|
||||
|
||||
|
||||
def download_from_gcs(gcs_uri: str, local_path: Path):
|
||||
# Download a file or folder from GCS
|
||||
# Parse gs://bucket/path
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
prefix = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blobs = list(bucket.list_blobs(prefix=prefix))
|
||||
|
||||
for blob in blobs:
|
||||
# Get relative path from prefix
|
||||
rel_path = blob.name[len(prefix):].lstrip("/")
|
||||
if not rel_path:
|
||||
continue
|
||||
dest = local_path / rel_path
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
blob.download_to_filename(str(dest))
|
||||
logger.info(f"Downloaded: {blob.name} -> {dest}")
|
||||
|
||||
|
||||
def upload_to_gcs(local_path: Path, gcs_uri: str):
|
||||
# Upload a file or folder to GCS
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
|
||||
if local_path.is_file():
|
||||
blob_name = f"{prefix}/{local_path.name}" if prefix else local_path.name
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(local_path))
|
||||
logger.info(f"Uploaded: {local_path} -> gs://{bucket_name}/{blob_name}")
|
||||
else:
|
||||
for file in local_path.rglob("*"):
|
||||
if file.is_file():
|
||||
rel_path = file.relative_to(local_path)
|
||||
blob_name = f"{prefix}/{rel_path}" if prefix else str(rel_path)
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(file))
|
||||
logger.info(f"Uploaded: {file} -> gs://{bucket_name}/{blob_name}")
|
||||
|
||||
|
||||
def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
# Run FFmpeg encoding for all video formats
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Find input files (search recursively since GCS downloads preserve directory structure)
|
||||
input_files = list(work_dir.glob("**/*.mkv")) + list(work_dir.glob("**/*.mp4")) + list(work_dir.glob("**/*.mov"))
|
||||
# Exclude files in the outputs directory to avoid re-encoding previous outputs
|
||||
input_files = [f for f in input_files if "outputs" not in str(f)]
|
||||
if not input_files:
|
||||
raise ValueError(f"No video files found in {work_dir}")
|
||||
|
||||
# Use first video file as input (With Vocals or Karaoke)
|
||||
input_video = input_files[0]
|
||||
logger.info(f"Using input video: {input_video}")
|
||||
|
||||
# Find instrumental audio (search recursively, case-insensitive patterns)
|
||||
# GCS stores as: instrumental_clean.flac, instrumental_with_backing.flac
|
||||
instrumental_files = (
|
||||
list(work_dir.glob("**/*instrumental*.flac")) +
|
||||
list(work_dir.glob("**/*instrumental*.wav")) +
|
||||
list(work_dir.glob("**/*Instrumental*.flac")) +
|
||||
list(work_dir.glob("**/*Instrumental*.wav"))
|
||||
)
|
||||
instrumental = instrumental_files[0] if instrumental_files else None
|
||||
|
||||
output_dir = work_dir / "outputs"
|
||||
output_dir.mkdir(exist_ok=True)
|
||||
|
||||
output_files = []
|
||||
formats = config.get("formats", ["mp4_4k", "mp4_720p"])
|
||||
total_formats = len(formats)
|
||||
|
||||
for i, fmt in enumerate(formats):
|
||||
progress = 10 + int((i / total_formats) * 80)
|
||||
jobs[job_id]["progress"] = progress
|
||||
|
||||
if fmt == "mp4_4k_lossless" or fmt == "mp4_4k":
|
||||
# High quality 4K - CRF 18 (visually lossless)
|
||||
output_file = output_dir / "output_4k_lossless.mp4"
|
||||
cmd = [
|
||||
"ffmpeg", "-y",
|
||||
"-i", str(input_video),
|
||||
]
|
||||
if instrumental:
|
||||
cmd.extend(["-i", str(instrumental)])
|
||||
cmd.extend([
|
||||
"-map", "0:v",
|
||||
"-map", "1:a" if instrumental else "0:a",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "medium",
|
||||
"-crf", "18",
|
||||
"-c:a", "aac",
|
||||
"-b:a", "256k",
|
||||
"-threads", "24",
|
||||
str(output_file)
|
||||
])
|
||||
elif fmt == "mp4_4k_lossy":
|
||||
# Smaller 4K file - CRF 23 (good quality, smaller size)
|
||||
output_file = output_dir / "output_4k_lossy.mp4"
|
||||
cmd = [
|
||||
"ffmpeg", "-y",
|
||||
"-i", str(input_video),
|
||||
]
|
||||
if instrumental:
|
||||
cmd.extend(["-i", str(instrumental)])
|
||||
cmd.extend([
|
||||
"-map", "0:v",
|
||||
"-map", "1:a" if instrumental else "0:a",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "medium",
|
||||
"-crf", "23",
|
||||
"-c:a", "aac",
|
||||
"-b:a", "192k",
|
||||
"-threads", "24",
|
||||
str(output_file)
|
||||
])
|
||||
elif fmt == "mp4_720p":
|
||||
output_file = output_dir / "output_720p.mp4"
|
||||
cmd = [
|
||||
"ffmpeg", "-y",
|
||||
"-i", str(input_video),
|
||||
]
|
||||
if instrumental:
|
||||
cmd.extend(["-i", str(instrumental)])
|
||||
cmd.extend([
|
||||
"-map", "0:v",
|
||||
"-map", "1:a" if instrumental else "0:a",
|
||||
"-vf", "scale=-1:720",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "medium",
|
||||
"-crf", "20",
|
||||
"-c:a", "aac",
|
||||
"-b:a", "192k",
|
||||
"-threads", "24",
|
||||
str(output_file)
|
||||
])
|
||||
else:
|
||||
logger.warning(f"Unknown format: {fmt}")
|
||||
continue
|
||||
|
||||
logger.info(f"Running: {' '.join(cmd)}")
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
|
||||
if result.returncode != 0:
|
||||
logger.error(f"FFmpeg failed: {result.stderr}")
|
||||
raise RuntimeError(f"FFmpeg encoding failed: {result.stderr[-500:]}")
|
||||
|
||||
output_files.append(str(output_file))
|
||||
logger.info(f"Encoded: {output_file}")
|
||||
|
||||
jobs[job_id]["output_files"] = output_files
|
||||
jobs[job_id]["progress"] = 90
|
||||
return output_dir
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
async def process_job(job_id: str, request: EncodeRequest):
|
||||
# Process an encoding job asynchronously
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Download input files
|
||||
jobs[job_id]["progress"] = 5
|
||||
logger.info(f"Downloading from {request.input_gcs_path}")
|
||||
download_from_gcs(request.input_gcs_path, work_dir)
|
||||
|
||||
# Run encoding in thread pool (CPU-bound)
|
||||
loop = asyncio.get_event_loop()
|
||||
output_dir = await loop.run_in_executor(
|
||||
executor,
|
||||
run_encoding,
|
||||
job_id,
|
||||
work_dir,
|
||||
request.encoding_config
|
||||
)
|
||||
|
||||
# Upload outputs
|
||||
jobs[job_id]["progress"] = 95
|
||||
logger.info(f"Uploading to {request.output_gcs_path}")
|
||||
upload_to_gcs(output_dir, request.output_gcs_path)
|
||||
|
||||
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
|
||||
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
|
||||
# We need paths like "jobs/id/encoded/output_4k_lossless.mp4"
|
||||
gcs_path = request.output_gcs_path.replace("gs://", "")
|
||||
parts = gcs_path.split("/", 1)
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
local_output_files = jobs[job_id].get("output_files", [])
|
||||
blob_paths = []
|
||||
for local_path in local_output_files:
|
||||
filename = Path(local_path).name
|
||||
blob_path = f"{prefix}/{filename}" if prefix else filename
|
||||
blob_paths.append(blob_path)
|
||||
jobs[job_id]["output_files"] = blob_paths
|
||||
logger.info(f"Output files (blob paths): {blob_paths}")
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
@app.post("/encode")
|
||||
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit an encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
if job_id in jobs:
|
||||
raise HTTPException(status_code=409, detail=f"Job {job_id} already exists")
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.get("/status/{job_id}")
|
||||
async def get_job_status(job_id: str, _auth: bool = Depends(verify_api_key)) -> JobStatus:
|
||||
# Get the status of an encoding job
|
||||
if job_id not in jobs:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
return JobStatus(**jobs[job_id])
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
# Health check endpoint
|
||||
active_jobs = sum(1 for j in jobs.values() if j["status"] == "running")
|
||||
return {
|
||||
"status": "ok",
|
||||
"active_jobs": active_jobs,
|
||||
"queue_length": sum(1 for j in jobs.values() if j["status"] == "pending"),
|
||||
"ffmpeg_version": subprocess.run(["ffmpeg", "-version"], capture_output=True, text=True).stdout.split("\\n")[0],
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8080)
|
||||
PYTHON_CODE
|
||||
|
||||
# Create systemd service (use heredoc without quotes to expand ENCODING_API_KEY)
|
||||
cat > /etc/systemd/system/encoding-worker.service << SYSTEMD
|
||||
[Unit]
|
||||
Description=Encoding Worker HTTP API Service
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/encoding-worker
|
||||
ExecStart=/opt/encoding-worker/venv/bin/python main.py
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
|
||||
Environment="ENCODING_API_KEY=${ENCODING_API_KEY}"
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
SYSTEMD
|
||||
|
||||
systemctl daemon-reload
|
||||
systemctl enable encoding-worker
|
||||
systemctl start encoding-worker
|
||||
|
||||
echo "Encoding worker service started at $(date)"
|
||||
echo "Service listening on port 8080"
|
||||
"""
|
||||
|
||||
# Encoding worker VM instance (C4-standard-96 for maximum performance)
|
||||
encoding_worker_vm = gcp.compute.Instance(
|
||||
"encoding-worker",
|
||||
name="encoding-worker",
|
||||
machine_type="c4-standard-8", # 8 vCPUs, Intel Granite Rapids 3.9 GHz (quota limit: 24 CPUs)
|
||||
zone="us-central1-a",
|
||||
boot_disk=gcp.compute.InstanceBootDiskArgs(
|
||||
initialize_params=gcp.compute.InstanceBootDiskInitializeParamsArgs(
|
||||
image="debian-cloud/debian-12",
|
||||
size=100, # GB - for temp video files during encoding
|
||||
type="hyperdisk-balanced", # C4 machines require Hyperdisk
|
||||
),
|
||||
),
|
||||
network_interfaces=[gcp.compute.InstanceNetworkInterfaceArgs(
|
||||
network="default",
|
||||
access_configs=[gcp.compute.InstanceNetworkInterfaceAccessConfigArgs(
|
||||
nat_ip=encoding_worker_ip.address, # Static external IP
|
||||
)],
|
||||
)],
|
||||
service_account=gcp.compute.InstanceServiceAccountArgs(
|
||||
email=encoding_worker_sa.email,
|
||||
scopes=["cloud-platform"],
|
||||
),
|
||||
metadata_startup_script=ENCODING_WORKER_STARTUP_SCRIPT,
|
||||
tags=["encoding-worker"],
|
||||
allow_stopping_for_update=True,
|
||||
# Enable ALL_CORE_MAX for maximum sustained frequency
|
||||
advanced_machine_features=gcp.compute.InstanceAdvancedMachineFeaturesArgs(
|
||||
threads_per_core=2, # Enable hyperthreading
|
||||
),
|
||||
)
|
||||
|
||||
# Firewall rule for encoding worker
|
||||
# Note: Cloud Run doesn't have VPC access by default, so we allow external access
|
||||
# with API key authentication at the application level
|
||||
encoding_worker_firewall = gcp.compute.Firewall(
|
||||
"encoding-worker-firewall",
|
||||
name="encoding-worker-allow-http",
|
||||
network="default",
|
||||
allows=[
|
||||
# HTTP API
|
||||
gcp.compute.FirewallAllowArgs(protocol="tcp", ports=["8080"]),
|
||||
],
|
||||
# Allow from anywhere - auth is handled at application level via API key
|
||||
source_ranges=["0.0.0.0/0"],
|
||||
target_tags=["encoding-worker"],
|
||||
description="Allow HTTP access to encoding worker (auth required)",
|
||||
)
|
||||
|
||||
# API key secret for encoding worker authentication
|
||||
encoding_worker_api_key_secret = secretmanager.Secret(
|
||||
"encoding-worker-api-key",
|
||||
secret_id="encoding-worker-api-key",
|
||||
replication=secretmanager.SecretReplicationArgs(
|
||||
auto=secretmanager.SecretReplicationAutoArgs(),
|
||||
),
|
||||
)
|
||||
|
||||
# Export encoding worker values
|
||||
pulumi.export("encoding_worker_external_ip", encoding_worker_ip.address)
|
||||
pulumi.export("encoding_worker_service_url", encoding_worker_ip.address.apply(
|
||||
lambda ip: f"http://{ip}:8080"
|
||||
))
|
||||
pulumi.export("encoding_worker_service_account", encoding_worker_sa.email)
|
||||
pulumi.export("encoding_worker_vm_name", encoding_worker_vm.name)
|
||||
|
||||
# ==================== Cloud Monitoring Alerting ====================
|
||||
# Alert policies for proactive monitoring and incident response
|
||||
|
||||
@@ -1537,13 +2106,44 @@ echo "Python 3.13 added to tool cache"
|
||||
|
||||
# ==================== Docker cleanup cron ====================
|
||||
# Prevent disk from filling up with old Docker images
|
||||
# Run hourly instead of daily, with threshold-based cleanup
|
||||
echo "Setting up Docker cleanup cron..."
|
||||
cat > /etc/cron.daily/docker-cleanup << 'CRON'
|
||||
cat > /etc/cron.hourly/docker-cleanup << 'CRON'
|
||||
#!/bin/bash
|
||||
# Clean up Docker resources older than 7 days
|
||||
docker system prune -af --filter "until=168h"
|
||||
# Docker disk space cleanup for GitHub Actions runners
|
||||
# Runs hourly, with aggressive cleanup when disk usage exceeds threshold
|
||||
|
||||
# Log to syslog
|
||||
exec 1> >(logger -t docker-cleanup) 2>&1
|
||||
|
||||
THRESHOLD=70 # Cleanup when disk usage exceeds this percentage
|
||||
|
||||
# Get current disk usage percentage (remove % sign)
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
|
||||
if [ "$USAGE" -gt "$THRESHOLD" ]; then
|
||||
echo "Disk usage ${USAGE}% exceeds threshold ${THRESHOLD}%, running cleanup..."
|
||||
|
||||
# Remove ALL unused images (not just old ones) - this is the key fix
|
||||
# The previous 168h filter let recent images accumulate
|
||||
docker system prune -af
|
||||
|
||||
# Also clean builder cache which can grow large
|
||||
docker builder prune -af
|
||||
|
||||
# Report new usage
|
||||
NEW_USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Cleanup complete. Disk usage: ${NEW_USAGE}%"
|
||||
else
|
||||
echo "Disk usage ${USAGE}% is below threshold ${THRESHOLD}%, skipping cleanup"
|
||||
fi
|
||||
CRON
|
||||
chmod +x /etc/cron.daily/docker-cleanup
|
||||
chmod +x /etc/cron.hourly/docker-cleanup
|
||||
|
||||
# Remove old daily cleanup if it exists (migrating to hourly)
|
||||
rm -f /etc/cron.daily/docker-cleanup
|
||||
|
||||
# ==================== Docker pre-warm cron ====================
|
||||
# Keep base images fresh by pulling hourly
|
||||
|
||||
@@ -786,8 +786,8 @@
|
||||
|
||||
if (waveformRes.ok) {
|
||||
waveformData = await waveformRes.json();
|
||||
// API returns duration_seconds, not duration
|
||||
duration = waveformData.duration_seconds || 0;
|
||||
// API may return duration_seconds (cloud) or duration (local)
|
||||
duration = waveformData.duration_seconds || waveformData.duration || 0;
|
||||
}
|
||||
|
||||
// Set initial selection based on recommendation
|
||||
|
||||
@@ -3,6 +3,7 @@ import re
|
||||
import logging
|
||||
import shutil
|
||||
import json
|
||||
from typing import Optional
|
||||
from lyrics_transcriber import LyricsTranscriber, OutputConfig, TranscriberConfig, LyricsConfig
|
||||
from lyrics_transcriber.core.controller import LyricsControllerResult
|
||||
from dotenv import load_dotenv
|
||||
@@ -252,10 +253,11 @@ class LyricsProcessor:
|
||||
"See README.md 'Transcription Providers' section for detailed setup instructions."
|
||||
)
|
||||
|
||||
def transcribe_lyrics(self, input_audio_wav, artist, title, track_output_dir, lyrics_artist=None, lyrics_title=None):
|
||||
def transcribe_lyrics(self, input_audio_wav, artist, title, track_output_dir, lyrics_artist=None, lyrics_title=None,
|
||||
agentic_deadline: Optional[float] = None):
|
||||
"""
|
||||
Transcribe lyrics for a track.
|
||||
|
||||
|
||||
Args:
|
||||
input_audio_wav: Path to the audio file
|
||||
artist: Original artist name (used for filename generation)
|
||||
@@ -263,7 +265,9 @@ class LyricsProcessor:
|
||||
track_output_dir: Output directory path
|
||||
lyrics_artist: Artist name for lyrics processing (defaults to artist if None)
|
||||
lyrics_title: Title for lyrics processing (defaults to title if None)
|
||||
|
||||
agentic_deadline: Optional Unix timestamp. If agentic correction is still
|
||||
running after this time, it will abort and return uncorrected results.
|
||||
|
||||
Raises:
|
||||
ValueError: If transcription is enabled but no providers are configured
|
||||
"""
|
||||
@@ -423,8 +427,8 @@ class LyricsProcessor:
|
||||
logger=self.logger,
|
||||
)
|
||||
|
||||
# Process and get results
|
||||
results: LyricsControllerResult = transcriber.process()
|
||||
# Process and get results (pass deadline for agentic timeout)
|
||||
results: LyricsControllerResult = transcriber.process(agentic_deadline=agentic_deadline)
|
||||
self.logger.info(f"Transcriber Results Filepaths:")
|
||||
for key, value in results.__dict__.items():
|
||||
if key.endswith("_filepath"):
|
||||
|
||||
@@ -289,8 +289,13 @@ class LyricsTranscriber:
|
||||
"""Initialize output generation service."""
|
||||
return OutputGenerator(config=self.output_config, logger=self.logger)
|
||||
|
||||
def process(self) -> LyricsControllerResult:
|
||||
"""Main processing method that orchestrates the entire workflow."""
|
||||
def process(self, agentic_deadline: Optional[float] = None) -> LyricsControllerResult:
|
||||
"""Main processing method that orchestrates the entire workflow.
|
||||
|
||||
Args:
|
||||
agentic_deadline: Optional Unix timestamp. If agentic correction is still
|
||||
running after this time, it will abort and return uncorrected results.
|
||||
"""
|
||||
|
||||
self.logger.info(f"LyricsTranscriber controller beginning processing for {self.artist} - {self.title}")
|
||||
|
||||
@@ -390,7 +395,7 @@ class LyricsTranscriber:
|
||||
|
||||
# Step 3: Process and correct lyrics if enabled AND we have transcription results
|
||||
if self.output_config.run_correction and self.results.transcription_results:
|
||||
self.correct_lyrics()
|
||||
self.correct_lyrics(agentic_deadline=agentic_deadline)
|
||||
elif self.output_config.run_correction:
|
||||
self.logger.info("Skipping lyrics correction - no transcription results available")
|
||||
|
||||
@@ -494,8 +499,13 @@ class LyricsTranscriber:
|
||||
else:
|
||||
self.logger.debug(" - LocalWhisper: DISABLED (enable_local_whisper=False)")
|
||||
|
||||
def correct_lyrics(self) -> None:
|
||||
"""Run lyrics correction using transcription and internet lyrics."""
|
||||
def correct_lyrics(self, agentic_deadline: Optional[float] = None) -> None:
|
||||
"""Run lyrics correction using transcription and internet lyrics.
|
||||
|
||||
Args:
|
||||
agentic_deadline: Optional Unix timestamp. If agentic correction is still
|
||||
running after this time, it will abort and return uncorrected results.
|
||||
"""
|
||||
self.logger.info("Starting lyrics correction process")
|
||||
|
||||
# Check if we have reference lyrics to work with
|
||||
@@ -553,6 +563,7 @@ class LyricsTranscriber:
|
||||
transcription_results=self.results.transcription_results,
|
||||
lyrics_results=self.results.lyrics_results,
|
||||
metadata=metadata,
|
||||
agentic_deadline=agentic_deadline,
|
||||
)
|
||||
|
||||
# Store corrected results
|
||||
|
||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
import time
|
||||
from typing import Dict, Any, List, Optional
|
||||
|
||||
from .providers.base import BaseAIProvider
|
||||
@@ -120,7 +121,7 @@ class AgenticCorrector:
|
||||
title: Optional[str] = None
|
||||
) -> Optional[GapClassification]:
|
||||
"""Classify a gap using the AI provider.
|
||||
|
||||
|
||||
Args:
|
||||
gap_id: Unique identifier for the gap
|
||||
gap_text: The text of the gap
|
||||
@@ -129,10 +130,13 @@ class AgenticCorrector:
|
||||
reference_contexts: Dictionary of reference lyrics from each source
|
||||
artist: Song artist name
|
||||
title: Song title
|
||||
|
||||
|
||||
Returns:
|
||||
GapClassification object or None if classification fails
|
||||
"""
|
||||
logger.info(f"🤖 Classifying gap {gap_id} ({len(gap_text)} chars)")
|
||||
start_time = time.time()
|
||||
|
||||
# Build classification prompt
|
||||
prompt = build_classification_prompt(
|
||||
gap_text=gap_text,
|
||||
@@ -143,7 +147,7 @@ class AgenticCorrector:
|
||||
title=title,
|
||||
gap_id=gap_id
|
||||
)
|
||||
|
||||
|
||||
# Call AI provider to get classification
|
||||
try:
|
||||
data = self._provider.generate_correction_proposals(
|
||||
@@ -151,17 +155,25 @@ class AgenticCorrector:
|
||||
schema=GapClassification.model_json_schema(),
|
||||
session_id=self._session_id
|
||||
)
|
||||
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
|
||||
# Extract first result
|
||||
if data and len(data) > 0:
|
||||
item = data[0]
|
||||
if isinstance(item, dict) and "error" not in item:
|
||||
classification = GapClassification.model_validate(item)
|
||||
logger.debug(f"🤖 Classified gap {gap_id} as {classification.category} (confidence: {classification.confidence})")
|
||||
logger.info(
|
||||
f"🤖 Classified gap {gap_id} as {classification.category} "
|
||||
f"(confidence: {classification.confidence:.2f}) in {elapsed:.2f}s"
|
||||
)
|
||||
return classification
|
||||
else:
|
||||
logger.warning(f"🤖 Classification returned error for gap {gap_id}: {item}")
|
||||
except Exception as e:
|
||||
logger.warning(f"🤖 Failed to classify gap {gap_id}: {e}")
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.warning(f"🤖 Failed to classify gap {gap_id} after {elapsed:.2f}s: {e}")
|
||||
|
||||
return None
|
||||
|
||||
def propose_for_gap(
|
||||
|
||||
@@ -23,13 +23,23 @@ class ProviderConfig:
|
||||
gcp_project_id: Optional[str] = None
|
||||
gcp_location: str = "global"
|
||||
|
||||
request_timeout_seconds: float = 30.0
|
||||
# Timeout increased to 120s to handle Vertex AI connection establishment
|
||||
# and potential network latency. The 499 "operation cancelled" errors seen
|
||||
# at ~60s suggest internal timeouts; 120s provides headroom.
|
||||
request_timeout_seconds: float = 120.0
|
||||
max_retries: int = 2
|
||||
retry_backoff_base_seconds: float = 0.2
|
||||
# Backoff increased from 0.2s to 2.0s base - if a request times out,
|
||||
# retrying immediately is unlikely to help. Give the service time to recover.
|
||||
retry_backoff_base_seconds: float = 2.0
|
||||
retry_backoff_factor: float = 2.0
|
||||
circuit_breaker_failure_threshold: int = 3
|
||||
circuit_breaker_open_seconds: int = 60
|
||||
|
||||
# Initialization timeouts - fail fast instead of hanging forever
|
||||
# These are separate from request_timeout to catch connection establishment issues
|
||||
initialization_timeout_seconds: float = 30.0 # Model creation + warm-up
|
||||
warmup_timeout_seconds: float = 15.0 # Just the warm-up call
|
||||
|
||||
@staticmethod
|
||||
def from_env(cache_dir: Optional[str] = None) -> "ProviderConfig":
|
||||
"""Create config from environment variables.
|
||||
@@ -53,12 +63,14 @@ class ProviderConfig:
|
||||
cache_dir=cache_dir,
|
||||
gcp_project_id=os.getenv("GOOGLE_CLOUD_PROJECT") or os.getenv("GCP_PROJECT_ID"),
|
||||
gcp_location=os.getenv("GCP_LOCATION", "global"),
|
||||
request_timeout_seconds=float(os.getenv("AGENTIC_TIMEOUT_SECONDS", "30.0")),
|
||||
request_timeout_seconds=float(os.getenv("AGENTIC_TIMEOUT_SECONDS", "120.0")),
|
||||
max_retries=int(os.getenv("AGENTIC_MAX_RETRIES", "2")),
|
||||
retry_backoff_base_seconds=float(os.getenv("AGENTIC_BACKOFF_BASE_SECONDS", "0.2")),
|
||||
retry_backoff_base_seconds=float(os.getenv("AGENTIC_BACKOFF_BASE_SECONDS", "2.0")),
|
||||
retry_backoff_factor=float(os.getenv("AGENTIC_BACKOFF_FACTOR", "2.0")),
|
||||
circuit_breaker_failure_threshold=int(os.getenv("AGENTIC_CIRCUIT_THRESHOLD", "3")),
|
||||
circuit_breaker_open_seconds=int(os.getenv("AGENTIC_CIRCUIT_OPEN_SECONDS", "60")),
|
||||
initialization_timeout_seconds=float(os.getenv("AGENTIC_INIT_TIMEOUT_SECONDS", "30.0")),
|
||||
warmup_timeout_seconds=float(os.getenv("AGENTIC_WARMUP_TIMEOUT_SECONDS", "15.0")),
|
||||
)
|
||||
|
||||
def validate_environment(self, logger: Optional[object] = None) -> None:
|
||||
|
||||
+103
-13
@@ -13,6 +13,8 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
|
||||
from typing import List, Dict, Any, Optional
|
||||
from datetime import datetime
|
||||
|
||||
@@ -33,6 +35,14 @@ from .constants import (
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Error constant for initialization timeout
|
||||
INIT_TIMEOUT_ERROR = "initialization_timeout"
|
||||
|
||||
|
||||
class InitializationTimeoutError(Exception):
|
||||
"""Raised when model initialization exceeds the configured timeout."""
|
||||
pass
|
||||
|
||||
|
||||
class LangChainBridge(BaseAIProvider):
|
||||
"""Provider bridge using LangChain ChatModels with reliability patterns.
|
||||
@@ -87,6 +97,7 @@ class LangChainBridge(BaseAIProvider):
|
||||
|
||||
# Lazy-initialized chat model
|
||||
self._chat_model: Optional[Any] = None
|
||||
self._warmed_up: bool = False
|
||||
|
||||
def name(self) -> str:
|
||||
"""Return provider name for logging."""
|
||||
@@ -130,13 +141,45 @@ class LangChainBridge(BaseAIProvider):
|
||||
"until": open_until
|
||||
}]
|
||||
|
||||
# Step 2: Get or create chat model
|
||||
# Step 2: Get or create chat model with initialization timeout
|
||||
if not self._chat_model:
|
||||
timeout = self._config.initialization_timeout_seconds
|
||||
logger.info(f"🤖 Initializing model {self._model} with {timeout}s timeout...")
|
||||
init_start = time.time()
|
||||
|
||||
try:
|
||||
self._chat_model = self._factory.create_chat_model(
|
||||
self._model,
|
||||
self._config
|
||||
)
|
||||
# Use ThreadPoolExecutor for cross-platform timeout
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(
|
||||
self._factory.create_chat_model,
|
||||
self._model,
|
||||
self._config
|
||||
)
|
||||
try:
|
||||
self._chat_model = future.result(timeout=timeout)
|
||||
except FuturesTimeoutError:
|
||||
raise InitializationTimeoutError(
|
||||
f"Model initialization timed out after {timeout}s. "
|
||||
f"This may indicate network issues or service unavailability."
|
||||
) from None
|
||||
|
||||
init_elapsed = time.time() - init_start
|
||||
logger.info(f"🤖 Model created in {init_elapsed:.2f}s, starting warm-up...")
|
||||
|
||||
# Warm up the model to establish connection before real work
|
||||
self._warm_up_model()
|
||||
|
||||
total_elapsed = time.time() - init_start
|
||||
logger.info(f"🤖 Model initialization complete in {total_elapsed:.2f}s")
|
||||
|
||||
except InitializationTimeoutError as e:
|
||||
self._circuit_breaker.record_failure(self._model)
|
||||
logger.exception("🤖 Model initialization timeout")
|
||||
return [{
|
||||
"error": INIT_TIMEOUT_ERROR,
|
||||
"message": str(e),
|
||||
"timeout_seconds": timeout
|
||||
}]
|
||||
except Exception as e:
|
||||
self._circuit_breaker.record_failure(self._model)
|
||||
logger.error(f"🤖 Failed to initialize chat model: {e}")
|
||||
@@ -146,24 +189,27 @@ class LangChainBridge(BaseAIProvider):
|
||||
}]
|
||||
|
||||
# Step 3: Execute with retry logic
|
||||
logger.debug(
|
||||
f"🤖 [LangChain] Sending prompt to {self._model}: "
|
||||
f"{prompt[:PROMPT_LOG_LENGTH]}..."
|
||||
logger.info(
|
||||
f"🤖 [LangChain] Sending prompt to {self._model} ({len(prompt)} chars)"
|
||||
)
|
||||
|
||||
logger.debug(f"🤖 [LangChain] Prompt preview: {prompt[:PROMPT_LOG_LENGTH]}...")
|
||||
|
||||
invoke_start = time.time()
|
||||
result = self._executor.execute_with_retry(
|
||||
operation=lambda: self._invoke_model(prompt),
|
||||
operation_name=f"invoke_{self._model}"
|
||||
)
|
||||
|
||||
invoke_elapsed = time.time() - invoke_start
|
||||
|
||||
# Step 4: Handle result and update circuit breaker
|
||||
if result.success:
|
||||
self._circuit_breaker.record_success(self._model)
|
||||
|
||||
|
||||
logger.info(
|
||||
f"🤖 [LangChain] Got response from {self._model}: "
|
||||
f"{result.value[:RESPONSE_LOG_LENGTH]}..."
|
||||
f"🤖 [LangChain] Got response from {self._model} in {invoke_elapsed:.2f}s "
|
||||
f"({len(result.value)} chars)"
|
||||
)
|
||||
logger.debug(f"🤖 [LangChain] Response preview: {result.value[:RESPONSE_LOG_LENGTH]}...")
|
||||
|
||||
# Step 5: Cache the raw response for future use
|
||||
self._cache.set(
|
||||
@@ -225,3 +271,47 @@ class LangChainBridge(BaseAIProvider):
|
||||
|
||||
return content
|
||||
|
||||
def _warm_up_model(self) -> None:
|
||||
"""Send a lightweight request to warm up the model connection.
|
||||
|
||||
This helps establish the REST connection and potentially warm up any
|
||||
server-side resources before processing real correction requests.
|
||||
The warm-up uses a timeout to fail fast if the service is unresponsive.
|
||||
"""
|
||||
if self._warmed_up:
|
||||
return
|
||||
|
||||
timeout = self._config.warmup_timeout_seconds
|
||||
# Use print with flush=True for visibility when output is redirected
|
||||
print(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...", flush=True)
|
||||
logger.info(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...")
|
||||
|
||||
warmup_start = time.time()
|
||||
try:
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
# Minimal prompt that requires almost no processing
|
||||
warm_up_prompt = 'Respond with exactly: {"status":"ready"}'
|
||||
|
||||
# Use ThreadPoolExecutor for timeout on warm-up call
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(
|
||||
self._chat_model.invoke,
|
||||
[HumanMessage(content=warm_up_prompt)]
|
||||
)
|
||||
try:
|
||||
future.result(timeout=timeout)
|
||||
except FuturesTimeoutError:
|
||||
raise TimeoutError(f"Warm-up timed out after {timeout}s") from None
|
||||
|
||||
elapsed = time.time() - warmup_start
|
||||
self._warmed_up = True
|
||||
print(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s", flush=True)
|
||||
logger.info(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s")
|
||||
except Exception as e:
|
||||
elapsed = time.time() - warmup_start
|
||||
# Don't fail the actual request if warm-up fails
|
||||
# Just log and continue - the real request might still work
|
||||
print(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)", flush=True)
|
||||
logger.warning(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)")
|
||||
|
||||
+55
-14
@@ -3,12 +3,19 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from typing import Any, Optional, List
|
||||
|
||||
from .config import ProviderConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Error message constant for TRY003 compliance
|
||||
GOOGLE_API_KEY_MISSING_ERROR = (
|
||||
"GOOGLE_API_KEY environment variable is required for Google/Gemini models. "
|
||||
"Get an API key from https://aistudio.google.com/app/apikey"
|
||||
)
|
||||
|
||||
|
||||
class ModelFactory:
|
||||
"""Creates and configures LangChain ChatModels with observability.
|
||||
@@ -203,22 +210,56 @@ class ModelFactory:
|
||||
def _create_vertexai_model(
|
||||
self, model_name: str, callbacks: List[Any], config: ProviderConfig
|
||||
) -> Any:
|
||||
"""Create ChatVertexAI model for Google Gemini via Vertex AI.
|
||||
"""Create ChatGoogleGenerativeAI model for Google Gemini.
|
||||
|
||||
Uses Application Default Credentials (ADC) for authentication.
|
||||
In Cloud Run, this uses the service account automatically.
|
||||
Locally, run: gcloud auth application-default login
|
||||
Uses the unified langchain-google-genai package which supports both:
|
||||
- Vertex AI backend (service account / ADC auth) - when project is set
|
||||
- Google AI Studio backend (API key auth) - when only api_key is set
|
||||
|
||||
On Cloud Run, ADC (Application Default Credentials) are used automatically
|
||||
when the project parameter is provided, using the service account attached
|
||||
to the Cloud Run service.
|
||||
|
||||
This is a REST-based API that avoids the gRPC connection issues
|
||||
seen with the deprecated langchain-google-vertexai package.
|
||||
"""
|
||||
from langchain_google_vertexai import ChatVertexAI
|
||||
from langchain_google_genai import ChatGoogleGenerativeAI
|
||||
|
||||
model = ChatVertexAI(
|
||||
model=model_name,
|
||||
project=config.gcp_project_id,
|
||||
location=config.gcp_location,
|
||||
timeout=config.request_timeout_seconds,
|
||||
max_retries=config.max_retries,
|
||||
callbacks=callbacks,
|
||||
)
|
||||
logger.debug(f"🤖 Created Vertex AI model: {model_name} (project={config.gcp_project_id})")
|
||||
start_time = time.time()
|
||||
|
||||
# Determine authentication method
|
||||
api_key = config.google_api_key
|
||||
project = config.gcp_project_id
|
||||
|
||||
# Prefer Vertex AI (service account) if project is set, otherwise require API key
|
||||
if not project and not api_key:
|
||||
raise ValueError(GOOGLE_API_KEY_MISSING_ERROR)
|
||||
|
||||
if project:
|
||||
logger.info(f"🤖 Creating Google Gemini model via Vertex AI (project={project}): {model_name}")
|
||||
else:
|
||||
logger.info(f"🤖 Creating Google Gemini model via AI Studio API: {model_name}")
|
||||
|
||||
# Build kwargs - only include api_key if set (otherwise ADC is used)
|
||||
model_kwargs = {
|
||||
"model": model_name,
|
||||
"convert_system_message_to_human": True, # Gemini doesn't support system messages
|
||||
"max_retries": config.max_retries,
|
||||
"timeout": config.request_timeout_seconds,
|
||||
"callbacks": callbacks,
|
||||
}
|
||||
|
||||
# Add project to trigger Vertex AI backend with ADC
|
||||
if project:
|
||||
model_kwargs["project"] = project
|
||||
|
||||
# Add API key if available (can be used with or without project)
|
||||
if api_key:
|
||||
model_kwargs["google_api_key"] = api_key
|
||||
|
||||
model = ChatGoogleGenerativeAI(**model_kwargs)
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"🤖 Google Gemini model created in {elapsed:.2f}s: {model_name}")
|
||||
return model
|
||||
|
||||
+18
-6
@@ -51,22 +51,34 @@ class ResponseParser:
|
||||
|
||||
def _attempt_json_fix(self, content: str) -> str:
|
||||
"""Attempt to fix common JSON formatting issues.
|
||||
|
||||
|
||||
Args:
|
||||
content: Raw JSON string
|
||||
|
||||
|
||||
Returns:
|
||||
Fixed JSON string (or original if no fixes applied)
|
||||
"""
|
||||
import re
|
||||
|
||||
# Fix 0: Strip markdown code fences (```json ... ``` or ``` ... ```)
|
||||
# Models often wrap JSON in markdown code blocks
|
||||
fixed = content.strip()
|
||||
if fixed.startswith("```"):
|
||||
# Remove opening fence (with optional language identifier)
|
||||
fixed = re.sub(r'^```\w*\s*\n?', '', fixed)
|
||||
# Remove closing fence
|
||||
fixed = re.sub(r'\n?```\s*$', '', fixed)
|
||||
fixed = fixed.strip()
|
||||
logger.debug("🤖 Stripped markdown code fences from response")
|
||||
|
||||
# Fix 1: Replace invalid escape sequences like \' with '
|
||||
# (JSON only allows \", \\, \/, \b, \f, \n, \r, \t)
|
||||
fixed = content.replace("\\'", "'")
|
||||
|
||||
fixed = fixed.replace("\\'", "'")
|
||||
|
||||
# Fix 2: Remove any trailing commas before } or ]
|
||||
import re
|
||||
fixed = re.sub(r',\s*}', '}', fixed)
|
||||
fixed = re.sub(r',\s*]', ']', fixed)
|
||||
|
||||
|
||||
return fixed
|
||||
|
||||
def _normalize_json_response(self, data: Any) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -4,6 +4,7 @@ from pathlib import Path
|
||||
from copy import deepcopy
|
||||
import os
|
||||
import shortuuid
|
||||
import time
|
||||
|
||||
from lyrics_transcriber.correction.handlers.levenshtein import LevenshteinHandler
|
||||
from lyrics_transcriber.correction.handlers.no_space_punct_match import NoSpacePunctuationMatchHandler
|
||||
@@ -107,8 +108,24 @@ class LyricsCorrector:
|
||||
transcription_results: List[TranscriptionResult],
|
||||
lyrics_results: Dict[str, LyricsData],
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
agentic_deadline: Optional[float] = None,
|
||||
) -> CorrectionResult:
|
||||
"""Execute the correction process."""
|
||||
"""Execute the correction process.
|
||||
|
||||
Args:
|
||||
transcription_results: List of transcription results to correct.
|
||||
lyrics_results: Dictionary of lyrics data from various sources.
|
||||
metadata: Optional metadata including artist, title, audio file hash.
|
||||
agentic_deadline: Optional Unix timestamp (from time.time()). If agentic
|
||||
correction is still running after this time, it will abort and return
|
||||
uncorrected results for human review.
|
||||
|
||||
Note:
|
||||
The deadline is checked between gap iterations, not during LLM processing.
|
||||
A single long-running LLM call may exceed the deadline. The caller should
|
||||
wrap this method with an outer timeout (e.g., asyncio.wait_for) as a safety
|
||||
net for hung operations.
|
||||
"""
|
||||
# Optional agentic routing flag from environment; default off for safety
|
||||
agentic_enabled = os.getenv("USE_AGENTIC_AI", "").lower() in {"1", "true", "yes"}
|
||||
self.logger.info(f"🤖 AGENTIC MODE: {'ENABLED' if agentic_enabled else 'DISABLED'} (USE_AGENTIC_AI={os.getenv('USE_AGENTIC_AI', 'NOT_SET')})")
|
||||
@@ -132,9 +149,9 @@ class LyricsCorrector:
|
||||
# Store anchor sequences for use in correction handlers
|
||||
self._anchor_sequences = anchor_sequences
|
||||
|
||||
# Process corrections with metadata
|
||||
# Process corrections with metadata and optional deadline for agentic timeout
|
||||
corrections, corrected_segments, correction_steps, word_id_map, segment_id_map = self._process_corrections(
|
||||
primary_transcription.segments, gap_sequences, metadata=metadata
|
||||
primary_transcription.segments, gap_sequences, metadata=metadata, deadline=agentic_deadline
|
||||
)
|
||||
|
||||
# Calculate correction ratio
|
||||
@@ -178,10 +195,22 @@ class LyricsCorrector:
|
||||
return leading_space + new_word.strip() + trailing_space
|
||||
|
||||
def _process_corrections(
|
||||
self, segments: List[LyricsSegment], gap_sequences: List[GapSequence], metadata: Optional[Dict[str, Any]] = None
|
||||
self, segments: List[LyricsSegment], gap_sequences: List[GapSequence], metadata: Optional[Dict[str, Any]] = None,
|
||||
deadline: Optional[float] = None
|
||||
) -> Tuple[List[WordCorrection], List[LyricsSegment], List[CorrectionStep], Dict[str, str], Dict[str, str]]:
|
||||
"""Process corrections using handlers.
|
||||
|
||||
Args:
|
||||
segments: List of lyrics segments to process.
|
||||
gap_sequences: List of gap sequences to correct.
|
||||
metadata: Optional metadata including artist, title, audio file hash.
|
||||
deadline: Optional Unix timestamp (from time.time()). When agentic mode is
|
||||
enabled and this deadline is exceeded, remaining gaps are skipped and
|
||||
the method returns with whatever corrections have been made (likely none).
|
||||
|
||||
Returns:
|
||||
Tuple of (corrections, corrected_segments, correction_steps, word_id_map, segment_id_map).
|
||||
|
||||
The correction flow works as follows:
|
||||
1. First pass: Process all gaps
|
||||
- Iterate through each gap sequence
|
||||
@@ -415,6 +444,16 @@ class LyricsCorrector:
|
||||
# === END TEMPORARY CODE ===
|
||||
|
||||
for i, gap in enumerate(gap_sequences, 1):
|
||||
# Check deadline before processing each gap (agentic mode only)
|
||||
# This allows us to abort early and return uncorrected results for human review
|
||||
if deadline and use_agentic_env and time.time() > deadline:
|
||||
self.logger.warning(
|
||||
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {i-1}/{len(gap_sequences)} gaps. "
|
||||
"Skipping remaining gaps - human review will correct any issues."
|
||||
)
|
||||
# Break out of loop - continue with whatever corrections we have (likely none)
|
||||
break
|
||||
|
||||
self.logger.info(f"Processing gap {i}/{len(gap_sequences)} at position {gap.transcription_position}")
|
||||
|
||||
# Get the actual words for logging
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "lyrics-transcriber"
|
||||
version = "0.83.0"
|
||||
version = "0.84.0"
|
||||
description = "Automatically create synchronised lyrics files in ASS and MidiCo LRC formats with word-level timestamps, using Whisper and lyrics from Genius and Spotify"
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
@@ -55,7 +55,7 @@ langchain-core = ">=0.3.0"
|
||||
langchain-openai = ">=0.2.0"
|
||||
langchain-anthropic = ">=0.2.0"
|
||||
langchain-ollama = ">=0.2.0"
|
||||
langchain-google-vertexai = ">=3.1.1"
|
||||
langchain-google-genai = ">=2.0.0"
|
||||
langfuse = ">=3.0.0"
|
||||
pydantic = ">=2.7.0"
|
||||
jiwer = ">=3.0.0"
|
||||
|
||||
Generated
+795
-458
File diff suppressed because it is too large.
Load diff
+3
-2
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.82.0"
|
||||
version = "0.87.1"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
@@ -66,7 +66,7 @@ langchain-core = ">=0.3.0"
|
||||
langchain-openai = ">=0.2.0"
|
||||
langchain-anthropic = ">=0.2.0"
|
||||
langchain-ollama = ">=0.2.0"
|
||||
langchain-google-vertexai = ">=3.1.1"
|
||||
langchain-google-genai = ">=2.0.0"
|
||||
langfuse = ">=3.0.0"
|
||||
jiwer = ">=3.0.0"
|
||||
kbputils = "^0.0.16"
|
||||
@@ -108,6 +108,7 @@ opentelemetry-exporter-gcp-trace = ">=1.6.0"
|
||||
opentelemetry-resourcedetector-gcp = ">=1.6.0a0"
|
||||
# Optional: Local Whisper transcription (install with: pip install karaoke-gen[local-whisper])
|
||||
whisper-timestamped = { version = ">=1.15.0", optional = true }
|
||||
aiohttp = "^3.13.2"
|
||||
|
||||
[tool.poetry.group.dev.dependencies]
|
||||
black = ">=23"
|
||||
|
||||
Executable
+464
@@ -0,0 +1,464 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Karaoke-Gen Log Timing Analyzer
|
||||
|
||||
Parses karaoke-gen log output (both local CLI and cloud job logs) and extracts
|
||||
timing information for each stage of the processing pipeline. Identifies both
|
||||
automated processing time and human interaction wait times.
|
||||
|
||||
Supports two log formats:
|
||||
- Local CLI: "2025-12-30 16:22:28.657 - INFO - logger - message"
|
||||
- Cloud Job: "2025-12-30T20:52:21.293 - INFO - backend.service - message"
|
||||
|
||||
Usage:
|
||||
python analyze_log_timing.py <log_file>
|
||||
python analyze_log_timing.py <log_file> --json # Output as JSON
|
||||
python analyze_log_timing.py <log_file> --verbose # Show all events
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import dataclass, field, asdict
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class Stage:
|
||||
"""Represents a processing stage with timing information."""
|
||||
name: str
|
||||
start_time: Optional[datetime] = None
|
||||
end_time: Optional[datetime] = None
|
||||
is_human_wait: bool = False
|
||||
sub_stages: list = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def duration(self) -> Optional[timedelta]:
|
||||
if self.start_time and self.end_time:
|
||||
return self.end_time - self.start_time
|
||||
return None
|
||||
|
||||
@property
|
||||
def duration_seconds(self) -> Optional[float]:
|
||||
d = self.duration
|
||||
return d.total_seconds() if d else None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
'name': self.name,
|
||||
'start_time': self.start_time.isoformat() if self.start_time else None,
|
||||
'end_time': self.end_time.isoformat() if self.end_time else None,
|
||||
'duration_seconds': self.duration_seconds,
|
||||
'is_human_wait': self.is_human_wait,
|
||||
'sub_stages': [s.to_dict() for s in self.sub_stages]
|
||||
}
|
||||
|
||||
|
||||
# Timestamp patterns for both formats
|
||||
# Local: 2025-12-30 16:22:28.657
|
||||
# Cloud: 2025-12-30T20:52:21.293
|
||||
TIMESTAMP_PATTERN_LOCAL = re.compile(r'^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3})')
|
||||
TIMESTAMP_PATTERN_CLOUD = re.compile(r'^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3})')
|
||||
|
||||
# Stage detection patterns
|
||||
STAGE_PATTERNS = {
|
||||
# Initial setup
|
||||
'flacfetch_search': (r'Using remote flacfetch API at:', r'Found \d+ results'),
|
||||
'human_track_select': (r'Select \(1-\d+\)', r'Downloading:.*from'),
|
||||
'audio_download': (r'\[RemoteFlacFetcher\] Downloading:', r'Audio downloaded from'),
|
||||
|
||||
# Parallel processing phase
|
||||
'parallel_start': (r'=== Starting Parallel Processing ===', None),
|
||||
|
||||
# Lyrics track
|
||||
'lyrics_fetch_lrclib': (r'lrclib - Searching LRCLIB for', r'Successfully fetched lyrics from lrclib'),
|
||||
'lyrics_fetch_genius': (r'genius - Trying RapidAPI for', r'Successfully fetched lyrics from genius'),
|
||||
'lyrics_fetch_spotify': (r'spotify - Trying RapidAPI for', r'Successfully fetched lyrics from spotify'),
|
||||
'lyrics_transcription': (r'audioshake - Starting transcription for', r'audioshake - All targets completed'),
|
||||
'lyrics_correction': (r'corrector - Starting correction process', r'Correction process completed'),
|
||||
|
||||
# Audio track
|
||||
'audio_separation_stage1': (r'Stage 1: Submitting audio separation job', r'Stage 1 completed'),
|
||||
'audio_separation_stage2': (r'Stage 2: Processing clean vocals', r'Stage 2 completed'),
|
||||
'audio_normalization': (r'Normalizing clean instrumental', r'Audio normalization process completed'),
|
||||
|
||||
# Human review
|
||||
'human_lyrics_review': (r'Opening review UI:', r'Human review completed'),
|
||||
|
||||
# Post-review rendering
|
||||
'video_render_main': (r'Rendering karaoke video with synchronized lyrics', r'Video rendered successfully:'),
|
||||
'title_screen_gen': (r'Creating title video', r'Creating end screen video'),
|
||||
'end_screen_gen': (r'Creating end screen video', r'Audio separation was already completed'),
|
||||
|
||||
# Human instrumental selection
|
||||
'human_instrumental_select': (r'Starting instrumental review UI', r'User selected:'),
|
||||
|
||||
# KaraokeFinalise phase
|
||||
'human_finalise_confirm': (r'Confirm features enabled log messages above match', r'Finding input file ending in'),
|
||||
'cdg_package': (r'Creating CDG package', r'CDG package created successfully'),
|
||||
'txt_package': (r'Creating TXT package', r'TXT package created successfully'),
|
||||
|
||||
# Video encoding (the big one)
|
||||
'encode_step1_remux': (r'\[Step 1/6\] Remuxing video', r'\[Step 2/6\]'),
|
||||
'encode_step2_mp4_convert': (r'\[Step 2/6\] Converting karaoke video', r'\[Step 3/6\]'),
|
||||
'encode_step3_lossless_4k': (r'\[Step 3/6\] Encoding lossless 4K', r'\[Step 4/6\]'),
|
||||
'encode_step4_lossy_4k': (r'\[Step 4/6\] Encoding lossy 4K', r'\[Step 5/6\]'),
|
||||
'encode_step5_mkv_flac': (r'\[Step 5/6\] Creating MKV', r'\[Step 6/6\]'),
|
||||
'encode_step6_720p': (r'\[Step 6/6\] Encoding 720p', r'Video encoding completed'),
|
||||
|
||||
# Human video check
|
||||
'human_video_check': (r'Please check them! Proceed\?', r'Video encoding completed successfully'),
|
||||
|
||||
# Modal file downloads (often overlooked)
|
||||
'modal_download_stage1': (r'📥 Downloading 8 output files', r'🎉 Successfully downloaded 8 files'),
|
||||
'modal_download_stage2': (r'📥 Downloading 2 output files', r'🎉 Successfully downloaded 2 files'),
|
||||
|
||||
# Distribution
|
||||
'youtube_upload': (r'Uploading final MKV to YouTube', r'Uploaded video to YouTube:'),
|
||||
'discord_notification': (r'Posting Discord notification', r'Message posted to Discord'),
|
||||
'rclone_copy': (r'Copying to cloud destination', r'Command completed successfully'),
|
||||
}
|
||||
|
||||
# Cloud-specific stage patterns (for backend workers)
|
||||
CLOUD_STAGE_PATTERNS = {
|
||||
# Job lifecycle
|
||||
'job_created': (r'transitioned to searching_audio', None),
|
||||
'audio_search': (r'transitioned to searching_audio', r'transitioned to awaiting_audio_selection'),
|
||||
'human_audio_select': (r'transitioned to awaiting_audio_selection', r'transitioned to downloading_audio'),
|
||||
'flacfetch_download': (r'transitioned to downloading_audio', r'transitioned to downloading'),
|
||||
|
||||
# Audio worker (cloud) - overall
|
||||
'cloud_audio_worker': (r'=== AUDIO WORKER STARTED ===', r'Audio separation complete!'),
|
||||
|
||||
# Audio worker stages
|
||||
'cloud_audio_gcs_download': (r'Downloading audio file\.\.\.', r'Audio downloaded:'),
|
||||
'cloud_modal_stage1': (r'Stage 1: Submitting audio separation job', r'Stage 1 completed'),
|
||||
'cloud_modal_stage1_download': (r'📥 Downloading 8 output files', r'🎉 Successfully downloaded 8 files'),
|
||||
'cloud_modal_stage2': (r'Stage 2: Processing clean vocals', r'Stage 2 completed'),
|
||||
'cloud_modal_stage2_download': (r'📥 Downloading 2 output files', r'🎉 Successfully downloaded 2 files'),
|
||||
'cloud_audio_normalize': (r'Normalizing clean instrumental', r'Audio normalization process completed'),
|
||||
'cloud_stems_upload': (r'Uploading separation results to GCS', r'All stems uploaded successfully'),
|
||||
|
||||
# Lyrics worker (cloud) - overall
|
||||
'cloud_lyrics_worker': (r'=== LYRICS WORKER STARTED ===', r'=== LYRICS WORKER COMPLETE ==='),
|
||||
|
||||
# Lyrics worker stages
|
||||
'cloud_lyrics_gcs_download': (r'Downloading audio file from GCS', r'Audio downloaded:.*lyrics'),
|
||||
'cloud_style_download': (r'Loading style configuration for lyrics', r'Using custom style params:'),
|
||||
'cloud_lyrics_fetch_lrclib': (r'Searching LRCLIB for', r'Successfully fetched lyrics from lrclib'),
|
||||
'cloud_lyrics_fetch_genius': (r'Trying RapidAPI for.*genius|Trying RapidAPI for', r'Successfully fetched lyrics from genius'),
|
||||
'cloud_lyrics_fetch_spotify': (r'Successfully fetched lyrics from genius', r'Successfully fetched lyrics from spotify'),
|
||||
'cloud_audioshake_transcribe': (r'Uploading.*to AudioShake', r'Getting task result for task'),
|
||||
'cloud_lyrics_correction': (r'Starting correction process', r'Correction process completed'),
|
||||
'cloud_lyrics_upload': (r'Uploading lyrics results to GCS', r'Successfully uploaded lyrics'),
|
||||
|
||||
# Screens worker (cloud)
|
||||
'cloud_screens_worker': (r'=== SCREENS WORKER STARTED ===', r'=== SCREENS WORKER COMPLETE ==='),
|
||||
'cloud_title_screen': (r'Creating title screen', r'Title screen created'),
|
||||
'cloud_end_screen': (r'Creating end screen', r'End screen created'),
|
||||
|
||||
# Render worker (cloud)
|
||||
'cloud_render_worker': (r'=== RENDER WORKER STARTED ===', r'=== RENDER WORKER COMPLETE ==='),
|
||||
'cloud_video_render': (r'Rendering karaoke video|Starting video render', r'Video rendered successfully|Video generation complete'),
|
||||
'cloud_render_upload': (r'Uploading rendered video', r'Successfully uploaded.*video'),
|
||||
|
||||
# Video worker (cloud)
|
||||
'cloud_video_worker': (r'=== VIDEO WORKER STARTED ===', r'=== VIDEO WORKER COMPLETE ==='),
|
||||
'cloud_video_download': (r'Downloading files for video encoding', r'All files downloaded'),
|
||||
'cloud_video_encode': (r'Starting video encoding', r'Video encoding completed'),
|
||||
}
|
||||
|
||||
# Human wait stages
|
||||
HUMAN_STAGES = {
|
||||
'human_track_select',
|
||||
'human_lyrics_review',
|
||||
'human_instrumental_select',
|
||||
'human_finalise_confirm',
|
||||
'human_video_check',
|
||||
'human_audio_select', # Cloud
|
||||
}
|
||||
|
||||
|
||||
def parse_timestamp(line: str) -> Optional[datetime]:
|
||||
"""Extract timestamp from log line. Handles both local and cloud formats."""
|
||||
# Try local format first: 2025-12-30 16:22:28.657
|
||||
match = TIMESTAMP_PATTERN_LOCAL.match(line)
|
||||
if match:
|
||||
return datetime.strptime(match.group(1), '%Y-%m-%d %H:%M:%S.%f')
|
||||
|
||||
# Try cloud format: 2025-12-30T20:52:21.293
|
||||
match = TIMESTAMP_PATTERN_CLOUD.match(line)
|
||||
if match:
|
||||
return datetime.strptime(match.group(1), '%Y-%m-%dT%H:%M:%S.%f')
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def detect_log_type(lines: list) -> str:
|
||||
"""Detect whether log is from local CLI or cloud backend."""
|
||||
for line in lines[:50]: # Check first 50 lines
|
||||
if TIMESTAMP_PATTERN_CLOUD.match(line):
|
||||
if 'backend.' in line or 'WORKER_START' in line or 'Firestore' in line:
|
||||
return 'cloud'
|
||||
if TIMESTAMP_PATTERN_LOCAL.match(line):
|
||||
if 'karaoke_gen' in line or 'nomadauto' in line.lower():
|
||||
return 'local'
|
||||
# Default based on timestamp format
|
||||
for line in lines[:10]:
|
||||
if TIMESTAMP_PATTERN_CLOUD.match(line):
|
||||
return 'cloud'
|
||||
if TIMESTAMP_PATTERN_LOCAL.match(line):
|
||||
return 'local'
|
||||
return 'unknown'
|
||||
|
||||
|
||||
def find_pattern_match(line: str, pattern: str) -> bool:
|
||||
"""Check if line matches pattern."""
|
||||
return bool(re.search(pattern, line))
|
||||
|
||||
|
||||
def analyze_log(log_path: Path, verbose: bool = False) -> dict:
|
||||
"""
|
||||
Analyze a karaoke-gen log file and extract timing information.
|
||||
|
||||
Returns:
|
||||
Dictionary with stage timings and summary statistics.
|
||||
"""
|
||||
with open(log_path, 'r', encoding='utf-8', errors='replace') as f:
|
||||
lines = f.readlines()
|
||||
|
||||
# Detect log type
|
||||
log_type = detect_log_type(lines)
|
||||
if verbose:
|
||||
print(f"Detected log type: {log_type}")
|
||||
|
||||
# Select appropriate patterns based on log type
|
||||
if log_type == 'cloud':
|
||||
patterns_to_use = {**CLOUD_STAGE_PATTERNS}
|
||||
else:
|
||||
patterns_to_use = {**STAGE_PATTERNS}
|
||||
|
||||
# Track stage states
|
||||
stages: dict[str, Stage] = {}
|
||||
active_stages: set[str] = set()
|
||||
|
||||
first_timestamp = None
|
||||
last_timestamp = None
|
||||
|
||||
for line in lines:
|
||||
ts = parse_timestamp(line)
|
||||
if ts:
|
||||
if first_timestamp is None:
|
||||
first_timestamp = ts
|
||||
last_timestamp = ts
|
||||
|
||||
# Check each stage pattern
|
||||
for stage_name, (start_pattern, end_pattern) in patterns_to_use.items():
|
||||
# Check for stage start
|
||||
if start_pattern and find_pattern_match(line, start_pattern):
|
||||
if stage_name not in stages:
|
||||
stages[stage_name] = Stage(
|
||||
name=stage_name,
|
||||
is_human_wait=stage_name in HUMAN_STAGES
|
||||
)
|
||||
if ts and not stages[stage_name].start_time:
|
||||
stages[stage_name].start_time = ts
|
||||
active_stages.add(stage_name)
|
||||
if verbose:
|
||||
print(f"[START] {stage_name} at {ts}")
|
||||
|
||||
# Check for stage end
|
||||
if end_pattern and find_pattern_match(line, end_pattern):
|
||||
if stage_name in stages and stage_name in active_stages:
|
||||
if ts:
|
||||
stages[stage_name].end_time = ts
|
||||
active_stages.discard(stage_name)
|
||||
if verbose:
|
||||
print(f"[END] {stage_name} at {ts} ({stages[stage_name].duration})")
|
||||
|
||||
# Calculate summary statistics
|
||||
total_duration = (last_timestamp - first_timestamp) if (first_timestamp and last_timestamp) else None
|
||||
|
||||
human_time = timedelta()
|
||||
processing_time = timedelta()
|
||||
|
||||
for stage in stages.values():
|
||||
if stage.duration:
|
||||
if stage.is_human_wait:
|
||||
human_time += stage.duration
|
||||
else:
|
||||
processing_time += stage.duration
|
||||
|
||||
# Group stages by category (different categories for local vs cloud)
|
||||
if log_type == 'cloud':
|
||||
categories = {
|
||||
'job_setup': ['job_created', 'audio_search', 'flacfetch_download', 'cloud_tasks_setup'],
|
||||
'audio_worker': ['cloud_audio_worker', 'cloud_audio_gcs_download', 'cloud_modal_api',
|
||||
'cloud_modal_stage1_download', 'cloud_modal_stage2_download', 'cloud_stems_upload'],
|
||||
'lyrics_worker': ['cloud_lyrics_worker', 'cloud_lyrics_gcs_download', 'cloud_style_download',
|
||||
'cloud_audioshake_upload', 'cloud_lyrics_correction', 'cloud_lyrics_upload'],
|
||||
'screens_worker': ['cloud_screens_worker', 'cloud_title_screen', 'cloud_end_screen'],
|
||||
'render_worker': ['cloud_render_worker', 'cloud_video_render', 'cloud_render_upload'],
|
||||
'video_worker': ['cloud_video_worker', 'cloud_video_download', 'cloud_video_encode'],
|
||||
'human_interaction': ['human_audio_select'],
|
||||
}
|
||||
else:
|
||||
categories = {
|
||||
'setup': ['flacfetch_search', 'audio_download'],
|
||||
'lyrics_processing': ['lyrics_fetch_lrclib', 'lyrics_fetch_genius', 'lyrics_fetch_spotify',
|
||||
'lyrics_transcription', 'lyrics_correction'],
|
||||
'audio_separation': ['audio_separation_stage1', 'audio_separation_stage2', 'audio_normalization',
|
||||
'modal_download_stage1', 'modal_download_stage2'],
|
||||
'video_rendering': ['video_render_main', 'title_screen_gen', 'end_screen_gen'],
|
||||
'packaging': ['cdg_package', 'txt_package'],
|
||||
'video_encoding': ['encode_step1_remux', 'encode_step2_mp4_convert', 'encode_step3_lossless_4k',
|
||||
'encode_step4_lossy_4k', 'encode_step5_mkv_flac', 'encode_step6_720p'],
|
||||
'distribution': ['youtube_upload', 'discord_notification', 'rclone_copy'],
|
||||
'human_interaction': ['human_track_select', 'human_lyrics_review', 'human_instrumental_select',
|
||||
'human_finalise_confirm', 'human_video_check'],
|
||||
}
|
||||
|
||||
category_totals = {}
|
||||
for cat_name, stage_names in categories.items():
|
||||
total = timedelta()
|
||||
for sn in stage_names:
|
||||
if sn in stages and stages[sn].duration:
|
||||
total += stages[sn].duration
|
||||
category_totals[cat_name] = total.total_seconds()
|
||||
|
||||
return {
|
||||
'log_file': str(log_path),
|
||||
'log_type': log_type,
|
||||
'first_timestamp': first_timestamp.isoformat() if first_timestamp else None,
|
||||
'last_timestamp': last_timestamp.isoformat() if last_timestamp else None,
|
||||
'total_wall_clock_seconds': total_duration.total_seconds() if total_duration else None,
|
||||
'total_human_wait_seconds': human_time.total_seconds(),
|
||||
'total_processing_seconds': processing_time.total_seconds(),
|
||||
'automated_processing_seconds': (total_duration.total_seconds() - human_time.total_seconds()) if total_duration else None,
|
||||
'category_totals': category_totals,
|
||||
'stages': {name: stage.to_dict() for name, stage in stages.items()},
|
||||
}
|
||||
|
||||
|
||||
def format_duration(seconds: Optional[float]) -> str:
|
||||
"""Format seconds as human-readable duration."""
|
||||
if seconds is None:
|
||||
return "N/A"
|
||||
minutes, secs = divmod(int(seconds), 60)
|
||||
if minutes > 0:
|
||||
return f"{minutes}m {secs}s"
|
||||
return f"{secs}s"
|
||||
|
||||
|
||||
def print_report(analysis: dict) -> None:
|
||||
"""Print a human-readable timing report."""
|
||||
print("=" * 70)
|
||||
print("KARAOKE-GEN LOG TIMING ANALYSIS")
|
||||
print("=" * 70)
|
||||
log_type = analysis.get('log_type', 'unknown')
|
||||
print(f"\nLog file: {analysis['log_file']}")
|
||||
print(f"Log type: {log_type.upper()}")
|
||||
print(f"Start: {analysis['first_timestamp']}")
|
||||
print(f"End: {analysis['last_timestamp']}")
|
||||
|
||||
print("\n" + "-" * 70)
|
||||
print("SUMMARY")
|
||||
print("-" * 70)
|
||||
print(f"Total wall-clock time: {format_duration(analysis['total_wall_clock_seconds'])}")
|
||||
print(f"Human interaction time: {format_duration(analysis['total_human_wait_seconds'])}")
|
||||
print(f"Automated processing time: {format_duration(analysis['automated_processing_seconds'])}")
|
||||
|
||||
print("\n" + "-" * 70)
|
||||
print("CATEGORY BREAKDOWN")
|
||||
print("-" * 70)
|
||||
for cat_name, seconds in analysis['category_totals'].items():
|
||||
if seconds > 0:
|
||||
label = cat_name.replace('_', ' ').title()
|
||||
print(f" {label:25} {format_duration(seconds):>10}")
|
||||
|
||||
print("\n" + "-" * 70)
|
||||
print("DETAILED STAGE TIMINGS")
|
||||
print("-" * 70)
|
||||
|
||||
# Sort stages by start time
|
||||
sorted_stages = sorted(
|
||||
[(name, data) for name, data in analysis['stages'].items() if data['start_time']],
|
||||
key=lambda x: x[1]['start_time']
|
||||
)
|
||||
|
||||
for name, data in sorted_stages:
|
||||
duration = format_duration(data['duration_seconds'])
|
||||
human_marker = " [HUMAN]" if data['is_human_wait'] else ""
|
||||
label = name.replace('_', ' ').title()
|
||||
print(f" {label:35} {duration:>10}{human_marker}")
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
|
||||
# Performance insights
|
||||
print("\nPERFORMANCE INSIGHTS")
|
||||
print("-" * 70)
|
||||
|
||||
# Find slowest automated stages
|
||||
automated_stages = [
|
||||
(name, data['duration_seconds'])
|
||||
for name, data in analysis['stages'].items()
|
||||
if data['duration_seconds'] and not data['is_human_wait']
|
||||
]
|
||||
automated_stages.sort(key=lambda x: x[1], reverse=True)
|
||||
|
||||
print("\nSlowest automated stages:")
|
||||
for name, duration in automated_stages[:5]:
|
||||
label = name.replace('_', ' ').title()
|
||||
print(f" {label:35} {format_duration(duration):>10}")
|
||||
|
||||
# Calculate encoding total
|
||||
encoding_total = sum(
|
||||
analysis['stages'].get(f'encode_step{i}_{suffix}', {}).get('duration_seconds', 0) or 0
|
||||
for i, suffix in [(1, 'remux'), (2, 'mp4_convert'), (3, 'lossless_4k'),
|
||||
(4, 'lossy_4k'), (5, 'mkv_flac'), (6, '720p')]
|
||||
)
|
||||
print(f"\nTotal video encoding time: {format_duration(encoding_total)}")
|
||||
|
||||
# Parallelizable encoding stages (steps 4, 5, 6)
|
||||
parallel_steps = ['encode_step4_lossy_4k', 'encode_step5_mkv_flac', 'encode_step6_720p']
|
||||
parallel_times = [
|
||||
analysis['stages'].get(step, {}).get('duration_seconds', 0) or 0
|
||||
for step in parallel_steps
|
||||
]
|
||||
current_sequential = sum(parallel_times)
|
||||
potential_parallel = max(parallel_times) if parallel_times else 0
|
||||
|
||||
if current_sequential > 0:
|
||||
print(f"\nEncoding optimization potential:")
|
||||
print(f" Steps 4-6 currently run sequentially: {format_duration(current_sequential)}")
|
||||
print(f" If parallelized, could take: {format_duration(potential_parallel)}")
|
||||
print(f" Potential time savings: {format_duration(current_sequential - potential_parallel)}")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Analyze karaoke-gen log files for timing information'
|
||||
)
|
||||
parser.add_argument('log_file', type=Path, help='Path to log file')
|
||||
parser.add_argument('--json', action='store_true', help='Output as JSON')
|
||||
parser.add_argument('--verbose', '-v', action='store_true', help='Show all detected events')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.log_file.exists():
|
||||
print(f"Error: Log file not found: {args.log_file}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
analysis = analyze_log(args.log_file, verbose=args.verbose)
|
||||
|
||||
if args.json:
|
||||
print(json.dumps(analysis, indent=2))
|
||||
else:
|
||||
print_report(analysis)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,299 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Isolated FFmpeg encoding benchmark.
|
||||
|
||||
Creates a test video and runs the same encoding commands used by karaoke-gen
|
||||
to measure raw encoding performance. Can be run locally and compared with
|
||||
cloud results.
|
||||
|
||||
Usage:
|
||||
# Run locally
|
||||
python scripts/benchmark_ffmpeg.py
|
||||
|
||||
# Compare with cloud (run from Cloud Run shell)
|
||||
python scripts/benchmark_ffmpeg.py --cloud
|
||||
|
||||
Output:
|
||||
Timing results for each encoding step
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def run_timed_command(description: str, command: str, timeout: int = 600) -> dict:
|
||||
"""Run a command and return timing info."""
|
||||
print(f"\n{'='*60}")
|
||||
print(f"BENCHMARK: {description}")
|
||||
print(f"{'='*60}")
|
||||
print(f"Command: {command[:100]}...")
|
||||
|
||||
start = time.time()
|
||||
result = subprocess.run(
|
||||
command,
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout
|
||||
)
|
||||
elapsed = time.time() - start
|
||||
|
||||
success = result.returncode == 0
|
||||
print(f"Result: {'SUCCESS' if success else 'FAILED'}")
|
||||
print(f"Time: {elapsed:.2f}s")
|
||||
|
||||
if not success:
|
||||
print(f"Error: {result.stderr[:500]}")
|
||||
|
||||
return {
|
||||
"description": description,
|
||||
"success": success,
|
||||
"elapsed_seconds": elapsed,
|
||||
"command": command,
|
||||
"stderr": result.stderr[:1000] if not success else None,
|
||||
}
|
||||
|
||||
|
||||
def get_video_info(file_path: str) -> dict:
|
||||
"""Get video file info using ffprobe."""
|
||||
cmd = f'ffprobe -v quiet -print_format json -show_format -show_streams "{file_path}"'
|
||||
result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
|
||||
if result.returncode == 0:
|
||||
return json.loads(result.stdout)
|
||||
return {}
|
||||
|
||||
|
||||
def create_test_video(output_path: str, duration: int = 60) -> dict:
|
||||
"""Create a test video similar to karaoke output (4K, ~1 minute)."""
|
||||
# Generate a 4K test pattern video with audio
|
||||
# This simulates the karaoke video before final encoding
|
||||
command = (
|
||||
f'ffmpeg -y -f lavfi -i "testsrc2=size=3840x2160:rate=30:duration={duration}" '
|
||||
f'-f lavfi -i "sine=frequency=440:duration={duration}" '
|
||||
f'-c:v libx264 -preset ultrafast -crf 18 -pix_fmt yuv420p '
|
||||
f'-c:a pcm_s16le -ar 48000 '
|
||||
f'"{output_path}"'
|
||||
)
|
||||
return run_timed_command(f"Create test video ({duration}s, 4K)", command)
|
||||
|
||||
|
||||
def benchmark_lossless_4k_encode(input_file: str, output_file: str) -> dict:
|
||||
"""
|
||||
Benchmark Step 3: Lossless 4K MP4 encoding.
|
||||
This is the most CPU-intensive step in karaoke-gen.
|
||||
"""
|
||||
command = (
|
||||
f'ffmpeg -y -hide_banner -nostats -loglevel warning '
|
||||
f'-i "{input_file}" '
|
||||
f'-c:v libx264 -preset medium -crf 18 -pix_fmt yuv420p '
|
||||
f'-c:a pcm_s16le -ar 48000 '
|
||||
f'"{output_file}"'
|
||||
)
|
||||
return run_timed_command("Encode lossless 4K MP4 (Step 3 equivalent)", command)
|
||||
|
||||
|
||||
def benchmark_720p_encode(input_file: str, output_file: str) -> dict:
|
||||
"""
|
||||
Benchmark Step 6: 720p encoding with scaling.
|
||||
This is the second most CPU-intensive step.
|
||||
"""
|
||||
command = (
|
||||
f'ffmpeg -y -hide_banner -nostats -loglevel warning '
|
||||
f'-i "{input_file}" '
|
||||
f'-c:v libx264 -vf "scale=1280:720" -b:v 2000k -preset medium -tune animation '
|
||||
f'-c:a aac -ar 48000 -b:a 128k -pix_fmt yuv420p '
|
||||
f'"{output_file}"'
|
||||
)
|
||||
return run_timed_command("Encode 720p MP4 (Step 6 equivalent)", command)
|
||||
|
||||
|
||||
def benchmark_lossy_4k_encode(input_file: str, output_file: str) -> dict:
|
||||
"""
|
||||
Benchmark Step 4: Lossy 4K with AAC audio.
|
||||
Mostly stream copy, should be fast.
|
||||
"""
|
||||
command = (
|
||||
f'ffmpeg -y -hide_banner -nostats -loglevel warning '
|
||||
f'-i "{input_file}" '
|
||||
f'-c:v copy -c:a aac -ar 48000 -b:a 320k '
|
||||
f'"{output_file}"'
|
||||
)
|
||||
return run_timed_command("Encode lossy 4K MP4 (Step 4 equivalent)", command)
|
||||
|
||||
|
||||
def benchmark_mkv_flac_encode(input_file: str, output_file: str) -> dict:
|
||||
"""
|
||||
Benchmark Step 5: MKV with FLAC audio.
|
||||
Video copy, audio re-encode to FLAC.
|
||||
"""
|
||||
command = (
|
||||
f'ffmpeg -y -hide_banner -nostats -loglevel warning '
|
||||
f'-i "{input_file}" '
|
||||
f'-c:v copy -c:a flac '
|
||||
f'"{output_file}"'
|
||||
)
|
||||
return run_timed_command("Encode MKV with FLAC (Step 5 equivalent)", command)
|
||||
|
||||
|
||||
def benchmark_title_screen_generation(output_file: str) -> dict:
|
||||
"""
|
||||
Benchmark title screen generation (5 second video).
|
||||
This simulates the screens worker.
|
||||
"""
|
||||
# Create a title screen similar to what karaoke-gen creates
|
||||
command = (
|
||||
f'ffmpeg -y -hide_banner -nostats -loglevel warning '
|
||||
f'-f lavfi -i "color=black:s=3840x2160:d=5:r=30" '
|
||||
f'-f lavfi -i "anullsrc=r=48000:cl=stereo:d=5" '
|
||||
f'-vf "drawtext=text=\'Artist - Title\':fontsize=120:fontcolor=white:'
|
||||
f'x=(w-text_w)/2:y=(h-text_h)/2" '
|
||||
f'-c:v libx264 -preset medium -pix_fmt yuv420p '
|
||||
f'-c:a pcm_s16le '
|
||||
f'"{output_file}"'
|
||||
)
|
||||
return run_timed_command("Generate title screen (5s, 4K)", command)
|
||||
|
||||
|
||||
def get_system_info() -> dict:
|
||||
"""Get system information for context."""
|
||||
info = {
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"python_version": sys.version,
|
||||
}
|
||||
|
||||
# Get FFmpeg version
|
||||
result = subprocess.run("ffmpeg -version", shell=True, capture_output=True, text=True)
|
||||
if result.returncode == 0:
|
||||
info["ffmpeg_version"] = result.stdout.split('\n')[0]
|
||||
|
||||
# Get CPU info
|
||||
if sys.platform == "darwin":
|
||||
result = subprocess.run("sysctl -n machdep.cpu.brand_string", shell=True, capture_output=True, text=True)
|
||||
if result.returncode == 0:
|
||||
info["cpu"] = result.stdout.strip()
|
||||
result = subprocess.run("sysctl -n hw.ncpu", shell=True, capture_output=True, text=True)
|
||||
if result.returncode == 0:
|
||||
info["cpu_cores"] = int(result.stdout.strip())
|
||||
else:
|
||||
result = subprocess.run("nproc", shell=True, capture_output=True, text=True)
|
||||
if result.returncode == 0:
|
||||
info["cpu_cores"] = int(result.stdout.strip())
|
||||
result = subprocess.run("cat /proc/cpuinfo | grep 'model name' | head -1", shell=True, capture_output=True, text=True)
|
||||
if result.returncode == 0:
|
||||
info["cpu"] = result.stdout.split(':')[-1].strip() if ':' in result.stdout else "Unknown"
|
||||
|
||||
return info
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="FFmpeg encoding benchmark")
|
||||
parser.add_argument("--duration", type=int, default=60, help="Test video duration in seconds")
|
||||
parser.add_argument("--cloud", action="store_true", help="Running on cloud (skip some checks)")
|
||||
parser.add_argument("--output-dir", type=Path, help="Output directory for results")
|
||||
parser.add_argument("--existing-video", type=Path, help="Use existing video instead of creating test")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Setup output directory
|
||||
if args.output_dir:
|
||||
output_dir = args.output_dir
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
else:
|
||||
output_dir = Path(tempfile.mkdtemp(prefix="ffmpeg_benchmark_"))
|
||||
|
||||
print("="*70)
|
||||
print("FFMPEG ENCODING BENCHMARK")
|
||||
print("="*70)
|
||||
print(f"Output directory: {output_dir}")
|
||||
print(f"Test duration: {args.duration}s")
|
||||
print(f"Environment: {'Cloud' if args.cloud else 'Local'}")
|
||||
|
||||
# Get system info
|
||||
sys_info = get_system_info()
|
||||
print(f"\nSystem Info:")
|
||||
for k, v in sys_info.items():
|
||||
print(f" {k}: {v}")
|
||||
|
||||
results = {
|
||||
"system_info": sys_info,
|
||||
"environment": "cloud" if args.cloud else "local",
|
||||
"benchmarks": [],
|
||||
}
|
||||
|
||||
# Create or use test video
|
||||
if args.existing_video and args.existing_video.exists():
|
||||
test_video = str(args.existing_video)
|
||||
print(f"\nUsing existing video: {test_video}")
|
||||
else:
|
||||
test_video = str(output_dir / "test_input.mp4")
|
||||
result = create_test_video(test_video, args.duration)
|
||||
results["benchmarks"].append(result)
|
||||
|
||||
if not result["success"]:
|
||||
print("ERROR: Failed to create test video")
|
||||
sys.exit(1)
|
||||
|
||||
# Get video info
|
||||
video_info = get_video_info(test_video)
|
||||
if video_info:
|
||||
duration = float(video_info.get("format", {}).get("duration", args.duration))
|
||||
print(f"\nTest video duration: {duration:.1f}s")
|
||||
results["video_duration"] = duration
|
||||
|
||||
# Run benchmarks
|
||||
benchmarks = [
|
||||
("title_screen", benchmark_title_screen_generation, output_dir / "title_screen.mp4"),
|
||||
("lossless_4k", benchmark_lossless_4k_encode, output_dir / "output_lossless_4k.mp4"),
|
||||
("lossy_4k", benchmark_lossy_4k_encode, output_dir / "output_lossy_4k.mp4"),
|
||||
("mkv_flac", benchmark_mkv_flac_encode, output_dir / "output_lossless_4k.mkv"),
|
||||
("720p", benchmark_720p_encode, output_dir / "output_720p.mp4"),
|
||||
]
|
||||
|
||||
for name, func, output_path in benchmarks:
|
||||
if name == "title_screen":
|
||||
result = func(str(output_path))
|
||||
else:
|
||||
result = func(test_video, str(output_path))
|
||||
results["benchmarks"].append(result)
|
||||
|
||||
# Summary
|
||||
print("\n" + "="*70)
|
||||
print("BENCHMARK SUMMARY")
|
||||
print("="*70)
|
||||
print(f"\n{'Step':<40} {'Time':>12} {'Status':>10}")
|
||||
print("-"*64)
|
||||
|
||||
total_time = 0
|
||||
for bench in results["benchmarks"]:
|
||||
status = "OK" if bench["success"] else "FAILED"
|
||||
time_str = f"{bench['elapsed_seconds']:.2f}s"
|
||||
print(f"{bench['description']:<40} {time_str:>12} {status:>10}")
|
||||
if bench["success"]:
|
||||
total_time += bench["elapsed_seconds"]
|
||||
|
||||
print("-"*64)
|
||||
print(f"{'Total encoding time':<40} {total_time:.2f}s")
|
||||
|
||||
# Save results
|
||||
results_file = output_dir / "benchmark_results.json"
|
||||
with open(results_file, "w") as f:
|
||||
json.dump(results, f, indent=2)
|
||||
print(f"\nResults saved to: {results_file}")
|
||||
|
||||
# Cleanup temp files (keep results)
|
||||
if not args.output_dir:
|
||||
print(f"\nTemp files in: {output_dir}")
|
||||
print("Delete manually when done reviewing")
|
||||
|
||||
return results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+360
@@ -0,0 +1,360 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Performance comparison script: Local vs Remote karaoke generation.
|
||||
|
||||
Runs the same audio file through both local and remote pipelines in
|
||||
non-interactive mode, capturing detailed timing logs for analysis.
|
||||
|
||||
Usage:
|
||||
python scripts/compare_local_vs_remote.py <audio_file> [--local-only] [--remote-only]
|
||||
|
||||
# Run both tests
|
||||
python scripts/compare_local_vs_remote.py input/minutemen/Search.flac
|
||||
|
||||
# Run only local test
|
||||
python scripts/compare_local_vs_remote.py input/minutemen/Search.flac --local-only
|
||||
|
||||
# Run only remote test (useful if local already done)
|
||||
python scripts/compare_local_vs_remote.py input/minutemen/Search.flac --remote-only
|
||||
|
||||
Output:
|
||||
- output/perf-test-{timestamp}/local/ - Local CLI output + logs
|
||||
- output/perf-test-{timestamp}/remote/ - Remote CLI output + logs
|
||||
- output/perf-test-{timestamp}/comparison.txt - Side-by-side timing analysis
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def run_command(cmd: list[str], log_file: Path, cwd: Path) -> tuple[int, float]:
|
||||
"""
|
||||
Run a command and capture output to a log file.
|
||||
|
||||
Returns:
|
||||
Tuple of (return_code, elapsed_seconds)
|
||||
"""
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Running: {' '.join(cmd)}")
|
||||
print(f"Logging to: {log_file}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
with open(log_file, 'w') as f:
|
||||
# Write command header
|
||||
f.write(f"# Command: {' '.join(cmd)}\n")
|
||||
f.write(f"# Started: {datetime.now().isoformat()}\n")
|
||||
f.write(f"# CWD: {cwd}\n")
|
||||
f.write("#" + "=" * 79 + "\n\n")
|
||||
f.flush()
|
||||
|
||||
# Run with real-time output to both console and file
|
||||
process = subprocess.Popen(
|
||||
cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
cwd=cwd,
|
||||
text=True,
|
||||
bufsize=1,
|
||||
)
|
||||
|
||||
for line in process.stdout:
|
||||
# Add timestamp to each line for log analysis
|
||||
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
|
||||
timestamped_line = f"{timestamp} - {line}"
|
||||
|
||||
# Write to file
|
||||
f.write(timestamped_line if not line.startswith("20") else line)
|
||||
f.flush()
|
||||
|
||||
# Echo to console (without extra timestamp)
|
||||
print(line, end='')
|
||||
|
||||
process.wait()
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
|
||||
# Append timing summary to log
|
||||
with open(log_file, 'a') as f:
|
||||
f.write(f"\n# Completed: {datetime.now().isoformat()}\n")
|
||||
f.write(f"# Return code: {process.returncode}\n")
|
||||
f.write(f"# Total elapsed: {elapsed:.2f} seconds ({elapsed/60:.1f} minutes)\n")
|
||||
|
||||
return process.returncode, elapsed
|
||||
|
||||
|
||||
def extract_job_id_from_log(log_file: Path) -> str | None:
|
||||
"""Extract the job ID from remote CLI log output."""
|
||||
content = log_file.read_text()
|
||||
|
||||
# Look for job ID in various formats
|
||||
patterns = [
|
||||
r'Job ID: ([a-f0-9-]+)',
|
||||
r'job_id["\']?\s*[:=]\s*["\']?([a-f0-9-]+)',
|
||||
r'Created job: ([a-f0-9-]+)',
|
||||
r'/jobs/([a-f0-9-]+)',
|
||||
]
|
||||
|
||||
for pattern in patterns:
|
||||
match = re.search(pattern, content, re.IGNORECASE)
|
||||
if match:
|
||||
return match.group(1)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def run_local_test(audio_file: Path, output_dir: Path, artist: str, title: str) -> tuple[int, float, Path]:
|
||||
"""
|
||||
Run local karaoke-gen CLI test.
|
||||
|
||||
Returns:
|
||||
Tuple of (return_code, elapsed_seconds, log_file_path)
|
||||
"""
|
||||
print("\n" + "=" * 70)
|
||||
print("RUNNING LOCAL TEST (karaoke-gen)")
|
||||
print("=" * 70)
|
||||
|
||||
log_file = output_dir / "local_cli.log"
|
||||
|
||||
cmd = [
|
||||
"poetry", "run", "karaoke-gen",
|
||||
str(audio_file.absolute()),
|
||||
artist,
|
||||
title,
|
||||
"--output_dir", str(output_dir.absolute()),
|
||||
"--log_level", "DEBUG",
|
||||
"--skip_transcription_review", # Non-interactive: skip lyrics review
|
||||
"--skip_instrumental_review", # Non-interactive: skip instrumental selection
|
||||
"-y", # Auto-confirm prompts
|
||||
]
|
||||
|
||||
# Get project root for cwd
|
||||
project_root = Path(__file__).parent.parent
|
||||
|
||||
return_code, elapsed = run_command(cmd, log_file, project_root)
|
||||
|
||||
return return_code, elapsed, log_file
|
||||
|
||||
|
||||
def run_remote_test(audio_file: Path, output_dir: Path, artist: str, title: str) -> tuple[int, float, Path, str | None]:
|
||||
"""
|
||||
Run remote karaoke-gen-remote CLI test.
|
||||
|
||||
Returns:
|
||||
Tuple of (return_code, elapsed_seconds, log_file_path, job_id)
|
||||
"""
|
||||
print("\n" + "=" * 70)
|
||||
print("RUNNING REMOTE TEST (karaoke-gen-remote)")
|
||||
print("=" * 70)
|
||||
|
||||
log_file = output_dir / "remote_cli.log"
|
||||
|
||||
cmd = [
|
||||
"poetry", "run", "karaoke-gen-remote",
|
||||
str(audio_file.absolute()),
|
||||
artist,
|
||||
title,
|
||||
"--output_dir", str(output_dir.absolute()),
|
||||
"--log_level", "DEBUG",
|
||||
"--skip_transcription_review", # Non-interactive: skip lyrics review
|
||||
"--skip_instrumental_review", # Non-interactive: skip instrumental selection
|
||||
"-y", # Auto-confirm prompts
|
||||
]
|
||||
|
||||
# Get project root for cwd
|
||||
project_root = Path(__file__).parent.parent
|
||||
|
||||
return_code, elapsed = run_command(cmd, log_file, project_root)
|
||||
|
||||
# Try to extract job ID for fetching cloud logs
|
||||
job_id = extract_job_id_from_log(log_file)
|
||||
|
||||
return return_code, elapsed, log_file, job_id
|
||||
|
||||
|
||||
def analyze_logs(local_log: Path | None, remote_log: Path | None, output_dir: Path) -> None:
|
||||
"""Run timing analysis on the logs and generate comparison."""
|
||||
project_root = Path(__file__).parent.parent
|
||||
analyzer = project_root / "scripts" / "analyze_log_timing.py"
|
||||
|
||||
results = {}
|
||||
|
||||
if local_log and local_log.exists():
|
||||
print("\n" + "=" * 70)
|
||||
print("ANALYZING LOCAL LOG")
|
||||
print("=" * 70)
|
||||
|
||||
# Run analyzer with JSON output
|
||||
result = subprocess.run(
|
||||
["python", str(analyzer), str(local_log), "--json"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
try:
|
||||
results['local'] = json.loads(result.stdout)
|
||||
except json.JSONDecodeError:
|
||||
print(f"Warning: Could not parse local log analysis")
|
||||
|
||||
# Also print human-readable output
|
||||
subprocess.run(
|
||||
["python", str(analyzer), str(local_log)],
|
||||
cwd=project_root,
|
||||
)
|
||||
|
||||
if remote_log and remote_log.exists():
|
||||
print("\n" + "=" * 70)
|
||||
print("ANALYZING REMOTE LOG")
|
||||
print("=" * 70)
|
||||
|
||||
# Run analyzer with JSON output
|
||||
result = subprocess.run(
|
||||
["python", str(analyzer), str(remote_log), "--json"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
try:
|
||||
results['remote'] = json.loads(result.stdout)
|
||||
except json.JSONDecodeError:
|
||||
print(f"Warning: Could not parse remote log analysis")
|
||||
|
||||
# Also print human-readable output
|
||||
subprocess.run(
|
||||
["python", str(analyzer), str(remote_log)],
|
||||
cwd=project_root,
|
||||
)
|
||||
|
||||
# Generate comparison summary
|
||||
if results:
|
||||
comparison_file = output_dir / "comparison.json"
|
||||
comparison_file.write_text(json.dumps(results, indent=2))
|
||||
print(f"\nComparison data saved to: {comparison_file}")
|
||||
|
||||
# Print comparison summary
|
||||
if 'local' in results and 'remote' in results:
|
||||
print("\n" + "=" * 70)
|
||||
print("PERFORMANCE COMPARISON SUMMARY")
|
||||
print("=" * 70)
|
||||
|
||||
local_total = results['local'].get('total_wall_clock_seconds', 0)
|
||||
remote_total = results['remote'].get('total_wall_clock_seconds', 0)
|
||||
local_auto = results['local'].get('automated_processing_seconds', 0)
|
||||
remote_auto = results['remote'].get('automated_processing_seconds', 0)
|
||||
|
||||
print(f"\n{'Metric':<35} {'Local':>12} {'Remote':>12} {'Diff':>12}")
|
||||
print("-" * 73)
|
||||
|
||||
if local_total and remote_total:
|
||||
diff = remote_total - local_total
|
||||
print(f"{'Total wall-clock time':35} {local_total:>10.1f}s {remote_total:>10.1f}s {diff:>+10.1f}s")
|
||||
|
||||
if local_auto and remote_auto:
|
||||
diff = remote_auto - local_auto
|
||||
print(f"{'Automated processing':35} {local_auto:>10.1f}s {remote_auto:>10.1f}s {diff:>+10.1f}s")
|
||||
|
||||
# Compare category totals
|
||||
print("\nCategory breakdown:")
|
||||
local_cats = results['local'].get('category_totals', {})
|
||||
remote_cats = results['remote'].get('category_totals', {})
|
||||
all_cats = set(local_cats.keys()) | set(remote_cats.keys())
|
||||
|
||||
for cat in sorted(all_cats):
|
||||
local_time = local_cats.get(cat, 0)
|
||||
remote_time = remote_cats.get(cat, 0)
|
||||
if local_time > 0 or remote_time > 0:
|
||||
diff = remote_time - local_time
|
||||
cat_label = cat.replace('_', ' ').title()
|
||||
print(f" {cat_label:<33} {local_time:>10.1f}s {remote_time:>10.1f}s {diff:>+10.1f}s")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compare local vs remote karaoke generation performance"
|
||||
)
|
||||
parser.add_argument("audio_file", type=Path, help="Path to audio file to process")
|
||||
parser.add_argument("--artist", default="Minutemen", help="Artist name (default: Minutemen)")
|
||||
parser.add_argument("--title", default="Search", help="Track title (default: Search)")
|
||||
parser.add_argument("--local-only", action="store_true", help="Only run local test")
|
||||
parser.add_argument("--remote-only", action="store_true", help="Only run remote test")
|
||||
parser.add_argument("--output-dir", type=Path, help="Custom output directory")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Validate audio file
|
||||
if not args.audio_file.exists():
|
||||
print(f"Error: Audio file not found: {args.audio_file}")
|
||||
sys.exit(1)
|
||||
|
||||
# Create timestamped output directory
|
||||
timestamp = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
if args.output_dir:
|
||||
output_base = args.output_dir
|
||||
else:
|
||||
output_base = Path(__file__).parent.parent / "output" / f"perf-test-{timestamp}"
|
||||
|
||||
output_base.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
local_output = output_base / "local"
|
||||
remote_output = output_base / "remote"
|
||||
|
||||
print(f"\n{'='*70}")
|
||||
print(f"KARAOKE GENERATION PERFORMANCE TEST")
|
||||
print(f"{'='*70}")
|
||||
print(f"Audio file: {args.audio_file}")
|
||||
print(f"Artist: {args.artist}")
|
||||
print(f"Title: {args.title}")
|
||||
print(f"Output directory: {output_base}")
|
||||
print(f"{'='*70}")
|
||||
|
||||
local_log = None
|
||||
remote_log = None
|
||||
|
||||
# Run local test
|
||||
if not args.remote_only:
|
||||
local_output.mkdir(parents=True, exist_ok=True)
|
||||
return_code, elapsed, local_log = run_local_test(
|
||||
args.audio_file, local_output, args.artist, args.title
|
||||
)
|
||||
print(f"\nLocal test completed in {elapsed:.1f}s ({elapsed/60:.1f} minutes)")
|
||||
print(f"Return code: {return_code}")
|
||||
|
||||
# Run remote test
|
||||
if not args.local_only:
|
||||
remote_output.mkdir(parents=True, exist_ok=True)
|
||||
return_code, elapsed, remote_log, job_id = run_remote_test(
|
||||
args.audio_file, remote_output, args.artist, args.title
|
||||
)
|
||||
print(f"\nRemote test completed in {elapsed:.1f}s ({elapsed/60:.1f} minutes)")
|
||||
print(f"Return code: {return_code}")
|
||||
|
||||
if job_id:
|
||||
print(f"Job ID: {job_id}")
|
||||
print(f"\nTo fetch detailed cloud logs:")
|
||||
print(f" python scripts/fetch_job_logs.py {job_id}")
|
||||
|
||||
# Analyze logs
|
||||
analyze_logs(local_log, remote_log, output_base)
|
||||
|
||||
print(f"\n{'='*70}")
|
||||
print(f"TEST COMPLETE")
|
||||
print(f"{'='*70}")
|
||||
print(f"Output directory: {output_base}")
|
||||
if local_log:
|
||||
print(f"Local log: {local_log}")
|
||||
if remote_log:
|
||||
print(f"Remote log: {remote_log}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+170
@@ -0,0 +1,170 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Fetch cloud job logs from Firestore and save to a local file.
|
||||
|
||||
This script extracts worker_logs and timeline entries from a cloud job
|
||||
and formats them into a log file compatible with analyze_log_timing.py.
|
||||
|
||||
Usage:
|
||||
python scripts/fetch_job_logs.py <job_id> [output_file]
|
||||
python scripts/fetch_job_logs.py 7ccfcd64
|
||||
python scripts/fetch_job_logs.py 7ccfcd64 ./output/job-7ccfcd64.log
|
||||
|
||||
Requires:
|
||||
- google-cloud-firestore
|
||||
- GOOGLE_APPLICATION_CREDENTIALS or gcloud auth application-default login
|
||||
"""
|
||||
import sys
|
||||
import json
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
|
||||
def get_job(job_id: str) -> dict | None:
|
||||
"""Fetch job document from Firestore."""
|
||||
db = firestore.Client(project='nomadkaraoke')
|
||||
doc = db.collection('jobs').document(job_id).get()
|
||||
|
||||
if doc.exists:
|
||||
return doc.to_dict()
|
||||
return None
|
||||
|
||||
|
||||
def format_worker_log(log: dict) -> str:
|
||||
"""Format a worker_log entry as a log line."""
|
||||
timestamp = log.get('timestamp', '')
|
||||
# Handle both string and datetime timestamps
|
||||
if hasattr(timestamp, 'isoformat'):
|
||||
timestamp = timestamp.isoformat()
|
||||
|
||||
worker = log.get('worker', 'unknown')
|
||||
level = log.get('level', 'INFO')
|
||||
message = log.get('message', '')
|
||||
|
||||
# Format: TIMESTAMP [WORKER] LEVEL: MESSAGE
|
||||
return f"{timestamp} [{worker}] {level}: {message}"
|
||||
|
||||
|
||||
def format_timeline_entry(entry: dict) -> str:
|
||||
"""Format a timeline entry as a log line."""
|
||||
timestamp = entry.get('timestamp', '')
|
||||
# Handle both string and datetime timestamps
|
||||
if hasattr(timestamp, 'isoformat'):
|
||||
timestamp = timestamp.isoformat()
|
||||
|
||||
status = entry.get('status', 'unknown')
|
||||
message = entry.get('message', '')
|
||||
progress = entry.get('progress', '')
|
||||
|
||||
# Format: TIMESTAMP [timeline] STATUS: MESSAGE (progress%)
|
||||
progress_str = f" ({progress}%)" if progress else ""
|
||||
return f"{timestamp} [timeline] {status}: {message}{progress_str}"
|
||||
|
||||
|
||||
def fetch_job_logs(job_id: str, output_path: str | None = None) -> str:
|
||||
"""
|
||||
Fetch job logs from Firestore and save to a file.
|
||||
|
||||
Args:
|
||||
job_id: The job ID to fetch logs for
|
||||
output_path: Optional path to save logs. If None, uses ./output/job-{job_id}.log
|
||||
|
||||
Returns:
|
||||
Path to the saved log file
|
||||
"""
|
||||
data = get_job(job_id)
|
||||
|
||||
if data is None:
|
||||
print(f"Error: Job {job_id} not found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Collect all log entries with timestamps
|
||||
log_entries = []
|
||||
|
||||
# Extract worker_logs
|
||||
worker_logs = data.get('worker_logs', [])
|
||||
for log in worker_logs:
|
||||
timestamp = log.get('timestamp', '')
|
||||
if hasattr(timestamp, 'isoformat'):
|
||||
timestamp = timestamp.isoformat()
|
||||
log_entries.append({
|
||||
'timestamp': timestamp,
|
||||
'line': format_worker_log(log)
|
||||
})
|
||||
|
||||
# Extract timeline entries
|
||||
timeline = data.get('timeline', [])
|
||||
for entry in timeline:
|
||||
timestamp = entry.get('timestamp', '')
|
||||
if hasattr(timestamp, 'isoformat'):
|
||||
timestamp = timestamp.isoformat()
|
||||
log_entries.append({
|
||||
'timestamp': timestamp,
|
||||
'line': format_timeline_entry(entry)
|
||||
})
|
||||
|
||||
# Sort all entries by timestamp
|
||||
log_entries.sort(key=lambda x: x['timestamp'])
|
||||
|
||||
# Generate header with job metadata
|
||||
header_lines = [
|
||||
f"# Cloud Job Logs for {job_id}",
|
||||
f"# Fetched at: {datetime.now().isoformat()}",
|
||||
f"# Status: {data.get('status', 'unknown')}",
|
||||
f"# Artist: {data.get('artist', 'unknown')}",
|
||||
f"# Title: {data.get('title', 'unknown')}",
|
||||
f"# Created: {data.get('created_at', 'unknown')}",
|
||||
f"# Worker logs: {len(worker_logs)}",
|
||||
f"# Timeline entries: {len(timeline)}",
|
||||
"#" + "=" * 79,
|
||||
""
|
||||
]
|
||||
|
||||
# Combine header and log lines
|
||||
all_lines = header_lines + [entry['line'] for entry in log_entries]
|
||||
content = '\n'.join(all_lines)
|
||||
|
||||
# Determine output path
|
||||
if output_path is None:
|
||||
output_dir = Path('./output')
|
||||
output_dir.mkdir(exist_ok=True)
|
||||
output_path = output_dir / f"job-{job_id}.log"
|
||||
else:
|
||||
output_path = Path(output_path)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Write to file
|
||||
output_path.write_text(content)
|
||||
|
||||
return str(output_path)
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print(__doc__)
|
||||
print("\nExamples:")
|
||||
print(" python scripts/fetch_job_logs.py 7ccfcd64")
|
||||
print(" python scripts/fetch_job_logs.py 7ccfcd64 ./logs/my-job.log")
|
||||
print("\nThe output file can be analyzed with:")
|
||||
print(" python scripts/analyze_log_timing.py ./output/job-7ccfcd64.log")
|
||||
sys.exit(1)
|
||||
|
||||
job_id = sys.argv[1]
|
||||
output_path = sys.argv[2] if len(sys.argv) > 2 else None
|
||||
|
||||
print(f"Fetching logs for job {job_id}...", file=sys.stderr)
|
||||
|
||||
saved_path = fetch_job_logs(job_id, output_path)
|
||||
|
||||
print(f"Logs saved to: {saved_path}", file=sys.stderr)
|
||||
print(f"\nTo analyze timing, run:", file=sys.stderr)
|
||||
print(f" python scripts/analyze_log_timing.py {saved_path}", file=sys.stderr)
|
||||
|
||||
# Print the path to stdout for scripting
|
||||
print(saved_path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Migration script to copy karaoke-gen users from 'users' collection to 'gen_users' collection.
|
||||
|
||||
This script:
|
||||
1. Identifies karaoke-gen users in the 'users' collection (by presence of 'credits' field)
|
||||
2. Copies them to the 'gen_users' collection (preserving document ID = email)
|
||||
3. Optionally deletes the old documents from 'users' collection
|
||||
|
||||
Run with:
|
||||
python scripts/migrate_users_to_gen_users.py --dry-run # Preview only
|
||||
python scripts/migrate_users_to_gen_users.py # Execute migration
|
||||
python scripts/migrate_users_to_gen_users.py --delete # Also delete old docs
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from google.cloud import firestore
|
||||
|
||||
|
||||
def is_karaoke_gen_user(doc_data: dict) -> bool:
|
||||
"""
|
||||
Determine if a user document belongs to karaoke-gen.
|
||||
|
||||
karaoke-gen users have:
|
||||
- Document ID = email address (contains @)
|
||||
- 'credits' field
|
||||
- 'role' field with value 'user' or 'admin'
|
||||
- 'is_active' field
|
||||
|
||||
karaoke-decide users have:
|
||||
- Document ID = hash or guest_xxx
|
||||
- 'user_id' field
|
||||
- 'is_guest' field
|
||||
"""
|
||||
# Check for karaoke-gen specific fields
|
||||
has_credits = 'credits' in doc_data
|
||||
has_role = doc_data.get('role') in ['user', 'admin']
|
||||
has_is_active = 'is_active' in doc_data
|
||||
|
||||
# Check for karaoke-decide specific fields
|
||||
has_user_id = 'user_id' in doc_data
|
||||
has_is_guest = 'is_guest' in doc_data
|
||||
|
||||
# karaoke-gen users have credits/role/is_active but NOT user_id/is_guest
|
||||
return has_credits and has_role and has_is_active and not has_user_id and not has_is_guest
|
||||
|
||||
|
||||
def migrate_users(dry_run: bool = True, delete_old: bool = False):
|
||||
"""
|
||||
Migrate karaoke-gen users from 'users' to 'gen_users' collection.
|
||||
"""
|
||||
db = firestore.Client(project='nomadkaraoke')
|
||||
|
||||
users_collection = db.collection('users')
|
||||
gen_users_collection = db.collection('gen_users')
|
||||
|
||||
# Get all documents from users collection
|
||||
print("Fetching all documents from 'users' collection...")
|
||||
all_docs = list(users_collection.stream())
|
||||
print(f"Found {len(all_docs)} total documents in 'users' collection")
|
||||
|
||||
# Filter to karaoke-gen users
|
||||
gen_users = []
|
||||
decide_users = []
|
||||
unknown_users = []
|
||||
|
||||
for doc in all_docs:
|
||||
doc_data = doc.to_dict()
|
||||
doc_id = doc.id
|
||||
|
||||
if is_karaoke_gen_user(doc_data):
|
||||
gen_users.append((doc_id, doc_data))
|
||||
elif 'user_id' in doc_data or 'is_guest' in doc_data:
|
||||
decide_users.append(doc_id)
|
||||
else:
|
||||
unknown_users.append(doc_id)
|
||||
|
||||
print(f"\nClassification results:")
|
||||
print(f" - karaoke-gen users: {len(gen_users)}")
|
||||
print(f" - karaoke-decide users: {len(decide_users)}")
|
||||
print(f" - Unknown/other: {len(unknown_users)}")
|
||||
|
||||
if unknown_users:
|
||||
print(f"\nUnknown users (first 5): {unknown_users[:5]}")
|
||||
|
||||
if dry_run:
|
||||
print(f"\n[DRY RUN] Would migrate {len(gen_users)} users to 'gen_users' collection:")
|
||||
for doc_id, doc_data in gen_users[:10]:
|
||||
email = doc_data.get('email', doc_id)
|
||||
credits = doc_data.get('credits', 0)
|
||||
print(f" - {email} (credits: {credits})")
|
||||
if len(gen_users) > 10:
|
||||
print(f" ... and {len(gen_users) - 10} more")
|
||||
return
|
||||
|
||||
# Execute migration
|
||||
print(f"\nMigrating {len(gen_users)} users to 'gen_users' collection...")
|
||||
migrated = 0
|
||||
errors = 0
|
||||
|
||||
for doc_id, doc_data in gen_users:
|
||||
try:
|
||||
# Use the same document ID (email) in the new collection
|
||||
gen_users_collection.document(doc_id).set(doc_data)
|
||||
migrated += 1
|
||||
|
||||
if migrated % 10 == 0:
|
||||
print(f" Migrated {migrated}/{len(gen_users)}...")
|
||||
except Exception as e:
|
||||
print(f" ERROR migrating {doc_id}: {e}")
|
||||
errors += 1
|
||||
|
||||
print(f"\nMigration complete: {migrated} migrated, {errors} errors")
|
||||
|
||||
# Optionally delete old documents
|
||||
if delete_old and errors == 0:
|
||||
print(f"\nDeleting old documents from 'users' collection...")
|
||||
deleted = 0
|
||||
for doc_id, _ in gen_users:
|
||||
try:
|
||||
users_collection.document(doc_id).delete()
|
||||
deleted += 1
|
||||
except Exception as e:
|
||||
print(f" ERROR deleting {doc_id}: {e}")
|
||||
print(f"Deleted {deleted} documents from 'users' collection")
|
||||
elif delete_old and errors > 0:
|
||||
print("\nSkipping deletion due to migration errors")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Migrate karaoke-gen users to gen_users collection')
|
||||
parser.add_argument('--dry-run', action='store_true', help='Preview migration without making changes')
|
||||
parser.add_argument('--delete', action='store_true', help='Delete old documents after successful migration')
|
||||
args = parser.parse_args()
|
||||
|
||||
# Default to dry-run if neither flag is set
|
||||
dry_run = args.dry_run or not args.delete
|
||||
|
||||
if not dry_run:
|
||||
print("=" * 60)
|
||||
print("WARNING: This will modify production data!")
|
||||
print("=" * 60)
|
||||
response = input("Type 'yes' to continue: ")
|
||||
if response.lower() != 'yes':
|
||||
print("Aborted")
|
||||
sys.exit(1)
|
||||
|
||||
migrate_users(dry_run=dry_run, delete_old=args.delete)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,359 @@
|
||||
"""
|
||||
Unit tests for LyricsCacheService.
|
||||
|
||||
Tests the GCS cache synchronization service for LyricsTranscriber.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from backend.services.lyrics_cache_service import (
|
||||
LyricsCacheService,
|
||||
TRANSCRIPTION_PROVIDERS,
|
||||
LYRICS_PROVIDERS,
|
||||
CACHE_SUFFIXES,
|
||||
)
|
||||
|
||||
|
||||
class TestHashComputation:
|
||||
"""Tests for hash computation methods."""
|
||||
|
||||
def test_compute_audio_hash_matches_lyrics_transcriber(self, temp_dir):
|
||||
"""Audio hash should match LyricsTranscriber's _get_file_hash implementation."""
|
||||
# Create a test audio file
|
||||
audio_path = os.path.join(temp_dir, "test_audio.wav")
|
||||
test_content = b"fake audio content for testing hash computation"
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(test_content)
|
||||
|
||||
# Compute hash using our service
|
||||
mock_storage = MagicMock()
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
computed_hash = service.compute_audio_hash(audio_path)
|
||||
|
||||
# Compute expected hash (same algorithm as LyricsTranscriber)
|
||||
md5_hash = hashlib.md5()
|
||||
with open(audio_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
md5_hash.update(chunk)
|
||||
expected_hash = md5_hash.hexdigest()
|
||||
|
||||
assert computed_hash == expected_hash
|
||||
|
||||
def test_compute_lyrics_hash_matches_lyrics_transcriber(self):
|
||||
"""Lyrics hash should match LyricsTranscriber's _get_artist_title_hash implementation."""
|
||||
mock_storage = MagicMock()
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
artist = "Test Artist"
|
||||
title = "Test Song"
|
||||
|
||||
computed_hash = service.compute_lyrics_hash(artist, title)
|
||||
|
||||
# Expected hash (same algorithm as LyricsTranscriber)
|
||||
combined = f"{artist.lower()}_{title.lower()}"
|
||||
expected_hash = hashlib.md5(combined.encode()).hexdigest()
|
||||
|
||||
assert computed_hash == expected_hash
|
||||
|
||||
def test_compute_lyrics_hash_case_insensitive(self):
|
||||
"""Lyrics hash should be case-insensitive."""
|
||||
mock_storage = MagicMock()
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
hash1 = service.compute_lyrics_hash("Artist", "Title")
|
||||
hash2 = service.compute_lyrics_hash("ARTIST", "TITLE")
|
||||
hash3 = service.compute_lyrics_hash("artist", "title")
|
||||
|
||||
assert hash1 == hash2 == hash3
|
||||
|
||||
|
||||
class TestCacheFilenames:
|
||||
"""Tests for cache filename generation."""
|
||||
|
||||
def test_get_cache_filenames_transcription_providers(self):
|
||||
"""Should generate correct filenames for transcription providers."""
|
||||
mock_storage = MagicMock()
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
filenames = service._get_cache_filenames(TRANSCRIPTION_PROVIDERS, "abc123")
|
||||
|
||||
expected = []
|
||||
for provider in TRANSCRIPTION_PROVIDERS:
|
||||
for suffix in CACHE_SUFFIXES:
|
||||
expected.append(f"{provider}_abc123_{suffix}.json")
|
||||
|
||||
assert sorted(filenames) == sorted(expected)
|
||||
|
||||
def test_get_cache_filenames_lyrics_providers(self):
|
||||
"""Should generate correct filenames for lyrics providers."""
|
||||
mock_storage = MagicMock()
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
filenames = service._get_cache_filenames(LYRICS_PROVIDERS, "def456")
|
||||
|
||||
expected = []
|
||||
for provider in LYRICS_PROVIDERS:
|
||||
for suffix in CACHE_SUFFIXES:
|
||||
expected.append(f"{provider}_def456_{suffix}.json")
|
||||
|
||||
assert sorted(filenames) == sorted(expected)
|
||||
|
||||
|
||||
class TestSyncCacheFromGCS:
|
||||
"""Tests for downloading cache from GCS."""
|
||||
|
||||
def test_sync_cache_from_gcs_downloads_existing_files(self, temp_dir):
|
||||
"""Should download files that exist in GCS."""
|
||||
mock_storage = MagicMock()
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
# Configure mock to simulate some files exist in GCS
|
||||
def file_exists(path):
|
||||
# Simulate audioshake_abc123_raw.json exists
|
||||
return "audioshake_abc123_raw.json" in path
|
||||
|
||||
mock_storage.file_exists.side_effect = file_exists
|
||||
|
||||
stats = service.sync_cache_from_gcs(
|
||||
local_cache_dir=temp_dir,
|
||||
audio_hash="abc123",
|
||||
lyrics_hash="def456",
|
||||
)
|
||||
|
||||
# Should have downloaded 1 file (audioshake_abc123_raw.json)
|
||||
assert stats["downloaded"] == 1
|
||||
# Should have not found the rest
|
||||
assert stats["not_found"] > 0
|
||||
assert stats["errors"] == 0
|
||||
|
||||
# Verify download_file was called for the existing file
|
||||
mock_storage.download_file.assert_called_once()
|
||||
|
||||
def test_sync_cache_from_gcs_creates_directory(self, temp_dir):
|
||||
"""Should create the local cache directory if it doesn't exist."""
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.file_exists.return_value = False
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
cache_dir = os.path.join(temp_dir, "new_cache_dir")
|
||||
assert not os.path.exists(cache_dir)
|
||||
|
||||
service.sync_cache_from_gcs(
|
||||
local_cache_dir=cache_dir,
|
||||
audio_hash="abc123",
|
||||
lyrics_hash="def456",
|
||||
)
|
||||
|
||||
assert os.path.exists(cache_dir)
|
||||
|
||||
def test_sync_cache_from_gcs_handles_errors(self, temp_dir):
|
||||
"""Should handle download errors gracefully."""
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.file_exists.return_value = True
|
||||
mock_storage.download_file.side_effect = Exception("Network error")
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
stats = service.sync_cache_from_gcs(
|
||||
local_cache_dir=temp_dir,
|
||||
audio_hash="abc123",
|
||||
lyrics_hash="def456",
|
||||
)
|
||||
|
||||
# All files should have errors
|
||||
assert stats["errors"] > 0
|
||||
assert stats["downloaded"] == 0
|
||||
|
||||
|
||||
class TestSyncCacheToGCS:
|
||||
"""Tests for uploading cache to GCS."""
|
||||
|
||||
def test_sync_cache_to_gcs_uploads_new_files(self, temp_dir):
|
||||
"""Should upload files that don't exist in GCS."""
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.file_exists.return_value = False
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
# Create a cache file in temp_dir
|
||||
cache_file = os.path.join(temp_dir, "audioshake_abc123_raw.json")
|
||||
with open(cache_file, "w") as f:
|
||||
json.dump({"test": "data"}, f)
|
||||
|
||||
stats = service.sync_cache_to_gcs(
|
||||
local_cache_dir=temp_dir,
|
||||
audio_hash="abc123",
|
||||
lyrics_hash="def456",
|
||||
)
|
||||
|
||||
assert stats["uploaded"] == 1
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
def test_sync_cache_to_gcs_skips_existing_files(self, temp_dir):
|
||||
"""Should skip files that already exist in GCS."""
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.file_exists.return_value = True
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
# Create a cache file in temp_dir
|
||||
cache_file = os.path.join(temp_dir, "audioshake_abc123_raw.json")
|
||||
with open(cache_file, "w") as f:
|
||||
json.dump({"test": "data"}, f)
|
||||
|
||||
stats = service.sync_cache_to_gcs(
|
||||
local_cache_dir=temp_dir,
|
||||
audio_hash="abc123",
|
||||
lyrics_hash="def456",
|
||||
)
|
||||
|
||||
assert stats["uploaded"] == 0
|
||||
assert stats["skipped"] == 1
|
||||
mock_storage.upload_file.assert_not_called()
|
||||
|
||||
def test_sync_cache_to_gcs_ignores_unexpected_files(self, temp_dir):
|
||||
"""Should ignore files that don't match expected cache patterns."""
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.file_exists.return_value = False
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
# Create an unexpected file
|
||||
unexpected_file = os.path.join(temp_dir, "random_file.txt")
|
||||
with open(unexpected_file, "w") as f:
|
||||
f.write("not a cache file")
|
||||
|
||||
# Create a file with wrong hash
|
||||
wrong_hash_file = os.path.join(temp_dir, "audioshake_wrong_hash_raw.json")
|
||||
with open(wrong_hash_file, "w") as f:
|
||||
json.dump({"test": "data"}, f)
|
||||
|
||||
stats = service.sync_cache_to_gcs(
|
||||
local_cache_dir=temp_dir,
|
||||
audio_hash="abc123",
|
||||
lyrics_hash="def456",
|
||||
)
|
||||
|
||||
# Should not upload any files
|
||||
assert stats["uploaded"] == 0
|
||||
mock_storage.upload_file.assert_not_called()
|
||||
|
||||
def test_sync_cache_to_gcs_handles_missing_directory(self, temp_dir):
|
||||
"""Should handle missing cache directory gracefully."""
|
||||
mock_storage = MagicMock()
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
stats = service.sync_cache_to_gcs(
|
||||
local_cache_dir=os.path.join(temp_dir, "nonexistent"),
|
||||
audio_hash="abc123",
|
||||
lyrics_hash="def456",
|
||||
)
|
||||
|
||||
assert stats["uploaded"] == 0
|
||||
assert stats["skipped"] == 0
|
||||
assert stats["errors"] == 0
|
||||
|
||||
def test_sync_cache_to_gcs_handles_upload_errors(self, temp_dir):
|
||||
"""Should handle upload errors gracefully."""
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.file_exists.return_value = False
|
||||
mock_storage.upload_file.side_effect = Exception("Upload failed")
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
# Create a cache file
|
||||
cache_file = os.path.join(temp_dir, "audioshake_abc123_raw.json")
|
||||
with open(cache_file, "w") as f:
|
||||
json.dump({"test": "data"}, f)
|
||||
|
||||
stats = service.sync_cache_to_gcs(
|
||||
local_cache_dir=temp_dir,
|
||||
audio_hash="abc123",
|
||||
lyrics_hash="def456",
|
||||
)
|
||||
|
||||
assert stats["errors"] == 1
|
||||
assert stats["uploaded"] == 0
|
||||
|
||||
|
||||
class TestGCSPathStructure:
|
||||
"""Tests for GCS path structure."""
|
||||
|
||||
def test_gcs_cache_prefix(self):
|
||||
"""GCS cache prefix should be correct."""
|
||||
assert LyricsCacheService.GCS_CACHE_PREFIX == "lyrics-transcriber-cache/"
|
||||
|
||||
def test_sync_uses_correct_gcs_paths(self, temp_dir):
|
||||
"""Should use correct GCS paths when downloading/uploading."""
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.file_exists.return_value = True
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
service.sync_cache_from_gcs(
|
||||
local_cache_dir=temp_dir,
|
||||
audio_hash="abc123",
|
||||
lyrics_hash="def456",
|
||||
)
|
||||
|
||||
# Check that file_exists was called with correct GCS path prefix
|
||||
calls = mock_storage.file_exists.call_args_list
|
||||
for call in calls:
|
||||
gcs_path = call[0][0]
|
||||
assert gcs_path.startswith("lyrics-transcriber-cache/")
|
||||
|
||||
|
||||
class TestEndToEnd:
|
||||
"""End-to-end tests for cache sync workflow."""
|
||||
|
||||
def test_round_trip_cache_sync(self, temp_dir):
|
||||
"""Test uploading and then downloading the same cache file."""
|
||||
# We'll simulate this with two separate temp dirs
|
||||
upload_dir = os.path.join(temp_dir, "upload")
|
||||
download_dir = os.path.join(temp_dir, "download")
|
||||
os.makedirs(upload_dir)
|
||||
os.makedirs(download_dir)
|
||||
|
||||
# Create a test cache file
|
||||
test_data = {"transcription": "test data", "words": [1, 2, 3]}
|
||||
cache_filename = "audioshake_abc123_raw.json"
|
||||
upload_path = os.path.join(upload_dir, cache_filename)
|
||||
with open(upload_path, "w") as f:
|
||||
json.dump(test_data, f)
|
||||
|
||||
# Create mock that tracks uploaded files
|
||||
uploaded_files = {}
|
||||
|
||||
def mock_upload(local_path, gcs_path):
|
||||
with open(local_path, "r") as f:
|
||||
uploaded_files[gcs_path] = f.read()
|
||||
return gcs_path
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
if gcs_path in uploaded_files:
|
||||
with open(local_path, "w") as f:
|
||||
f.write(uploaded_files[gcs_path])
|
||||
return local_path
|
||||
|
||||
def mock_exists(gcs_path):
|
||||
return gcs_path in uploaded_files
|
||||
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.upload_file.side_effect = mock_upload
|
||||
mock_storage.download_file.side_effect = mock_download
|
||||
mock_storage.file_exists.side_effect = mock_exists
|
||||
|
||||
service = LyricsCacheService(storage=mock_storage)
|
||||
|
||||
# Upload
|
||||
upload_stats = service.sync_cache_to_gcs(upload_dir, "abc123", "def456")
|
||||
assert upload_stats["uploaded"] == 1
|
||||
|
||||
# Download to new location
|
||||
download_stats = service.sync_cache_from_gcs(download_dir, "abc123", "def456")
|
||||
assert download_stats["downloaded"] == 1
|
||||
|
||||
# Verify downloaded content matches
|
||||
downloaded_path = os.path.join(download_dir, cache_filename)
|
||||
with open(downloaded_path, "r") as f:
|
||||
downloaded_data = json.load(f)
|
||||
assert downloaded_data == test_data
|
||||
Reference in new issue
Block a user