Compare commits

...
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude 16e8c25d22 fix(email): Fix YouTube URL missing in job completion email (#229)
## Summary
- Fix race condition where YouTube URL was missing from job completion
emails
- Remove '🎤 Nomad Karaoke' header from email templates
- Add custom HTML email signature to job completion and action reminder
emails

## Problem
When a job completed, the automatic completion email was sent with
"[YouTube URL not available]" instead of the actual YouTube link. This
was because:
1. The job transitioned to COMPLETE status
2. During the transition, `_trigger_state_notifications()` was called
3. The completion email was sent using the current job data
4. **BUT** the youtube_url was being stored in state_data AFTER the
transition

## Solution
Move the `update_job()` call that stores `youtube_url`, `dropbox_link`,
and other metadata BEFORE calling `transition_to_state(COMPLETE)`. This
ensures the data is in the database when the completion email is
triggered.

Fixed in both code paths:
- Orchestrated path (`generate_video_orchestrated`)
- Legacy path (`generate_video_legacy`)

## Test plan
- [x] Backend tests pass (1071 passed)
- [ ] Deploy and verify next job completion email contains YouTube URL
- [ ] Verify email signature renders correctly

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-07 04:42:44 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 8e602a8173 feat(frontend): Add encoding worker status to footer (#228)
## Summary

Display encoding worker health and version in the footer alongside
frontend and backend versions, providing visibility into whether the
high-performance encoder is healthy and up-to-date.

## Changes

- **Backend**: Add `GET /health/encoding-worker` lightweight endpoint
for frontend consumption
- **Encoding Worker**: Enhance `/health` endpoint to include
`wheel_version` (karaoke-gen package version)
- **Frontend**: Update `VersionFooter` to fetch and display encoder
status with visual indicator
- **Version**: Bump to 0.95.1

## Footer Display

When healthy:
```
Frontend: v0.95.1 | Backend: v0.95.1 | ● Encoder: v0.95.1
```

With active jobs:
```
Frontend: v0.95.1 | Backend: v0.95.1 | ● Encoder: v0.95.1 (2 active)
```

When offline:
```
Frontend: v0.95.1 | Backend: v0.95.1 | ○ Encoder offline
```

## Testing

- [x] Frontend builds successfully
- [x] Backend tests pass (1071 passed)
- [x] TypeScript compiles without errors in changed files

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 23:18:25 -05:00
Andrew Beveridge ☄️andClaude 2783c2cfcf feat(audio-search): Add display artist/title override for search (#227)
## Summary
- Add optional "Display As" fields to audio search, allowing users to
search for audio by one artist/title but display a different
artist/title on title screens and filenames
- Useful for Broadway casts, covers, remixes where tracker metadata
differs from desired display

## Changes
- Add `display_artist`/`display_title` fields to `AudioSearchRequest`
model
- Update job creation to use display values for output (title screens,
filenames, YouTube, distribution)
- Add "Display As" optional fields to frontend search UI
- Update API documentation

## Example Use Case
- **Search For**: Artist = "Jeremy Kushnier", Title = "I Can't Stand
Still"
- **Display As**: Artist = "Footloose (Broadway Cast)"
- **Result**: Audio found using "Jeremy Kushnier", but title screen
shows "Footloose (Broadway Cast) - I Can't Stand Still"

## Test plan
- [ ] Search with display override → verify title screen shows display
values
- [ ] Search without display override → verify search values used
(backwards compatible)
- [ ] Verify filenames use display values

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-06 22:36:27 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 f6e91b0014 fix(infrastructure): Increase systemd startup timeout for wheel install (#226)
## Summary

Fix encoding worker startup timeout issue discovered after #225
deployment.

## Problem

The ExecStartPre script that installs the karaoke-gen wheel can take 2+
minutes on first boot. The default systemd timeout (90s) caused multiple
retries before eventually succeeding.

## Solution

Add `TimeoutStartSec=300` to the systemd service definition to allow up
to 5 minutes for the startup script to complete.

## Test Plan

- [ ] Merge triggers Packer image rebuild
- [ ] New image has longer timeout configured
- [ ] Next VM recreation starts without timeout retries

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 15:18:57 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 4b58e9c040 feat(infrastructure): Add Packer custom image for encoding worker (#225)
## Summary

- Reduce encoding worker VM startup time from **~10 minutes to ~30
seconds** by pre-installing dependencies in a custom GCE image
- Add Packer template and provisioning script for building the custom
image
- Simplify startup script from ~700 lines to ~30 lines

## Changes

### New Files
- `infrastructure/packer/encoding-worker.pkr.hcl` - Packer template
- `infrastructure/packer/scripts/provision.sh` - Provisioning script
- `infrastructure/packer/README.md` - Documentation

### Modified Files
- `infrastructure/compute/encoding_worker_vm.py` - Use custom image
family `encoding-worker`
- `infrastructure/compute/startup_scripts/encoding_worker.sh` -
Simplified to just start service

## What's Pre-installed in the Image

| Component | Location |
|-----------|----------|
| Python 3.13.1 (PGO+LTO) | `/opt/python313` |
| FFmpeg 7.x | `/usr/local/bin/ffmpeg` |
| Noto fonts (incl. CJK) | System fonts |
| Virtual environment | `/opt/encoding-worker/venv` |
| Systemd service | `encoding-worker.service` |

## Startup Time Improvement

| Phase | Before | After |
|-------|--------|-------|
| Python 3.13 build | ~7 min | 0 |
| FFmpeg install | ~1 min | 0 |
| System packages | ~1 min | 0 |
| API key + wheel | ~30 sec | ~30 sec |
| **Total** | **~10 min** | **~30 sec** |

## Test Plan

- [x] Packer image builds successfully (`encoding-worker-1767723849`)
- [x] `pulumi preview` shows expected changes (VM replacement only)
- [ ] After merge: Verify VM starts in ~30 seconds
- [ ] After merge: Verify encoding jobs complete successfully

## Pulumi Preview

```
+- gcp:compute:Instance encoding-worker replace [diff: ~bootDisk,metadataStartupScript]
Resources: +-1 to replace, 131 unchanged
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 14:21:56 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 ad37306704 refactor(infrastructure): Modularize Pulumi IaC from 2600-line monolith (#223)
## Summary

- Refactor monolithic `infrastructure/__main__.py` from **2,602 lines to
339 lines** (87% reduction)
- Split resources into organized, purpose-specific modules following
SOLID principles
- Extract VM startup scripts to separate `.sh` files for easier
maintenance
- Extract embedded Python encoding worker app to proper package

## Changes

### New Module Structure
```
infrastructure/
├── __main__.py              # Entry point (339 lines, was 2,602)
├── config.py                # Centralized configuration
├── modules/                 # Core GCP resources
│   ├── database.py          # Firestore + indexes
│   ├── storage.py           # GCS bucket + lifecycle
│   ├── artifact_registry.py # Docker repository
│   ├── secrets.py           # Secret Manager secrets
│   ├── cloud_tasks.py       # Task queues
│   ├── cloud_run.py         # Domain mapping, Cloud Run Jobs
│   ├── monitoring.py        # Alert policies
│   └── iam/                 # Service accounts by purpose
├── compute/                 # VM-based resources
│   ├── flacfetch_vm.py
│   ├── encoding_worker_vm.py
│   ├── github_runners.py
│   └── startup_scripts/     # External .sh files
└── docs/
    └── PHASE5-PACKER-IMAGE-PLAN.md  # Future optimization plan
```

### Extracted Code
- **Startup scripts**: Moved from embedded strings to separate `.sh`
files
- **GCE encoding service**: Extracted ~600-line Python app to
`backend/services/gce_encoding/`

### Documentation
- Updated `infrastructure/README.md` with new structure
- Added `docs/PHASE5-PACKER-IMAGE-PLAN.md` for future Packer image
optimization

## Verification

```bash
pulumi preview --diff
# Result: 120 resources unchanged
# Only 10 GitHub runner replacements (expected - startup script improvements from main)
```

## Test Plan

- [x] `pulumi preview` shows no unintended infrastructure changes
- [x] All module imports resolve correctly
- [x] Startup scripts produce identical output when read from files

## Notes

- Phase 5 (Packer custom images) documented but not implemented -
reduces encoding worker startup from ~10min to ~30sec
- `deploy.sh` for gdrive_validator kept - handles source upload to GCS
which Pulumi doesn't do automatically

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 18:01:07 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 0170e88652 refactor: Unify preview encoding with LocalPreviewEncodingService (#222)
## Summary
- Create `LocalPreviewEncodingService` as single source of truth for
preview video encoding
- Update GCE worker to use the service from the installed wheel (like
`LocalEncodingService`)
- Update `VideoGenerator` to delegate preview encoding to the new
service
- Remove ~140 lines of duplicated FFmpeg logic

## Changes
- **New:** `backend/services/local_preview_encoding_service.py` -
Standalone preview encoding service with NVENC support, FFmpeg filter
path escaping, and font directory support
- **New:** `backend/tests/test_local_preview_encoding_service.py` - 37
comprehensive tests
- **Modified:** `infrastructure/__main__.py` - GCE worker now uses
`LocalPreviewEncodingService` from wheel
- **Modified:**
`lyrics_transcriber_temp/lyrics_transcriber/output/video.py` - Delegates
to `LocalPreviewEncodingService`, removed duplicate
`_build_preview_ffmpeg_command`
- **Modified:** `docs/README.md`, `docs/LESSONS-LEARNED.md` -
Documentation updates

## Why This Matters
Previously, preview encoding had THREE implementations:
1. `VideoGenerator._build_preview_ffmpeg_command` (local CLI)
2. `run_preview_encoding` in GCE worker (hardcoded settings)
3. ~~Inline FFmpeg in old code~~ (already removed)

This created risk of inconsistent preview videos between environments.
Now there's ONE source of truth.

## Testing
- [x] 37 new tests for `LocalPreviewEncodingService` pass
- [x] 28 existing `LocalEncodingService` tests pass
- [x] 22 `VideoGenerator` tests pass  
- [x] Full backend test suite: 1042 passed

## Review
- [x] CodeRabbit CLI attempted (rate limited, tests verified instead)
- [x] Documentation reviewed and updated

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 12:50:17 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 4ad8018773 infra: Grant Editor role to GitHub Actions service account (#224)
## Summary
- Adds `roles/editor` to the GitHub Actions deployer service account
- Prevents permission errors when adding new GCP resources to
infrastructure
- Fixes the failed workflow that couldn't create `idle-reminder-queue`
due to missing `cloudtasks.queues.create` permission

## Test plan
- [x] Manually bootstrapped the permission via `gcloud` CLI
- [ ] Verify CI passes after merge (the Editor role is already applied)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 12:06:42 -05:00
Andrew Beveridge ☄️andClaude 560c3e4904 feat: Add email notification system with admin controls (#220)
## Summary

- Add automated email notification system for job completion and action
reminders
- Admin UI buttons to copy completion message or send email directly
- Comprehensive test suite (81 new tests)
- **Email subjects include release ID and song info** for easy
identification

## Changes

### Backend
- **Email Service** (`email_service.py`): Extended with SendGrid CC
support, `send_job_completion()` and `send_action_reminder()` methods
- **Job Notification Service** (`job_notification_service.py`): New
service orchestrating completion emails and action reminder scheduling
- **Template Service** (`template_service.py`): GCS-backed HTML email
templates with fallback defaults
- **Worker Service** (`worker_service.py`): Cloud Tasks integration for
delayed idle reminders (5-min delay)
- **Job Manager** (`job_manager.py`): State transition triggers for
automated notifications
- **Admin Routes** (`admin.py`): New endpoints `GET
/api/admin/jobs/{id}/completion-message` and `POST
/api/admin/jobs/{id}/send-completion-email`
- **Internal Routes** (`internal.py`): Idle reminder check endpoint for
Cloud Tasks

### Email Subject Format
Email subjects now include brand code and song info for easy
identification:
- With brand code: `NOMAD-1178: Seether - Tonight (Your karaoke video is
ready!)`
- Without brand code: `Seether - Tonight (Your karaoke video is ready!)`
- Without song info: `Your karaoke video is ready!`

### Frontend
- **OutputLinks Component**: Admin-only "Copy Message" and "Send Email"
buttons with dialog UI
- **E2E Helpers** (`email-testing.ts`): `waitForCompletionEmail()` and
`isCompletionEmail()` methods for testmail.app integration
- **Happy Path Test**: Step 9.5 verifies completion email arrives after
job completion

### Documentation
- Updated API.md with new endpoints
- Updated ARCHITECTURE.md with notification system architecture
- Updated README.md Recent Changes

## Testing

- [x] 81 new unit tests covering:
- `test_email_service.py` (27 tests) - providers, CC support, job
completion/reminder methods
- `test_job_notification_service.py` (21 tests) - URL building, message
generation
- `test_job_manager_notifications.py` (15 tests) - state transition
triggers
  - `test_admin_email_endpoints.py` (12 tests) - admin API endpoints
- `test_template_service.py` (6 tests) - GCS template loading, fallbacks
- [x] E2E test extended to verify completion email delivery
- [x] All backend tests pass (`make test`)
- [x] TypeScript compiles

## Feature Flags

- `ENABLE_AUTO_EMAILS` (default: `false`) - Controls automatic email
sending on job state changes

## Review

- [x] Local CodeRabbit review completed (3 cycles)
- [x] Review feedback addressed:
  - Fixed potential None state_data in internal.py
  - Masked user emails in logs (PII protection)
  - Fixed React setTimeout memory leak
  - Changed placeholder URL default to empty
  - Added CC email deduplication

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-06 11:53:11 -05:00
Andrew Beveridge ☄️andClaude 07cde54b3f feat: Upgrade GCE encoding worker to Python 3.13 (#221)
## Summary

Upgrades the GCE encoding worker VM from Debian's default Python 3.11 to
Python 3.13 built from source, aligning it with CI and Cloud Run Python
version.

## Changes

- Build Python 3.13.1 from source with `--enable-optimizations
--with-lto` for best performance
- Install to `/opt/python313` on the Debian 12 VM
- Create virtual environment using Python 3.13 instead of system Python
3.11
- Add build dependencies (build-essential, libssl-dev, zlib1g-dev, etc.)
- Remove `python3-pip` and `python3-venv` apt packages (no longer
needed)
- Update docs/README.md with recent changes entry

## Testing

- [x] Pulumi preview shows only encoding-worker VM replacement
(expected)
- [x] Backend tests pass (`make test` - 1006 passed)
- [x] Frontend tests pass (`npm run test:all` - 65 passed)
- [x] Python and bash syntax validation pass

## Review

- [x] Local CodeRabbit review completed
- [x] No actionable issues found

## Notes

- The VM will be **replaced** (not updated in-place) when this deploys
- Python build from source takes ~5-10 minutes during VM startup
- Brief encoding downtime expected during deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-06 10:43:47 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 bd26d742a2 fix: Include backend package in wheel for GCE worker LocalEncodingService (#219)
## Summary

- Include `backend` package in pyproject.toml packages list for wheel
distribution
- This enables the GCE encoding worker to import `LocalEncodingService`

## Problem

The GCE encoding worker failed with:
```
ERROR:__main__:LocalEncodingService not available: No module named 'backend'
```

The wheel was being installed correctly, but `backend/` wasn't included
in the package.

## Changes

- Added `{ include = "backend" }` to packages list in pyproject.toml
- Bumped version to 0.93.0

## Test Plan

- [x] Verify pyproject.toml changes are correct
- [ ] CI builds wheel with backend included
- [ ] GCE worker downloads new wheel at job start
- [ ] E2E happy path test passes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 09:58:39 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 f308a7231c fix: Upgrade github-runner storage role to objectAdmin (#218)
## Summary

Wheel upload needs to overwrite existing files which requires
`storage.objects.delete` permission. Changed from `objectCreator` to
`objectAdmin` role.

## Previous Error
```
AccessDeniedException: 403 github-runner@nomadkaraoke.iam.gserviceaccount.com does not have storage.objects.delete access
```

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 09:09:04 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 53ade5b9dc fix: Pre-install wheel at VM startup and increase install timeout (#217)
## Summary

Fixes GCE encoding worker failing because wheel installation timed out
after 2 minutes.

**Root cause**: The karaoke-gen wheel (4.4MB) has many dependencies
(pytorch, langchain, etc.) that take several minutes to resolve and
install.

## Changes

1. **Pre-install wheel during VM startup** - Added to bash startup
script so wheel is installed before the service starts
2. **Increased runtime install timeout** - From 2 to 5 minutes for hot
code updates at job start

## Why This Works

- VM startup: Full install with all dependencies (takes ~3-4 minutes,
but that's OK during boot)
- Job start: Quick check for newer wheel, reinstall only if needed
(usually fast since deps cached)

## Test Plan

- [x] Syntax check passes
- [ ] CI passes
- [ ] After merge + Pulumi deploy: Restart GCE worker, trigger E2E happy
path test

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 09:03:20 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 c3b8cf880d fix: CI failures - escape triple quotes and add storage permission (#216)
## Summary

Fixes two CI failures from PR #214 merge:

1. **SyntaxError in Pulumi deployment**: The embedded Python code in the
GCE startup script used `"""` docstrings which prematurely terminated
the outer triple-quoted string. Changed to `'''` single-quoted
docstrings.

2. **Storage permission denied**: `github-runner` service account lacked
`storage.objects.create` permission to upload wheels to GCS. Added
`roles/storage.objectCreator` IAM binding.

## Changes

- Changed 3 docstrings in embedded Python code from `"""` to `'''`
- Added `github_runner_storage_iam` IAM binding with
`roles/storage.objectCreator`

## Test Plan

- [x] `python3 -m py_compile infrastructure/__main__.py` passes
- [x] `make test` passes locally
- [ ] CI should pass with these fixes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 01:39:30 -05:00
Andrew Beveridge ☄️ 4ebf811c3f Update pyproject.toml (#215) 2026-01-06 06:26:32 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 b345135be1 feat: Unify GCE encoding with LocalEncodingService (single code path) (#214)
## Summary
- Unifies GCE encoding worker with LocalEncodingService (same code as
local CLI)
- Removes all fallback/simplified encoding logic
- Produces proper filenames: `Artist - Title (Final Karaoke Lossless
4k).mp4`
- Includes concatenated title + karaoke + end screens
- CI uploads wheel to GCS for hot code updates without restart

## Problem
Cloud/remote karaoke generation was using simplified, duplicated
encoding logic that:
- Used generic filenames like `output_4k_lossless.mp4`
- Did NOT concatenate title + karaoke + end screens
- Was missing MKV format for YouTube uploads

## Solution
GCE worker now:
1. Downloads latest karaoke-gen wheel from GCS at job start
2. Imports and uses `LocalEncodingService` (same as local CLI)
3. Produces identical output to local karaoke-gen

## Test plan
- [ ] Run a cloud generation job after merging
- [ ] Verify output files have proper names
- [ ] Verify videos include title/end screens
- [ ] Verify Dropbox/GDrive uploads work correctly

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 01:25:45 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 1114bd5554 fix: Use FFmpeg filter escaping instead of shell escaping for paths (#213)
## Summary
- Fixes video generation for songs with apostrophes in titles (e.g.,
"I'm With You")
- PR #212 introduced shell-style escaping (`'\''`) which doesn't work
with subprocess command lists
- Changed to FFmpeg's native filter escaping rules

## Test plan
- [x] All 22 video.py unit tests pass
- [x] All 68 backend tests pass  
- [x] End-to-end FFmpeg test with apostrophe in path succeeds
- [ ] Manual test with "Avril Lavigne - I'm With You"

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 23:45:27 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 60af90de31 fix: Escape FFmpeg filter paths using single-quote wrapping (#212)
## Summary
- Fix FFmpeg failing on paths with apostrophes (e.g., "I'm With You")
- Use single-quote wrapping with `'\''` pattern for embedded quotes
- Update both `video.py` and `video_background_processor.py`

## Test plan
- [x] Unit tests updated and passing (1006 tests)
- [x] E2E tests passing (68 tests)
- [x] Manually tested with "Avril Lavigne - I'm With You" - video
generates successfully

## Technical details
FFmpeg filter syntax interprets apostrophes as special characters.
Simple backslash escaping (`\'`) doesn't work because FFmpeg's filter
parser consumes the backslash. The solution is to wrap paths in single
quotes and escape embedded quotes using the `'\''` pattern (end quote,
literal quote, start quote).

Example: `I'm With You` becomes `'I'\''m With You'`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 03:50:24 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 2110d0f84b perf: Optimize agentic correction with parallel gap processing (~5-6x speedup) (#211)
## Summary

- Fix anti-pattern of creating new LLM model instance per gap (caused
repeated 2s warm-up overhead)
- Create agentic corrector once and reuse for all gaps
- Process gaps in parallel using ThreadPoolExecutor (default 5 workers)
- Remove unnecessary warm-up code from langchain_bridge.py
- Add `AGENTIC_MAX_PARALLEL_GAPS` env var to configure parallelism

**Performance improvement**: ~5-6x speedup (20 gaps: ~55s vs ~5+ minutes
previously)

## Test plan

- [x] Ran local test with 20 gaps - completed in ~55 seconds
- [x] Verified parallel processing logs show concurrent gap processing
- [x] Verified model initialization only happens once at start
- [x] CodeRabbit review passed locally

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 22:43:13 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 4b0dd6169f ci: Propagate base image to all runners after rebuild (#210)
## Summary
- Adds new `propagate-base-image` job that runs after base image rebuild
- Spawns 20 parallel jobs to pull the new base image to all self-hosted
runners
- Eliminates ~8 minute pull delay on subsequent app builds

## Problem
When the base image is rebuilt (due to `poetry.lock` or
`Dockerfile.base` changes), other runners still have the old image
cached. The first app build on each runner must pull the new ~5GB image,
adding ~8 minutes to deployment time.

Example:
https://github.com/nomadkaraoke/karaoke-gen/actions/runs/20735879195/job/59532914508

## Solution
After base image is pushed, trigger 20 parallel jobs that pull the new
image to all runners immediately. Future app builds will already have
the latest base image cached.

## Implementation
- Added `base_rebuilt` output to `deploy-backend` job
- New `propagate-base-image` job with matrix strategy (20 parallel jobs)
- `fail-fast: false` so one offline runner doesn't block others
- Uses same Workload Identity auth as other deployment jobs

## Test plan
- [ ] Verify workflow syntax is valid (YAML parses correctly)
- [ ] Merge and observe next base image rebuild triggers propagation
- [ ] Confirm subsequent app builds don't need to pull base image

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 21:54:55 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 8e3e3e4713 fix: Check base image hash from registry instead of stale local cache (#209)
## Summary
- Fix GitHub Actions base image cache check to query registry directly
instead of trusting potentially stale local Docker cache
- Prevent unnecessary base image rebuilds caused by runners with
outdated cached images
- Ensure app image is always built on the correct base image

## Changes
- Fetch image hash label via lightweight registry API (~10KB config
blob) instead of pulling full image
- Add step to verify local cache matches registry before building app
image
- Compare config digests consistently for reliable image matching

## Problem
GitHub Actions runners can retain Docker images between runs. When the
runner's local cache has an old base image, the previous logic would:
1. Check local cache first → find old image with wrong hash
2. Conclude base image needs rebuild → unnecessary ~10 min rebuild
3. Or worse: build app image on stale base if hashes happened to match
older state

## Testing
- [x] Verified registry API returns correct hash label
- [x] CodeRabbit CLI review completed locally

## Review
- [x] Local CodeRabbit review completed
- [x] Review feedback addressed (fixed digest comparison types)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 21:27:02 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 fef9a45739 docs: Add Pulumi CI skip-preview lesson learned (#208)
## Summary
- Add Pulumi CI skip-preview lesson to `LESSONS-LEARNED.md`
- Update `CLAUDE.md` to emphasize infrastructure changes require PRs
- Add detailed Infrastructure section to `DEVELOPMENT.md` documenting:
  - What's managed by Pulumi
  - The PR-based workflow
  - Why not to run `pulumi up` locally

## Changes
| File | Change |
|------|--------|
| `docs/LESSONS-LEARNED.md` | Added "Pulumi CI Deployments Need
Skip-Preview" section |
| `CLAUDE.md` | Strengthened Infrastructure section with PR workflow |
| `docs/DEVELOPMENT.md` | Added comprehensive Infrastructure Changes
section |

## Test plan
- [x] Documentation only - no code changes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 21:09:45 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 69b87c232a fix: Add IAM permissions for Pulumi to refresh state in CI (#207)
## Summary
- Add `roles/resourcemanager.projectIamAdmin` for IAM binding state
refresh
- Add `roles/secretmanager.admin` for secret state refresh
- Use `--skip-preview` instead of `--refresh` to avoid needing read
permissions on all GCP resource types

The service account would need many additional roles (Cloud Tasks,
Firestore, Monitoring, etc.) for Pulumi to refresh all resources. Using
`--skip-preview` is the standard approach for CI/CD where Pulumi state
is authoritative.

## Test plan
- [ ] CI passes on this PR
- [ ] After merge, Pulumi deployment succeeds

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 18:57:10 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 0530e5f226 fix: Use correct Pulumi stack name in CI infrastructure deploy (#206)
## Summary
- Fixed Pulumi stack reference from `nomadkaraoke/karaoke-gen/prod` to
`nomadkaraoke/karaoke-gen-infrastructure/prod` (matches project name in
`Pulumi.yaml`)
- Added GCP Workload Identity authentication for Pulumi deployment step

## Test plan
- [ ] CI passes on this PR
- [ ] After merge, infrastructure deployment job succeeds

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 15:34:00 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 d8710bc3d4 ci: Add infrastructure deployment job for GCE encoding worker (#205)
## Summary
Adds a Pulumi deployment job that runs on main branch pushes to
auto-deploy infrastructure changes.

## Problem
Infrastructure changes (like GCE encoding worker code) weren't being
deployed automatically. This caused issues where the backend was
deployed with new code but the GCE worker still had old code.

## Solution
New `deploy-infrastructure` job that:
- Runs Pulumi to deploy infrastructure changes
- Restarts the encoding worker service to pick up new code

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 15:07:41 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 c83ac4376c fix: Handle cached preview responses from GCE worker (#204)
## Summary
- Fixes issue where subsequent preview requests hang indefinitely
- GCE worker now returns "cached" status for already-encoded previews
instead of 409 Conflict
- Backend handles cached response by returning immediately without
polling

## Problem
When users clicked preview multiple times without changing lyrics:
1. First preview worked correctly
2. Subsequent previews hung forever because GCE returned 409 Conflict
3. Backend raised an error instead of handling the cached case

## Solution
- GCE `/encode-preview` endpoint now returns `{"status": "cached", ...}`
for completed jobs
- Backend `encode_preview_video()` checks submit response and returns
immediately for cached
- Also handles "in_progress" status when another request is already
encoding

## Test plan
- [x] First preview generates correctly
- [ ] Subsequent previews (unchanged lyrics) return cached video
instantly
- [ ] Changed lyrics generate new preview video

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 19:47:35 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 e70ce2b558 chore: Enable USE_GCE_PREVIEW_ENCODING in production CI (#203)
## Summary
Enable the GCE preview encoding feature in production CI deployments.

## Changes
- Add `USE_GCE_PREVIEW_ENCODING=true` to Cloud Run environment variables
in CI deploy

## Note
To enable immediately (before next deploy), run:
```bash
gcloud run services update karaoke-backend \
  --region us-central1 \
  --project nomadkaraoke \
  --update-env-vars "USE_GCE_PREVIEW_ENCODING=true"
```

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 18:17:40 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 506db844c2 fix: Double-escape backslashes in GCE embedded Python code (#202)
## Summary
Fix syntax error in GCE encoding worker startup script caused by
incorrect backslash escaping.

## Problem
The `escape_ffmpeg_filter_path` function in the GCE worker startup
script had a syntax error:
```
SyntaxError: unexpected character after line continuation character
```

The issue was that backslashes in the embedded Python code weren't
properly escaped for the triple-quoted string context in Pulumi.

## Solution
Double the backslashes in the Pulumi source:
- 4 backslashes in source → 2 in string value → 2 on VM (valid Python)

## Testing
- GCE encoding worker now starts successfully
- Health endpoint responds correctly
- Already deployed via `pulumi up` and verified working

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 02:03:33 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 be04d3efaf feat: Offload preview video encoding to GCE for faster generation (#200)
## Summary
- Move preview video encoding from Cloud Run to GCE encoding worker
- Reduce preview generation time from 60+ seconds to ~15-20 seconds
- Improve UX during lyrics review by dramatically speeding up preview
generation

## Changes
- **GCE Worker** (`infrastructure/__main__.py`): Add `/encode-preview`
endpoint with `EncodePreviewRequest` model, FFmpeg command for 480x270
preview encoding, and FFmpeg filter path escaping
- **EncodingService** (`backend/services/encoding_service.py`): Add
`is_preview_enabled` property, `submit_preview_encoding_job()`, and
`encode_preview_video()` methods
- **Config** (`backend/config.py`): Add `USE_GCE_PREVIEW_ENCODING`
setting (default: false)
- **Review Route** (`backend/api/routes/review.py`): Modify
`generate_preview_video()` to use GCE with fallback to local encoding
- **LyricsTranscriber** (`operations.py`, `generator.py`): Add
`ass_only` parameter to generate ASS subtitles without video encoding
- **Documentation**: Update ARCHITECTURE.md and LESSONS-LEARNED.md

## Technical Approach
The optimization splits preview generation into two phases:
1. **ASS Generation (Cloud Run)** - Fast text processing (~2s)
2. **Video Encoding (GCE)** - Offloaded to C4-standard-8 VM (~15-20s)

If GCE encoding fails or is disabled, falls back to local Cloud Run
encoding.

## Testing
- [x] Unit tests pass (1004 tests)
- [x] Emulator tests pass (68 tests)
- [x] CodeRabbit CLI review completed locally

## Deployment
1. Deploy infrastructure changes (`pulumi up` or restart GCE worker)
2. Deploy backend to Cloud Run
3. Enable via `USE_GCE_PREVIEW_ENCODING=true` in Cloud Run env

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 01:04:35 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 f2639754a9 fix: Download GCE-encoded files locally for YouTube upload (#201)
## Summary
- Fix YouTube upload failing when using GCE encoding backend
- GCE encoding returns GCS blob paths, but YouTube upload expected local
files
- Added `_download_gce_encoded_files()` to download encoded videos from
GCS before distribution

## Root Cause
Job `62a95e01` completed but didn't upload to YouTube. Investigation
showed:
1. GCE encoding stores files in GCS and returns blob paths like
`jobs/{id}/finals/output_4k_lossless.mp4`
2. The orchestrator stored these paths directly without downloading
3. YouTube upload checks `os.path.isfile()` which returns `False` for
GCS paths
4. Result: "No video file available for YouTube upload" warning

## Changes
| File | Change |
|------|--------|
| `backend/workers/video_worker_orchestrator.py` | Added
`_download_gce_encoded_files()` method called after GCE encoding |
| `backend/tests/test_video_worker_orchestrator.py` | Added 2 tests for
GCE download behavior |
| `docs/LESSONS-LEARNED.md` | Documented the issue and fix |

## Test plan
- [x] All 30 orchestrator tests pass
- [x] New test `test_run_encoding_gce_downloads_files` verifies GCE
paths are downloaded
- [x] New test `test_run_encoding_local_does_not_download` verifies
local encoding skips download
- [ ] Manual test: Create job via web UI and verify YouTube upload works

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 05:51:52 +00:00
Andrew Beveridge ☄️ 94182277fa Update pyproject.toml (#199) 2026-01-05 04:14:37 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 e16da0fa5f fix: Allow explicit override of YouTube upload server default (#198)
## Summary
- Changed `enable_youtube_upload` from `bool=False` to
`Optional[bool]=None` in request models
- Allows explicit `false` to override server default when
`DEFAULT_ENABLE_YOUTUBE_UPLOAD=true`
- E2E tests now intercept requests to disable YouTube upload for test
jobs

## Changes
- **Request models**: Changed to `Optional[bool] = None` to support
three states (None/True/False)
- **API routes**: All job creation endpoints now compute effective value
using `if is not None else` pattern
- **E2E test**: Added Playwright request interception to set
`enable_youtube_upload=false` for test jobs
- **Tests**: Updated to expect `None` instead of `False` for request
model defaults
- **Docs**: Added lesson learned about Python `or` operator gotcha with
boolean defaults

## Testing
- [x] Backend tests pass (`make test`)
- [x] No TypeScript errors in frontend changes

## Review
- [x] CodeRabbit CLI review completed locally
- [x] No issues found

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 23:04:06 -05:00
Andrew Beveridge ☄️andClaude b8e15b2af5 fix: Ensure user data is fetched after client-side navigation to /app (#197)
## Summary

- Fix E2E happy path test failure at Step 3 (Verify Credits)
- After beta enrollment, client-side navigation to `/app` didn't fetch
user data
- Credits weren't displayed because zustand auth store had `user: null`

## Changes

- **frontend/app/app/page.tsx**: Added `useEffect` that calls
`fetchUser()` when there's a token but no user data loaded. This ensures
user data is fetched regardless of how the user navigated to the page
(page reload vs client-side navigation).

- **frontend/e2e/regression/authentication.spec.ts**: Added new test
`'beta enrollment shows credits after redirect to app'` that
specifically verifies credits are displayed after the beta enrollment
flow.

- **docs/LESSONS-LEARNED.md**: Documented the root cause (zustand
module-level initialization only runs once, not on client-side
navigation) for future reference.

## Root Cause

The auth module's initialization code (`fetchUser()` call) only runs
once when the module is first imported. After beta enrollment,
`router.push('/app')` performs client-side navigation which doesn't
reload modules. The user data was never fetched, leaving `user: null` in
the store.

## Testing

- [x] New regression test passes locally (9/9 auth tests pass)
- [x] Build succeeds
- [x] CodeRabbit CLI review completed locally

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 21:33:01 -05:00
Andrew Beveridge ☄️ 2ee3cabc92 Added admin API script (#196) 2026-01-05 01:33:17 +00:00
99 changed files with 17226 additions and 7890 deletions

No files matched your search

@@ -0,0 +1,218 @@
# Plan: Job Email Notifications & Admin Message Tools
**Created:** 2026-01-06
**Branch:** feat/sess-20260106-0116-add-email-button
**Status:** Draft
## Overview
Add comprehensive email notification system for karaoke-gen that:
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
2. **Uses cloud-stored templates** for the job completion message
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
## Requirements
### Functional Requirements
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
- [ ] Store email templates in GCS (editable without code deploy)
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
### Non-Functional Requirements
- [ ] Email sending must not block job completion (async/background)
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
- [ ] Template changes should take effect without restart (no long-term caching)
## Technical Approach
### 1. Template Storage (GCS)
Store templates in `gs://karaoke-gen-storage/templates/`:
- `job-completion.txt` - Plain text template for job completion
- `job-completion.html` - HTML version (optional, for rich emails)
- `action-needed-lyrics.txt` - Reminder for lyrics review
- `action-needed-instrumental.txt` - Reminder for instrumental selection
**Template Variables:**
- `{name}` - User's display name or "there" if unknown
- `{youtube_url}` - YouTube video URL
- `{dropbox_url}` - Dropbox folder URL
- `{artist}` - Artist name
- `{title}` - Song title
- `{job_id}` - Job ID
- `{review_url}` - Lyrics review URL (for action-needed emails)
- `{instrumental_url}` - Instrumental selection URL
- `{feedback_url}` - Feedback form URL (for completion emails)
### 2. Email Service Extensions
Extend `backend/services/email_service.py`:
- Add `send_job_completion()` method with CC support
- Add `send_action_reminder()` method
- Add `render_template()` helper to fetch & render GCS templates
### 3. Auto-Email Triggers
**Job Completion:**
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
- Implementation: Call email service async in background task
**Action Reminders (5-min idle detection):**
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
- Record `blocking_state_entered_at` timestamp in `state_data`
- Schedule a Cloud Tasks task for 5 minutes later
- The scheduled task checks if job is still in blocking state and user hasn't started review
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
### 4. Admin UI (Frontend)
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
- "Copy Message" button - fetches rendered template, copies to clipboard
- "Send Email" button - opens dialog for email input, calls API to send
### 5. Backend API Endpoints
New endpoints in `backend/api/routes/admin.py`:
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
New internal endpoint for scheduled reminder:
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
## Implementation Steps
### Phase 1: Template Infrastructure
1. [ ] Upload initial template to GCS from user's existing template file
2. [ ] Create `template_service.py` to fetch/render templates from GCS
3. [ ] Add unit tests for template rendering
### Phase 2: Email Extensions
4. [ ] Extend `email_service.py` with CC support in `send_email()`
5. [ ] Add `send_job_completion()` method
6. [ ] Add `send_action_reminder()` method
7. [ ] Add unit tests for new email methods
### Phase 3: Auto-Email on Completion
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
9. [ ] Add integration test for completion email trigger
### Phase 4: Idle Reminder System
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
12. [ ] Schedule 5-minute delayed task when entering blocking states
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
14. [ ] Add tests for idle reminder flow
### Phase 5: Admin API
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
17. [ ] Add tests for admin endpoints
### Phase 6: Admin UI
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
19. [ ] Add "Copy Message" button (admin-only) with loading state
20. [ ] Add "Send Email" button (admin-only) with email input dialog
21. [ ] Add E2E test for admin buttons
### Phase 7: Documentation
22. [ ] Document template management in docs/DEVELOPMENT.md
23. [ ] Update docs/API.md with new endpoints
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
## Testing Strategy
**Unit Tests:**
- Template rendering with all variable combinations
- Email service CC support
- Completion/reminder email generation
- Idle detection logic
**Integration Tests:**
- Full flow: job completion → email sent
- Idle reminder: job enters blocking state → 5 min → reminder sent
- Admin endpoint authentication
**Manual Testing:**
- Verify emails render correctly in Gmail/Outlook
- Test admin buttons on completed job
- Test clipboard copy works across browsers
## Open Questions
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
## Rollback Plan
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
2. **Template fetch failures** - Fall back to hardcoded template in code
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
## Template Content
Initial template (from user's file, with feedback link added):
```
Hi {name},
Thanks for your order!
Here's the link for the karaoke video published to YouTube:
{youtube_url}
Here's the dropbox folder with all the finished files and source files, including:
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
- "(Title).mov"/"(End).mov": title card and end screen videos
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
- lyrics/*.txt song lyrics from various sources in plain text format
{dropbox_url}
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
{feedback_url}
Thanks again and have a great day!
-Andrew
```
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
## Updating Templates
To update email templates after deployment:
```bash
# Upload new template
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
# Templates are fetched fresh on each use (no caching in code)
# Changes take effect immediately
```
@@ -0,0 +1,115 @@
# Build Encoding Worker GCE Image
#
# Rebuilds the custom Packer image when provisioning changes.
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
#
# Triggers:
# - Push to main with changes to infrastructure/packer/**
# - Manual dispatch (workflow_dispatch)
#
# After image build, the encoding-worker VM will use the new image
# on next recreation (e.g., via pulumi up or manual restart).
name: Build Encoding Worker Image
on:
push:
branches: [main]
paths:
- 'infrastructure/packer/**'
workflow_dispatch:
inputs:
python_version:
description: 'Python version to install'
required: false
default: '3.13.1'
concurrency:
group: build-encoding-worker-image
cancel-in-progress: false # Don't cancel in-progress builds
jobs:
build-image:
name: Build Packer Image
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write # Required for Workload Identity Federation
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Install Packer
run: |
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y packer
- name: Initialize Packer
working-directory: infrastructure/packer
run: packer init encoding-worker.pkr.hcl
- name: Validate Packer Template
working-directory: infrastructure/packer
run: packer validate encoding-worker.pkr.hcl
- name: Build Image
working-directory: infrastructure/packer
env:
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
run: |
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
packer build \
-var "python_version=${PYTHON_VERSION}" \
encoding-worker.pkr.hcl
- name: Get Latest Image Info
id: image-info
run: |
IMAGE_NAME=$(gcloud compute images list \
--project=nomadkaraoke \
--filter="family:encoding-worker" \
--sort-by="~creationTimestamp" \
--limit=1 \
--format="value(name)")
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
echo "Built image: ${IMAGE_NAME}"
- name: Summary
run: |
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
notify-on-failure:
name: Notify on Failure
runs-on: ubuntu-latest
needs: build-image
if: failure()
steps:
- name: Log Failure
run: |
echo "::error::Encoding worker image build failed. Check the build-image job logs."
+197 -21
View File
@@ -903,11 +903,14 @@ jobs:
deploy-backend:
name: "Deploy - Backend (Cloud Run)"
needs: [package-build]
runs-on: [self-hosted, linux, gcp]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
outputs:
base_rebuilt: ${{ steps.check-base.outputs.needs_rebuild }}
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
@@ -965,6 +968,32 @@ jobs:
with:
version: "latest"
# ============================================
# Upload wheel to GCS for GCE encoding worker
# The GCE worker installs this wheel at job start for hot code updates
# ============================================
- name: Download wheel artifact
uses: actions/download-artifact@v4
with:
name: package-dist
path: dist/
- name: Upload wheel to GCS for GCE worker
run: |
echo "📦 Uploading wheel to GCS for GCE encoding worker..."
WHEEL_FILE=$(ls dist/*.whl | head -1)
if [ -z "$WHEEL_FILE" ]; then
echo "::error::No wheel file found in dist/"
exit 1
fi
echo "Found wheel: $WHEEL_FILE"
# Upload to GCS (GCE worker downloads latest at job start)
gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/
echo "✅ Wheel uploaded - GCE worker will pick up changes on next job"
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
@@ -990,32 +1019,32 @@ jobs:
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
# Try to get the existing base image's hash label
# First check if image exists locally (fast, no network call)
# Only pull from registry if not cached locally
# Fetch hash label directly from registry (10KB config blob, not full image)
# This avoids issues with stale local Docker cache on GitHub runners
EXISTING_HASH=""
if docker image inspect $BASE_IMAGE >/dev/null 2>&1; then
echo "✅ Base image found in local cache (skipping slow registry pull)"
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
if [ -z "$EXISTING_HASH" ]; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
fi
echo "Existing base image hash: $EXISTING_HASH"
elif docker pull $BASE_IMAGE 2>/dev/null; then
echo "📥 Base image pulled from registry"
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
if [ -z "$EXISTING_HASH" ]; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
fi
echo "Existing base image hash: $EXISTING_HASH"
# Get the manifest (small JSON, ~1KB) - includes config digest
MANIFEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null || echo "")
CONFIG_DIGEST=$(echo "$MANIFEST" | jq -r '.config.digest // ""' 2>/dev/null || echo "")
if [ -n "$CONFIG_DIGEST" ] && [ "$CONFIG_DIGEST" != "null" ] && [ "$CONFIG_DIGEST" != "" ]; then
echo "📋 Found manifest, config digest: $CONFIG_DIGEST"
# Fetch just the config blob (~10KB) to get labels - much faster than pulling full image
TOKEN=$(gcloud auth print-access-token)
CONFIG_JSON=$(curl -sL -H "Authorization: Bearer $TOKEN" \
"https://${{ env.REGISTRY }}/v2/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}/blobs/$CONFIG_DIGEST" 2>/dev/null)
EXISTING_HASH=$(echo "$CONFIG_JSON" | jq -r '.config.Labels["base.content.hash"] // .config.Labels["poetry.lock.hash"] // ""' 2>/dev/null || echo "")
echo "📋 Registry base image hash: $EXISTING_HASH"
else
echo "⚠️ No existing base image found - will build"
echo "⚠️ No existing base image found in registry - will build"
fi
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
echo " Current: ${{ steps.base-hash.outputs.hash }}"
echo " Registry: $EXISTING_HASH"
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
else
echo "✅ Base image is up to date"
@@ -1043,6 +1072,31 @@ jobs:
# Provenance disabled for smaller image and faster push
provenance: false
- name: Ensure local base image is current (if not rebuilt)
if: steps.check-base.outputs.needs_rebuild == 'false'
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
# Get the config digest from registry manifest (identifies the image content)
# This is the same value as Docker's Image ID for the same image
REGISTRY_CONFIG_DIGEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null | jq -r '.config.digest' || echo "")
# Get local image's ID (which is the config digest)
LOCAL_IMAGE_ID=$(docker image inspect $BASE_IMAGE --format='{{.Id}}' 2>/dev/null || echo "")
echo "Registry config digest: $REGISTRY_CONFIG_DIGEST"
echo "Local image ID: $LOCAL_IMAGE_ID"
if [ -z "$LOCAL_IMAGE_ID" ]; then
echo "📥 Base image not in local cache - pulling from registry"
docker pull $BASE_IMAGE
elif [ "$LOCAL_IMAGE_ID" != "$REGISTRY_CONFIG_DIGEST" ]; then
echo "🔄 Local cache has different image - pulling latest"
docker pull $BASE_IMAGE
else
echo "✅ Local cache matches registry - skipping pull"
fi
- name: Build and push app image
uses: docker/build-push-action@v6
with:
@@ -1085,7 +1139,7 @@ jobs:
--concurrency 1 \
--max-instances 50 \
--min-instances 4 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true" \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest"
- name: Verify deployment and version
@@ -1139,3 +1193,125 @@ jobs:
echo ""
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
# ============================================
# PROPAGATE BASE IMAGE TO ALL RUNNERS
# When base image is rebuilt, pull it to all 20 runners
# so subsequent app builds don't wait 8 min for the pull
#
# NOTE: This is best-effort distribution. GitHub Actions doesn't
# support targeting specific runners by name, so we spawn 20 jobs
# and rely on the scheduler to distribute them across idle runners.
# In practice, with all runners idle after deploy-backend, distribution
# is good. Runners that miss this pull will get the image on their
# next app build (with the 8 min delay).
# ============================================
propagate-base-image:
name: "Propagate Base Image (${{ matrix.runner_index }})"
needs: deploy-backend
if: github.ref == 'refs/heads/main' && github.event_name == 'push' && needs.deploy-backend.outputs.base_rebuilt == 'true'
runs-on: [self-hosted, linux, gcp]
permissions:
contents: read
id-token: write
strategy:
matrix:
# 20 parallel jobs - GitHub distributes across available runners
runner_index: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
max-parallel: 20
fail-fast: false # Don't cancel other runners if one fails
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
REPOSITORY: karaoke-repo
BASE_IMAGE_NAME: karaoke-backend-base
steps:
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
- name: Pull latest base image
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
RUNNER_NAME=$(hostname)
echo "📥 Pulling latest base image on runner $RUNNER_NAME..."
docker pull $BASE_IMAGE
echo "✅ Base image cached on $RUNNER_NAME"
deploy-infrastructure:
name: "Deploy - Infrastructure (Pulumi)"
runs-on: [self-hosted, linux, gcp]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: '3.13'
- name: Install Pulumi
run: |
curl -fsSL https://get.pulumi.com | sh
echo "$HOME/.pulumi/bin" >> $GITHUB_PATH
- name: Install Python dependencies
working-directory: infrastructure
run: |
pip install pulumi pulumi-gcp pulumi-docker
- name: Configure Pulumi
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
pulumi login
- name: Deploy Infrastructure
working-directory: infrastructure
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
echo "🚀 Deploying infrastructure changes..."
# Skip preview to avoid needing broad read permissions on all GCP resources
# Pulumi state is authoritative since infrastructure is only managed via Pulumi
pulumi up --yes --skip-preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
- name: Restart GCE Encoding Worker
run: |
echo "🔄 Restarting encoding worker to pick up new code..."
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker" || true
echo "✅ Encoding worker restart triggered"
- name: Summary
if: success()
run: |
echo "🎉 Infrastructure deployment successful!"
echo "Commit: ${{ github.sha }}"
echo "Branch: ${{ github.ref_name }}"
+4 -2
View File
@@ -48,8 +48,10 @@ cd frontend && npm run test:all 2>&1 | tail -n 200
- Skip for docs-only changes
### Infrastructure
- All GCP changes via Pulumi in `infrastructure/`
- `gcloud` CLI for reading/debugging only
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
- Never modify GCP resources directly via console or `gcloud` CLI
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
- Stop and notify user on auth issues
## API Authentication (for Agents)
+156 -1
View File
@@ -579,4 +579,159 @@ async def get_flacfetch_cache_stats(
status_code=502,
detail=f"Failed to get cache stats: {e}"
)
# Trigger deployment
# =============================================================================
# Job Completion Message Endpoints (for admin copy/send functionality)
# =============================================================================
class CompletionMessageResponse(BaseModel):
"""Response containing the rendered completion message."""
job_id: str
message: str
subject: str
youtube_url: Optional[str] = None
dropbox_url: Optional[str] = None
class SendCompletionEmailRequest(BaseModel):
"""Request to send a completion email."""
to_email: str
cc_admin: bool = True
class SendCompletionEmailResponse(BaseModel):
"""Response from sending a completion email."""
success: bool
job_id: str
to_email: str
message: str
@router.get("/jobs/{job_id}/completion-message", response_model=CompletionMessageResponse)
async def get_job_completion_message(
job_id: str,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Get the rendered completion message for a job.
Returns the plain text message that would be sent to the user,
rendered using the job completion template with the job's details.
This is useful for:
- Copying the message to clipboard (e.g., for Fiverr)
- Previewing the email before sending
Requires admin authentication.
"""
from backend.services.job_notification_service import get_job_notification_service
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
state_data = job.state_data or {}
youtube_url = state_data.get('youtube_url')
dropbox_url = state_data.get('dropbox_link')
brand_code = state_data.get('brand_code')
# Render the completion message
notification_service = get_job_notification_service()
message = notification_service.get_completion_message(
job_id=job.job_id,
user_name=None, # Use default "there"
artist=job.artist,
title=job.title,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
)
# Build subject: "NOMAD-1178: Artist - Title (Your karaoke video is ready!)"
if brand_code and job.artist and job.title:
subject = f"{brand_code}: {job.artist} - {job.title} (Your karaoke video is ready!)"
elif job.artist and job.title:
subject = f"{job.artist} - {job.title} (Your karaoke video is ready!)"
else:
subject = "Your karaoke video is ready!"
return CompletionMessageResponse(
job_id=job_id,
message=message,
subject=subject,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
)
@router.post("/jobs/{job_id}/send-completion-email", response_model=SendCompletionEmailResponse)
async def send_job_completion_email(
job_id: str,
request: SendCompletionEmailRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
):
"""
Send a completion email for a job to a specified email address.
This allows admins to manually send (or re-send) completion emails,
useful for:
- Sending to customers who didn't have an email on file
- Re-sending if the original email was lost
- Sending to alternate email addresses
Requires admin authentication.
"""
from backend.services.job_notification_service import get_job_notification_service
from backend.services.email_service import get_email_service
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
state_data = job.state_data or {}
youtube_url = state_data.get('youtube_url')
dropbox_url = state_data.get('dropbox_link')
brand_code = state_data.get('brand_code')
# Render the completion message
notification_service = get_job_notification_service()
message = notification_service.get_completion_message(
job_id=job.job_id,
user_name=None, # Use default "there"
artist=job.artist,
title=job.title,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
)
# Send the email
email_service = get_email_service()
success = email_service.send_job_completion(
to_email=request.to_email,
message_content=message,
artist=job.artist,
title=job.title,
brand_code=brand_code,
cc_admin=request.cc_admin,
)
if success:
logger.info(f"Admin sent completion email for job {job_id} to {request.to_email}")
return SendCompletionEmailResponse(
success=True,
job_id=job_id,
to_email=request.to_email,
message=f"Completion email sent to {request.to_email}",
)
else:
logger.error(f"Failed to send completion email for job {job_id}")
raise HTTPException(
status_code=500,
detail="Failed to send email. Check email service configuration."
)
+15 -4
View File
@@ -95,7 +95,7 @@ class AudioSearchRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
@@ -107,6 +107,10 @@ class AudioSearchRequest(BaseModel):
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
# Display overrides (optional - if empty, search values are used for display)
display_artist: Optional[str] = Field(None, description="Artist name for title screens/filenames. If empty, uses search artist.")
display_title: Optional[str] = Field(None, description="Title for title screens/filenames. If empty, uses search title.")
# Audio separation model configuration
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
@@ -485,7 +489,8 @@ async def search_audio(
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
# Apply defaults for YouTube/Dropbox distribution (for web service)
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
# Use explicit value if provided, otherwise fall back to server default
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
effective_youtube_description = body.youtube_description or settings.default_youtube_description
@@ -529,10 +534,16 @@ async def search_audio(
# Use authenticated user's email
effective_user_email = auth_result.user_email
# Determine display values - use display_* if provided, otherwise fall back to search values
# Display values are used for title screens, filenames, YouTube, etc.
# Search values (body.artist, body.title) are used for audio search
effective_display_artist = body.display_artist.strip() if body.display_artist else body.artist
effective_display_title = body.display_title.strip() if body.display_title else body.title
# Create job
job_create = JobCreate(
artist=body.artist,
title=body.title,
artist=effective_display_artist, # Display value for title screens, filenames
title=effective_display_title, # Display value for title screens, filenames
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
+36 -13
View File
@@ -69,7 +69,7 @@ class CreateJobFromUrlRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
@@ -123,7 +123,7 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
@@ -406,7 +406,7 @@ async def upload_and_create_job(
enable_txt: Optional[bool] = Form(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise"),
# Finalisation options
brand_prefix: Optional[str] = Form(None, description="Brand code prefix (e.g., NOMAD)"),
enable_youtube_upload: bool = Form(False, description="Upload to YouTube"),
enable_youtube_upload: Optional[bool] = Form(None, description="Upload to YouTube. None = use server default"),
youtube_description: Optional[str] = Form(None, description="YouTube video description text"),
discord_webhook_url: Optional[str] = Form(None, description="Discord webhook URL for notifications"),
webhook_url: Optional[str] = Form(None, description="Generic webhook URL"),
@@ -510,12 +510,19 @@ async def upload_and_create_job(
if dist.brand_prefix and not brand_prefix:
logger.info(f"Using default brand_prefix: {dist.brand_prefix}")
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload
if effective_enable_youtube_upload and enable_youtube_upload is None:
logger.info("Using default enable_youtube_upload: True (from env)")
# Validate credentials for requested distribution services (including defaults)
# This prevents accepting jobs that will fail later due to missing credentials
invalid_services = []
credential_manager = get_credential_manager()
if enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -594,7 +601,7 @@ async def upload_and_create_job(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=youtube_description,
discord_webhook_url=dist.discord_webhook_url,
webhook_url=webhook_url,
@@ -815,11 +822,12 @@ async def upload_and_create_job(
"using_default": gdrive_folder_id is None,
}
if enable_youtube_upload:
if effective_enable_youtube_upload:
youtube_result = credential_manager.check_youtube_credentials()
distribution_services["youtube"] = {
"enabled": True,
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
"using_default": enable_youtube_upload is None,
}
if dist.discord_webhook_url:
@@ -908,7 +916,7 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code from folder name")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
@@ -1048,11 +1056,16 @@ async def create_job_with_upload_urls(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1105,7 +1118,7 @@ async def create_job_with_upload_urls(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
youtube_description_template=body.youtube_description, # video_worker reads this field
discord_webhook_url=dist.discord_webhook_url,
@@ -1466,11 +1479,16 @@ async def create_job_from_url(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1522,7 +1540,7 @@ async def create_job_from_url(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
discord_webhook_url=dist.discord_webhook_url,
webhook_url=body.webhook_url,
@@ -1712,11 +1730,16 @@ async def create_finalise_only_job(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate distribution credentials if services are requested
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1766,7 +1789,7 @@ async def create_finalise_only_job(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
discord_webhook_url=dist.discord_webhook_url,
dropbox_path=dist.dropbox_path,
+34
View File
@@ -227,6 +227,7 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
"active_jobs": health.get("active_jobs", 0),
"queue_length": health.get("queue_length", 0),
"ffmpeg_version": health.get("ffmpeg_version"),
"wheel_version": health.get("wheel_version"),
}
except Exception as e:
return {
@@ -237,6 +238,39 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
}
@router.get("/health/encoding-worker")
async def encoding_worker_health() -> Dict[str, Any]:
"""
Lightweight endpoint to check encoding worker status.
Returns minimal info for frontend footer display.
No authentication required.
"""
status = await check_encoding_worker_status()
# Return simplified response for frontend
if not status.get("configured"):
return {
"available": False,
"status": "not_configured",
}
if not status.get("available"):
return {
"available": False,
"status": "offline",
"error": status.get("error"),
}
return {
"available": True,
"status": "ok",
"version": status.get("wheel_version"),
"active_jobs": status.get("active_jobs", 0),
"queue_length": status.get("queue_length", 0),
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
+106 -1
View File
@@ -316,13 +316,118 @@ async def trigger_render_video_worker(
)
@router.post("/jobs/{job_id}/check-idle-reminder")
async def check_idle_reminder(
job_id: str,
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Check if a job needs an idle reminder email.
This endpoint is called by a Cloud Tasks scheduled task 5 minutes after
a job enters a blocking state (AWAITING_REVIEW or AWAITING_INSTRUMENTAL_SELECTION).
If the job is still in the blocking state and no reminder has been sent yet,
sends a reminder email to the user.
Idempotency: Only one reminder per job (tracked via reminder_sent flag).
"""
from backend.models.job import JobStatus
from backend.services.job_notification_service import get_job_notification_service
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] IDLE_REMINDER_CHECK starting")
add_span_attribute("job_id", job_id)
add_span_attribute("operation", "idle_reminder_check")
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
logger.warning(f"[job:{job_id}] Job not found for idle reminder check")
add_span_event("job_not_found")
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
# Check if job is still in a blocking state
blocking_states = [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]
if job.status not in [s.value for s in blocking_states]:
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
add_span_event("not_blocking", {"current_status": job.status})
return {
"status": "skipped",
"job_id": job_id,
"message": f"Job not in blocking state (current: {job.status})"
}
# Normalize state_data to prevent None errors
state_data = job.state_data or {}
# Check if reminder was already sent (idempotency)
if state_data.get('reminder_sent'):
logger.info(f"[job:{job_id}] Reminder already sent, skipping")
add_span_event("already_sent")
return {"status": "already_sent", "job_id": job_id, "message": "Reminder already sent"}
# Check if user has an email
if not job.user_email:
logger.warning(f"[job:{job_id}] No user email, cannot send reminder")
add_span_event("no_email")
return {"status": "no_email", "job_id": job_id, "message": "No user email configured"}
# Determine action type
action_type = state_data.get('blocking_action_type')
if not action_type:
action_type = "lyrics" if job.status == JobStatus.AWAITING_REVIEW.value else "instrumental"
# Send the reminder email
try:
notification_service = get_job_notification_service()
success = await notification_service.send_action_reminder_email(
job_id=job.job_id,
user_email=job.user_email,
action_type=action_type,
user_name=None, # Could fetch from user service if needed
artist=job.artist,
title=job.title,
audio_hash=job.audio_hash,
review_token=job.review_token,
instrumental_token=job.instrumental_token,
)
if success:
# Mark reminder as sent (prevents duplicate sends)
job_manager.firestore.update_job(job_id, {
'state_data': {**state_data, 'reminder_sent': True}
})
logger.info(f"[job:{job_id}] Sent {action_type} reminder email to {job.user_email}")
add_span_event("reminder_sent", {"action_type": action_type})
return {
"status": "sent",
"job_id": job_id,
"message": f"Sent {action_type} reminder to {job.user_email}"
}
else:
logger.error(f"[job:{job_id}] Failed to send reminder email")
add_span_event("send_failed")
return {"status": "failed", "job_id": job_id, "message": "Failed to send reminder"}
except Exception as e:
logger.exception(f"[job:{job_id}] Error sending reminder: {e}")
add_span_event("error", {"error": str(e)})
return {"status": "error", "job_id": job_id, "message": str(e)}
@router.get("/health")
async def internal_health(
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Internal health check endpoint.
Used to verify the internal API is responsive.
Requires admin authentication.
"""
+6 -1
View File
@@ -87,6 +87,11 @@ async def create_job(
user_email = request.user_email
logger.info(f"Admin {auth_result.user_email} creating job on behalf of {user_email}")
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = request.enable_youtube_upload if request.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Create job with all preferences
job_create = JobCreate(
url=str(request.url),
@@ -94,7 +99,7 @@ async def create_job(
title=request.title,
enable_cdg=request.enable_cdg,
enable_txt=request.enable_txt,
enable_youtube_upload=request.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=request.youtube_description,
webhook_url=request.webhook_url,
user_email=user_email
+122 -40
View File
@@ -27,8 +27,10 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_logging import job_log_context, JobLogger
from backend.services.tracing import create_span, add_span_attribute, add_span_event
from backend.services.encoding_service import get_encoding_service
from backend.api.dependencies import require_auth, require_review_auth
from backend.services.auth_service import UserType
from backend.config import get_settings
# LyricsTranscriber imports for preview generation
from lyrics_transcriber.types import CorrectionResult
@@ -385,28 +387,35 @@ async def generate_preview_video(
):
"""
Generate a preview video with the current corrections.
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
video with the user's current corrections applied.
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
the high-performance GCE worker for faster generation (15-20s vs 60+s).
"""
job_manager = JobManager()
storage = StorageService()
settings = get_settings()
encoding_service = get_encoding_service()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Job must be in review state to generate preview
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
logger.info(f"Job {job_id}: Generating preview video")
# Check if GCE preview encoding is enabled
use_gce_preview = encoding_service.is_preview_enabled
logger.info(f"Job {job_id}: Generating preview video (GCE preview: {use_gce_preview})")
# Use tracing and job_log_context for full observability
with create_span("generate-preview-video", {"job_id": job_id}) as span:
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
with job_log_context(job_id, worker="preview"):
try:
# Create temp directory for this preview operation
@@ -416,19 +425,19 @@ async def generate_preview_video(
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
corrections_path = os.path.join(temp_dir, "corrections.json")
storage.download_file(corrections_gcs, corrections_path)
with open(corrections_path, 'r', encoding='utf-8') as f:
original_data = json.load(f)
# 2. Download input audio
audio_path = os.path.join(temp_dir, "audio.flac")
storage.download_file(job.input_media_gcs_path, audio_path)
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
# 3. Load original as CorrectionResult
correction_result = CorrectionResult.from_dict(original_data)
add_span_event("corrections_loaded")
# 4. Get or create styles file for preview using unified style loader
with create_span("load-styles") as styles_span:
styles_path, _ = load_styles_from_gcs(
@@ -439,54 +448,127 @@ async def generate_preview_video(
logger=logger,
)
styles_span.set_attribute("styles_path", styles_path)
# 5. Set up output config for preview
output_dir = os.path.join(temp_dir, "output")
cache_dir = os.path.join(temp_dir, "cache")
os.makedirs(output_dir, exist_ok=True)
os.makedirs(cache_dir, exist_ok=True)
output_config = OutputConfig(
output_styles_json=styles_path,
output_dir=output_dir,
cache_dir=cache_dir,
video_resolution="360p",
)
# 6. Generate preview video using CorrectionOperations (heavy operation)
with create_span("render-preview-video") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger
)
add_span_event("render_complete")
# 7. Upload preview video to GCS for serving
preview_hash = result["preview_hash"]
video_path = result["video_path"]
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# 6. Generate preview (ASS-only if using GCE, or full video if local)
preview_gcs_path = None
if use_gce_preview:
# GCE path: Generate ASS only, then offload encoding to GCE
try:
with create_span("generate-ass-subtitles") as ass_span:
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger,
ass_only=True, # Only generate ASS, skip video encoding
)
preview_hash = result["preview_hash"]
ass_path = result["ass_path"]
ass_span.set_attribute("ass_path", ass_path)
add_span_event("ass_generated")
# Upload ASS to GCS
with create_span("upload-ass-to-gcs") as upload_ass_span:
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
storage.upload_file(ass_path, ass_gcs_path)
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
# Call GCE encoding service
with create_span("gce-preview-encoding") as gce_span:
bucket_name = settings.gcs_bucket_name
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
# Get background image and font from style assets if available
style_assets = job.style_assets or {}
background_image_gcs_path = None
for key in ["karaoke_background", "style_karaoke_background"]:
if key in style_assets:
background_image_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
gce_span.set_attribute("background_image", background_image_gcs_path)
break
font_gcs_path = None
for key in ["font", "style_font"]:
if key in style_assets:
font_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
gce_span.set_attribute("font", font_gcs_path)
break
gce_result = await encoding_service.encode_preview_video(
job_id=f"preview_{job_id}_{preview_hash}",
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
background_color="black",
background_image_gcs_path=background_image_gcs_path,
font_gcs_path=font_gcs_path,
)
gce_span.set_attribute("gce_status", gce_result.get("status"))
add_span_event("gce_encoding_complete")
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
except Exception as gce_error:
# Fall back to local encoding if GCE fails
logger.warning(
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
)
span.set_attribute("gce_fallback", True)
use_gce_preview = False # Fall through to local encoding below
if not use_gce_preview:
# Local path: Generate full preview video locally
with create_span("render-preview-video-local") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger,
ass_only=False, # Generate full video locally
)
preview_hash = result["preview_hash"]
video_path = result["video_path"]
add_span_event("render_complete")
# Upload preview video to GCS
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# Store the GCS path for serving
if job_id not in _preview_videos:
_preview_videos[job_id] = {}
_preview_videos[job_id][preview_hash] = preview_gcs_path
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
span.set_attribute("preview_hash", preview_hash)
span.set_attribute("success", True)
return {"status": "success", "preview_hash": preview_hash}
except Exception as e:
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
span.set_attribute("error", str(e))
+6
View File
@@ -68,6 +68,12 @@ class Settings(BaseSettings):
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
# GCE Preview Encoding (for faster preview video generation)
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
+2 -2
View File
@@ -14,7 +14,7 @@ class URLSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
enable_youtube_upload: Optional[bool] = None # None = use server default
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
@@ -28,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
enable_youtube_upload: Optional[bool] = None # None = use server default
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
+418 -6
View File
@@ -14,7 +14,7 @@ import html
import logging
import os
from abc import ABC, abstractmethod
from typing import Optional
from typing import Optional, List
from backend.config import get_settings
@@ -31,7 +31,8 @@ class EmailProvider(ABC):
to_email: str,
subject: str,
html_content: str,
text_content: Optional[str] = None
text_content: Optional[str] = None,
cc_emails: Optional[List[str]] = None,
) -> bool:
"""Send an email. Returns True if successful."""
pass
@@ -49,10 +50,13 @@ class ConsoleEmailProvider(EmailProvider):
to_email: str,
subject: str,
html_content: str,
text_content: Optional[str] = None
text_content: Optional[str] = None,
cc_emails: Optional[List[str]] = None,
) -> bool:
logger.info("=" * 60)
logger.info(f"EMAIL TO: {to_email}")
if cc_emails:
logger.info(f"CC: {', '.join(cc_emails)}")
logger.info(f"SUBJECT: {subject}")
logger.info("-" * 60)
logger.info(text_content or html_content)
@@ -77,12 +81,13 @@ class SendGridEmailProvider(EmailProvider):
to_email: str,
subject: str,
html_content: str,
text_content: Optional[str] = None
text_content: Optional[str] = None,
cc_emails: Optional[List[str]] = None,
) -> bool:
try:
# Import here to avoid requiring sendgrid in all environments
from sendgrid import SendGridAPIClient
from sendgrid.helpers.mail import Mail, Email, To, Content
from sendgrid.helpers.mail import Mail, Email, To, Content, Cc
sg = SendGridAPIClient(api_key=self.api_key)
@@ -93,13 +98,28 @@ class SendGridEmailProvider(EmailProvider):
html_content=Content("text/html", html_content)
)
# Add CC recipients if provided (deduplicated and normalized)
if cc_emails:
# Normalize emails (lowercase, strip whitespace) and deduplicate
seen = set()
unique_cc_emails = []
for cc_email in cc_emails:
normalized = cc_email.strip().lower()
if normalized and normalized not in seen:
seen.add(normalized)
unique_cc_emails.append(cc_email.strip())
for cc_email in unique_cc_emails:
message.add_cc(Cc(cc_email))
if text_content:
message.add_content(Content("text/plain", text_content))
response = sg.send(message)
if response.status_code >= 200 and response.status_code < 300:
logger.info(f"Email sent to {to_email} via SendGrid")
cc_info = f" (CC: {', '.join(cc_emails)})" if cc_emails else ""
logger.info(f"Email sent to {to_email}{cc_info} via SendGrid")
return True
else:
logger.error(f"SendGrid returned status {response.status_code}")
@@ -816,6 +836,398 @@ Thanks for being part of making Nomad Karaoke better!
from datetime import datetime
return datetime.now().year
def _get_email_signature(self) -> str:
"""Get the HTML email signature for Nomad Karaoke."""
return """
<table cellpadding="0" cellspacing="0" border="0" style="font-size: medium; font-family: Trebuchet MS;">
<tbody>
<tr>
<td>
<table cellpadding="0" cellspacing="0" border="0" style="font-size: medium; font-family: Trebuchet MS;">
<tbody>
<tr>
<td style="vertical-align: top;">
<table cellpadding="0" cellspacing="0" border="0"
style="font-size: medium; font-family: Trebuchet MS;">
<tbody>
<tr>
<td style="text-align: center;"><a
href="https://www.linkedin.com/in/andrewbeveridge"
target="_blank"><img
src="https://beveradb.github.io/public-images/andrew-buildspace-circle-150px.png"
role="presentation" style="display: block; max-width: 128px;"
width="130"></a></td>
</tr>
<tr>
<td height="5"></td>
</tr>
<tr>
<td style="text-align: center;"><a href="https://nomadkaraoke.com"
target="_blank"><img role="presentation" width="130"
style="display: block; max-width: 130px; border-radius: 7px;"
src="https://beveradb.github.io/public-images/Nomad-Karaoke-Logo-small-indexed-websafe-rectangle.gif"></a>
</td>
</tr>
</tbody>
</table>
</td>
<td width="10">
<div></div>
</td>
<td style="padding: 0px; vertical-align: middle;">
<h2 color="#000000"
style="margin: 0px; font-size: 18px; color: rgb(0, 0, 0); font-weight: 600;">
<span>Andrew</span><span>&nbsp;</span><span>Beveridge</span>
</h2>
<p color="#000000" font-size="medium"
style="margin: 0px; color: rgb(0, 0, 0); font-size: 14px; line-height: 22px;">
<span>Founder</span>
</p>
<p color="#000000" font-size="medium"
style="margin: 0px; font-weight: 500; color: rgb(0, 0, 0); font-size: 14px; line-height: 22px;">
<span>Nomad Karaoke
LLC</span>
</p>
<table cellpadding="0" cellspacing="0" border="0"
style="width: 100%; font-size: medium; font-family: Trebuchet MS; margin-top: 3px; margin-bottom: 3px;">
<tbody>
<tr>
<td color="#ff7acc" direction="horizontal" width="auto" height="15"
style="width: 100%; display: block; line-height:0; font-size:0;">
</td>
</tr>
<tr>
<td color="#ff7acc" direction="horizontal" width="auto" height="1"
style="width: 100%; border-bottom: 1px solid rgb(255, 122, 204); border-left: medium; display: block; line-height:0; font-size:0;">
</td>
</tr>
</tbody>
</table>
<table cellpadding="0" cellspacing="0" border="0"
style="width: 100%; font-size: medium; font-family: Trebuchet MS; margin-top: 3px; margin-bottom: 3px;">
<tbody>
<tr>
<td style="text-align: center; line-height:0; font-size:0;">
<table cellpadding="0" cellspacing="0" border="0"
style="display: inline-block; font-size: medium; font-family: Trebuchet MS;">
<tbody>
<tr style="text-align: center;">
<td><a href="https://www.youtube.com/@nomadkaraoke"
color="#230a89"
style="display: inline-block; padding: 0px; background-color: rgb(35, 10, 137);"
target="_blank"><img
src="https://beveradb.github.io/public-images/youtube-icon-2x.png"
alt="youtube" color="#230a89" width="24"
style="background-color: rgb(35, 10, 137); max-width: 135px; display: block;"></a>
</td>
<td width="5">
<div></div>
</td>
<td><a href="https://github.com/nomadkaraoke"
color="#230a89"
style="display: inline-block; padding: 0px; background-color: rgb(35, 10, 137);"
target="_blank"><img
src="https://beveradb.github.io/public-images/github-icon-2x.png"
alt="github" color="#230a89" width="24"
style="background-color: rgb(35, 10, 137); max-width: 135px; display: block;"></a>
</td>
<td width="5">
<div></div>
</td>
<td><a href="https://www.linkedin.com/in/andrewbeveridge"
color="#230a89"
style="display: inline-block; padding: 0px; background-color: rgb(35, 10, 137);"
target="_blank"><img
src="https://beveradb.github.io/public-images/linkedin-icon-2x.png"
alt="linkedin" color="#230a89" width="24"
style="background-color: rgb(35, 10, 137); max-width: 135px; display: block;"></a>
</td>
<td width="5">
<div></div>
</td>
<td><a href="https://twitter.com/beveradb" color="#230a89"
style="display: inline-block; padding: 0px; background-color: rgb(35, 10, 137);"><img
src="https://beveradb.github.io/public-images/twitter-icon-2x.png"
alt="twitter" color="#230a89" width="24"
style="background-color: rgb(35, 10, 137); max-width: 135px; display: block;"></a>
</td>
<td width="5">
<div></div>
</td>
<td><a href="https://www.instagram.com/beveradb/"
color="#230a89"
style="display: inline-block; padding: 0px; background-color: rgb(35, 10, 137);"><img
src="https://beveradb.github.io/public-images/instagram-icon-2x.png"
alt="instagram" color="#230a89" width="24"
style="background-color: rgb(35, 10, 137); max-width: 135px; display: block;"></a>
</td>
<td width="5">
<div></div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<table cellpadding="0" cellspacing="0" border="0"
style="width: 100%; font-size: medium; font-family: Trebuchet MS; margin-top: 3px; margin-bottom: 3px;">
<tbody>
<tr>
<td color="#ff7acc" direction="horizontal" width="auto" height="1"
style="width: 100%; border-bottom: 1px solid rgb(255, 122, 204); border-left: medium; display: block; line-height:0; font-size:0;">
</td>
</tr>
<tr>
<td color="#ff7acc" direction="horizontal" width="auto" height="15"
style="width: 100%; display: block;">
</td>
</tr>
</tbody>
</table>
<table cellpadding="0" cellspacing="0" border="0"
style="font-size: medium; font-family: Trebuchet MS;">
<tbody>
<tr style="vertical-align: middle;" height="20">
<td width="30" style="vertical-align: middle;">
<table cellpadding="0" cellspacing="0" border="0"
style="font-size: medium; font-family: Trebuchet MS;">
<tbody>
<tr>
<td style="vertical-align: bottom;"><span color="#ff7acc"
width="11"
style="display: inline-block; background-color: rgb(255, 122, 204);"><img
src="https://beveradb.github.io/public-images/phone-icon-2x.png"
color="#ff7acc" alt="mobilePhone" width="13"
style="display: block; background-color: rgb(255, 122, 204);"></span>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding: 0px; color: rgb(0, 0, 0);"><a href="tel:8036363267"
color="#000000"
style="text-decoration: none; color: rgb(0, 0, 0); font-size: 12px;"><span>+1 (803) 636-3267</span></a> | <a href="tel:07835171222"
color="#000000"
style="text-decoration: none; color: rgb(0, 0, 0); font-size: 12px;"><span>+44 07835171222</span></a></td>
</tr>
<tr style="vertical-align: middle;" height="20">
<td width="30" style="vertical-align: middle;">
<table cellpadding="0" cellspacing="0" border="0"
style="font-size: medium; font-family: Trebuchet MS;">
<tbody>
<tr>
<td style="vertical-align: bottom;"><span color="#ff7acc"
width="11"
style="display: inline-block; background-color: rgb(255, 122, 204);"><img
src="https://beveradb.github.io/public-images/email-icon-2x.png"
color="#ff7acc" alt="emailAddress" width="13"
style="display: block; background-color: rgb(255, 122, 204);"></span>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding: 0px;"><a href="mailto:andrew@nomadkaraoke.com"
color="#000000"
style="text-decoration: none; color: rgb(0, 0, 0); font-size: 12px;"><span>andrew@nomadkaraoke.com</span></a>
</td>
</tr>
<tr style="vertical-align: middle;" height="20">
<td width="30" style="vertical-align: middle;">
<table cellpadding="0" cellspacing="0" border="0"
style="font-size: medium; font-family: Trebuchet MS;">
<tbody>
<tr>
<td style="vertical-align: bottom;"><span color="#ff7acc"
width="11"
style="display: inline-block; background-color: rgb(255, 122, 204);"><img
src="https://beveradb.github.io/public-images/link-icon-2x.png"
color="#ff7acc" alt="website" width="13"
style="display: block; background-color: rgb(255, 122, 204);"></span>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding: 0px;"><a href="https://nomadkaraoke.com" target="_blank"
color="#000000"
style="text-decoration: none; color: rgb(0, 0, 0); font-size: 12px;"><span>nomadkaraoke.com</span></a>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
"""
def send_job_completion(
self,
to_email: str,
message_content: str,
artist: Optional[str] = None,
title: Optional[str] = None,
brand_code: Optional[str] = None,
cc_admin: bool = True,
) -> bool:
"""
Send job completion email with the rendered template content.
Args:
to_email: User's email address
message_content: Pre-rendered message content (plain text)
artist: Artist name for subject line
title: Song title for subject line
brand_code: Release ID (e.g., "NOMAD-1178") for subject line
cc_admin: Whether to CC gen@nomadkaraoke.com
Returns:
True if email was sent successfully
"""
# Build subject: "NOMAD-1178: Artist - Title (Your karaoke video is ready!)"
if brand_code and artist and title:
subject = f"{brand_code}: {artist} - {title} (Your karaoke video is ready!)"
elif artist and title:
subject = f"{artist} - {title} (Your karaoke video is ready!)"
else:
subject = "Your karaoke video is ready!"
# Convert plain text to simple HTML (preserve line breaks)
html_content = f"""
<!DOCTYPE html>
<html>
<head>
<style>
body {{
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
line-height: 1.6;
color: #333;
max-width: 600px;
margin: 0 auto;
padding: 20px;
white-space: pre-wrap;
}}
.content {{
white-space: pre-wrap;
font-size: 14px;
}}
.signature {{
margin-top: 30px;
padding-top: 20px;
}}
</style>
</head>
<body>
<div class="content">{html.escape(message_content)}</div>
<div class="signature">
{self._get_email_signature()}
</div>
</body>
</html>
"""
cc_emails = ["gen@nomadkaraoke.com"] if cc_admin else None
return self.provider.send_email(
to_email=to_email,
subject=subject,
html_content=html_content,
text_content=message_content,
cc_emails=cc_emails,
)
def send_action_reminder(
self,
to_email: str,
message_content: str,
action_type: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> bool:
"""
Send action-needed reminder email.
Args:
to_email: User's email address
message_content: Pre-rendered message content (plain text)
action_type: Type of action needed ("lyrics" or "instrumental")
artist: Artist name for subject line
title: Song title for subject line
Returns:
True if email was sent successfully
"""
# Build subject based on action type
song_info = f" for {artist} - {title}" if artist and title else ""
if action_type == "lyrics":
subject = f"Action needed: Review lyrics{song_info}"
elif action_type == "instrumental":
subject = f"Action needed: Select instrumental{song_info}"
else:
subject = f"Action needed{song_info}"
# Convert plain text to simple HTML
html_content = f"""
<!DOCTYPE html>
<html>
<head>
<style>
body {{
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
line-height: 1.6;
color: #333;
max-width: 600px;
margin: 0 auto;
padding: 20px;
}}
.alert {{
background-color: #fef3c7;
border: 1px solid #fcd34d;
border-radius: 8px;
padding: 16px;
margin: 20px 0;
text-align: center;
}}
.content {{
white-space: pre-wrap;
font-size: 14px;
}}
.signature {{
margin-top: 30px;
padding-top: 20px;
}}
</style>
</head>
<body>
<div class="alert">
⏰ Your karaoke video is waiting for you!
</div>
<div class="content">{html.escape(message_content)}</div>
<div class="signature">
{self._get_email_signature()}
</div>
</body>
</html>
"""
return self.provider.send_email(
to_email=to_email,
subject=subject,
html_content=html_content,
text_content=message_content,
)
# Global instance
_email_service = None
+11 -8
View File
@@ -351,8 +351,8 @@ class GCEEncodingBackend(EncodingBackend):
result = {}
raw_output_files = result.get("output_files", {})
# Convert output_files from list of paths to dict if needed
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
# Convert output_files from list of paths to dict
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
if isinstance(raw_output_files, list):
output_files = {}
@@ -360,15 +360,18 @@ class GCEEncodingBackend(EncodingBackend):
if not isinstance(path, str):
continue
filename = path.split("/")[-1] if "/" in path else path
filename_lower = filename.lower()
# Map filename patterns to output format keys
if "4k_lossless" in filename:
output_files["mp4_4k_lossless"] = path
elif "4k_lossy" in filename:
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
if "lossless 4k" in filename_lower:
if filename.endswith(".mkv"):
output_files["mkv_4k"] = path
else:
output_files["mp4_4k_lossless"] = path
elif "lossy 4k" in filename_lower:
output_files["mp4_4k_lossy"] = path
elif "720p" in filename:
elif "720p" in filename_lower:
output_files["mp4_720p"] = path
elif filename.endswith(".mkv"):
output_files["mkv_4k"] = path
self.logger.info(f"Converted output_files list to dict: {output_files}")
else:
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
+126
View File
@@ -63,6 +63,11 @@ class EncodingService:
"""Check if GCE encoding is enabled and configured."""
return self.settings.use_gce_encoding and self.is_configured
@property
def is_preview_enabled(self) -> bool:
"""Check if GCE preview encoding is enabled and configured."""
return self.settings.use_gce_preview_encoding and self.is_configured
async def submit_encoding_job(
self,
job_id: str,
@@ -242,6 +247,127 @@ class EncodingService:
job_id, progress_callback=progress_callback
)
async def submit_preview_encoding_job(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
font_gcs_path: Optional[str] = None,
) -> Dict[str, Any]:
"""
Submit a preview video encoding job to the GCE worker.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
font_gcs_path: Optional GCS path to custom font file
Returns:
Response from the encoding worker
Raises:
Exception: If submission fails
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._url}/encode-preview"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
"ass_gcs_path": ass_gcs_path,
"audio_gcs_path": audio_gcs_path,
"output_gcs_path": output_gcs_path,
"background_color": background_color,
}
if background_image_gcs_path:
payload["background_image_gcs_path"] = background_image_gcs_path
if font_gcs_path:
payload["font_gcs_path"] = font_gcs_path
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
async with aiohttp.ClientSession() as session:
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
if resp.status == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp.status == 409:
raise RuntimeError(f"Preview encoding job {job_id} already exists")
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"Failed to submit preview encoding job: {resp.status} - {text}")
return await resp.json()
async def encode_preview_video(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
font_gcs_path: Optional[str] = None,
timeout: float = 90.0,
poll_interval: float = 2.0,
) -> Dict[str, Any]:
"""
Submit preview encoding job and wait for completion.
This is a convenience method that combines submit + wait with shorter
timeouts suitable for preview videos.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
font_gcs_path: Optional GCS path to custom font file
timeout: Maximum time to wait (default 90s for preview)
poll_interval: Seconds between status checks (default 2s)
Returns:
Final job status with output files
"""
# Submit the job
submit_result = await self.submit_preview_encoding_job(
job_id=job_id,
ass_gcs_path=ass_gcs_path,
audio_gcs_path=audio_gcs_path,
output_gcs_path=output_gcs_path,
background_color=background_color,
background_image_gcs_path=background_image_gcs_path,
font_gcs_path=font_gcs_path,
)
# If cached, return immediately - video already exists in GCS
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
return {"status": "complete", "output_path": submit_result.get("output_path")}
# If in_progress, another request is encoding it - just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
# Wait for completion with shorter timeout
return await self.wait_for_completion(
job_id=job_id,
poll_interval=poll_interval,
timeout=timeout,
)
async def health_check(self) -> Dict[str, Any]:
"""
Check the health of the encoding worker.
+72
View File
@@ -0,0 +1,72 @@
# GCE Encoding Worker
HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.
## Overview
This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the `encoding-worker` GCE VM instance.
## Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/encode` | POST | Submit a full encoding job |
| `/encode-preview` | POST | Submit a preview encoding job (480x270, fast) |
| `/status/{job_id}` | GET | Get job status and progress |
| `/health` | GET | Health check |
## Authentication
All endpoints (except `/health`) require an API key via the `X-API-Key` header. The key is stored in Secret Manager (`encoding-worker-api-key`).
## Encoding Process
### Full Encoding (`/encode`)
1. Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
2. Uses `LocalEncodingService` from the karaoke-gen wheel to produce:
- Lossless 4K MP4
- Lossy 4K MP4
- Lossless MKV
- 720p MP4
3. Uploads results to GCS
4. Reports progress via status endpoint
### Preview Encoding (`/encode-preview`)
1. Downloads ASS subtitle and audio files from GCS
2. Runs FFmpeg to create a quick 480x270 preview video
3. Uploads result to GCS
4. Used for real-time preview in the web UI
## Dependencies
- **FFmpeg**: Static build from John Van Sickle
- **Python 3.13**: Built from source (required for karaoke-gen)
- **karaoke-gen wheel**: Downloaded from GCS at job start (enables hot updates)
## Deployment
This service is deployed via the VM's startup script. The script:
1. Installs system dependencies (FFmpeg, fonts, Python build deps)
2. Builds Python 3.13 from source
3. Creates venv and installs dependencies
4. Writes `main.py` to `/opt/encoding-worker/`
5. Creates systemd service
6. Starts the service on port 8080
## Local Development
This code is extracted from the infrastructure for maintainability. To test locally:
```bash
cd backend/services/gce_encoding
pip install -r requirements.txt
pip install ../../../ # Install karaoke-gen wheel
ENCODING_API_KEY=test python main.py
```
## Architecture Notes
- **Hot code updates**: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
- **In-memory job tracking**: Job state is stored in memory. Restart clears queue.
- **Parallel encoding**: ThreadPoolExecutor with 4 workers
- **GCS integration**: Direct download/upload via google-cloud-storage client
+22
View File
@@ -0,0 +1,22 @@
"""
GCE Encoding Worker Service.
This package contains the FastAPI application that runs on the GCE encoding worker VM.
It provides HTTP endpoints for video encoding jobs.
The service:
- Downloads input files from GCS
- Runs FFmpeg encoding via LocalEncodingService
- Uploads results back to GCS
- Provides job status tracking
Endpoints:
- POST /encode - Submit a full encoding job
- POST /encode-preview - Submit a preview encoding job (fast, low-res)
- GET /status/{job_id} - Get job status
- GET /health - Health check
"""
from .main import app
__all__ = ["app"]
+589
View File
@@ -0,0 +1,589 @@
import asyncio
import glob as glob_module
import json
import logging
import os
import subprocess
import sys
import tempfile
import uuid
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from typing import Optional
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
from google.cloud import storage
from pydantic import BaseModel
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
app = FastAPI(title="Encoding Worker", version="1.0.0")
# API key authentication
API_KEY = os.environ.get("ENCODING_API_KEY", "")
async def verify_api_key(x_api_key: str = Header(None)):
# Verify API key for authentication
if not API_KEY:
logger.warning("No API key configured - authentication disabled")
return True
if x_api_key != API_KEY:
raise HTTPException(status_code=401, detail="Invalid API key")
return True
# Job tracking
jobs: dict[str, dict] = {}
executor = ThreadPoolExecutor(max_workers=4) # 4 parallel encoding jobs
# GCS client
storage_client = storage.Client()
class EncodeRequest(BaseModel):
job_id: str
input_gcs_path: str # gs://bucket/path/to/inputs/
output_gcs_path: str # gs://bucket/path/to/outputs/
encoding_config: dict # Video formats to generate
class EncodePreviewRequest(BaseModel):
job_id: str
ass_gcs_path: str # gs://bucket/path/to/subtitles.ass
audio_gcs_path: str # gs://bucket/path/to/audio.flac
output_gcs_path: str # gs://bucket/path/to/output.mp4
background_color: str = "black"
background_image_gcs_path: Optional[str] = None
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
class JobStatus(BaseModel):
job_id: str
status: str # pending, running, complete, failed
progress: int # 0-100
error: Optional[str] = None
output_files: Optional[list[str]] = None
def download_from_gcs(gcs_uri: str, local_path: Path):
# Download a file or folder from GCS
# Parse gs://bucket/path
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
prefix = parts[1] if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
blobs = list(bucket.list_blobs(prefix=prefix))
for blob in blobs:
# Get relative path from prefix
rel_path = blob.name[len(prefix):].lstrip("/")
if not rel_path:
continue
dest = local_path / rel_path
dest.parent.mkdir(parents=True, exist_ok=True)
blob.download_to_filename(str(dest))
logger.info(f"Downloaded: {blob.name} -> {dest}")
def upload_to_gcs(local_path: Path, gcs_uri: str):
# Upload a file or folder to GCS
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
if local_path.is_file():
blob_name = f"{prefix}/{local_path.name}" if prefix else local_path.name
blob = bucket.blob(blob_name)
blob.upload_from_filename(str(local_path))
logger.info(f"Uploaded: {local_path} -> gs://{bucket_name}/{blob_name}")
else:
for file in local_path.rglob("*"):
if file.is_file():
rel_path = file.relative_to(local_path)
blob_name = f"{prefix}/{rel_path}" if prefix else str(rel_path)
blob = bucket.blob(blob_name)
blob.upload_from_filename(str(file))
logger.info(f"Uploaded: {file} -> gs://{bucket_name}/{blob_name}")
def download_single_file_from_gcs(gcs_uri: str, local_path: Path):
# Download a single file from GCS
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
blob_name = parts[1] if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
local_path.parent.mkdir(parents=True, exist_ok=True)
blob.download_to_filename(str(local_path))
logger.info(f"Downloaded: {gcs_uri} -> {local_path}")
def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
# Upload a single file to a specific GCS path
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
blob_name = parts[1] if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
blob.upload_from_filename(str(local_path))
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
# Run FFmpeg encoding for preview video (480x270, fast settings)
jobs[job_id]["status"] = "running"
jobs[job_id]["progress"] = 10
try:
# Download input files
ass_path = work_dir / "subtitles.ass"
audio_path = work_dir / "audio.flac"
download_single_file_from_gcs(request.ass_gcs_path, ass_path)
jobs[job_id]["progress"] = 20
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
jobs[job_id]["progress"] = 30
# Download background image if provided
bg_image_path = None
if request.background_image_gcs_path:
bg_image_path = work_dir / "background.png"
download_single_file_from_gcs(request.background_image_gcs_path, bg_image_path)
# Download custom font if provided and register with fontconfig
if request.font_gcs_path:
# Use standard fontconfig location that's already in the search path
fonts_dir = Path("/usr/local/share/fonts/custom")
fonts_dir.mkdir(parents=True, exist_ok=True)
font_filename = request.font_gcs_path.split("/")[-1]
font_path = fonts_dir / font_filename
download_single_file_from_gcs(request.font_gcs_path, font_path)
logger.info(f"Downloaded custom font: {font_path}")
# Update fontconfig cache so libass can find the font
subprocess.run(["fc-cache", "-fv"], capture_output=True)
logger.info(f"Updated fontconfig cache with custom font: {font_filename}")
# Build FFmpeg command
output_path = work_dir / "preview.mp4"
# Escape special characters in path for FFmpeg filter syntax
# FFmpeg filter parsing requires escaping: \ : , [ ] ;
def escape_ffmpeg_filter_path(path: str) -> str:
# Note: Extra escaping needed since this is inside a triple-quoted string in Pulumi
return path.replace("\\", "\\\\").replace(":", "\\:").replace(",", "\\,").replace("[", "\\[").replace("]", "\\]").replace(";", "\\;")
escaped_ass_path = escape_ffmpeg_filter_path(str(ass_path))
# Base command with frame rate
cmd = ["ffmpeg", "-y", "-r", "24"]
# Video input: background image or solid color
if bg_image_path and bg_image_path.exists():
cmd.extend(["-loop", "1", "-i", str(bg_image_path)])
# Scale and pad background to 480x270
vf = f"scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)/2:(oh-ih)/2,ass={escaped_ass_path}"
else:
# Solid color background
color = request.background_color or "black"
cmd.extend(["-f", "lavfi", "-i", f"color=c={color}:s=480x270:r=24"])
vf = f"ass={escaped_ass_path}"
# Audio input
cmd.extend(["-i", str(audio_path)])
# Video filter and encoding settings
cmd.extend([
"-vf", vf,
"-c:a", "aac", "-b:a", "96k",
"-c:v", "libx264",
"-preset", "superfast",
"-crf", "28",
"-pix_fmt", "yuv420p",
"-movflags", "+faststart",
"-threads", "8",
"-shortest",
str(output_path)
])
jobs[job_id]["progress"] = 40
logger.info(f"Running preview encoding: {' '.join(cmd)}")
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
logger.error(f"FFmpeg failed: {result.stderr}")
raise RuntimeError(f"FFmpeg preview encoding failed: {result.stderr[-500:]}")
jobs[job_id]["progress"] = 80
logger.info(f"Preview encoded: {output_path}")
# Upload output to GCS
upload_single_file_to_gcs(output_path, request.output_gcs_path)
jobs[job_id]["progress"] = 95
jobs[job_id]["output_files"] = [request.output_gcs_path]
return output_path
except Exception as e:
logger.error(f"Preview encoding failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
raise
def ensure_latest_wheel():
'''Download and install latest karaoke-gen wheel from GCS.
Called at the start of each job to enable hot code updates without restart.
In-progress jobs continue with their version, new jobs get latest code.
'''
try:
logger.info("Checking for latest karaoke-gen wheel in GCS...")
# Download latest wheel
result = subprocess.run(
["gsutil", "cp", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl", "/tmp/"],
capture_output=True, text=True, timeout=60
)
# Find the downloaded wheel (get the latest by version sorting)
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
if not wheels:
logger.warning("No wheel found in GCS, using fallback encoding logic")
return False
# Sort to get latest version
wheel_path = sorted(wheels)[-1]
logger.info(f"Installing wheel: {wheel_path}")
# Install (or upgrade) the wheel
# Use 5-minute timeout - first install at job start may need to resolve dependencies
# Subsequent installs are faster since dependencies are cached
install_result = subprocess.run(
[sys.executable, "-m", "pip", "install", "--upgrade", "--quiet", wheel_path],
capture_output=True, text=True, timeout=300
)
if install_result.returncode != 0:
logger.warning(f"Wheel installation failed: {install_result.stderr}")
return False
logger.info(f"Successfully installed wheel: {wheel_path}")
return True
except subprocess.TimeoutExpired:
logger.warning("Wheel download/install timed out, using fallback")
return False
except Exception as e:
logger.warning(f"Failed to ensure latest wheel: {e}")
return False
def find_file(work_dir: Path, *patterns):
'''Find a file matching any of the given glob patterns.'''
for pattern in patterns:
matches = list(work_dir.glob(f"**/{pattern}"))
if matches:
return matches[0]
return None
def run_encoding(job_id: str, work_dir: Path, config: dict):
'''Run encoding using LocalEncodingService (single source of truth).
Uses LocalEncodingService from the installed karaoke-gen wheel to ensure
output files match local CLI exactly:
- Proper names like "Artist - Title (Final Karaoke Lossless 4k).mp4"
- Concatenated title + karaoke + end screens
- All formats: lossless 4K MP4, lossy 4K MP4, lossless MKV, 720p MP4
Requires the karaoke-gen wheel to be installed (done by ensure_latest_wheel).
'''
jobs[job_id]["status"] = "running"
jobs[job_id]["progress"] = 10
try:
# Import LocalEncodingService from installed wheel (required, no fallback)
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
logger.info("Using LocalEncodingService from installed wheel")
# Get artist/title from config for proper naming
artist = config.get("artist", "Unknown Artist")
title = config.get("title", "Unknown Title")
base_name = f"{artist} - {title}"
logger.info(f"Encoding for: {base_name}")
# Find input files in work_dir
# Title/end screens are in screens/ subdirectory
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
# Karaoke video - search for With Vocals or main karaoke video
karaoke_video = find_file(
work_dir,
"*With Vocals*.mov", "*With Vocals*.mkv",
"*Vocals*.mov", "*Vocals*.mkv",
"*.mkv", "*.mov"
)
# Exclude title/end/output videos
if karaoke_video:
name_lower = karaoke_video.name.lower()
if "title" in name_lower or "end" in name_lower or "outputs" in str(karaoke_video):
# Search more specifically for karaoke video
karaoke_video = find_file(work_dir, "*Karaoke*.mkv", "*Karaoke*.mov", "*vocals*.mkv")
# Instrumental audio
instrumental = find_file(
work_dir,
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
logger.info(f"Found files:")
logger.info(f" Title video: {title_video}")
logger.info(f" Karaoke video: {karaoke_video}")
logger.info(f" End video: {end_video}")
logger.info(f" Instrumental: {instrumental}")
# Validate required files
if not title_video:
raise ValueError(f"No title video found in {work_dir}. Check screens/ subdirectory.")
if not karaoke_video:
raise ValueError(f"No karaoke video found in {work_dir}")
if not instrumental:
raise ValueError(f"No instrumental audio found in {work_dir}")
output_dir = work_dir / "outputs"
output_dir.mkdir(exist_ok=True)
jobs[job_id]["progress"] = 20
# Build encoding config with proper file names
encoding_config = EncodingConfig(
title_video=str(title_video),
karaoke_video=str(karaoke_video),
instrumental_audio=str(instrumental),
end_video=str(end_video) if end_video else None,
output_karaoke_mp4=str(output_dir / f"{base_name} (Karaoke).mp4"),
output_with_vocals_mp4=str(output_dir / f"{base_name} (With Vocals).mp4"),
output_lossless_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mp4"),
output_lossy_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 720p).mp4"),
)
# Create service and run encoding
service = LocalEncodingService(logger=logger)
jobs[job_id]["progress"] = 30
logger.info("Starting LocalEncodingService.encode_all_formats()")
result = service.encode_all_formats(encoding_config)
if not result.success:
raise RuntimeError(f"Encoding failed: {result.error}")
jobs[job_id]["progress"] = 90
# Collect output files
output_files = [str(f) for f in output_dir.glob("*") if f.is_file()]
jobs[job_id]["output_files"] = output_files
logger.info(f"Encoding complete. Output files: {output_files}")
return output_dir
except ImportError as e:
# No fallback - wheel must be installed
error_msg = (
f"LocalEncodingService not available: {e}. "
"The karaoke-gen wheel must be installed. "
"Check that ensure_latest_wheel() succeeded and wheel exists in GCS."
)
logger.error(error_msg)
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = error_msg
raise RuntimeError(error_msg) from e
except Exception as e:
logger.error(f"Encoding failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
raise
async def process_job(job_id: str, request: EncodeRequest):
# Process an encoding job asynchronously
try:
# Download and install latest wheel at job start (allows hot updates without restart)
# This means in-progress jobs continue with their version, new jobs get latest code
ensure_latest_wheel()
with tempfile.TemporaryDirectory() as temp_dir:
work_dir = Path(temp_dir) / "work"
work_dir.mkdir()
# Download input files
jobs[job_id]["progress"] = 5
logger.info(f"Downloading from {request.input_gcs_path}")
download_from_gcs(request.input_gcs_path, work_dir)
# Run encoding in thread pool (CPU-bound)
loop = asyncio.get_event_loop()
output_dir = await loop.run_in_executor(
executor,
run_encoding,
job_id,
work_dir,
request.encoding_config
)
# Upload outputs
jobs[job_id]["progress"] = 95
logger.info(f"Uploading to {request.output_gcs_path}")
upload_to_gcs(output_dir, request.output_gcs_path)
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
# We need paths like "jobs/id/encoded/Artist - Title (Final Karaoke Lossless 4k).mp4"
gcs_path = request.output_gcs_path.replace("gs://", "")
parts = gcs_path.split("/", 1)
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
local_output_files = jobs[job_id].get("output_files", [])
blob_paths = []
for local_path in local_output_files:
filename = Path(local_path).name
blob_path = f"{prefix}/{filename}" if prefix else filename
blob_paths.append(blob_path)
jobs[job_id]["output_files"] = blob_paths
logger.info(f"Output files (blob paths): {blob_paths}")
jobs[job_id]["status"] = "complete"
jobs[job_id]["progress"] = 100
logger.info(f"Job {job_id} complete")
except Exception as e:
logger.error(f"Job {job_id} failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
async def process_preview_job(job_id: str, request: EncodePreviewRequest):
# Process a preview encoding job asynchronously
try:
with tempfile.TemporaryDirectory() as temp_dir:
work_dir = Path(temp_dir) / "work"
work_dir.mkdir()
# Run preview encoding in thread pool (CPU-bound)
# Note: run_preview_encoding handles download/upload internally
loop = asyncio.get_event_loop()
await loop.run_in_executor(
executor,
run_preview_encoding,
job_id,
work_dir,
request
)
jobs[job_id]["status"] = "complete"
jobs[job_id]["progress"] = 100
logger.info(f"Preview job {job_id} complete")
except Exception as e:
logger.error(f"Preview job {job_id} failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
@app.post("/encode-preview")
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit a preview encoding job
job_id = request.job_id
# If job already exists, return cached result or current status
if job_id in jobs:
existing_job = jobs[job_id]
if existing_job["status"] == "complete":
# Return cached result - preview already encoded
return {"status": "cached", "job_id": job_id, "output_path": existing_job.get("output_path")}
elif existing_job["status"] == "failed":
# Previous attempt failed, allow retry by replacing the job
pass
else:
# Job is still in progress
return {"status": "in_progress", "job_id": job_id}
jobs[job_id] = {
"job_id": job_id,
"status": "pending",
"progress": 0,
"error": None,
"output_files": None,
}
background_tasks.add_task(process_preview_job, job_id, request)
return {"status": "accepted", "job_id": job_id}
@app.post("/encode")
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit an encoding job
job_id = request.job_id
if job_id in jobs:
raise HTTPException(status_code=409, detail=f"Job {job_id} already exists")
jobs[job_id] = {
"job_id": job_id,
"status": "pending",
"progress": 0,
"error": None,
"output_files": None,
}
background_tasks.add_task(process_job, job_id, request)
return {"status": "accepted", "job_id": job_id}
@app.get("/status/{job_id}")
async def get_job_status(job_id: str, _auth: bool = Depends(verify_api_key)) -> JobStatus:
# Get the status of an encoding job
if job_id not in jobs:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
return JobStatus(**jobs[job_id])
@app.get("/health")
async def health_check():
# Health check endpoint
active_jobs = sum(1 for j in jobs.values() if j["status"] == "running")
# Get karaoke-gen wheel version if installed
wheel_version = None
try:
from importlib.metadata import version as get_version
wheel_version = get_version("karaoke-gen")
except Exception:
pass
return {
"status": "ok",
"active_jobs": active_jobs,
"queue_length": sum(1 for j in jobs.values() if j["status"] == "pending"),
"ffmpeg_version": subprocess.run(["ffmpeg", "-version"], capture_output=True, text=True).stdout.split("\n")[0],
"wheel_version": wheel_version,
}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8080)
@@ -0,0 +1,16 @@
# GCE Encoding Worker dependencies
# These are installed in the VM's Python virtual environment
# Web framework
fastapi>=0.109.0
uvicorn>=0.27.0
# GCS client
google-cloud-storage>=2.14.0
# Async utilities (optional, for future improvements)
aiofiles>=23.2.0
aiohttp>=3.9.0
# Note: The karaoke-gen wheel is installed separately from GCS.
# It provides LocalEncodingService which does the actual encoding.
+139 -1
View File
@@ -359,9 +359,147 @@ class JobManager:
'instrumental_token': updates['instrumental_token'],
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
})
logger.info(f"Job {job_id} transitioned to {new_status}")
# Trigger notifications asynchronously (fire-and-forget)
self._trigger_state_notifications(job_id, new_status)
return True
def _trigger_state_notifications(self, job_id: str, new_status: JobStatus) -> None:
"""
Trigger email notifications based on state transitions.
This is fire-and-forget - notification failures don't affect job processing.
Args:
job_id: Job ID
new_status: New job status
"""
import asyncio
try:
# Get the job to access user info
job = self.get_job(job_id)
if not job or not job.user_email:
logger.debug(f"No user email for job {job_id}, skipping notifications")
return
# Job completion notification
if new_status == JobStatus.COMPLETE:
self._schedule_completion_email(job)
# Idle reminder scheduling for blocking states
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]:
self._schedule_idle_reminder(job, new_status)
except Exception as e:
# Never let notification failures affect job processing
logger.error(f"Error triggering notifications for job {job_id}: {e}")
def _schedule_completion_email(self, job: Job) -> None:
"""
Schedule sending a job completion email.
Uses asyncio to fire-and-forget the email sending.
"""
import asyncio
import threading
try:
from backend.services.job_notification_service import get_job_notification_service
notification_service = get_job_notification_service()
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
state_data = job.state_data or {}
youtube_url = state_data.get('youtube_url')
dropbox_url = state_data.get('dropbox_link')
brand_code = state_data.get('brand_code')
# Create async task (fire-and-forget)
async def send_email():
await notification_service.send_job_completion_email(
job_id=job.job_id,
user_email=job.user_email,
user_name=None, # Could fetch from user service if needed
artist=job.artist,
title=job.title,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
brand_code=brand_code,
)
# Try to get existing event loop, create new one if none exists
try:
loop = asyncio.get_running_loop()
# If we're in an async context, create a task
loop.create_task(send_email())
except RuntimeError:
# No event loop - we're likely in a sync context
# Use daemon thread to avoid blocking job completion
def run_in_thread():
asyncio.run(send_email())
thread = threading.Thread(target=run_in_thread, daemon=True)
thread.start()
logger.info(f"Scheduled completion email for job {job.job_id}")
except Exception as e:
logger.error(f"Failed to schedule completion email for job {job.job_id}: {e}")
def _schedule_idle_reminder(self, job: Job, new_status: JobStatus) -> None:
"""
Schedule an idle reminder for a blocking state.
Records the timestamp when the blocking state was entered and
schedules a Cloud Tasks task for 5 minutes later.
"""
import asyncio
import threading
try:
# Record when we entered blocking state (for idle detection)
blocking_entered_at = datetime.utcnow().isoformat()
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
# Update state_data with blocking state info (handle None state_data)
existing_state_data = job.state_data or {}
state_data_update = {
'blocking_state_entered_at': blocking_entered_at,
'blocking_action_type': action_type,
'reminder_sent': False, # Will be set to True after reminder is sent
}
self.firestore.update_job(job.job_id, {
'state_data': {**existing_state_data, **state_data_update}
})
# Schedule the idle reminder check via worker service (5 min delay)
from backend.services.worker_service import get_worker_service
async def schedule_reminder():
worker_service = get_worker_service()
await worker_service.schedule_idle_reminder(job.job_id)
# Try to get existing event loop, create new one if none exists
try:
loop = asyncio.get_running_loop()
loop.create_task(schedule_reminder())
except RuntimeError:
# No event loop - we're in a sync context
# Use daemon thread to avoid blocking job processing
def run_in_thread():
asyncio.run(schedule_reminder())
thread = threading.Thread(target=run_in_thread, daemon=True)
thread.start()
logger.info(f"Scheduled idle reminder for job {job.job_id} ({action_type})")
except Exception as e:
logger.error(f"Failed to schedule idle reminder for job {job.job_id}: {e}")
def update_state_data(self, job_id: str, key: str, value: Any) -> None:
"""
@@ -0,0 +1,271 @@
"""
Job notification service for sending emails on job state changes.
Handles:
- Job completion emails (when job enters COMPLETE status)
- Action reminder emails (when user is idle at blocking states)
This service orchestrates between the template service and email service,
building the complete notification flow for jobs.
"""
import logging
import os
from typing import Optional, Dict, Any
from backend.services.email_service import get_email_service
from backend.services.template_service import get_template_service
logger = logging.getLogger(__name__)
# Environment variable to enable/disable auto emails
ENABLE_AUTO_EMAILS = os.getenv("ENABLE_AUTO_EMAILS", "true").lower() == "true"
# Feedback form URL (configured per environment, empty by default to avoid placeholder in emails)
FEEDBACK_FORM_URL = os.getenv("FEEDBACK_FORM_URL", "")
def _mask_email(email: str) -> str:
"""Mask email for logging to protect PII. Shows first char + domain."""
if not email or "@" not in email:
return "***"
local, domain = email.split("@", 1)
if len(local) <= 1:
return f"*@{domain}"
return f"{local[0]}***@{domain}"
class JobNotificationService:
"""
Service for sending job-related email notifications.
Coordinates between template service (for message rendering) and
email service (for sending).
"""
def __init__(self):
"""Initialize notification service."""
self.email_service = get_email_service()
self.template_service = get_template_service()
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
self.backend_url = os.getenv("BACKEND_URL", "https://api.nomadkaraoke.com")
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None) -> str:
"""Build the lyrics review URL for a job."""
import urllib.parse
review_ui_url = os.getenv("REVIEW_UI_URL", f"{self.frontend_url}/lyrics/")
base_api_url = f"{self.backend_url}/api/review/{job_id}"
encoded_api_url = urllib.parse.quote(base_api_url, safe='')
url = f"{review_ui_url}?baseApiUrl={encoded_api_url}"
if audio_hash:
url += f"&audioHash={urllib.parse.quote(audio_hash, safe='')}"
if review_token:
url += f"&reviewToken={urllib.parse.quote(review_token, safe='')}"
return url
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None) -> str:
"""Build the instrumental selection URL for a job."""
base_api_url = f"{self.backend_url}/api/jobs/{job_id}"
import urllib.parse
encoded_api_url = urllib.parse.quote(base_api_url, safe='')
url = f"{self.frontend_url}/instrumental/?baseApiUrl={encoded_api_url}"
if instrumental_token:
url += f"&instrumentalToken={urllib.parse.quote(instrumental_token, safe='')}"
return url
async def send_job_completion_email(
self,
job_id: str,
user_email: str,
user_name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
brand_code: Optional[str] = None,
) -> bool:
"""
Send job completion email to user.
Args:
job_id: Job ID
user_email: User's email address
user_name: User's display name (optional)
artist: Artist name
title: Song title
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
brand_code: Release ID (e.g., "NOMAD-1178")
Returns:
True if email was sent successfully
"""
if not ENABLE_AUTO_EMAILS:
logger.info(f"Auto emails disabled, skipping completion email for job {job_id}")
return False
if not user_email:
logger.warning(f"No user email for job {job_id}, skipping completion email")
return False
try:
# Render the completion message using template service
message_content = self.template_service.render_job_completion(
name=user_name,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
artist=artist,
title=title,
job_id=job_id,
feedback_url=FEEDBACK_FORM_URL,
)
# Send the email with CC to admin
success = self.email_service.send_job_completion(
to_email=user_email,
message_content=message_content,
artist=artist,
title=title,
brand_code=brand_code,
cc_admin=True,
)
if success:
logger.info(f"Sent completion email for job {job_id} to {_mask_email(user_email)}")
else:
logger.error(f"Failed to send completion email for job {job_id}")
return success
except Exception as e:
logger.exception(f"Error sending completion email for job {job_id}: {e}")
return False
async def send_action_reminder_email(
self,
job_id: str,
user_email: str,
action_type: str,
user_name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
audio_hash: Optional[str] = None,
review_token: Optional[str] = None,
instrumental_token: Optional[str] = None,
) -> bool:
"""
Send action-needed reminder email to user.
Args:
job_id: Job ID
user_email: User's email address
action_type: Type of action needed ("lyrics" or "instrumental")
user_name: User's display name
artist: Artist name
title: Song title
audio_hash: Audio hash for review URL
review_token: Review token for URL
instrumental_token: Instrumental token for URL
Returns:
True if email was sent successfully
"""
if not ENABLE_AUTO_EMAILS:
logger.info(f"Auto emails disabled, skipping reminder for job {job_id}")
return False
if not user_email:
logger.warning(f"No user email for job {job_id}, skipping reminder")
return False
try:
# Render the appropriate template
if action_type == "lyrics":
review_url = self._build_review_url(job_id, audio_hash, review_token)
message_content = self.template_service.render_action_needed_lyrics(
name=user_name,
artist=artist,
title=title,
review_url=review_url,
)
elif action_type == "instrumental":
instrumental_url = self._build_instrumental_url(job_id, instrumental_token)
message_content = self.template_service.render_action_needed_instrumental(
name=user_name,
artist=artist,
title=title,
instrumental_url=instrumental_url,
)
else:
logger.error(f"Unknown action type: {action_type}")
return False
# Send the reminder email (no CC)
success = self.email_service.send_action_reminder(
to_email=user_email,
message_content=message_content,
action_type=action_type,
artist=artist,
title=title,
)
if success:
logger.info(f"Sent {action_type} reminder for job {job_id} to {_mask_email(user_email)}")
else:
logger.error(f"Failed to send {action_type} reminder for job {job_id}")
return success
except Exception as e:
logger.exception(f"Error sending {action_type} reminder for job {job_id}: {e}")
return False
def get_completion_message(
self,
job_id: str,
user_name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
) -> str:
"""
Get the rendered completion message for a job (for admin copy functionality).
Args:
job_id: Job ID
user_name: User's display name
artist: Artist name
title: Song title
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
Returns:
Rendered message content as plain text
"""
return self.template_service.render_job_completion(
name=user_name,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
artist=artist,
title=title,
job_id=job_id,
feedback_url=FEEDBACK_FORM_URL,
)
# Global instance
_job_notification_service: Optional[JobNotificationService] = None
def get_job_notification_service() -> JobNotificationService:
"""Get the global job notification service instance."""
global _job_notification_service
if _job_notification_service is None:
_job_notification_service = JobNotificationService()
return _job_notification_service
@@ -0,0 +1,407 @@
"""
Local Preview Encoding Service.
Provides local FFmpeg-based preview video encoding functionality, extracted from
VideoGenerator for use by the GCE worker. This ensures the same encoding logic
is used across local CLI, Cloud Run, and GCE worker environments.
This service handles:
- Generating preview videos with ASS subtitle overlay
- Hardware acceleration detection (NVENC) with fallback to software encoding
- Background image/color support
- Custom font support
- Optimized settings for fast preview generation
"""
import logging
import os
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Optional, List
logger = logging.getLogger(__name__)
@dataclass
class PreviewEncodingConfig:
"""Configuration for preview video encoding."""
ass_path: str # Path to ASS subtitles file
audio_path: str # Path to audio file
output_path: str # Path for output video file
# Optional background settings
background_image_path: Optional[str] = None # Path to background image
background_color: str = "black" # Fallback background color
# Optional font settings
font_path: Optional[str] = None # Path to custom font file
@dataclass
class PreviewEncodingResult:
"""Result of preview encoding operation."""
success: bool
output_path: Optional[str] = None
error: Optional[str] = None
class LocalPreviewEncodingService:
"""
Service for local FFmpeg-based preview video encoding.
This is the single source of truth for preview encoding logic, used by:
- Local CLI (via VideoGenerator which delegates here)
- Cloud Run (when GCE is unavailable)
- GCE worker (via installed wheel)
Supports hardware acceleration (NVENC) with automatic fallback
to software encoding (libx264) when unavailable.
"""
# Preview video settings - these are the canonical values
PREVIEW_WIDTH = 480
PREVIEW_HEIGHT = 270
PREVIEW_FPS = 24
PREVIEW_AUDIO_BITRATE = "96k"
def __init__(
self,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the local preview encoding service.
Args:
logger: Optional logger instance
"""
self.logger = logger or logging.getLogger(__name__)
# Hardware acceleration settings (detected on first use)
self._nvenc_available: Optional[bool] = None
self._video_encoder: Optional[str] = None
self._hwaccel_flags: Optional[List[str]] = None
def _detect_nvenc_support(self) -> bool:
"""
Detect if NVENC hardware encoding is available.
Returns:
True if NVENC is available, False otherwise
"""
try:
self.logger.info("Detecting NVENC hardware acceleration...")
# Test h264_nvenc encoder directly
test_cmd = [
"ffmpeg", "-hide_banner", "-loglevel", "error",
"-f", "lavfi", "-i", "testsrc=duration=1:size=320x240:rate=1",
"-c:v", "h264_nvenc", "-f", "null", "-"
]
result = subprocess.run(
test_cmd,
capture_output=True,
text=True,
timeout=30
)
if result.returncode == 0:
self.logger.info("NVENC hardware acceleration available")
return True
# Try alternative test with different source
alt_test_cmd = [
"ffmpeg", "-hide_banner", "-loglevel", "error",
"-f", "lavfi", "-i", "color=red:size=320x240:duration=0.1",
"-c:v", "h264_nvenc", "-preset", "fast", "-f", "null", "-"
]
alt_result = subprocess.run(
alt_test_cmd,
capture_output=True,
text=True,
timeout=30
)
if alt_result.returncode == 0:
self.logger.info("NVENC hardware acceleration available")
return True
self.logger.info("NVENC not available, using software encoding")
return False
except subprocess.TimeoutExpired:
self.logger.debug("NVENC detection timed out")
return False
except Exception as e:
self.logger.debug(f"NVENC detection failed: {e}")
return False
def _configure_hardware_acceleration(self) -> None:
"""Configure hardware acceleration settings based on detected capabilities."""
if self._nvenc_available is None:
self._nvenc_available = self._detect_nvenc_support()
if self._nvenc_available:
self._video_encoder = "h264_nvenc"
self._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
self.logger.info("Using NVENC hardware acceleration for preview encoding")
else:
self._video_encoder = "libx264"
self._hwaccel_flags = []
self.logger.info("Using software encoding (libx264) for preview")
@property
def nvenc_available(self) -> bool:
"""Check if NVENC hardware acceleration is available."""
if self._nvenc_available is None:
self._configure_hardware_acceleration()
return self._nvenc_available
@property
def video_encoder(self) -> str:
"""Get the video encoder to use."""
if self._video_encoder is None:
self._configure_hardware_acceleration()
return self._video_encoder
@property
def hwaccel_flags(self) -> List[str]:
"""Get hardware acceleration flags."""
if self._hwaccel_flags is None:
self._configure_hardware_acceleration()
return self._hwaccel_flags
def _escape_ffmpeg_filter_path(self, path: str) -> str:
"""
Escape a path for FFmpeg filter expressions (for subprocess without shell).
When using subprocess with a command list (no shell), FFmpeg receives the
filter string directly. FFmpeg's filter parser requires escaping:
- Backslashes: double them (\\ -> \\\\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\\\')
- Spaces: escape with backslash ( -> \\ )
- Special characters: :,[];
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
"""
# First escape existing backslashes
escaped = path.replace("\\", "\\\\")
# Escape single quotes
escaped = escaped.replace("'", "\\\\\\'")
# Escape spaces
escaped = escaped.replace(" ", "\\ ")
# Escape FFmpeg filter special characters
escaped = escaped.replace(":", "\\:")
escaped = escaped.replace(",", "\\,")
escaped = escaped.replace("[", "\\[")
escaped = escaped.replace("]", "\\]")
escaped = escaped.replace(";", "\\;")
return escaped
def _build_ass_filter(self, ass_path: str, font_path: Optional[str] = None) -> str:
"""
Build ASS filter with optional font directory support.
Args:
ass_path: Path to ASS subtitles file
font_path: Optional path to custom font file
Returns:
FFmpeg ASS filter string
"""
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
ass_filter = f"ass={escaped_ass_path}"
if font_path and os.path.isfile(font_path):
font_dir = os.path.dirname(font_path)
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
ass_filter += f":fontsdir={escaped_font_dir}"
self.logger.debug(f"Using font directory: {font_dir}")
return ass_filter
def _build_preview_ffmpeg_command(self, config: PreviewEncodingConfig) -> List[str]:
"""
Build FFmpeg command for preview video generation.
This is the canonical preview encoding command, ensuring consistency
across all environments (local CLI, Cloud Run, GCE worker).
Args:
config: Preview encoding configuration
Returns:
FFmpeg command as a list of arguments
"""
width, height = self.PREVIEW_WIDTH, self.PREVIEW_HEIGHT
cmd = [
"ffmpeg",
"-hide_banner",
"-loglevel", "error",
"-r", str(self.PREVIEW_FPS),
]
# Add hardware acceleration flags if available
cmd.extend(self.hwaccel_flags)
# Input source (background image or solid color)
if config.background_image_path and os.path.isfile(config.background_image_path):
self.logger.debug(f"Using background image: {config.background_image_path}")
cmd.extend([
"-loop", "1",
"-i", config.background_image_path,
])
# Build video filter with scaling and ASS subtitles
video_filter = (
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
f"{self._build_ass_filter(config.ass_path, config.font_path)}"
)
else:
self.logger.debug(f"Using solid {config.background_color} background")
cmd.extend([
"-f", "lavfi",
"-i", f"color=c={config.background_color}:s={width}x{height}:r={self.PREVIEW_FPS}",
])
# Just ASS subtitles, no scaling needed
video_filter = self._build_ass_filter(config.ass_path, config.font_path)
cmd.extend([
"-i", config.audio_path,
"-vf", video_filter,
"-c:a", "aac",
"-b:a", self.PREVIEW_AUDIO_BITRATE,
"-c:v", self.video_encoder,
])
# Add encoder-specific settings optimized for speed
if self.nvenc_available:
cmd.extend([
"-preset", "p1", # Fastest NVENC preset
"-tune", "ll", # Low latency
"-rc", "cbr", # Constant bitrate for speed
"-b:v", "800k", # Lower bitrate for speed
"-profile:v", "baseline", # Most compatible profile
"-level", "3.1", # Lower level for speed
])
self.logger.debug("Using NVENC with maximum speed settings")
else:
cmd.extend([
"-profile:v", "baseline",
"-level", "3.0",
"-preset", "superfast",
"-tune", "fastdecode",
"-b:v", "600k",
"-maxrate", "800k",
"-bufsize", "1200k",
"-crf", "28",
])
self.logger.debug("Using software encoding with maximum speed settings")
cmd.extend([
"-pix_fmt", "yuv420p", # Required for browser compatibility
"-movflags", "+faststart+frag_keyframe+empty_moov+dash",
"-g", "48", # Keyframe every 48 frames (2 seconds at 24fps)
"-keyint_min", "48",
"-sc_threshold", "0", # Disable scene change detection for speed
"-threads", "0", # Use all available CPU threads
"-shortest",
"-y",
config.output_path,
])
return cmd
def encode_preview(self, config: PreviewEncodingConfig) -> PreviewEncodingResult:
"""
Encode a preview video.
Args:
config: Preview encoding configuration
Returns:
PreviewEncodingResult with success status and output path
"""
self.logger.info(f"Encoding preview video: {config.output_path}")
# Validate input files
if not os.path.isfile(config.ass_path):
return PreviewEncodingResult(
success=False,
error=f"ASS subtitles file not found: {config.ass_path}"
)
if not os.path.isfile(config.audio_path):
return PreviewEncodingResult(
success=False,
error=f"Audio file not found: {config.audio_path}"
)
# Ensure output directory exists
output_dir = os.path.dirname(config.output_path)
if output_dir:
os.makedirs(output_dir, exist_ok=True)
try:
# Build and execute FFmpeg command
cmd = self._build_preview_ffmpeg_command(config)
self.logger.debug(f"FFmpeg command: {' '.join(cmd)}")
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=300 # 5 minute timeout for preview encoding
)
if result.returncode != 0:
error_msg = result.stderr[-500:] if result.stderr else "Unknown error"
self.logger.error(f"FFmpeg failed: {error_msg}")
return PreviewEncodingResult(
success=False,
error=f"FFmpeg preview encoding failed: {error_msg}"
)
self.logger.info(f"Preview encoded successfully: {config.output_path}")
return PreviewEncodingResult(
success=True,
output_path=config.output_path
)
except subprocess.TimeoutExpired:
self.logger.error("Preview encoding timed out")
return PreviewEncodingResult(
success=False,
error="Preview encoding timed out after 5 minutes"
)
except Exception as e:
self.logger.error(f"Preview encoding failed: {e}")
return PreviewEncodingResult(
success=False,
error=str(e)
)
# Singleton instance and factory function
_local_preview_encoding_service: Optional[LocalPreviewEncodingService] = None
def get_local_preview_encoding_service(**kwargs) -> LocalPreviewEncodingService:
"""
Get a local preview encoding service instance.
Args:
**kwargs: Arguments passed to LocalPreviewEncodingService
Returns:
LocalPreviewEncodingService instance
"""
global _local_preview_encoding_service
if _local_preview_encoding_service is None:
_local_preview_encoding_service = LocalPreviewEncodingService(**kwargs)
return _local_preview_encoding_service
+330
View File
@@ -0,0 +1,330 @@
"""
Template service for managing email templates stored in GCS.
Templates are stored in GCS bucket and rendered with job-specific variables.
This allows updating email content without code deployment.
Template locations:
- gs://{bucket}/templates/job-completion.txt - Job completion email (plain text)
- gs://{bucket}/templates/action-needed-lyrics.txt - Lyrics review reminder
- gs://{bucket}/templates/action-needed-instrumental.txt - Instrumental selection reminder
"""
import logging
import re
from typing import Optional, Dict, Any
from backend.config import get_settings
logger = logging.getLogger(__name__)
# Default templates (fallback if GCS fetch fails)
DEFAULT_JOB_COMPLETION_TEMPLATE = """Hi {name},
Thanks for your order!
Here's the link for the karaoke video published to YouTube:
{youtube_url}
Here's the dropbox folder with all the finished files and source files, including:
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
- "(Title).mov"/"(End).mov": title card and end screen videos
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
- lyrics/*.txt song lyrics from various sources in plain text format
{dropbox_url}
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
{feedback_url}
Thanks again and have a great day!
-Andrew
"""
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE = """Hi {name},
Your karaoke video for "{artist} - {title}" is ready for lyrics review!
The AI has transcribed and synchronized the lyrics, but they may need some corrections. Please review and approve them so we can finish generating your video.
Review your lyrics here:
{review_url}
This usually takes just a few minutes.
Thanks!
-Nomad Karaoke
"""
DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE = """Hi {name},
Your karaoke video for "{artist} - {title}" is almost done!
We've separated the audio into different versions. Please select which instrumental track you'd like to use for the final video.
Select your instrumental here:
{instrumental_url}
Thanks!
-Nomad Karaoke
"""
class TemplateService:
"""
Service for fetching and rendering email templates from GCS.
Templates support the following variables:
- {name} - User's display name or "there" if unknown
- {youtube_url} - YouTube video URL
- {dropbox_url} - Dropbox folder URL
- {artist} - Artist name
- {title} - Song title
- {job_id} - Job ID
- {review_url} - Lyrics review URL
- {instrumental_url} - Instrumental selection URL
- {feedback_url} - Feedback form URL
"""
TEMPLATE_PREFIX = "templates/"
def __init__(self):
"""Initialize template service."""
self.settings = get_settings()
self._storage_client = None
self._bucket = None
@property
def storage_client(self):
"""Lazy-initialize storage client."""
if self._storage_client is None:
from google.cloud import storage
self._storage_client = storage.Client(project=self.settings.google_cloud_project)
return self._storage_client
@property
def bucket(self):
"""Get the GCS bucket for templates."""
if self._bucket is None:
self._bucket = self.storage_client.bucket(self.settings.gcs_bucket_name)
return self._bucket
def _fetch_template_from_gcs(self, template_name: str) -> Optional[str]:
"""
Fetch a template from GCS.
Args:
template_name: Name of template file (e.g., "job-completion.txt")
Returns:
Template content as string, or None if not found
"""
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
try:
blob = self.bucket.blob(blob_path)
if blob.exists():
content = blob.download_as_text()
logger.debug(f"Fetched template from GCS: {blob_path}")
return content
else:
logger.warning(f"Template not found in GCS: {blob_path}")
return None
except Exception as e:
logger.error(f"Failed to fetch template {blob_path}: {e}")
return None
def get_job_completion_template(self) -> str:
"""Get the job completion email template."""
template = self._fetch_template_from_gcs("job-completion.txt")
if template is None:
logger.info("Using default job completion template")
return DEFAULT_JOB_COMPLETION_TEMPLATE
return template
def get_action_needed_lyrics_template(self) -> str:
"""Get the lyrics review reminder template."""
template = self._fetch_template_from_gcs("action-needed-lyrics.txt")
if template is None:
logger.info("Using default lyrics reminder template")
return DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
return template
def get_action_needed_instrumental_template(self) -> str:
"""Get the instrumental selection reminder template."""
template = self._fetch_template_from_gcs("action-needed-instrumental.txt")
if template is None:
logger.info("Using default instrumental reminder template")
return DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
return template
def render_template(self, template: str, variables: Dict[str, Any]) -> str:
"""
Render a template with the given variables.
Missing variables are replaced with empty strings.
Handles conditional sections like feedback URL.
Args:
template: Template string with {variable} placeholders
variables: Dictionary of variable values
Returns:
Rendered template string
"""
result = template
# Handle feedback URL section - remove if not provided
if not variables.get("feedback_url"):
# Remove feedback section (lines containing feedback_url placeholder and surrounding text)
result = re.sub(
r'\n*If you have a moment.*?\{feedback_url\}\n*',
'\n',
result,
flags=re.DOTALL
)
variables["feedback_url"] = ""
# Replace all variables
for key, value in variables.items():
placeholder = "{" + key + "}"
result = result.replace(placeholder, str(value) if value else "")
# Clean up any remaining unreplaced placeholders
result = re.sub(r'\{[a-z_]+\}', '', result)
return result.strip()
def render_job_completion(
self,
name: Optional[str] = None,
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
job_id: Optional[str] = None,
feedback_url: Optional[str] = None,
) -> str:
"""
Render the job completion email template.
Args:
name: User's display name (defaults to "there")
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
artist: Artist name
title: Song title
job_id: Job ID
feedback_url: Feedback form URL (optional)
Returns:
Rendered email content
"""
template = self.get_job_completion_template()
variables = {
"name": name or "there",
"youtube_url": youtube_url or "[YouTube URL not available]",
"dropbox_url": dropbox_url or "[Dropbox URL not available]",
"artist": artist or "Unknown Artist",
"title": title or "Unknown Title",
"job_id": job_id or "",
"feedback_url": feedback_url,
}
return self.render_template(template, variables)
def render_action_needed_lyrics(
self,
name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
review_url: str = "",
) -> str:
"""
Render the lyrics review reminder template.
Args:
name: User's display name
artist: Artist name
title: Song title
review_url: Lyrics review URL
Returns:
Rendered email content
"""
template = self.get_action_needed_lyrics_template()
variables = {
"name": name or "there",
"artist": artist or "Unknown Artist",
"title": title or "Unknown Title",
"review_url": review_url,
}
return self.render_template(template, variables)
def render_action_needed_instrumental(
self,
name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
instrumental_url: str = "",
) -> str:
"""
Render the instrumental selection reminder template.
Args:
name: User's display name
artist: Artist name
title: Song title
instrumental_url: Instrumental selection URL
Returns:
Rendered email content
"""
template = self.get_action_needed_instrumental_template()
variables = {
"name": name or "there",
"artist": artist or "Unknown Artist",
"title": title or "Unknown Title",
"instrumental_url": instrumental_url,
}
return self.render_template(template, variables)
def upload_template(self, template_name: str, content: str) -> bool:
"""
Upload a template to GCS.
Args:
template_name: Name of template file (e.g., "job-completion.txt")
content: Template content
Returns:
True if upload successful
"""
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
try:
blob = self.bucket.blob(blob_path)
blob.upload_from_string(content, content_type="text/plain")
logger.info(f"Uploaded template to GCS: {blob_path}")
return True
except Exception as e:
logger.error(f"Failed to upload template {blob_path}: {e}")
return False
# Global instance
_template_service: Optional[TemplateService] = None
def get_template_service() -> TemplateService:
"""Get the global template service instance."""
global _template_service
if _template_service is None:
_template_service = TemplateService()
return _template_service
+98
View File
@@ -37,6 +37,7 @@ WORKER_QUEUES = {
"screens": "screens-worker-queue",
"render-video": "render-worker-queue",
"video": "video-worker-queue",
"idle-reminder": "idle-reminder-queue", # For delayed idle reminder checks
}
# Dispatch deadlines for each worker type (in seconds)
@@ -49,8 +50,12 @@ WORKER_DISPATCH_DEADLINES = {
"screens": 600, # 10 min - Screen generation is fast
"render-video": 1800, # 30 min - Video encoding can be slow
"video": 1800, # 30 min - Video encoding can be slow
"idle-reminder": 60, # 1 min - Quick check and potential email send
}
# Default delay for idle reminders (seconds)
IDLE_REMINDER_DELAY_SECONDS = 5 * 60 # 5 minutes
class WorkerService:
"""
@@ -430,6 +435,99 @@ class WorkerService:
"""Trigger render video worker (post-review)."""
return await self.trigger_worker("render-video", job_id)
async def schedule_idle_reminder(
self,
job_id: str,
delay_seconds: int = IDLE_REMINDER_DELAY_SECONDS
) -> bool:
"""
Schedule an idle reminder check for a job.
The reminder task will be delivered after the specified delay.
When the task executes, it checks if the job is still in a blocking
state and sends a reminder email if the user hasn't taken action.
Args:
job_id: Job ID to check
delay_seconds: Delay before executing the check (default: 5 minutes)
Returns:
True if task was scheduled successfully, False otherwise
"""
if not self._use_cloud_tasks:
# In development mode, log and skip (no delayed execution support)
logger.info(
f"[job:{job_id}] Idle reminder not scheduled (Cloud Tasks disabled). "
f"Would have fired in {delay_seconds}s."
)
return True
try:
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import time
queue_name = WORKER_QUEUES.get("idle-reminder")
if not queue_name:
logger.error("Idle reminder queue not configured")
return False
project = self.settings.google_cloud_project
if not project:
logger.error("GOOGLE_CLOUD_PROJECT not set")
return False
location = self.settings.gcp_region
# Build queue path
parent = self.tasks_client.queue_path(project, location, queue_name)
# Build headers
headers = {
"Content-Type": "application/json",
}
if self._admin_token:
headers["X-Admin-Token"] = self._admin_token
# Inject trace context
headers = inject_trace_context(headers)
# Calculate schedule time
schedule_time = timestamp_pb2.Timestamp()
schedule_time.FromSeconds(int(time.time()) + delay_seconds)
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get("idle-reminder", 60)
# Build task payload
task = {
"http_request": {
"http_method": tasks_v2.HttpMethod.POST,
"url": f"{self._base_url}/api/internal/jobs/{job_id}/check-idle-reminder",
"headers": headers,
"body": json.dumps({"job_id": job_id}).encode(),
"oidc_token": {
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
},
},
"schedule_time": schedule_time,
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
}
# Create task
response = self.tasks_client.create_task(parent=parent, task=task)
logger.info(
f"[job:{job_id}] Scheduled idle reminder check in {delay_seconds}s: {response.name}"
)
return True
except Exception as e:
logger.error(
f"[job:{job_id}] Failed to schedule idle reminder: {e}",
exc_info=True
)
return False
# Global worker service instance
_worker_service: Optional[WorkerService] = None
+411
View File
@@ -0,0 +1,411 @@
"""
Unit tests for admin email endpoints.
Tests the completion message and send email API endpoints.
"""
import pytest
from unittest.mock import Mock, patch, MagicMock, AsyncMock
from fastapi.testclient import TestClient
from fastapi import FastAPI
from backend.api.routes.admin import router
from backend.api.dependencies import require_admin
from backend.models.job import Job, JobStatus
# Create a test app with the admin router
# The router already has prefix="/admin", so we add /api prefix
app = FastAPI()
app.include_router(router, prefix="/api")
def get_mock_admin():
"""Override for require_admin dependency."""
return ("admin@example.com", "admin", 1)
# Override the require_admin dependency
app.dependency_overrides[require_admin] = get_mock_admin
@pytest.fixture
def client():
"""Create a test client."""
return TestClient(app)
@pytest.fixture
def mock_job():
"""Create a mock job."""
job = Mock(spec=Job)
job.job_id = "test-job-123"
job.user_email = "user@example.com"
job.artist = "Test Artist"
job.title = "Test Song"
job.status = JobStatus.COMPLETE
job.audio_hash = "hash123"
job.review_token = "review123"
job.instrumental_token = "inst123"
job.state_data = {
"youtube_url": "https://youtube.com/watch?v=test123",
"dropbox_link": "https://dropbox.com/folder/test",
"brand_code": "NOMAD-1234",
}
return job
class TestGetCompletionMessage:
"""Tests for GET /api/admin/jobs/{job_id}/completion-message endpoint."""
def test_returns_completion_message(self, client, mock_job):
"""Test successful completion message retrieval."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
# Setup mocks
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Your video is ready!"
mock_get_ns.return_value = mock_ns
response = client.get(
"/api/admin/jobs/test-job-123/completion-message",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["job_id"] == "test-job-123"
assert data["message"] == "Your video is ready!"
# Subject format: "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
assert data["subject"] == "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
assert data["youtube_url"] == "https://youtube.com/watch?v=test123"
assert data["dropbox_url"] == "https://dropbox.com/folder/test"
def test_returns_404_when_job_not_found(self, client):
"""Test 404 when job doesn't exist."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = None
mock_jm_class.return_value = mock_jm
response = client.get(
"/api/admin/jobs/nonexistent-job/completion-message",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 404
def test_handles_none_state_data(self, client, mock_job):
"""Test handling of job with None state_data."""
mock_job.state_data = None
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Your video is ready!"
mock_get_ns.return_value = mock_ns
response = client.get(
"/api/admin/jobs/test-job-123/completion-message",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["youtube_url"] is None
assert data["dropbox_url"] is None
def test_default_subject_without_song_info(self, client, mock_job):
"""Test default subject when no artist/title."""
mock_job.artist = None
mock_job.title = None
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Message"
mock_get_ns.return_value = mock_ns
response = client.get(
"/api/admin/jobs/test-job-123/completion-message",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["subject"] == "Your karaoke video is ready!"
class TestSendCompletionEmail:
"""Tests for POST /api/admin/jobs/{job_id}/send-completion-email endpoint."""
def test_sends_email_successfully(self, client, mock_job):
"""Test successful email sending."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
patch('backend.services.email_service.get_email_service') as mock_get_es:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Your video is ready!"
mock_get_ns.return_value = mock_ns
mock_es = Mock()
mock_es.send_job_completion.return_value = True
mock_get_es.return_value = mock_es
response = client.post(
"/api/admin/jobs/test-job-123/send-completion-email",
json={"to_email": "customer@example.com", "cc_admin": True},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["success"] is True
assert data["job_id"] == "test-job-123"
assert data["to_email"] == "customer@example.com"
# Verify email was sent with correct params
mock_es.send_job_completion.assert_called_once_with(
to_email="customer@example.com",
message_content="Your video is ready!",
artist="Test Artist",
title="Test Song",
brand_code="NOMAD-1234",
cc_admin=True,
)
def test_sends_email_without_cc(self, client, mock_job):
"""Test email sending without CC."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
patch('backend.services.email_service.get_email_service') as mock_get_es:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Message"
mock_get_ns.return_value = mock_ns
mock_es = Mock()
mock_es.send_job_completion.return_value = True
mock_get_es.return_value = mock_es
response = client.post(
"/api/admin/jobs/test-job-123/send-completion-email",
json={"to_email": "customer@example.com", "cc_admin": False},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
# Verify CC was not included
call_kwargs = mock_es.send_job_completion.call_args.kwargs
assert call_kwargs["cc_admin"] is False
def test_returns_404_when_job_not_found(self, client):
"""Test 404 when job doesn't exist."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = None
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/nonexistent-job/send-completion-email",
json={"to_email": "customer@example.com"},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 404
def test_returns_500_when_email_fails(self, client, mock_job):
"""Test 500 when email sending fails."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
patch('backend.services.email_service.get_email_service') as mock_get_es:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Message"
mock_get_ns.return_value = mock_ns
mock_es = Mock()
mock_es.send_job_completion.return_value = False # Email failed
mock_get_es.return_value = mock_es
response = client.post(
"/api/admin/jobs/test-job-123/send-completion-email",
json={"to_email": "customer@example.com"},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 500
assert "Failed to send email" in response.json()["detail"]
def test_handles_none_state_data(self, client, mock_job):
"""Test handling of job with None state_data."""
mock_job.state_data = None
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
patch('backend.services.email_service.get_email_service') as mock_get_es:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Message"
mock_get_ns.return_value = mock_ns
mock_es = Mock()
mock_es.send_job_completion.return_value = True
mock_get_es.return_value = mock_es
response = client.post(
"/api/admin/jobs/test-job-123/send-completion-email",
json={"to_email": "customer@example.com"},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
def test_accepts_any_email_string(self, client, mock_job):
"""Test that any string is currently accepted as email (no validation).
Documents current behavior: FastAPI/Pydantic doesn't validate email
format by default. If validation is added later, this test should
be updated to expect 422 for invalid emails.
"""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
patch('backend.services.email_service.get_email_service') as mock_get_es:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Message"
mock_get_ns.return_value = mock_ns
mock_es = Mock()
mock_es.send_job_completion.return_value = True
mock_get_es.return_value = mock_es
response = client.post(
"/api/admin/jobs/test-job-123/send-completion-email",
json={"to_email": "invalid-email"}, # Not a valid email format
headers={"Authorization": "Bearer admin-token"}
)
# Currently accepts any string - endpoint succeeds
assert response.status_code == 200
# Verify the invalid email was passed through
mock_es.send_job_completion.assert_called_once()
call_kwargs = mock_es.send_job_completion.call_args.kwargs
assert call_kwargs["to_email"] == "invalid-email"
class TestIdleReminderEndpoint:
"""Tests for the internal idle reminder endpoint."""
def test_sends_reminder_when_still_idle(self, mock_job):
"""Test that reminder is sent when user is still idle."""
from backend.api.routes.internal import router as internal_router
from backend.api.dependencies import require_admin
# Create test app for internal router
# The router already has prefix="/internal", so we add /api prefix
test_app = FastAPI()
test_app.include_router(internal_router, prefix="/api")
test_app.dependency_overrides[require_admin] = get_mock_admin
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
mock_job.state_data = {
"blocking_state_entered_at": "2024-01-01T00:00:00",
"blocking_action_type": "lyrics",
"reminder_sent": False,
}
with patch('backend.api.routes.internal.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.firestore = Mock() # Mock firestore for update_job call
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.send_action_reminder_email = AsyncMock(return_value=True)
mock_get_ns.return_value = mock_ns
test_client = TestClient(test_app)
response = test_client.post(
"/api/internal/jobs/test-job-123/check-idle-reminder",
headers={"Authorization": "Bearer admin-token"}
)
# Should succeed and return "sent" status
assert response.status_code == 200
data = response.json()
assert data["status"] == "sent"
assert data["job_id"] == "test-job-123"
def test_skips_reminder_when_already_sent(self, mock_job):
"""Test that reminder is skipped if already sent."""
from backend.api.routes.internal import router as internal_router
from backend.api.dependencies import require_admin
# Create test app for internal router
# The router already has prefix="/internal", so we add /api prefix
test_app = FastAPI()
test_app.include_router(internal_router, prefix="/api")
test_app.dependency_overrides[require_admin] = get_mock_admin
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
mock_job.state_data = {
"blocking_state_entered_at": "2024-01-01T00:00:00",
"blocking_action_type": "lyrics",
"reminder_sent": True, # Already sent
}
with patch('backend.api.routes.internal.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
test_client = TestClient(test_app)
response = test_client.post(
"/api/internal/jobs/test-job-123/check-idle-reminder",
headers={"Authorization": "Bearer admin-token"}
)
# Should succeed with "already_sent" status
assert response.status_code == 200
data = response.json()
assert data["status"] == "already_sent"
assert data["job_id"] == "test-job-123"
+492
View File
@@ -0,0 +1,492 @@
"""
Unit tests for email service.
Tests the new job completion and action reminder email methods,
as well as CC support.
"""
import pytest
from unittest.mock import Mock, patch, MagicMock
from backend.services.email_service import (
EmailService,
EmailProvider,
ConsoleEmailProvider,
SendGridEmailProvider,
get_email_service,
)
class TestConsoleEmailProvider:
"""Tests for console email provider."""
def test_send_email_basic(self):
"""Test basic email logging."""
provider = ConsoleEmailProvider()
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test content</p>",
)
assert result is True
def test_send_email_with_text_content(self):
"""Test email with text content."""
provider = ConsoleEmailProvider()
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
text_content="Test plain text",
)
assert result is True
def test_send_email_with_cc(self):
"""Test email with CC recipients."""
provider = ConsoleEmailProvider()
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
cc_emails=["cc1@example.com", "cc2@example.com"],
)
assert result is True
class TestSendGridEmailProvider:
"""Tests for SendGrid email provider."""
def test_send_email_success(self):
"""Test successful email sending via SendGrid."""
provider = SendGridEmailProvider(
api_key="test-api-key",
from_email="from@example.com",
from_name="Test Sender",
)
mock_response = Mock()
mock_response.status_code = 202
# Patch at the sendgrid module level since it's imported inline
with patch('sendgrid.SendGridAPIClient') as mock_sg:
mock_client = Mock()
mock_client.send.return_value = mock_response
mock_sg.return_value = mock_client
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
)
assert result is True
mock_client.send.assert_called_once()
def test_send_email_with_cc(self):
"""Test email sending with CC via SendGrid."""
provider = SendGridEmailProvider(
api_key="test-api-key",
from_email="from@example.com",
)
mock_response = Mock()
mock_response.status_code = 202
with patch('sendgrid.SendGridAPIClient') as mock_sg:
mock_client = Mock()
mock_client.send.return_value = mock_response
mock_sg.return_value = mock_client
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
cc_emails=["cc@example.com"],
)
assert result is True
def test_send_email_failure_status(self):
"""Test email sending failure due to bad status."""
provider = SendGridEmailProvider(
api_key="test-api-key",
from_email="from@example.com",
)
mock_response = Mock()
mock_response.status_code = 400
with patch('sendgrid.SendGridAPIClient') as mock_sg:
mock_client = Mock()
mock_client.send.return_value = mock_response
mock_sg.return_value = mock_client
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
)
assert result is False
def test_send_email_exception(self):
"""Test email sending exception handling."""
provider = SendGridEmailProvider(
api_key="test-api-key",
from_email="from@example.com",
)
with patch('sendgrid.SendGridAPIClient') as mock_sg:
mock_sg.return_value.send.side_effect = Exception("API error")
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
)
assert result is False
class TestEmailServiceJobCompletion:
"""Tests for job completion email method."""
def test_send_job_completion_basic(self):
"""Test basic job completion email."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
result = service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
)
assert result is True
service.provider.send_email.assert_called_once()
def test_send_job_completion_with_song_info(self):
"""Test job completion email includes song in subject."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
artist="Test Artist",
title="Test Song",
)
call_args = service.provider.send_email.call_args
subject = call_args.kwargs.get('subject') or call_args[0][1]
assert "Test Artist" in subject
assert "Test Song" in subject
def test_send_job_completion_with_brand_code(self):
"""Test job completion email includes brand code in subject."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
artist="Seether",
title="Tonight",
brand_code="NOMAD-1178",
)
call_args = service.provider.send_email.call_args
subject = call_args.kwargs.get('subject') or call_args[0][1]
# Subject format: "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
assert subject == "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
def test_send_job_completion_default_subject(self):
"""Test job completion email default subject without song info."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
)
call_args = service.provider.send_email.call_args
subject = call_args.kwargs.get('subject') or call_args[0][1]
assert "karaoke video is ready" in subject.lower()
def test_send_job_completion_with_cc(self):
"""Test job completion email with CC to admin."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
cc_admin=True,
)
call_kwargs = service.provider.send_email.call_args.kwargs
assert "gen@nomadkaraoke.com" in call_kwargs.get('cc_emails', [])
def test_send_job_completion_without_cc(self):
"""Test job completion email without CC."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
cc_admin=False,
)
call_kwargs = service.provider.send_email.call_args.kwargs
assert call_kwargs.get('cc_emails') is None
def test_send_job_completion_escapes_html(self):
"""Test that message content is HTML-escaped."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Test <script>alert('xss')</script>",
)
call_kwargs = service.provider.send_email.call_args.kwargs
html_content = call_kwargs.get('html_content')
assert "<script>" not in html_content
assert "&lt;script&gt;" in html_content
def test_send_job_completion_includes_plain_text(self):
"""Test that plain text content is included."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
message = "Your video is ready!"
service.send_job_completion(
to_email="user@example.com",
message_content=message,
)
call_kwargs = service.provider.send_email.call_args.kwargs
assert call_kwargs.get('text_content') == message
class TestEmailServiceActionReminder:
"""Tests for action reminder email method."""
def test_send_action_reminder_lyrics(self):
"""Test lyrics action reminder email."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
result = service.send_action_reminder(
to_email="user@example.com",
message_content="Please review your lyrics",
action_type="lyrics",
)
assert result is True
call_kwargs = service.provider.send_email.call_args.kwargs
subject = call_kwargs.get('subject')
assert "lyrics" in subject.lower()
def test_send_action_reminder_instrumental(self):
"""Test instrumental action reminder email."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
result = service.send_action_reminder(
to_email="user@example.com",
message_content="Please select instrumental",
action_type="instrumental",
)
assert result is True
call_kwargs = service.provider.send_email.call_args.kwargs
subject = call_kwargs.get('subject')
assert "instrumental" in subject.lower()
def test_send_action_reminder_with_song_info(self):
"""Test action reminder includes song in subject."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_action_reminder(
to_email="user@example.com",
message_content="Please review",
action_type="lyrics",
artist="Test Artist",
title="Test Song",
)
call_kwargs = service.provider.send_email.call_args.kwargs
subject = call_kwargs.get('subject')
assert "Test Artist" in subject
assert "Test Song" in subject
def test_send_action_reminder_unknown_type(self):
"""Test action reminder with unknown type still sends."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
result = service.send_action_reminder(
to_email="user@example.com",
message_content="Please take action",
action_type="unknown",
)
assert result is True
call_kwargs = service.provider.send_email.call_args.kwargs
subject = call_kwargs.get('subject')
assert "Action needed" in subject
def test_send_action_reminder_escapes_html(self):
"""Test that message content is HTML-escaped."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_action_reminder(
to_email="user@example.com",
message_content="Test <script>alert('xss')</script>",
action_type="lyrics",
)
call_kwargs = service.provider.send_email.call_args.kwargs
html_content = call_kwargs.get('html_content')
assert "<script>" not in html_content
assert "&lt;script&gt;" in html_content
def test_send_action_reminder_no_cc(self):
"""Test that action reminders don't have CC."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_action_reminder(
to_email="user@example.com",
message_content="Please review",
action_type="lyrics",
)
call_kwargs = service.provider.send_email.call_args.kwargs
# Action reminders should not have CC
assert call_kwargs.get('cc_emails') is None
class TestEmailServiceConfiguration:
"""Tests for email service configuration."""
def test_uses_sendgrid_when_configured(self):
"""Test that SendGrid is used when API key is set."""
with patch.dict('os.environ', {'SENDGRID_API_KEY': 'test-key'}):
service = EmailService()
assert isinstance(service.provider, SendGridEmailProvider)
def test_uses_console_when_not_configured(self):
"""Test that console is used when no API key."""
with patch.dict('os.environ', {}, clear=True):
# Remove SENDGRID_API_KEY if it exists
import os
original = os.environ.pop('SENDGRID_API_KEY', None)
try:
service = EmailService()
assert isinstance(service.provider, ConsoleEmailProvider)
finally:
if original:
os.environ['SENDGRID_API_KEY'] = original
def test_is_configured_true_for_sendgrid(self):
"""Test is_configured returns True for SendGrid."""
service = EmailService()
service.provider = SendGridEmailProvider("key", "from@example.com")
assert service.is_configured() is True
def test_is_configured_false_for_console(self):
"""Test is_configured returns False for console."""
service = EmailService()
service.provider = ConsoleEmailProvider()
assert service.is_configured() is False
class TestGlobalInstance:
"""Tests for global instance management."""
def test_get_email_service_returns_same_instance(self):
"""Test that get_email_service returns singleton."""
# Reset global
import backend.services.email_service as es
es._email_service = None
service1 = get_email_service()
service2 = get_email_service()
assert service1 is service2
class TestCCFunctionality:
"""Integration tests for CC functionality across providers."""
def test_console_provider_logs_cc(self, caplog):
"""Test that console provider logs CC recipients."""
import logging
caplog.set_level(logging.INFO)
provider = ConsoleEmailProvider()
provider.send_email(
to_email="user@example.com",
subject="Test",
html_content="<p>Test</p>",
cc_emails=["cc1@example.com", "cc2@example.com"],
)
# Check that CC was logged
assert "cc1@example.com" in caplog.text or "CC:" in caplog.text
def test_sendgrid_provider_adds_cc_recipients(self):
"""Test that SendGrid provider adds CC recipients to message."""
provider = SendGridEmailProvider(
api_key="test-api-key",
from_email="from@example.com",
)
mock_response = Mock()
mock_response.status_code = 202
# Patch at sendgrid module level since imports are inline
with patch('sendgrid.SendGridAPIClient') as mock_sg:
mock_client = Mock()
mock_client.send.return_value = mock_response
mock_sg.return_value = mock_client
# We need to capture the Mail object
with patch('sendgrid.helpers.mail.Mail') as mock_mail_class:
mock_mail = MagicMock()
mock_mail_class.return_value = mock_mail
provider.send_email(
to_email="user@example.com",
subject="Test",
html_content="<p>Test</p>",
cc_emails=["cc@example.com"],
)
# Verify add_cc was called
mock_mail.add_cc.assert_called()
+2 -1
View File
@@ -1128,7 +1128,8 @@ class TestFinaliseOnlyModels:
assert request.enable_txt is None
assert request.brand_prefix is None
assert request.keep_brand_code is None
assert request.enable_youtube_upload is False
# YouTube upload default is None (server applies default_enable_youtube_upload)
assert request.enable_youtube_upload is None
assert request.dropbox_path is None
assert request.gdrive_folder_id is None
@@ -0,0 +1,329 @@
"""
Unit tests for job manager notification methods.
Tests the state transition triggers for email notifications.
"""
import pytest
from unittest.mock import Mock, patch, MagicMock, AsyncMock
from datetime import datetime
from backend.services.job_manager import JobManager
from backend.models.job import Job, JobStatus
class TestTriggerStateNotifications:
"""Tests for _trigger_state_notifications method."""
def test_triggers_completion_email_on_complete(self):
"""Test that completion email is triggered when job completes."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {"youtube_url": "https://youtube.com/123"}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
mock_schedule.assert_called_once_with(mock_job)
def test_triggers_idle_reminder_on_awaiting_review(self):
"""Test that idle reminder is triggered when job enters awaiting review."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
manager._trigger_state_notifications("job-123", JobStatus.AWAITING_REVIEW)
mock_schedule.assert_called_once_with(mock_job, JobStatus.AWAITING_REVIEW)
def test_triggers_idle_reminder_on_awaiting_instrumental(self):
"""Test that idle reminder is triggered when job enters awaiting instrumental."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
manager._trigger_state_notifications(
"job-123",
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
)
mock_schedule.assert_called_once_with(
mock_job,
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
)
def test_skips_notification_when_no_job(self):
"""Test that notification is skipped when job not found."""
manager = JobManager()
manager.firestore = Mock()
manager.get_job = Mock(return_value=None)
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
mock_schedule.assert_not_called()
def test_skips_notification_when_no_user_email(self):
"""Test that notification is skipped when job has no user email."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = None
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
mock_schedule.assert_not_called()
def test_does_not_trigger_on_other_states(self):
"""Test that no notification is triggered for non-notification states."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_schedule_completion_email') as mock_complete, \
patch.object(manager, '_schedule_idle_reminder') as mock_idle:
# Test various non-notification states
for status in [JobStatus.PENDING, JobStatus.DOWNLOADING, JobStatus.TRANSCRIBING,
JobStatus.RENDERING_VIDEO, JobStatus.FAILED]:
manager._trigger_state_notifications("job-123", status)
mock_complete.assert_not_called()
mock_idle.assert_not_called()
def test_handles_exception_gracefully(self):
"""Test that exceptions don't propagate from notifications."""
manager = JobManager()
manager.firestore = Mock()
manager.get_job = Mock(side_effect=Exception("Database error"))
# Should not raise
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
class TestScheduleCompletionEmail:
"""Tests for _schedule_completion_email method."""
def test_extracts_urls_from_state_data(self):
"""Test that YouTube and Dropbox URLs are extracted from state_data."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.artist = "Test Artist"
mock_job.title = "Test Song"
mock_job.state_data = {
"youtube_url": "https://youtube.com/watch?v=123",
"dropbox_link": "https://dropbox.com/folder/abc",
}
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
mock_service = Mock()
mock_service.send_job_completion_email = AsyncMock(return_value=True)
mock_get_service.return_value = mock_service
# Run in sync context - this will start a daemon thread
manager._schedule_completion_email(mock_job)
# Give the thread a moment to execute
import time
time.sleep(0.1)
def test_handles_none_state_data(self):
"""Test that None state_data is handled gracefully."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.artist = None
mock_job.title = None
mock_job.state_data = None # None state_data
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
mock_service = Mock()
mock_service.send_job_completion_email = AsyncMock(return_value=True)
mock_get_service.return_value = mock_service
# Should not raise
manager._schedule_completion_email(mock_job)
def test_handles_exception_gracefully(self):
"""Test that exceptions don't propagate."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
mock_get_service.side_effect = Exception("Service error")
# Should not raise
manager._schedule_completion_email(mock_job)
class TestScheduleIdleReminder:
"""Tests for _schedule_idle_reminder method."""
def test_updates_state_data_for_lyrics_review(self):
"""Test that state_data is updated with blocking state info for lyrics."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {"existing_key": "value"}
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
mock_service = Mock()
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
mock_get_service.return_value = mock_service
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
# Verify state_data was updated
update_call = manager.firestore.update_job.call_args
assert update_call is not None
updated_data = update_call[0][1] # Second positional arg
state_data = updated_data.get('state_data', {})
assert state_data.get('blocking_action_type') == 'lyrics'
assert state_data.get('reminder_sent') is False
assert 'blocking_state_entered_at' in state_data
assert state_data.get('existing_key') == 'value' # Preserved
def test_updates_state_data_for_instrumental_selection(self):
"""Test that state_data is updated with blocking state info for instrumental."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
mock_service = Mock()
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
mock_get_service.return_value = mock_service
manager._schedule_idle_reminder(
mock_job,
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
)
# Verify state_data was updated
update_call = manager.firestore.update_job.call_args
updated_data = update_call[0][1]
state_data = updated_data.get('state_data', {})
assert state_data.get('blocking_action_type') == 'instrumental'
def test_handles_none_state_data(self):
"""Test that None state_data is handled gracefully."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = None
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
mock_service = Mock()
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
mock_get_service.return_value = mock_service
# Should not raise
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
# Verify update was still called
manager.firestore.update_job.assert_called_once()
def test_handles_exception_gracefully(self):
"""Test that exceptions don't propagate."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
# Make firestore raise an exception
manager.firestore.update_job.side_effect = Exception("Database error")
# Should not raise
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
class TestTransitionTriggersNotifications:
"""Integration tests for transition_to_state triggering notifications."""
def test_transition_to_complete_triggers_notification(self):
"""Test that transitioning to COMPLETE triggers notification."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
# Use ENCODING as the starting state since ENCODING -> COMPLETE is valid
mock_job.status = JobStatus.ENCODING
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
manager.transition_to_state("job-123", JobStatus.COMPLETE)
mock_trigger.assert_called_once_with("job-123", JobStatus.COMPLETE)
def test_transition_to_awaiting_review_triggers_notification(self):
"""Test that transitioning to AWAITING_REVIEW triggers notification."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
# Use APPLYING_PADDING as the starting state since APPLYING_PADDING -> AWAITING_REVIEW is valid
mock_job.status = JobStatus.APPLYING_PADDING
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
manager.transition_to_state("job-123", JobStatus.AWAITING_REVIEW)
mock_trigger.assert_called_once_with("job-123", JobStatus.AWAITING_REVIEW)
@@ -0,0 +1,443 @@
"""
Unit tests for job notification service.
"""
import pytest
from unittest.mock import Mock, patch, AsyncMock
import urllib.parse
from backend.services.job_notification_service import (
JobNotificationService,
get_job_notification_service,
ENABLE_AUTO_EMAILS,
FEEDBACK_FORM_URL,
)
class TestURLBuilding:
"""Tests for URL building methods."""
def test_build_review_url_basic(self):
"""Test basic review URL building."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.backend_url = "https://api.nomadkaraoke.com"
url = service._build_review_url("job-123")
assert "gen.nomadkaraoke.com/lyrics/" in url
assert "baseApiUrl=" in url
# The API URL should be URL-encoded
assert urllib.parse.quote("https://api.nomadkaraoke.com/api/review/job-123", safe='') in url
def test_build_review_url_with_audio_hash(self):
"""Test review URL with audio hash parameter."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.backend_url = "https://api.nomadkaraoke.com"
url = service._build_review_url("job-123", audio_hash="abc123")
assert "audioHash=abc123" in url
def test_build_review_url_with_review_token(self):
"""Test review URL with review token parameter."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.backend_url = "https://api.nomadkaraoke.com"
url = service._build_review_url("job-123", review_token="token456")
assert "reviewToken=token456" in url
def test_build_review_url_with_all_params(self):
"""Test review URL with all parameters."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.backend_url = "https://api.nomadkaraoke.com"
url = service._build_review_url(
"job-123",
audio_hash="hash789",
review_token="token456"
)
assert "baseApiUrl=" in url
assert "audioHash=hash789" in url
assert "reviewToken=token456" in url
def test_build_review_url_encodes_special_chars(self):
"""Test that special characters in job ID are encoded."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.backend_url = "https://api.nomadkaraoke.com"
url = service._build_review_url("job/with/slashes")
# The baseApiUrl parameter should have encoded slashes
assert "%2F" in url
def test_build_instrumental_url_basic(self):
"""Test basic instrumental URL building."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.backend_url = "https://api.nomadkaraoke.com"
url = service._build_instrumental_url("job-123")
assert "gen.nomadkaraoke.com/instrumental/" in url
assert "baseApiUrl=" in url
def test_build_instrumental_url_with_token(self):
"""Test instrumental URL with token parameter."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.backend_url = "https://api.nomadkaraoke.com"
url = service._build_instrumental_url("job-123", instrumental_token="inst-token")
assert "instrumentalToken=inst-token" in url
class TestCompletionEmail:
"""Tests for job completion email sending."""
@pytest.mark.asyncio
async def test_send_completion_email_success(self):
"""Test successful completion email sending."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_job_completion.return_value = True
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
user_name="Test User",
artist="Test Artist",
title="Test Song",
youtube_url="https://youtube.com/watch?v=123",
dropbox_url="https://dropbox.com/folder/abc",
)
assert result is True
service.template_service.render_job_completion.assert_called_once()
service.email_service.send_job_completion.assert_called_once()
@pytest.mark.asyncio
async def test_send_completion_email_disabled(self):
"""Test that completion email is skipped when auto emails are disabled."""
service = JobNotificationService()
service.email_service = Mock()
service.template_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
result = await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
)
assert result is False
service.email_service.send_job_completion.assert_not_called()
@pytest.mark.asyncio
async def test_send_completion_email_no_email(self):
"""Test that completion email is skipped when no user email."""
service = JobNotificationService()
service.email_service = Mock()
service.template_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_job_completion_email(
job_id="job-123",
user_email=None,
)
assert result is False
service.email_service.send_job_completion.assert_not_called()
@pytest.mark.asyncio
async def test_send_completion_email_empty_email(self):
"""Test that completion email is skipped when user email is empty."""
service = JobNotificationService()
service.email_service = Mock()
service.template_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_job_completion_email(
job_id="job-123",
user_email="",
)
assert result is False
service.email_service.send_job_completion.assert_not_called()
@pytest.mark.asyncio
async def test_send_completion_email_send_failure(self):
"""Test handling of email send failure."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_job_completion.return_value = False
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
)
assert result is False
@pytest.mark.asyncio
async def test_send_completion_email_exception(self):
"""Test handling of exceptions during email sending."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_job_completion.side_effect = Exception("Send error")
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
)
assert result is False
@pytest.mark.asyncio
async def test_send_completion_email_passes_cc_admin(self):
"""Test that completion email is sent with CC to admin."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_job_completion.return_value = True
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
)
# Verify cc_admin=True was passed
call_kwargs = service.email_service.send_job_completion.call_args.kwargs
assert call_kwargs.get('cc_admin') is True
class TestActionReminderEmail:
"""Tests for action reminder email sending."""
@pytest.mark.asyncio
async def test_send_lyrics_reminder_success(self):
"""Test successful lyrics reminder email sending."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_action_reminder.return_value = True
service.template_service = Mock()
service.template_service.render_action_needed_lyrics.return_value = "Review your lyrics"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="lyrics",
artist="Test Artist",
title="Test Song",
)
assert result is True
service.template_service.render_action_needed_lyrics.assert_called_once()
service.email_service.send_action_reminder.assert_called_once()
@pytest.mark.asyncio
async def test_send_instrumental_reminder_success(self):
"""Test successful instrumental reminder email sending."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_action_reminder.return_value = True
service.template_service = Mock()
service.template_service.render_action_needed_instrumental.return_value = "Select instrumental"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="instrumental",
)
assert result is True
service.template_service.render_action_needed_instrumental.assert_called_once()
service.email_service.send_action_reminder.assert_called_once()
@pytest.mark.asyncio
async def test_send_reminder_disabled(self):
"""Test that reminder email is skipped when auto emails are disabled."""
service = JobNotificationService()
service.email_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
result = await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="lyrics",
)
assert result is False
service.email_service.send_action_reminder.assert_not_called()
@pytest.mark.asyncio
async def test_send_reminder_no_email(self):
"""Test that reminder email is skipped when no user email."""
service = JobNotificationService()
service.email_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_action_reminder_email(
job_id="job-123",
user_email=None,
action_type="lyrics",
)
assert result is False
@pytest.mark.asyncio
async def test_send_reminder_unknown_action_type(self):
"""Test handling of unknown action type."""
service = JobNotificationService()
service.email_service = Mock()
service.template_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="unknown",
)
assert result is False
service.email_service.send_action_reminder.assert_not_called()
@pytest.mark.asyncio
async def test_send_lyrics_reminder_includes_review_url(self):
"""Test that lyrics reminder includes correct review URL."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.backend_url = "https://api.nomadkaraoke.com"
service.email_service = Mock()
service.email_service.send_action_reminder.return_value = True
service.template_service = Mock()
service.template_service.render_action_needed_lyrics.return_value = "Review"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="lyrics",
audio_hash="hash123",
review_token="token456",
)
# Verify the review URL was passed to template
call_kwargs = service.template_service.render_action_needed_lyrics.call_args.kwargs
review_url = call_kwargs.get('review_url')
assert "audioHash=hash123" in review_url
assert "reviewToken=token456" in review_url
@pytest.mark.asyncio
async def test_send_instrumental_reminder_includes_url(self):
"""Test that instrumental reminder includes correct URL."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.backend_url = "https://api.nomadkaraoke.com"
service.email_service = Mock()
service.email_service.send_action_reminder.return_value = True
service.template_service = Mock()
service.template_service.render_action_needed_instrumental.return_value = "Select"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="instrumental",
instrumental_token="inst-token",
)
# Verify the instrumental URL was passed to template
call_kwargs = service.template_service.render_action_needed_instrumental.call_args.kwargs
instrumental_url = call_kwargs.get('instrumental_url')
assert "instrumentalToken=inst-token" in instrumental_url
class TestGetCompletionMessage:
"""Tests for get_completion_message method."""
def test_get_completion_message_basic(self):
"""Test basic completion message retrieval."""
service = JobNotificationService()
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Your video is ready!"
result = service.get_completion_message(
job_id="job-123",
user_name="Test User",
artist="Test Artist",
title="Test Song",
youtube_url="https://youtube.com/watch?v=123",
dropbox_url="https://dropbox.com/folder/abc",
)
assert result == "Your video is ready!"
service.template_service.render_job_completion.assert_called_once()
def test_get_completion_message_includes_feedback_url(self):
"""Test that completion message includes feedback URL."""
service = JobNotificationService()
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Message"
service.get_completion_message(job_id="job-123")
# Verify feedback URL was passed
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
assert 'feedback_url' in call_kwargs
def test_get_completion_message_with_defaults(self):
"""Test completion message with default values."""
service = JobNotificationService()
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Message"
service.get_completion_message(job_id="job-123")
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
assert call_kwargs.get('job_id') == "job-123"
# Other params should be None/default
assert call_kwargs.get('name') is None
assert call_kwargs.get('artist') is None
class TestGlobalInstance:
"""Tests for global instance management."""
def test_get_job_notification_service_returns_same_instance(self):
"""Test that get_job_notification_service returns singleton."""
# Reset global
import backend.services.job_notification_service as jns
jns._job_notification_service = None
service1 = get_job_notification_service()
service2 = get_job_notification_service()
assert service1 is service2
def test_service_initializes_with_dependencies(self):
"""Test that service initializes with email and template services."""
service = JobNotificationService()
assert service.email_service is not None
assert service.template_service is not None
assert service.frontend_url is not None
assert service.backend_url is not None
@@ -0,0 +1,567 @@
"""
Tests for LocalPreviewEncodingService.
Tests cover:
- Service initialization
- Hardware acceleration detection (NVENC)
- FFmpeg filter path escaping
- ASS filter building
- Preview FFmpeg command construction
- Preview encoding execution
- Error handling
"""
import os
import tempfile
import pytest
from unittest.mock import MagicMock, patch
from backend.services.local_preview_encoding_service import (
LocalPreviewEncodingService,
PreviewEncodingConfig,
PreviewEncodingResult,
get_local_preview_encoding_service,
)
class TestLocalPreviewEncodingServiceInit:
"""Test service initialization."""
def test_init_default_values(self):
"""Test default initialization."""
service = LocalPreviewEncodingService()
assert service._nvenc_available is None # Lazy detection
assert service._video_encoder is None # Lazy detection
assert service._hwaccel_flags is None # Lazy detection
def test_init_with_custom_logger(self):
"""Test initialization with custom logger."""
import logging
custom_logger = logging.getLogger("test")
service = LocalPreviewEncodingService(logger=custom_logger)
assert service.logger is custom_logger
class TestLocalPreviewEncodingServiceHWAccel:
"""Test hardware acceleration detection."""
@patch("subprocess.run")
def test_detect_nvenc_available(self, mock_run):
"""Test NVENC detection when available."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalPreviewEncodingService()
result = service._detect_nvenc_support()
assert result is True
@patch("subprocess.run")
def test_detect_nvenc_not_available(self, mock_run):
"""Test fallback when NVENC not available."""
import subprocess
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
service = LocalPreviewEncodingService()
result = service._detect_nvenc_support()
assert result is False
@patch("subprocess.run")
def test_detect_nvenc_timeout(self, mock_run):
"""Test NVENC detection timeout handling."""
import subprocess
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
service = LocalPreviewEncodingService()
result = service._detect_nvenc_support()
assert result is False
@patch.object(LocalPreviewEncodingService, "_detect_nvenc_support")
def test_configure_nvenc_available(self, mock_detect):
"""Test configuration when NVENC is available."""
mock_detect.return_value = True
service = LocalPreviewEncodingService()
service._configure_hardware_acceleration()
assert service._nvenc_available is True
assert service._video_encoder == "h264_nvenc"
assert "-hwaccel" in service._hwaccel_flags
@patch.object(LocalPreviewEncodingService, "_detect_nvenc_support")
def test_configure_no_nvenc(self, mock_detect):
"""Test configuration when NVENC not available."""
mock_detect.return_value = False
service = LocalPreviewEncodingService()
service._configure_hardware_acceleration()
assert service._nvenc_available is False
assert service._video_encoder == "libx264"
assert service._hwaccel_flags == []
class TestLocalPreviewEncodingServiceProperties:
"""Test property accessors that trigger lazy detection."""
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
def test_nvenc_available_triggers_detection(self, mock_config):
"""Test that nvenc_available property triggers detection."""
service = LocalPreviewEncodingService()
service._nvenc_available = None
# Access property
_ = service.nvenc_available
mock_config.assert_called_once()
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
def test_video_encoder_triggers_detection(self, mock_config):
"""Test that video_encoder property triggers detection."""
service = LocalPreviewEncodingService()
service._video_encoder = None
# Access property
_ = service.video_encoder
mock_config.assert_called_once()
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
def test_hwaccel_flags_triggers_detection(self, mock_config):
"""Test that hwaccel_flags property triggers detection."""
service = LocalPreviewEncodingService()
service._hwaccel_flags = None
# Access property
_ = service.hwaccel_flags
mock_config.assert_called_once()
class TestLocalPreviewEncodingServicePathEscaping:
"""Test FFmpeg filter path escaping."""
def test_escape_simple_path(self):
"""Test escaping a simple path."""
service = LocalPreviewEncodingService()
result = service._escape_ffmpeg_filter_path("/simple/path.ass")
assert result == "/simple/path.ass"
def test_escape_path_with_spaces(self):
"""Test escaping path with spaces."""
service = LocalPreviewEncodingService()
result = service._escape_ffmpeg_filter_path("/path/with spaces/file.ass")
assert "\\ " in result # Spaces should be escaped
def test_escape_path_with_apostrophe(self):
"""Test escaping path with apostrophe (common in song titles)."""
service = LocalPreviewEncodingService()
result = service._escape_ffmpeg_filter_path("/path/I'm With You/file.ass")
assert "\\\\\\'" in result # Apostrophe should be triple-backslash escaped
def test_escape_path_with_special_chars(self):
"""Test escaping path with FFmpeg special characters."""
service = LocalPreviewEncodingService()
result = service._escape_ffmpeg_filter_path("/path:with[special];chars,here.ass")
# Should escape :,[];
assert "\\:" in result
assert "\\[" in result
assert "\\]" in result
assert "\\;" in result
assert "\\," in result
def test_escape_path_with_backslashes(self):
"""Test escaping path with existing backslashes."""
service = LocalPreviewEncodingService()
result = service._escape_ffmpeg_filter_path("/path\\with\\backslashes.ass")
assert "\\\\" in result
class TestLocalPreviewEncodingServiceASSFilter:
"""Test ASS filter building."""
def test_build_ass_filter_simple(self):
"""Test building ASS filter without font."""
service = LocalPreviewEncodingService()
result = service._build_ass_filter("/path/to/file.ass")
assert result.startswith("ass=")
assert "fontsdir" not in result
def test_build_ass_filter_with_font(self):
"""Test building ASS filter with custom font."""
with tempfile.TemporaryDirectory() as tmpdir:
font_path = os.path.join(tmpdir, "custom.ttf")
with open(font_path, "w") as f:
f.write("fake font")
service = LocalPreviewEncodingService()
result = service._build_ass_filter("/path/to/file.ass", font_path)
assert "fontsdir=" in result
def test_build_ass_filter_nonexistent_font(self):
"""Test building ASS filter with nonexistent font file."""
service = LocalPreviewEncodingService()
result = service._build_ass_filter("/path/to/file.ass", "/nonexistent/font.ttf")
# Should not include fontsdir for nonexistent font
assert "fontsdir" not in result
class TestLocalPreviewEncodingServiceFFmpegCommand:
"""Test FFmpeg command building."""
def test_build_command_solid_background_libx264(self):
"""Test command building with solid background and libx264."""
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4",
background_color="black"
)
cmd = service._build_preview_ffmpeg_command(config)
assert "ffmpeg" in cmd
assert "-r" in cmd and "24" in cmd # Frame rate
assert "-c:v" in cmd
assert "libx264" in cmd
assert "-preset" in cmd and "superfast" in cmd
assert "-crf" in cmd and "28" in cmd
assert "color=c=black" in " ".join(cmd)
def test_build_command_with_nvenc(self):
"""Test command building with NVENC hardware acceleration."""
service = LocalPreviewEncodingService()
service._nvenc_available = True
service._video_encoder = "h264_nvenc"
service._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4"
)
cmd = service._build_preview_ffmpeg_command(config)
assert "h264_nvenc" in cmd
assert "-hwaccel" in cmd
assert "-preset" in cmd and "p1" in cmd # Fastest NVENC preset
def test_build_command_with_background_image(self):
"""Test command building with background image."""
with tempfile.TemporaryDirectory() as tmpdir:
bg_path = os.path.join(tmpdir, "background.png")
with open(bg_path, "wb") as f:
f.write(b"fake png")
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4",
background_image_path=bg_path
)
cmd = service._build_preview_ffmpeg_command(config)
assert "-loop" in cmd
assert "1" in cmd
assert bg_path in cmd
# Should have scale filter for background image
vf_index = cmd.index("-vf")
vf_value = cmd[vf_index + 1]
assert "scale=" in vf_value
assert "pad=" in vf_value
def test_build_command_resolution(self):
"""Test that command uses correct preview resolution."""
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4"
)
cmd = service._build_preview_ffmpeg_command(config)
cmd_str = " ".join(cmd)
# Resolution should be 480x270
assert "480" in cmd_str
assert "270" in cmd_str
def test_build_command_audio_settings(self):
"""Test that command uses correct audio settings."""
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4"
)
cmd = service._build_preview_ffmpeg_command(config)
assert "-c:a" in cmd
assert "aac" in cmd
assert "-b:a" in cmd
assert "96k" in cmd
class TestLocalPreviewEncodingServiceEncode:
"""Test preview encoding execution."""
@patch("subprocess.run")
def test_encode_preview_success(self, mock_run):
"""Test successful preview encoding."""
mock_run.return_value = MagicMock(returncode=0, stderr="")
with tempfile.TemporaryDirectory() as tmpdir:
ass_path = os.path.join(tmpdir, "subs.ass")
audio_path = os.path.join(tmpdir, "audio.flac")
output_path = os.path.join(tmpdir, "output.mp4")
# Create input files
with open(ass_path, "w") as f:
f.write("[Script Info]\n")
with open(audio_path, "wb") as f:
f.write(b"fake audio")
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path=ass_path,
audio_path=audio_path,
output_path=output_path
)
result = service.encode_preview(config)
assert result.success is True
assert result.output_path == output_path
assert result.error is None
mock_run.assert_called_once()
@patch("subprocess.run")
def test_encode_preview_ffmpeg_failure(self, mock_run):
"""Test preview encoding with FFmpeg failure."""
mock_run.return_value = MagicMock(returncode=1, stderr="FFmpeg error message")
with tempfile.TemporaryDirectory() as tmpdir:
ass_path = os.path.join(tmpdir, "subs.ass")
audio_path = os.path.join(tmpdir, "audio.flac")
output_path = os.path.join(tmpdir, "output.mp4")
with open(ass_path, "w") as f:
f.write("[Script Info]\n")
with open(audio_path, "wb") as f:
f.write(b"fake audio")
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path=ass_path,
audio_path=audio_path,
output_path=output_path
)
result = service.encode_preview(config)
assert result.success is False
assert "FFmpeg preview encoding failed" in result.error
def test_encode_preview_missing_ass_file(self):
"""Test preview encoding with missing ASS file."""
with tempfile.TemporaryDirectory() as tmpdir:
audio_path = os.path.join(tmpdir, "audio.flac")
output_path = os.path.join(tmpdir, "output.mp4")
with open(audio_path, "wb") as f:
f.write(b"fake audio")
service = LocalPreviewEncodingService()
config = PreviewEncodingConfig(
ass_path="/nonexistent/subs.ass",
audio_path=audio_path,
output_path=output_path
)
result = service.encode_preview(config)
assert result.success is False
assert "not found" in result.error
def test_encode_preview_missing_audio_file(self):
"""Test preview encoding with missing audio file."""
with tempfile.TemporaryDirectory() as tmpdir:
ass_path = os.path.join(tmpdir, "subs.ass")
output_path = os.path.join(tmpdir, "output.mp4")
with open(ass_path, "w") as f:
f.write("[Script Info]\n")
service = LocalPreviewEncodingService()
config = PreviewEncodingConfig(
ass_path=ass_path,
audio_path="/nonexistent/audio.flac",
output_path=output_path
)
result = service.encode_preview(config)
assert result.success is False
assert "not found" in result.error
@patch("subprocess.run")
def test_encode_preview_timeout(self, mock_run):
"""Test preview encoding timeout."""
import subprocess
mock_run.side_effect = subprocess.TimeoutExpired("ffmpeg", 300)
with tempfile.TemporaryDirectory() as tmpdir:
ass_path = os.path.join(tmpdir, "subs.ass")
audio_path = os.path.join(tmpdir, "audio.flac")
output_path = os.path.join(tmpdir, "output.mp4")
with open(ass_path, "w") as f:
f.write("[Script Info]\n")
with open(audio_path, "wb") as f:
f.write(b"fake audio")
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path=ass_path,
audio_path=audio_path,
output_path=output_path
)
result = service.encode_preview(config)
assert result.success is False
assert "timed out" in result.error
class TestPreviewEncodingConfig:
"""Test PreviewEncodingConfig dataclass."""
def test_config_required_fields(self):
"""Test that required fields must be provided."""
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4"
)
assert config.ass_path == "/path/to/subs.ass"
assert config.background_color == "black" # Default
assert config.background_image_path is None # Optional
def test_config_all_fields(self):
"""Test config with all fields."""
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4",
background_image_path="/path/to/bg.png",
background_color="red",
font_path="/path/to/font.ttf"
)
assert config.background_image_path == "/path/to/bg.png"
assert config.background_color == "red"
assert config.font_path == "/path/to/font.ttf"
class TestPreviewEncodingResult:
"""Test PreviewEncodingResult dataclass."""
def test_result_success(self):
"""Test successful result."""
result = PreviewEncodingResult(
success=True,
output_path="/path/to/output.mp4"
)
assert result.success is True
assert result.error is None
assert result.output_path == "/path/to/output.mp4"
def test_result_failure(self):
"""Test failure result."""
result = PreviewEncodingResult(
success=False,
error="Something went wrong"
)
assert result.success is False
assert result.error == "Something went wrong"
assert result.output_path is None
class TestGetLocalPreviewEncodingService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.local_preview_encoding_service as module
module._local_preview_encoding_service = None
service = get_local_preview_encoding_service()
assert service is not None
assert isinstance(service, LocalPreviewEncodingService)
def test_get_service_returns_singleton(self):
"""Test that factory function returns the same instance."""
import backend.services.local_preview_encoding_service as module
module._local_preview_encoding_service = None
service1 = get_local_preview_encoding_service()
service2 = get_local_preview_encoding_service()
assert service1 is service2
class TestPreviewEncodingConstants:
"""Test that encoding constants match expected values."""
def test_preview_resolution(self):
"""Test preview resolution constants."""
service = LocalPreviewEncodingService()
assert service.PREVIEW_WIDTH == 480
assert service.PREVIEW_HEIGHT == 270
def test_preview_fps(self):
"""Test preview frame rate constant."""
service = LocalPreviewEncodingService()
assert service.PREVIEW_FPS == 24
def test_preview_audio_bitrate(self):
"""Test preview audio bitrate constant."""
service = LocalPreviewEncodingService()
assert service.PREVIEW_AUDIO_BITRATE == "96k"
+2 -1
View File
@@ -149,7 +149,8 @@ class TestUploadSubmissionRequest:
# CDG/TXT disabled by default (requires style config)
assert request.enable_cdg is False
assert request.enable_txt is False
assert request.enable_youtube_upload is False
# YouTube upload default is None (use server default)
assert request.enable_youtube_upload is None
class TestStartReviewRequest:
+295
View File
@@ -0,0 +1,295 @@
"""
Unit tests for template service.
"""
import pytest
from unittest.mock import Mock, patch, MagicMock
from backend.services.template_service import (
TemplateService,
get_template_service,
DEFAULT_JOB_COMPLETION_TEMPLATE,
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE,
DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE,
)
class TestTemplateRendering:
"""Tests for template rendering functionality."""
def test_render_template_basic(self):
"""Test basic template variable replacement."""
service = TemplateService()
template = "Hello {name}, your job {job_id} is ready!"
variables = {"name": "Alice", "job_id": "123"}
result = service.render_template(template, variables)
assert result == "Hello Alice, your job 123 is ready!"
def test_render_template_missing_variables(self):
"""Test that missing variables are replaced with empty strings."""
service = TemplateService()
template = "Hello {name}, your {thing} is {status}!"
variables = {"name": "Bob"}
result = service.render_template(template, variables)
assert result == "Hello Bob, your is !"
def test_render_template_none_values(self):
"""Test that None values are handled correctly."""
service = TemplateService()
template = "Hello {name}!"
variables = {"name": None}
result = service.render_template(template, variables)
assert result == "Hello !"
def test_render_template_removes_feedback_section_when_no_url(self):
"""Test that feedback section is removed when feedback_url is not provided."""
service = TemplateService()
template = """Thanks for your order!
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
{feedback_url}
Have a great day!"""
variables = {"feedback_url": None}
result = service.render_template(template, variables)
assert "feedback" not in result.lower()
assert "Thanks for your order!" in result
assert "Have a great day!" in result
def test_render_template_keeps_feedback_section_when_url_provided(self):
"""Test that feedback section is kept when feedback_url is provided."""
service = TemplateService()
template = """Thanks!
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
{feedback_url}
Bye!"""
variables = {"feedback_url": "https://example.com/feedback"}
result = service.render_template(template, variables)
assert "https://example.com/feedback" in result
assert "feedback" in result.lower()
class TestJobCompletionTemplate:
"""Tests for job completion template rendering."""
def test_render_job_completion_all_fields(self):
"""Test rendering job completion with all fields."""
service = TemplateService()
# Mock _fetch_template_from_gcs to return None (use default)
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.render_job_completion(
name="Alice",
youtube_url="https://youtube.com/watch?v=123",
dropbox_url="https://dropbox.com/folder/abc",
artist="Test Artist",
title="Test Song",
job_id="job-123",
feedback_url="https://example.com/feedback",
)
assert "Alice" in result
assert "https://youtube.com/watch?v=123" in result
assert "https://dropbox.com/folder/abc" in result
assert "https://example.com/feedback" in result
def test_render_job_completion_defaults(self):
"""Test rendering job completion with default values."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.render_job_completion()
assert "there" in result # Default name
assert "[YouTube URL not available]" in result
assert "[Dropbox URL not available]" in result
def test_render_job_completion_no_feedback_url(self):
"""Test rendering job completion without feedback URL removes section."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.render_job_completion(
name="Bob",
youtube_url="https://youtube.com/123",
dropbox_url="https://dropbox.com/abc",
)
assert "Bob" in result
# Feedback section should be removed
assert "really appreciate your feedback" not in result
class TestActionNeededTemplates:
"""Tests for action-needed template rendering."""
def test_render_action_needed_lyrics(self):
"""Test rendering lyrics review reminder."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.render_action_needed_lyrics(
name="Charlie",
artist="Test Artist",
title="Test Song",
review_url="https://example.com/review",
)
assert "Charlie" in result
assert "Test Artist" in result
assert "Test Song" in result
assert "https://example.com/review" in result
def test_render_action_needed_instrumental(self):
"""Test rendering instrumental selection reminder."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.render_action_needed_instrumental(
name="Diana",
artist="Test Artist",
title="Test Song",
instrumental_url="https://example.com/instrumental",
)
assert "Diana" in result
assert "Test Artist" in result
assert "Test Song" in result
assert "https://example.com/instrumental" in result
class TestGCSFetching:
"""Tests for GCS template fetching."""
def test_fetch_template_from_gcs_success(self):
"""Test successful template fetch from GCS."""
service = TemplateService()
# Mock the bucket and blob
mock_blob = Mock()
mock_blob.exists.return_value = True
mock_blob.download_as_text.return_value = "Custom template {name}"
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
with patch.object(service, "_bucket", mock_bucket):
result = service._fetch_template_from_gcs("test.txt")
assert result == "Custom template {name}"
mock_bucket.blob.assert_called_once_with("templates/test.txt")
def test_fetch_template_from_gcs_not_found(self):
"""Test template fetch when file doesn't exist in GCS."""
service = TemplateService()
mock_blob = Mock()
mock_blob.exists.return_value = False
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
with patch.object(service, "_bucket", mock_bucket):
result = service._fetch_template_from_gcs("nonexistent.txt")
assert result is None
def test_fetch_template_from_gcs_error(self):
"""Test template fetch handles errors gracefully."""
service = TemplateService()
mock_bucket = Mock()
mock_bucket.blob.side_effect = Exception("GCS error")
with patch.object(service, "_bucket", mock_bucket):
result = service._fetch_template_from_gcs("test.txt")
assert result is None
def test_get_job_completion_template_fallback(self):
"""Test that default template is used when GCS fetch fails."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.get_job_completion_template()
assert result == DEFAULT_JOB_COMPLETION_TEMPLATE
def test_get_action_needed_lyrics_template_fallback(self):
"""Test that default lyrics template is used when GCS fetch fails."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.get_action_needed_lyrics_template()
assert result == DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
def test_get_action_needed_instrumental_template_fallback(self):
"""Test that default instrumental template is used when GCS fetch fails."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.get_action_needed_instrumental_template()
assert result == DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
class TestTemplateUpload:
"""Tests for template upload functionality."""
def test_upload_template_success(self):
"""Test successful template upload."""
service = TemplateService()
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
with patch.object(service, "_bucket", mock_bucket):
result = service.upload_template("test.txt", "Test content")
assert result is True
mock_bucket.blob.assert_called_once_with("templates/test.txt")
mock_blob.upload_from_string.assert_called_once_with(
"Test content", content_type="text/plain"
)
def test_upload_template_error(self):
"""Test template upload handles errors."""
service = TemplateService()
mock_blob = Mock()
mock_blob.upload_from_string.side_effect = Exception("Upload failed")
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
with patch.object(service, "_bucket", mock_bucket):
result = service.upload_template("test.txt", "Test content")
assert result is False
class TestGlobalInstance:
"""Tests for global instance management."""
def test_get_template_service_returns_same_instance(self):
"""Test that get_template_service returns singleton."""
# Reset global
import backend.services.template_service as ts
ts._template_service = None
service1 = get_template_service()
service2 = get_template_service()
assert service1 is service2
@@ -425,6 +425,111 @@ class TestVideoWorkerOrchestratorEncoding:
assert "Encoding failed" in str(exc_info.value)
@pytest.mark.asyncio
async def test_run_encoding_gce_downloads_files(self):
"""Test that GCE encoding downloads files from GCS to local directory.
This test verifies the fix for YouTube upload failure when using GCE encoding.
GCE encoding returns GCS blob paths, which need to be downloaded locally
before YouTube upload can access them.
"""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path=os.path.join(temp_dir, "title.mov"),
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
output_dir=temp_dir,
)
# Mock storage service
mock_storage = MagicMock()
mock_storage.download_file = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config,
storage=mock_storage,
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
# GCE backend returns GCS blob paths (not local paths)
mock_backend = MagicMock()
mock_backend.name = "gce" # Important: must be "gce" to trigger download
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path="jobs/test-job/finals/output_4k_lossless.mp4",
lossy_4k_mp4_path="jobs/test-job/finals/output_4k_lossy.mp4",
lossy_720p_mp4_path="jobs/test-job/finals/output_720p.mp4",
lossless_mkv_path="jobs/test-job/finals/output_4k.mkv",
encoding_time_seconds=60.0,
encoding_backend="gce",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# Verify download_file was called for each output file
assert mock_storage.download_file.call_count == 4
# Verify the result paths were updated to local paths
assert orchestrator.result.final_video.startswith(temp_dir)
assert orchestrator.result.final_video_mkv.startswith(temp_dir)
assert orchestrator.result.final_video_lossy.startswith(temp_dir)
assert orchestrator.result.final_video_720p.startswith(temp_dir)
@pytest.mark.asyncio
async def test_run_encoding_local_does_not_download(self):
"""Test that local encoding does NOT trigger GCS download.
Local encoding produces files directly in the output directory,
so no download is needed.
"""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path=os.path.join(temp_dir, "title.mov"),
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
output_dir=temp_dir,
)
# Mock storage service
mock_storage = MagicMock()
mock_storage.download_file = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config,
storage=mock_storage,
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
# Local backend returns local paths directly
mock_backend = MagicMock()
mock_backend.name = "local" # Local backend
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path=os.path.join(temp_dir, "lossless.mp4"),
lossy_4k_mp4_path=os.path.join(temp_dir, "lossy.mp4"),
lossy_720p_mp4_path=os.path.join(temp_dir, "720p.mp4"),
lossless_mkv_path=os.path.join(temp_dir, "lossless.mkv"),
encoding_time_seconds=120.0,
encoding_backend="local",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# Verify download_file was NOT called for local encoding
mock_storage.download_file.assert_not_called()
class TestVideoWorkerOrchestratorOrganization:
"""Test organization stage."""
+22 -17
View File
@@ -639,40 +639,45 @@ async def upload_lyrics_results(
logger.error(f"Job {job_id}: {error_msg}")
raise Exception(error_msg)
# Upload reference lyrics if available
# Upload ALL reference lyrics files (not just first) so they're available for distribution
# Note: Source names use .title() so "lrclib" -> "Lrclib", "genius" -> "Genius"
# Use sanitized artist/title to match LyricsProcessor's file naming
reference_files = [
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "Genius"),
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "Spotify"),
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "Musixmatch"),
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "LRCLib"), # Note: source is 'lrclib', .title() = 'Lrclib'
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "genius"),
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "spotify"),
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "musixmatch"),
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "lrclib"),
]
found_reference = False
for ref_filename, source_name in reference_files:
found_references = []
for ref_filename, source_key in reference_files:
ref_path = os.path.join(lyrics_dir, ref_filename)
if os.path.exists(ref_path):
# Upload with original filename to preserve proper naming
gcs_path = f"jobs/{job_id}/lyrics/{ref_filename}"
url = storage.upload_file(ref_path, gcs_path)
# Track in job.files so video_worker can download for distribution
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
if job_log:
job_log.info(f"Found reference lyrics from {source_name}")
job_log.info(f"Found reference lyrics from {source_key}")
logger.info(f"Job {job_id}: Uploaded reference lyrics: {ref_filename}")
found_reference = True
break # Only upload first available reference
if not found_reference:
found_references.append(source_key)
if not found_references:
if job_log:
job_log.warning("No reference lyrics found from any source (Genius, Spotify, Musixmatch, LRCLib)")
logger.warning(f"Job {job_id}: No reference lyrics found from any source")
else:
logger.info(f"Job {job_id}: Found {len(found_references)} reference lyrics sources: {found_references}")
# Upload uncorrected transcription if available (use sanitized names)
uncorrected_file = os.path.join(lyrics_dir, f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt")
# Upload uncorrected transcription if available (preserve original filename for distribution)
uncorrected_filename = f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt"
uncorrected_file = os.path.join(lyrics_dir, uncorrected_filename)
if os.path.exists(uncorrected_file):
gcs_path = f"jobs/{job_id}/lyrics/uncorrected.txt"
gcs_path = f"jobs/{job_id}/lyrics/{uncorrected_filename}"
url = storage.upload_file(uncorrected_file, gcs_path)
job_manager.update_file_url(job_id, 'lyrics', 'uncorrected', url)
logger.info(f"Job {job_id}: Uploaded uncorrected transcription")
logger.info(f"Job {job_id}: Uploaded uncorrected transcription: {uncorrected_filename}")
logger.info(f"Job {job_id}: All lyrics results uploaded successfully")
+81 -42
View File
@@ -160,14 +160,17 @@ async def _encode_via_gce(
storage.download_file(gcs_path, local_path)
# Map to result keys expected by _upload_results
if "4K Lossless" in filename or "4k_lossless" in filename.lower():
local_files["final_video"] = local_path
elif "4K Lossy" in filename or "4k_lossy" in filename.lower():
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
filename_lower = filename.lower()
if "lossless 4k" in filename_lower:
if filename.endswith(".mkv"):
local_files["final_video_mkv"] = local_path
else:
local_files["final_video"] = local_path
elif "lossy 4k" in filename_lower:
local_files["final_video_lossy"] = local_path
elif "720p" in filename:
elif "720p" in filename_lower:
local_files["final_video_720p"] = local_path
elif filename.endswith(".mkv"):
local_files["final_video_mkv"] = local_path
job_log.info(f"Downloaded {len(local_files)} encoded files")
@@ -332,16 +335,9 @@ async def generate_video_orchestrated(job_id: str) -> bool:
'final_karaoke_txt_zip': result.final_karaoke_txt_zip,
})
# Mark job as complete
# Store result metadata in job BEFORE transitioning to COMPLETE
# This ensures youtube_url is available when completion email is sent
logger.info(f"[job:{job_id}] Video generation complete")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
progress=100,
message="Karaoke generation complete!"
)
# Store result metadata in job
job_manager.update_job(job_id, {
'state_data': {
**job.state_data,
@@ -352,6 +348,14 @@ async def generate_video_orchestrated(job_id: str) -> bool:
}
})
# Mark job as complete (triggers completion email with youtube_url now available)
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
progress=100,
message="Karaoke generation complete!"
)
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.brand_code or '')
@@ -662,19 +666,12 @@ async def generate_video_legacy(job_id: str) -> bool:
with job_span("upload-results", job_id):
await _upload_results(job_id, job_manager, storage, temp_dir, result)
# Mark job as complete
logger.info(f"[job:{job_id}] Video generation complete")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
progress=100,
message="Karaoke generation complete!"
)
# Store result metadata in job
# Store result metadata in job BEFORE transitioning to COMPLETE
# This ensures youtube_url is available when completion email is sent
# NOTE: Must include dropbox_link and gdrive_files from result, since
# _handle_native_distribution already saved them but job.state_data
# is stale (fetched before distribution ran)
logger.info(f"[job:{job_id}] Video generation complete")
job_manager.update_job(job_id, {
'state_data': {
**job.state_data,
@@ -684,7 +681,15 @@ async def generate_video_legacy(job_id: str) -> bool:
'gdrive_files': result.get('gdrive_files'),
}
})
# Mark job as complete (triggers completion email with youtube_url now available)
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE,
progress=100,
message="Karaoke generation complete!"
)
duration = time.time() - start_time
root_span.set_attribute("duration_seconds", duration)
root_span.set_attribute("brand_code", result.get('brand_code', ''))
@@ -1108,29 +1113,63 @@ async def _prepare_distribution_directory(
log_progress(f"Downloaded {stem_key} to stems/")
# --- Download lyrics files to lyrics/ subfolder ---
# Use proper filenames matching local CLI output structure
# Common lyrics file types
lyrics_file_keys = ['lrc', 'ass', 'srt', 'txt', 'json', 'original_json', 'corrected_json']
for lyrics_key in lyrics_file_keys:
# Map job.files.lyrics keys to proper local filenames
# Format: (job_files_key, local_filename_pattern)
lyrics_file_mappings = [
# Karaoke output files
('lrc', f"{base_name} (Karaoke).lrc"),
('ass', f"{base_name} (Karaoke).ass"),
# Corrections and transcription files
('corrections', f"{base_name} (Lyrics Corrections).json"),
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
# Reference lyrics from various sources
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
('reference_spotify', f"{base_name} (Lyrics Spotify).txt"),
('reference_musixmatch', f"{base_name} (Lyrics Musixmatch).txt"),
('reference_lrclib', f"{base_name} (Lyrics Lrclib).txt"),
]
for lyrics_key, local_filename in lyrics_file_mappings:
if lyrics.get(lyrics_key):
# Determine proper extension
ext_map = {
'lrc': '.lrc',
'ass': '.ass',
'srt': '.srt',
'txt': '.txt',
'json': '.json',
'original_json': '_original.json',
'corrected_json': '_corrected.json',
}
ext = ext_map.get(lyrics_key, f'.{lyrics_key}')
dest_path = os.path.join(lyrics_dir, f"{base_name} (Karaoke){ext}")
dest_path = os.path.join(lyrics_dir, local_filename)
try:
storage.download_file(lyrics[lyrics_key], dest_path)
log_progress(f"Downloaded {lyrics_key} to lyrics/")
log_progress(f"Downloaded {lyrics_key} to lyrics/{local_filename}")
except Exception as e:
log_progress(f"Could not download {lyrics_key}: {e}")
# --- Download (With Vocals).mkv to lyrics/ subfolder ---
# Local CLI places the karaoke video with vocals in the lyrics folder
videos = job.file_urls.get('videos', {})
if videos.get('with_vocals'):
with_vocals_dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
try:
storage.download_file(videos['with_vocals'], with_vocals_dest)
log_progress(f"Downloaded with_vocals to lyrics/{base_name} (With Vocals).mkv")
except Exception as e:
log_progress(f"Could not download with_vocals to lyrics/: {e}")
# --- Download preview ASS files to lyrics/previews/ subfolder ---
# Preview files are stored in jobs/{job_id}/previews/ in GCS
previews_dir = os.path.join(lyrics_dir, "previews")
os.makedirs(previews_dir, exist_ok=True)
# List all preview files from GCS and download them
try:
preview_prefix = f"jobs/{job_id}/previews/"
preview_files = storage.list_files(preview_prefix)
for blob_name in preview_files:
if blob_name.endswith('.ass'):
filename = os.path.basename(blob_name)
dest_path = os.path.join(previews_dir, filename)
storage.download_file(blob_name, dest_path)
log_progress(f"Downloaded preview {filename} to lyrics/previews/")
except Exception as e:
log_progress(f"Could not download preview files: {e}")
log_progress("Distribution directory prepared with stems/ and lyrics/ subfolders")
+56 -1
View File
@@ -367,15 +367,70 @@ class VideoWorkerOrchestrator:
if not output.success:
raise Exception(f"Encoding failed: {output.error_message}")
# Store results
# Store results - for GCE backend, these are GCS blob paths that need to be downloaded
self.result.final_video = output.lossless_4k_mp4_path
self.result.final_video_mkv = output.lossless_mkv_path
self.result.final_video_lossy = output.lossy_4k_mp4_path
self.result.final_video_720p = output.lossy_720p_mp4_path
self.result.encoding_time_seconds = output.encoding_time_seconds
# For GCE encoding, download the encoded files from GCS to local directory
# This is required for YouTube upload and other local file operations
if encoding_backend.name == "gce" and self.storage:
await self._download_gce_encoded_files(output)
self.job_log.info(f"Encoding complete ({encoding_backend.name}) in {output.encoding_time_seconds:.1f}s")
async def _download_gce_encoded_files(self, output):
"""
Download GCE-encoded files from GCS to the local output directory.
GCE encoding stores files in GCS and returns blob paths like:
'jobs/{job_id}/finals/Artist - Title (Final Karaoke Lossless 4k).mp4'
This method downloads those files locally so that subsequent stages
(YouTube upload, etc.) can access them as local files.
Args:
output: EncodingOutput from the GCE backend with GCS blob paths
"""
self.job_log.info("Downloading GCE-encoded files from GCS")
# Map of result attributes to download
file_mappings = [
('lossless_4k_mp4_path', 'final_video'),
('lossless_mkv_path', 'final_video_mkv'),
('lossy_4k_mp4_path', 'final_video_lossy'),
('lossy_720p_mp4_path', 'final_video_720p'),
]
downloaded_count = 0
for output_attr, result_attr in file_mappings:
gcs_path = getattr(output, output_attr, None)
if not gcs_path:
continue
# Extract filename from GCS path
filename = os.path.basename(gcs_path)
local_path = os.path.join(self.config.output_dir, filename)
try:
self.job_log.info(f"Downloading {filename} from GCS")
self.storage.download_file(gcs_path, local_path)
# Update the result to point to local file
setattr(self.result, result_attr, local_path)
downloaded_count += 1
self.job_log.info(f"Downloaded {filename} to {local_path}")
except Exception as e:
self.job_log.error(f"Failed to download {filename}: {e}")
# Clear the result attribute so downstream doesn't try to use invalid GCS path
setattr(self.result, result_attr, None)
# Don't fail - some formats might not be generated
self.job_log.info(f"Downloaded {downloaded_count} encoded files from GCS")
async def _run_organization(self):
"""Run the organization stage (brand code generation)."""
self.job_log.info("Starting organization stage")
+127 -2
View File
@@ -20,6 +20,30 @@ GET /health
No auth required. Returns service status.
#### Encoding Worker Health
```http
GET /health/encoding-worker
```
No auth required. Returns GCE encoding worker status for frontend display.
Response:
```json
{
"available": true,
"status": "ok",
"version": "0.95.1",
"active_jobs": 0,
"queue_length": 0
}
```
Status values:
- `ok` - Worker is healthy and available
- `offline` - Worker is unavailable (includes `error` field)
- `not_configured` - Encoding worker not configured on backend
### Jobs
#### Create Job with Upload
@@ -114,11 +138,52 @@ Content-Type: application/json
### Audio Search
#### Search for Audio
```http
GET /api/audio-search?q=song+name
POST /api/audio-search/search
Content-Type: application/json
{
"artist": "Artist Name",
"title": "Song Title",
"auto_download": false,
"theme_id": "nomad",
"display_artist": "Display Artist (optional)",
"display_title": "Display Title (optional)"
}
```
Search for songs (flacfetch integration).
Searches for audio sources (flacfetch integration). Creates a job and returns search results.
**Key fields:**
- `artist`, `title` - Used to search for audio on trackers
- `display_artist`, `display_title` - Optional overrides for how artist/title appear on title screens and filenames. If omitted, search values are used.
- `auto_download` - If true, automatically selects best result and starts processing
- `theme_id` - Video theme to use
**Use case:** Search for "Jeremy Kushnier" but display "Footloose (Broadway Cast)" on the karaoke video.
#### Get Search Results
```http
GET /api/audio-search/{job_id}/results
```
Returns cached search results for a job awaiting audio selection.
#### Select Audio Source
```http
POST /api/audio-search/{job_id}/select
Content-Type: application/json
{
"selection_index": 0
}
```
Selects an audio source from the search results and starts processing.
### Themes
@@ -487,6 +552,66 @@ Response:
}
```
### Email Notifications (Admin)
#### Get Completion Message
```http
GET /api/admin/jobs/{job_id}/completion-message
Authorization: Bearer ADMIN_TOKEN
```
Returns the rendered completion message for a job (for copying to clipboard).
Response:
```json
{
"job_id": "abc123",
"message": "Hi there! Your karaoke video is ready...",
"subject": "Your Karaoke Video is Ready! 🎤",
"youtube_url": "https://youtube.com/watch?v=...",
"dropbox_url": "https://dropbox.com/..."
}
```
#### Send Completion Email
```http
POST /api/admin/jobs/{job_id}/send-completion-email
Authorization: Bearer ADMIN_TOKEN
Content-Type: application/json
{
"to_email": "customer@example.com",
"cc_admin": true
}
```
Sends the job completion email to the specified address. When `cc_admin` is true (default), CCs gen@nomadkaraoke.com.
Response:
```json
{
"success": true,
"job_id": "abc123",
"to_email": "customer@example.com",
"message": "Completion email sent successfully"
}
```
### Internal Email Endpoints
These endpoints are used by Cloud Tasks for automated notifications.
#### Check Idle Reminder
```http
POST /api/internal/jobs/{job_id}/check-idle-reminder
Authorization: Bearer ADMIN_TOKEN
```
Called by Cloud Tasks 5 minutes after a job enters a blocking state (awaiting_review or awaiting_instrumental_selection). Sends a reminder email if the user is still idle and no reminder has been sent yet.
## Rate Limits
No rate limits currently implemented.
+12 -3
View File
@@ -31,7 +31,8 @@
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
(*) GCE Encoding Worker: C4-standard-8 VM with Intel Granite Rapids for
high-performance FFmpeg encoding. Optional - falls back to Cloud Run.
high-performance FFmpeg encoding. Used for both final video encoding and
preview video generation. Uses LocalEncodingService via GCS wheel deployment.
```
## Processing Pipeline
@@ -57,7 +58,7 @@
▼
4. HUMAN REVIEW (AWAITING_REVIEW)
• React UI for lyrics correction
• Preview video generation
• Preview video generation (optionally via GCE for speed)
• Submit corrections
▼
5. VIDEO RENDERING
@@ -154,6 +155,8 @@ LyricsTranscriber LyricsTranscriber
| Vertex AI | Agentic AI correction (Gemini 3 Flash) | Default on |
| Genius | Reference lyrics | Yes |
| YouTube API | Video upload | Optional |
| SendGrid | Email notifications | Optional |
| Cloud Tasks | Delayed task scheduling (idle reminders) | Optional |
## Firestore Collections
@@ -190,7 +193,8 @@ The Video Worker uses an orchestrator pattern to ensure all features work regard
│ └─ YouTube upload │
│ │
│ Stage 5: NOTIFICATIONS │
│ └─ Discord webhooks │
│ ├─ Discord webhooks │
│ └─ Email notifications (completion, reminders) │
└─────────────────────────────────────────────────────────────┘
```
@@ -208,6 +212,9 @@ The Video Worker uses an orchestrator pattern to ensure all features work regard
| `local_encoding_service.py` | FFmpeg encoding (Cloud Run) |
| `encoding_interface.py` | Abstract interface + GCE backend |
| `video_worker_orchestrator.py` | Stage coordination |
| `email_service.py` | SendGrid email delivery with CC support |
| `template_service.py` | GCS-backed email templates |
| `job_notification_service.py` | Email orchestration (completion, reminders) |
## Tech Stack
@@ -216,5 +223,7 @@ The Video Worker uses an orchestrator pattern to ensure all features work regard
- **Database**: Firestore
- **Storage**: Google Cloud Storage
- **Secrets**: Google Secret Manager
- **Task Queues**: Cloud Tasks (idle reminders)
- **Email**: SendGrid
- **IaC**: Pulumi
- **CI/CD**: GitHub Actions
+29 -5
View File
@@ -110,17 +110,41 @@ Push to `main` triggers GitHub Actions:
3. Deploy to Cloud Run
4. Deploy frontend to Cloudflare Pages
### Manual Backend Deploy
### Infrastructure Changes (Pulumi)
**All infrastructure changes must go through PRs.** The CI pipeline automatically deploys Pulumi changes when PRs merge to main.
```bash
# Preview infrastructure changes
# Preview changes locally (requires GCP auth)
cd infrastructure
pulumi preview
pulumi preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
# Apply changes
pulumi up
# DO NOT run `pulumi up` locally - let CI handle deployment
# CI uses --skip-preview due to limited service account permissions
```
**What's managed by Pulumi:**
- Firestore database and indexes
- GCS buckets
- Cloud Run service configuration
- Cloud Tasks queues
- Secret Manager secrets
- Service accounts and IAM bindings
- Workload Identity Federation
- GCE instances (GitHub runners, encoding worker)
**Workflow:**
1. Make changes in the appropriate module under `infrastructure/`
- `modules/` - Core GCP resources (database, storage, secrets, IAM, etc.)
- `compute/` - VM definitions and startup scripts
- `config.py` - Shared configuration constants
- `__main__.py` - Entry point that wires modules together
2. Run `pulumi preview` locally to verify
3. Create PR and merge
4. CI deploys automatically
See `docs/LESSONS-LEARNED.md` for Pulumi CI gotchas.
### Manual Frontend Deploy
```bash
+464
View File
@@ -222,6 +222,52 @@ escaped = filename.replace("'", "\\'")
query = f"name='{escaped}' and '{parent_id}' in parents"
```
### FFmpeg Filter Path Escaping (subprocess without shell)
**Problem**: Song titles with apostrophes (e.g., "I'm With You") cause FFmpeg's ASS subtitle filter to fail because paths aren't escaped correctly.
```python
# BAD - FFmpeg strips the apostrophe, looks for wrong path
ass_filter = f"ass=./Avril Lavigne - I'm With You/cache/temp.ass"
# FFmpeg error: Could not create a libass track when reading file './Avril Lavigne - Im With You/cache/temp.ass'
```
**Wrong approach #1**: Simple backslash escaping like `\'` doesn't work because FFmpeg's filter parser requires more escaping.
**Wrong approach #2**: Shell-style escaping `'\''` (end quote, escaped quote, start quote) only works when passing through a shell. When using Python `subprocess` with a command list, there's no shell - arguments go directly to FFmpeg, so shell escaping patterns don't work.
**Solution**: Use FFmpeg filter escaping (for subprocess without shell):
```python
def _escape_ffmpeg_filter_path(path: str) -> str:
"""Escape a path for FFmpeg filter expressions (for subprocess without shell).
FFmpeg filter escaping rules:
- Backslashes: double them (\ -> \\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\\')
- Spaces: escape with backslash ( -> \ )
"""
# First escape existing backslashes (\ -> \\)
escaped = path.replace("\\", "\\\\")
# Escape single quotes (' -> \\\')
# In Python source: 6 backslashes + apostrophe = 3 backslashes + apostrophe in string
escaped = escaped.replace("'", "\\\\\\'")
# Escape spaces
escaped = escaped.replace(" ", "\\ ")
return escaped
# Result: "I'm With You" becomes "I\\\'m\ With\ You"
# FFmpeg correctly interprets this as: I'm With You
```
**Symptoms**:
- Video generation fails with "Could not create a libass track"
- Error path is subtly different from the path in your code (missing apostrophe)
- Works fine for songs without special characters
**Key insight**: FFmpeg filter escaping is different from shell escaping. When using `subprocess.run(cmd_list)` (no `shell=True`), there's no shell to interpret the arguments. FFmpeg receives the strings directly and applies its own filter expression parsing rules. The `'\''` shell idiom does NOT work in this context.
**Lesson**: When debugging FFmpeg filter path issues, check whether you're using shell or subprocess. For subprocess without shell, escape spaces with `\ ` and apostrophes with `\\\'` (3 backslashes + quote in the actual string).
### Duplicate Code Leads to Inconsistent Behavior
**Problem**: Multiple endpoints computing the same defaults independently. `audio_search.py` had `effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix` but `file_upload.py` was missing it for some endpoints.
@@ -247,6 +293,43 @@ def _get_effective_distribution_settings(...) -> EffectiveDistributionSettings:
**Lesson**: When you see the same logic repeated in multiple places, refactor into a helper function immediately. Otherwise bugs will be fixed in one place but not others.
### Python `or` Operator Can't Override True Defaults with False
**Problem**: Using Python's `or` operator for default logic prevents explicit `false` from overriding a `true` server default.
```python
# BAD - can't explicitly disable if server default is true
effective_value = body.enable_feature or settings.default_enable_feature
# When body.enable_feature=False and settings.default=True:
# False or True = True (ignores explicit False!)
```
**Symptoms**:
- Feature can be enabled but never disabled via API
- Works fine when server default is `false` but breaks when default is `true`
- No errors - just ignores the explicit `false` value
**Solution**: Use `Optional[bool] = None` with explicit `is not None` check:
```python
# Request model
enable_feature: Optional[bool] = None # None = use server default
# Default logic
effective_value = (
body.enable_feature
if body.enable_feature is not None
else settings.default_enable_feature
)
```
This allows three states:
- `None` (not specified) → use server default
- `True` → explicitly enable
- `False` → explicitly disable (overrides server default)
**Lesson**: When a boolean field needs to support "use server default" behavior, use `Optional[bool] = None` instead of `bool = False`. The `or` operator conflates "not specified" with "explicitly false".
### Field Name Mismatches Between Endpoints
**Problem**: Different endpoints setting different field names for the same logical value, where consumers check only one field name.
@@ -588,6 +671,43 @@ GitHub-hosted runners have ~14GB disk, which caused failures during Docker build
See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
### Pulumi CI Deployments Need Skip-Preview for Limited Service Accounts
**Problem**: Running `pulumi up --refresh` in CI requires the service account to have read permissions on ALL managed GCP resources. Pulumi refreshes state by querying each resource, and permission errors cause the deployment to fail before any changes are applied.
**Symptoms**:
```
Error when reading or editing Resource "project" with IAM Member: Role "roles/...":
Error retrieving IAM policy for project "...": googleapi: Error 403: The caller does not have permission
```
**Root cause**: `--refresh` (default behavior) makes read API calls to verify current state matches Pulumi's state. This requires permissions like:
- `resourcemanager.projects.getIamPolicy` (for IAM bindings)
- `secretmanager.secrets.get` (for secrets)
- `cloudtasks.queues.get` (for Cloud Tasks)
- `iam.workloadIdentityPools.get` (for Workload Identity)
- And many more for each resource type managed
**Solution**: Use `--skip-preview` for CI deployments where Pulumi state is authoritative:
```yaml
# In CI workflow
pulumi up --yes --skip-preview --stack org/project/stack
```
**Why this is safe**:
- Infrastructure is only modified through Pulumi (single source of truth)
- Preview/refresh is still useful during local development with broader permissions
- Reduces permissions surface for CI service account (principle of least privilege)
**Alternative** (if refresh is required): Grant the service account `roles/editor` or individual read roles for each resource type. This is less secure but allows drift detection.
**Lesson**: When setting up Pulumi in CI, either:
1. Grant broad read permissions and keep `--refresh` (detects drift)
2. Use minimal permissions and `--skip-preview` (trusts Pulumi state)
Choose based on your threat model. For most projects, trusting Pulumi state is acceptable since manual console changes are rare.
### Docker Disk Space Management on Self-Hosted Runners
**Problem**: Even with 200GB SSDs, self-hosted runners can fill up with Docker images. Each CI build creates new dangling images (~15GB each), and without aggressive cleanup, disk fills in days.
@@ -667,6 +787,61 @@ RUN apt-get update && apt-get install -y \
**Lesson**: When building Docker images for video/subtitle rendering, install fonts explicitly. The default slim images have nothing, and font fallback behavior differs from local development machines.
### Reuse LLM Model Instances Across Operations
**Problem**: Creating a new LLM model instance for each operation in a loop causes massive overhead. Each instance creation triggers model initialization, connection establishment, and optional "warm-up" calls - adding 2+ seconds per operation.
**Example**: Agentic lyrics correction was creating a new `AgenticCorrector` for each gap (lyric segment needing correction):
```python
# BAD - 31 gaps × 2s initialization = 62s overhead
for gap in gaps_to_correct:
agent = AgenticCorrector.from_model(model_id, ...) # New instance every time!
agent.correct_gap(gap)
```
**Symptoms**:
- Operations that should take ~1 minute take 5+ minutes
- Logs show "Initializing model..." or "Warming up..." repeatedly
- Each loop iteration has a consistent ~2s overhead before actual work
- Total time scales linearly with loop count, not with actual work
**Solution**: Create the model instance ONCE before the loop, then reuse:
```python
# GOOD - 1 initialization, reused for all gaps
agent = AgenticCorrector.from_model(model_id, ...) # Create once
for gap in gaps_to_correct:
agent.correct_gap(gap) # Reuse existing instance
```
**Additional optimization**: If operations are independent, process them in parallel:
```python
# BETTER - parallel processing with shared model
agent = AgenticCorrector.from_model(model_id, ...)
with ThreadPoolExecutor(max_workers=5) as executor:
futures = {executor.submit(agent.correct_gap, gap): gap for gap in gaps}
for future in as_completed(futures):
result = future.result()
```
**Results**:
- Before: 31 gaps → ~5 minutes (2s init + 3s work per gap)
- After: 20 gaps → ~55 seconds (1 init + parallel processing)
- Speedup: ~5-6x
**Why warm-up is often unnecessary**: The warm-up pattern (sending a trivial request to "prime" the model) made sense for cold-start scenarios in older systems. Modern LLM APIs:
- Don't have cold-start penalties per request
- Handle connection pooling internally
- May even rate-limit warm-up calls
If you're already making real requests, warm-up adds latency without benefit.
**Key insight**: LLM client instances are designed for reuse. They maintain connection pools, authentication state, and caching. Creating a new instance per request throws away these benefits.
**Configuration**: Added `AGENTIC_MAX_PARALLEL_GAPS` env var (default: 5) to control parallelism. Higher values increase throughput but may hit rate limits.
## Performance Observations
| Operation | Duration |
@@ -755,6 +930,48 @@ export function getToken() {
**Lesson**: For values that can change (auth tokens, user preferences), read fresh from storage on each access rather than caching in module scope.
### Zustand Module-Level Initialization Doesn't Re-run on Client Navigation
**Problem**: Module-level initialization code in zustand stores only runs once when the module is first imported. Client-side navigation (e.g., `router.push()`) doesn't reload modules, so initialization code doesn't re-run.
**Example bug**: Auth module had initialization that fetched user data if a token existed:
```typescript
// auth.ts - module level (runs once at import)
if (typeof window !== 'undefined') {
const token = getAccessToken()
if (token) {
useAuth.getState().fetchUser() // Only runs on initial page load!
}
}
```
After beta enrollment:
1. User lands on `/` (landing page) - auth.ts imported, no token yet, `fetchUser()` not called
2. User completes enrollment, token stored via `setAccessToken()`
3. `router.push('/app')` - client-side navigation, modules already loaded
4. Auth module initialization doesn't re-run - `user` stays `null`
5. UI shows "Login" button instead of user credits
**Symptoms**:
- Feature works after page reload but not after client-side navigation
- Zustand store has correct persisted data (token) but not computed data (user profile)
- E2E tests that use reload pass; tests that use `router.push` fail
**Solution**: Components that need fresh data should trigger fetches in useEffect:
```typescript
// In /app page component
const { user, fetchUser } = useAuth()
useEffect(() => {
const token = getAccessToken()
if (token && !user) {
fetchUser() // Ensure user data is loaded regardless of how we got here
}
}, [user, fetchUser])
```
**Lesson**: Don't rely on module-level initialization for data that might need to be fetched after client-side navigation. Use useEffect in components to ensure data is loaded when needed.
### GCP Organization Policies Block Service Account Keys
**Problem**: GCP organization policies can block service account key creation even when you have Owner access to the project.
@@ -1079,6 +1296,45 @@ logs_ref.document(log_id).set({
**Lesson**: Never use embedded arrays for unbounded data. If an array can grow indefinitely based on operation duration or user input, use a subcollection instead. The 1MB limit applies to the entire document, including all fields.
### Splitting Fast Generation from Slow Encoding
**Problem**: Preview video generation took 60+ seconds on Cloud Run, causing poor UX during lyrics review. The bottleneck was FFmpeg encoding (CPU-bound), not the subtitle generation (fast text processing).
**Solution**: Split the operation into two phases with optional offloading:
1. **ASS Generation (Cloud Run)**: Generate ASS subtitle file locally (~2s). This is pure text processing with no CPU-intensive encoding.
2. **Video Encoding (GCE)**: Upload ASS to GCS and offload FFmpeg encoding to the high-performance GCE worker (~15-20s on C4-standard-8).
```python
# Generate ASS only (fast, local)
result = CorrectionOperations.generate_preview_video(
...,
ass_only=True, # Skip video encoding
)
# Upload ASS to GCS
storage.upload_file(result["ass_path"], f"jobs/{job_id}/previews/{preview_hash}.ass")
# Offload encoding to GCE
gce_result = await encoding_service.encode_preview_video(
job_id=preview_hash,
ass_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.ass",
audio_gcs_path=audio_gcs_path,
output_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.mp4",
)
```
**Key insight**: When a pipeline has both fast (text generation) and slow (media encoding) steps, design for separability:
- Add `_only` parameters to skip slow steps
- Return intermediate outputs (paths to generated files)
- Allow external services to handle the slow part
- Keep fallback to local processing for reliability
**Configuration**: Controlled by `USE_GCE_PREVIEW_ENCODING` env var. Falls back to local encoding if GCE fails or is disabled.
**Result**: Preview generation dropped from 60+ seconds to ~15-20 seconds.
### External Service Response Format Mismatches
**Problem**: The GCE encoding worker returned responses in unexpected formats, causing `'list' object has no attribute 'get'` errors that were hard to debug through the async pipeline.
@@ -1127,6 +1383,214 @@ if isinstance(output_files, list):
**Debugging tip**: When hitting format errors in async pipelines, check the raw response from the external service directly (curl with API key) before assuming the bug is in your code.
### GCE Encoding Returns GCS Paths, Not Local Files
**Problem**: When using GCE encoding (remote high-performance encoder), the orchestrator received GCS blob paths like `jobs/{id}/finals/output_4k_lossless.mp4` instead of local file paths. Subsequent operations like YouTube upload checked `os.path.isfile()` which returned `False`, causing silent failures.
**Symptoms**:
- YouTube upload logged "No video file available for YouTube upload"
- Other file-based operations silently skipped
- Dropbox/GDrive uploads succeeded (they use different code paths)
- No errors in logs - just warnings that were easy to miss
**Root cause**: The `VideoWorkerOrchestrator._run_encoding()` method stored the paths returned by `GCEEncodingBackend` directly without downloading the files. For local encoding, the paths are already local. For GCE encoding, the paths are GCS blob paths that need to be downloaded first.
**Solution**: Added `_download_gce_encoded_files()` method that:
1. Detects when GCE backend was used (`encoding_backend.name == "gce"`)
2. Downloads each encoded file from GCS to the local temp directory
3. Updates result paths to point to local files
```python
# After encoding completes
if encoding_backend.name == "gce" and self.storage:
await self._download_gce_encoded_files(output)
```
**Key insight**: When an operation can run on multiple backends (local vs remote), ensure the outputs are normalized to a common format before subsequent stages consume them. The orchestrator pattern should abstract away backend differences, not leak them to downstream consumers.
**Testing tip**: Test the full pipeline with each backend type, not just the encoding step in isolation. The bug only manifested when YouTube upload ran after GCE encoding - unit tests of each stage passed.
### Unify Encoding Logic with GCS Wheel Deployment
**Problem**: The GCE encoding worker had simplified, duplicated encoding logic that:
- Didn't concatenate title + karaoke + end screens
- Used generic filenames like `output_4k_lossless.mp4` instead of proper names
- Lacked feature parity with the local CLI's `LocalEncodingService`
This caused cloud-generated videos to be incomplete (no title/end screens) and have unhelpful filenames that didn't match the local CLI output.
**Solution**: Deploy the karaoke-gen wheel to GCS, have the GCE worker install and use `LocalEncodingService`:
1. **CI uploads wheel to GCS**: `deploy-backend` job uploads wheel to `gs://bucket/wheels/`
2. **Dynamic wheel loading**: GCE worker downloads and installs latest wheel at job start:
```python
def ensure_latest_wheel():
"""Download and install latest karaoke-gen wheel from GCS."""
subprocess.run(["gsutil", "cp", "gs://bucket/wheels/karaoke_gen-*.whl", "/tmp/"])
wheel = sorted(glob.glob("/tmp/karaoke_gen-*.whl"))[-1]
subprocess.run([sys.executable, "-m", "pip", "install", "--upgrade", wheel])
async def process_job(job_id, request):
ensure_latest_wheel() # Get latest code at job start
# ... rest of processing
```
3. **Single encoding path**: GCE worker imports and uses `LocalEncodingService`:
```python
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
service = LocalEncodingService(logger=logger)
config = EncodingConfig(
title_video=str(title_video),
karaoke_video=str(karaoke_video),
output_lossless_4k_mp4=f"{base_name} (Final Karaoke Lossless 4k).mp4",
# ... proper naming for all formats
)
result = service.encode_all_formats(config)
```
**Key benefits**:
1. **Single source of truth**: Same encoding logic used by CLI, Cloud Run, and GCE
2. **Hot updates without restart**: New code deployed on next job, not VM restart
3. **In-progress jobs unaffected**: Jobs use the code version they started with
4. **Proper output**: Titles/end screens concatenated, files named correctly
**No fallback by design**: We removed all fallback logic to ensure there's only ONE code path. If the wheel fails to load, the job fails clearly rather than producing inconsistent output.
**Lesson**: When multiple deployment targets (CLI, Cloud Run, GCE) need the same complex logic, package it as a library and deploy via a shared mechanism (GCS wheel). Duplicating logic leads to feature drift and inconsistent behavior.
### Unify Preview Encoding with LocalPreviewEncodingService
**Problem**: Following the pattern of final encoding, the GCE preview encoding also had divergent code. The `run_preview_encoding` function in `infrastructure/__main__.py` was a 100-line "simplified" reimplementation that:
- Hardcoded resolution (480x270) instead of using constants
- Had its own FFmpeg filter path escaping function with different logic
- Didn't support hardware acceleration (NVENC)
- Lacked font directory support for custom fonts
- Had different movflags and encoding settings
While the final encoding was unified earlier, preview encoding was overlooked, creating a potential source of inconsistent preview videos between local development and cloud jobs.
**Solution**: Created `LocalPreviewEncodingService` in `backend/services/local_preview_encoding_service.py`:
1. **Extracted preview logic from VideoGenerator**: The service encapsulates the same FFmpeg command building from `VideoGenerator._build_preview_ffmpeg_command`
2. **Single source of truth for preview settings**:
```python
class LocalPreviewEncodingService:
PREVIEW_WIDTH = 480
PREVIEW_HEIGHT = 270
PREVIEW_FPS = 24
PREVIEW_AUDIO_BITRATE = "96k"
```
3. **GCE worker imports from installed wheel**:
```python
from backend.services.local_preview_encoding_service import (
LocalPreviewEncodingService,
PreviewEncodingConfig,
)
service = LocalPreviewEncodingService(logger=logger)
result = service.encode_preview(config)
```
4. **Process preview jobs call ensure_latest_wheel()**:
```python
async def process_preview_job(job_id, request):
ensure_latest_wheel() # Get latest code at job start
# ... rest of processing
```
**Benefits**:
- Consistent preview encoding across CLI, Cloud Run, and GCE
- Hardware acceleration (NVENC) when available
- Proper FFmpeg filter escaping for special characters in paths
- Custom font support via fontsdir filter
- Single place to modify preview settings
**Lesson**: After unifying a major component (final encoding), audit related features (preview encoding) that may have similar duplication. It's easy to overlook "simpler" variations when focusing on the main feature.
### Cloud Distribution Must Match Local CLI Output Structure
**Problem**: Cloud-generated karaoke outputs had different file organization than local CLI, causing confusion and missing files in Dropbox uploads:
1. **lyrics/ folder filename differences**:
- Cloud used generic `(Karaoke).json` instead of `(Lyrics Corrections).json`
- Cloud used `(Karaoke).txt` instead of `(Lyrics Corrected).txt`
- Reference lyrics files (`(Lyrics Genius).txt`, etc.) weren't downloaded
2. **Missing files in lyrics/ folder**:
- `(With Vocals).mkv` wasn't copied to lyrics/ subfolder
- `previews/` subfolder with preview ASS files wasn't created
- Reference lyrics from multiple sources weren't included
**Root causes**:
- Lyrics worker uploaded reference files to GCS but didn't track them in `job.files`
- Video worker used generic naming pattern for all lyrics files
- Video worker didn't copy `(With Vocals).mkv` or previews to lyrics folder
**Solution**:
1. **Track all files in job.files** (lyrics_worker.py):
```python
# Upload ALL reference lyrics and track them for distribution
for ref_filename, source_key in reference_files:
if os.path.exists(ref_path):
url = storage.upload_file(ref_path, gcs_path)
# Track in job.files so video_worker can download
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
```
2. **Use proper filenames in distribution** (video_worker.py):
```python
lyrics_file_mappings = [
('lrc', f"{base_name} (Karaoke).lrc"),
('corrections', f"{base_name} (Lyrics Corrections).json"),
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
# ... etc
]
```
3. **Copy (With Vocals).mkv to lyrics folder**:
```python
if videos.get('with_vocals'):
dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
storage.download_file(videos['with_vocals'], dest)
```
4. **Create previews/ subfolder**:
```python
previews_dir = os.path.join(lyrics_dir, "previews")
os.makedirs(previews_dir, exist_ok=True)
preview_files = storage.list_files(f"jobs/{job_id}/previews/")
for blob_name in preview_files:
if blob_name.endswith('.ass'):
storage.download_file(blob_name, dest_path)
```
**Expected output structure** (matches local CLI):
```
NOMAD-XXXX - Artist - Title/
├── stems/
│ ├── Artist - Title (Vocals model_bs_roformer_...).flac
│ └── ...
└── lyrics/
├── Artist - Title (Karaoke).lrc
├── Artist - Title (Karaoke).ass
├── Artist - Title (Lyrics Corrections).json
├── Artist - Title (Lyrics Corrected).txt
├── Artist - Title (Lyrics Uncorrected).txt
├── Artist - Title (Lyrics Genius).txt
├── Artist - Title (With Vocals).mkv
└── previews/
└── preview_abc123 (Karaoke).ass
```
**Lesson**: When building distribution packages from cloud storage, the organization code must replicate the exact structure users expect from local execution. Track ALL intermediate files in `job.files` even if they're "optional" - users may rely on them.
## What We'd Do Differently
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
+16
View File
@@ -36,6 +36,22 @@
## Recent Changes
- **Audio Search Display Override** (2026-01-06): Added optional "Display As" fields to audio search, allowing users to search for audio by one artist/title (e.g., "Jeremy Kushnier") but display a different artist/title on title screens and filenames (e.g., "Footloose (Broadway Cast)"). Useful for Broadway casts, covers, remixes where tracker metadata differs from desired display. See [API.md](API.md#audio-search).
- **Infrastructure Modularization** (2026-01-06): Refactored `infrastructure/__main__.py` from 2,602 lines to 339 lines (87% reduction). Split into organized modules: `modules/` for core GCP resources, `compute/` for VMs and startup scripts, `config.py` for shared constants. Extracted embedded encoding worker Python app to `backend/services/gce_encoding/`. See `infrastructure/README.md` for new structure and `infrastructure/docs/PHASE5-PACKER-IMAGE-PLAN.md` for planned Packer image optimization.
- **Email Notification System** (2026-01-06): Added automated email notifications for job completion and user action reminders. Features: GCS-backed HTML email templates with fallback defaults, SendGrid with CC support, auto-completion emails on job finish, idle reminder emails via Cloud Tasks (5-min delay for blocking states), admin UI buttons to copy message or send email manually. Endpoints: `GET /api/admin/jobs/{id}/completion-message`, `POST /api/admin/jobs/{id}/send-completion-email`. Feature flag `ENABLE_AUTO_EMAILS` (default: false). See [API.md](API.md#email-notifications-admin) and [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator).
- **GCE Preview Encoding Unified with LocalPreviewEncodingService** (2026-01-06): GCE preview encoding now uses the same `LocalPreviewEncodingService` from the installed wheel, eliminating 100 lines of duplicated FFmpeg logic. Preview videos are now identical across local CLI, Cloud Run, and GCE worker environments. Benefits include: consistent encoding settings (480x270, 24fps, crf 28), proper FFmpeg filter path escaping for special characters, hardware acceleration (NVENC) when available, and custom font support. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#unify-preview-encoding-with-localpreviewencodingservice).
- **GCE Encoding Worker Python 3.13** (2026-01-06): Upgraded GCE encoding worker from Debian's Python 3.11 to Python 3.13 built from source. Uses dedicated virtual environment at `/opt/encoding-worker/venv`, removing the need for `--break-system-packages`. Aligns the encoding worker with CI/Cloud Run Python version.
- **GCE Encoding Unified with LocalEncodingService** (2026-01-06): GCE encoding worker now uses the same `LocalEncodingService` as the local CLI, eliminating duplicated encoding logic. Output files now have proper names (`Artist - Title (Final Karaoke Lossless 4k).mp4` instead of `output_4k_lossless.mp4`) and include title/end screen concatenation. Wheel deployed to GCS; worker installs at job start for hot updates without VM restart. Removed all fallback logic - single code path for consistent output across CLI, Cloud Run, and GCE. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#unify-encoding-logic-with-gcs-wheel-deployment).
- **Cloud Distribution Matches Local CLI** (2026-01-06): Fixed cloud distribution (Dropbox uploads) to match local CLI output structure. The `lyrics/` folder now includes: proper filenames (`(Lyrics Corrections).json` instead of `(Karaoke).json`), all reference lyrics files (`(Lyrics Genius).txt`, `(Lyrics Spotify).txt`, etc.), `(With Vocals).mkv`, and `previews/` subfolder with preview ASS files. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#cloud-distribution-must-match-local-cli-output-structure).
- **Agentic Correction Performance** (2026-01-05): Optimized agentic AI correction from ~5 minutes to ~55 seconds for 20 gaps (~5-6x speedup). Fixed anti-pattern of creating new model instance per gap (caused repeated 2s warm-up overhead). Now creates model once and processes gaps in parallel using ThreadPoolExecutor (default 5 workers). Configure via `AGENTIC_MAX_PARALLEL_GAPS` env var. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#reuse-llm-model-instances-across-operations).
- **Worker Logs Subcollection** (2026-01-04): Moved `worker_logs` from embedded array in job documents to Firestore subcollection (`jobs/{job_id}/logs`). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flag `USE_LOG_SUBCOLLECTION=true` (default). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
+12
View File
@@ -3,6 +3,7 @@
import { useState, useEffect, useCallback } from "react"
import { useRouter } from "next/navigation"
import { api, Job, getAccessToken } from "@/lib/api"
import { useAuth } from "@/lib/auth"
import { useJobNotifications, useVisibilityRefresh } from "@/hooks/use-notifications"
import { Button } from "@/components/ui/button"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
@@ -28,6 +29,7 @@ export default function AppPage() {
const [isInitialLoad, setIsInitialLoad] = useState(true)
const [isAuthenticated, setIsAuthenticated] = useState<boolean | null>(null)
const { isDarkMode, toggleTheme, mounted } = useTheme()
const { user, fetchUser } = useAuth()
// Memoize loadJobs for use with visibility refresh
const loadJobs = useCallback(async () => {
@@ -67,6 +69,16 @@ export default function AppPage() {
setIsAuthenticated(true)
}, [router])
// Ensure user data is loaded after authentication
// This handles the case where user navigates via client-side navigation
// (e.g., after beta enrollment) and the module-level fetchUser() didn't run
useEffect(() => {
const token = getAccessToken()
if (token && !user) {
fetchUser()
}
}, [user, fetchUser])
// Load jobs on mount (only if authenticated)
useEffect(() => {
if (isAuthenticated !== true) {
+68 -24
View File
@@ -34,6 +34,9 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
// Audio search form
const [searchArtist, setSearchArtist] = useState("")
const [searchTitle, setSearchTitle] = useState("")
// Display As overrides (optional - if empty, search values used for display)
const [displayArtist, setDisplayArtist] = useState("")
const [displayTitle, setDisplayTitle] = useState("")
// Theme selection (shared across all tabs)
const [selectedTheme, setSelectedTheme] = useState<string | undefined>()
@@ -128,9 +131,14 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
theme_id: selectedTheme,
color_overrides: cleanColorOverrides(colorOverrides),
non_interactive: nonInteractive,
// Display overrides (empty string means use search values)
display_artist: displayArtist.trim() || undefined,
display_title: displayTitle.trim() || undefined,
})
setSearchArtist("")
setSearchTitle("")
setDisplayArtist("")
setDisplayTitle("")
onJobCreated()
} catch (err) {
if (err instanceof ApiError) {
@@ -407,33 +415,69 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
<TabsContent value="search" className="mt-4">
<form onSubmit={handleSearchSubmit} className="space-y-4">
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 sm:gap-4">
<div className="space-y-2">
<Label htmlFor="search-artist" style={{ color: 'var(--text)' }}>Artist</Label>
<Input
id="search-artist"
placeholder="Artist name"
value={searchArtist}
onChange={(e) => setSearchArtist(e.target.value)}
disabled={isSubmitting}
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
/>
{/* Search For section */}
<div className="space-y-2">
<Label style={{ color: 'var(--text)' }}>Search For</Label>
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 sm:gap-4">
<div className="space-y-2">
<Label htmlFor="search-artist" className="text-xs" style={{ color: 'var(--text-muted)' }}>Artist</Label>
<Input
id="search-artist"
placeholder="Artist name on trackers"
value={searchArtist}
onChange={(e) => setSearchArtist(e.target.value)}
disabled={isSubmitting}
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
/>
</div>
<div className="space-y-2">
<Label htmlFor="search-title" className="text-xs" style={{ color: 'var(--text-muted)' }}>Title</Label>
<Input
id="search-title"
placeholder="Song title on trackers"
value={searchTitle}
onChange={(e) => setSearchTitle(e.target.value)}
disabled={isSubmitting}
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
/>
</div>
</div>
<div className="space-y-2">
<Label htmlFor="search-title" style={{ color: 'var(--text)' }}>Title</Label>
<Input
id="search-title"
placeholder="Song title"
value={searchTitle}
onChange={(e) => setSearchTitle(e.target.value)}
disabled={isSubmitting}
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
/>
</div>
{/* Display As section (optional) */}
<div className="space-y-2">
<div className="flex items-center gap-2">
<Label style={{ color: 'var(--text)' }}>Display As</Label>
<span className="text-xs px-1.5 py-0.5 rounded" style={{ color: 'var(--text-muted)', backgroundColor: 'var(--secondary)' }}>optional</span>
</div>
<p className="text-xs" style={{ color: 'var(--text-muted)' }}>
Override how artist/title appear on the title card and filename. Leave empty to use search values.
</p>
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 sm:gap-4">
<div className="space-y-2">
<Label htmlFor="display-artist" className="text-xs" style={{ color: 'var(--text-muted)' }}>Display Artist</Label>
<Input
id="display-artist"
placeholder="e.g., Footloose (Broadway Cast)"
value={displayArtist}
onChange={(e) => setDisplayArtist(e.target.value)}
disabled={isSubmitting}
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
/>
</div>
<div className="space-y-2">
<Label htmlFor="display-title" className="text-xs" style={{ color: 'var(--text-muted)' }}>Display Title</Label>
<Input
id="display-title"
placeholder="e.g., I Can't Stand Still"
value={displayTitle}
onChange={(e) => setDisplayTitle(e.target.value)}
disabled={isSubmitting}
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
/>
</div>
</div>
</div>
<p className="text-xs" style={{ color: 'var(--text-muted)' }}>
<span className="text-amber-500 font-medium">Note:</span> Format these exactly as you want them on the title card and video filename.
</p>
{/* Theme Selection */}
<div className="space-y-4 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
+172 -3
View File
@@ -1,9 +1,20 @@
"use client"
import { useEffect, useState, useRef, useCallback } from "react"
import { api } from "@/lib/api"
import { api, adminApi } from "@/lib/api"
import { Button } from "@/components/ui/button"
import { Download, Loader2, ExternalLink, FolderOpen, Copy } from "lucide-react"
import { Download, Loader2, ExternalLink, FolderOpen, Copy, Mail, MessageSquare } from "lucide-react"
import { useAuth } from "@/lib/auth"
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog"
import { Input } from "@/components/ui/input"
import { Label } from "@/components/ui/label"
interface OutputLinksProps {
jobId: string
@@ -15,18 +26,35 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
const [dropboxUrl, setDropboxUrl] = useState<string | null>(null)
const [isLoading, setIsLoading] = useState(false)
const [copiedUrl, setCopiedUrl] = useState<string | null>(null)
const [copiedMessage, setCopiedMessage] = useState(false)
const [isLoadingMessage, setIsLoadingMessage] = useState(false)
const [showEmailDialog, setShowEmailDialog] = useState(false)
const [emailInput, setEmailInput] = useState("")
const [isSendingEmail, setIsSendingEmail] = useState(false)
const [emailSent, setEmailSent] = useState(false)
const copyTimeoutRef = useRef<NodeJS.Timeout | null>(null)
const messageTimeoutRef = useRef<NodeJS.Timeout | null>(null)
const emailTimeoutRef = useRef<NodeJS.Timeout | null>(null)
const { user } = useAuth()
const isAdmin = user?.role === 'admin'
useEffect(() => {
loadOutputLinks()
}, [jobId])
// Cleanup timeout on unmount
// Cleanup timeouts on unmount
useEffect(() => {
return () => {
if (copyTimeoutRef.current) {
clearTimeout(copyTimeoutRef.current)
}
if (messageTimeoutRef.current) {
clearTimeout(messageTimeoutRef.current)
}
if (emailTimeoutRef.current) {
clearTimeout(emailTimeoutRef.current)
}
}
}, [])
@@ -76,6 +104,60 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
}
}, [])
const copyCompletionMessage = useCallback(async () => {
if (!navigator?.clipboard?.writeText) {
console.error("Clipboard API not available")
return
}
setIsLoadingMessage(true)
try {
const response = await adminApi.getCompletionMessage(jobId)
await navigator.clipboard.writeText(response.message)
setCopiedMessage(true)
// Clear any existing timeout
if (messageTimeoutRef.current) {
clearTimeout(messageTimeoutRef.current)
}
// Set new timeout
messageTimeoutRef.current = setTimeout(() => {
setCopiedMessage(false)
}, 2000)
} catch (err) {
console.error("Failed to copy completion message:", err)
} finally {
setIsLoadingMessage(false)
}
}, [jobId])
const handleSendEmail = useCallback(async () => {
if (!emailInput.trim()) return
setIsSendingEmail(true)
try {
await adminApi.sendCompletionEmail(jobId, emailInput.trim(), true)
setEmailSent(true)
// Clear any existing timeout
if (emailTimeoutRef.current) {
clearTimeout(emailTimeoutRef.current)
}
// Set new timeout to close dialog after success message
emailTimeoutRef.current = setTimeout(() => {
setShowEmailDialog(false)
setEmailSent(false)
setEmailInput("")
}, 1500)
} catch (err) {
console.error("Failed to send email:", err)
alert("Failed to send email. Please check the email address and try again.")
} finally {
setIsSendingEmail(false)
}
}, [jobId, emailInput])
if (isLoading) {
return (
<div className="flex items-center gap-2 text-xs" style={{ color: 'var(--text-muted)' }}>
@@ -193,12 +275,99 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
</a>
)}
</div>
{/* Admin-only buttons */}
{isAdmin && (
<div className="flex flex-wrap gap-2 pt-1 border-t border-gray-700/50 mt-2">
<button
type="button"
onClick={(e) => { e.stopPropagation(); copyCompletionMessage() }}
disabled={isLoadingMessage}
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-amber-600 hover:bg-amber-500 text-white disabled:opacity-50"
title="Copy completion message to clipboard"
>
{isLoadingMessage ? (
<Loader2 className="w-3 h-3 animate-spin" />
) : (
<MessageSquare className="w-3 h-3" />
)}
{copiedMessage ? "Copied!" : "Copy Message"}
</button>
<button
type="button"
onClick={(e) => { e.stopPropagation(); setShowEmailDialog(true) }}
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-teal-600 hover:bg-teal-500 text-white"
title="Send completion email"
>
<Mail className="w-3 h-3" />
Send Email
</button>
</div>
)}
</div>
)}
{!hasOutputs && (
<p className="text-xs" style={{ color: 'var(--text-muted)' }}>No outputs available yet</p>
)}
{/* Send Email Dialog */}
<Dialog open={showEmailDialog} onOpenChange={setShowEmailDialog}>
<DialogContent className="sm:max-w-[425px]" onClick={(e) => e.stopPropagation()}>
<DialogHeader>
<DialogTitle>Send Completion Email</DialogTitle>
<DialogDescription>
Send the job completion message to an email address. The email will be CC'd to gen@nomadkaraoke.com.
</DialogDescription>
</DialogHeader>
<div className="grid gap-4 py-4">
<div className="grid grid-cols-4 items-center gap-4">
<Label htmlFor="email" className="text-right">
Email
</Label>
<Input
id="email"
type="email"
value={emailInput}
onChange={(e) => setEmailInput(e.target.value)}
placeholder="customer@example.com"
className="col-span-3"
onKeyDown={(e) => {
if (e.key === 'Enter') {
handleSendEmail()
}
}}
/>
</div>
</div>
<DialogFooter>
<Button
type="button"
variant="outline"
onClick={() => setShowEmailDialog(false)}
disabled={isSendingEmail}
>
Cancel
</Button>
<Button
type="button"
onClick={handleSendEmail}
disabled={isSendingEmail || !emailInput.trim() || emailSent}
>
{isSendingEmail ? (
<>
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
Sending...
</>
) : emailSent ? (
"Sent!"
) : (
"Send Email"
)}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
</div>
)
}
+50 -1
View File
@@ -1,10 +1,11 @@
"use client";
import { useEffect, useState } from "react";
import { api } from "@/lib/api";
import { api, EncodingWorkerHealth } from "@/lib/api";
export function VersionFooter() {
const [backendVersion, setBackendVersion] = useState<string | null>(null);
const [encodingWorker, setEncodingWorker] = useState<EncodingWorkerHealth | null>(null);
// Frontend version is injected at build time from pyproject.toml
const frontendVersion = process.env.NEXT_PUBLIC_APP_VERSION || "dev";
@@ -19,9 +20,56 @@ export function VersionFooter() {
}
};
const fetchEncodingWorkerHealth = async () => {
try {
const health = await api.getEncodingWorkerHealth();
setEncodingWorker(health);
} catch (error) {
console.error("Failed to fetch encoding worker health:", error);
setEncodingWorker(null);
}
};
fetchBackendVersion();
fetchEncodingWorkerHealth();
}, []);
// Render encoding worker status indicator
const renderEncodingWorkerStatus = () => {
if (!encodingWorker) {
return null;
}
if (encodingWorker.status === "not_configured") {
return null; // Don't show if not configured
}
const isHealthy = encodingWorker.available && encodingWorker.status === "ok";
const statusColor = isHealthy ? "text-green-500" : "text-red-500";
const statusDot = isHealthy ? "●" : "○";
// Build status text
let statusText = isHealthy ? "Encoder" : "Encoder offline";
if (isHealthy && encodingWorker.version) {
statusText = `Encoder: v${encodingWorker.version}`;
}
// Add active jobs indicator if there are any
if (isHealthy && encodingWorker.active_jobs && encodingWorker.active_jobs > 0) {
statusText += ` (${encodingWorker.active_jobs} active)`;
}
return (
<>
{" | "}
<span className={statusColor} title={isHealthy ? "Encoding worker healthy" : encodingWorker.error || "Encoding worker offline"}>
{statusDot}
</span>{" "}
{statusText}
</>
);
};
return (
<footer className="border-t mt-12 py-6" style={{ borderColor: 'var(--card-border)' }}>
<div className="px-4 text-center text-sm" style={{ color: 'var(--text-muted)' }}>
@@ -39,6 +87,7 @@ export function VersionFooter() {
Backend: v{backendVersion}
</>
)}
{renderEncodingWorkerStatus()}
</p>
</div>
</footer>
+139
View File
@@ -43,8 +43,10 @@ export interface EmailHelper {
isAvailable: boolean;
createInbox: () => Promise<InboxDto>;
waitForEmail: (inboxId: string, timeout?: number) => Promise<Email>;
waitForCompletionEmail: (inboxId: string, artist?: string, title?: string, timeout?: number) => Promise<Email>;
extractMagicLink: (email: Email) => string | null;
extractVerificationCode: (email: Email) => string | null;
isCompletionEmail: (email: Email) => boolean;
deleteInbox: (inboxId: string) => Promise<void>;
}
@@ -78,6 +80,37 @@ function generateUniqueTag(): string {
return `test-${timestamp}-${random}`;
}
/**
* Check if an email is a job completion email.
* Extracted to standalone function for TypeScript compatibility.
*/
function checkIsCompletionEmail(email: Email): boolean {
const subject = (email.subject || '').toLowerCase();
const body = (email.body || '').toLowerCase();
// Check subject for completion indicators
const subjectIndicators = [
'karaoke video is ready',
'your video is ready',
'video ready',
'complete',
];
// Check body for completion indicators
const bodyIndicators = [
'your karaoke video is ready',
'video has been created',
'youtube',
'dropbox',
'download your video',
];
const hasSubjectMatch = subjectIndicators.some(indicator => subject.includes(indicator));
const hasBodyMatch = bodyIndicators.some(indicator => body.includes(indicator));
return hasSubjectMatch || hasBodyMatch;
}
/**
* Creates an email testing helper.
* Returns a helper with isAvailable=false if testmail.app env vars are not set.
@@ -234,6 +267,110 @@ export async function createEmailHelper(): Promise<EmailHelper> {
return null;
},
/**
* Check if an email is a job completion email
*/
isCompletionEmail(email: Email): boolean {
return checkIsCompletionEmail(email);
},
/**
* Wait for a job completion email to arrive
* @param inboxId - The tag to check (returned from createInbox as id)
* @param artist - Optional artist name to verify in email
* @param title - Optional song title to verify in email
* @param timeout - Timeout in ms (default: 120000 = 2 minutes)
*/
async waitForCompletionEmail(
inboxId: string,
artist?: string,
title?: string,
timeout = 120000
): Promise<Email> {
console.log(`Waiting for completion email with tag ${inboxId}...`);
const pollInterval = Math.min(timeout, 30000);
const startTime = Date.now();
// Track emails we've already seen (by timestamp)
const seenTimestamps = new Set<number>();
while (Date.now() - startTime < timeout) {
const remainingTime = timeout - (Date.now() - startTime);
const waitTime = Math.min(pollInterval, remainingTime);
const waitSeconds = Math.ceil(waitTime / 1000);
const url = new URL(TESTMAIL_API_URL);
url.searchParams.set('apikey', TESTMAIL_API_KEY!);
url.searchParams.set('namespace', TESTMAIL_NAMESPACE!);
url.searchParams.set('tag', inboxId);
url.searchParams.set('livequery', 'true');
url.searchParams.set('wait', waitSeconds.toString());
try {
const response = await fetch(url.toString());
if (!response.ok) {
throw new Error(`testmail.app API error: ${response.status} ${response.statusText}`);
}
const data: TestmailResponse = await response.json();
if (data.result !== 'success') {
throw new Error(`testmail.app API error: ${data.message || 'Unknown error'}`);
}
if (data.emails && data.emails.length > 0) {
// Look for completion emails we haven't seen yet
for (const testmailEmail of data.emails.sort((a, b) => b.timestamp - a.timestamp)) {
// Skip if we've already processed this email
if (seenTimestamps.has(testmailEmail.timestamp)) {
continue;
}
seenTimestamps.add(testmailEmail.timestamp);
const email: Email = {
subject: testmailEmail.subject,
body: testmailEmail.html || testmailEmail.text,
from: testmailEmail.from,
timestamp: testmailEmail.timestamp,
};
// Check if this is a completion email
if (checkIsCompletionEmail(email)) {
console.log(`Found completion email: ${email.subject}`);
// Optionally verify artist/title in subject or body
if (artist && title) {
const content = `${email.subject} ${email.body}`.toLowerCase();
const hasArtist = content.includes(artist.toLowerCase());
const hasTitle = content.includes(title.toLowerCase());
if (hasArtist && hasTitle) {
console.log(` ✓ Verified artist "${artist}" and title "${title}" in email`);
} else {
console.log(` ⚠ Artist/title verification: artist=${hasArtist}, title=${hasTitle}`);
}
}
return email;
} else {
console.log(` Skipping non-completion email: ${testmailEmail.subject}`);
}
}
}
} catch (error) {
if (Date.now() - startTime < timeout) {
console.log(`Polling for completion email... (${Math.round((Date.now() - startTime) / 1000)}s elapsed)`);
continue;
}
throw error;
}
}
throw new Error(`Timed out waiting for completion email after ${timeout}ms`);
},
/**
* Delete an inbox after testing.
* In testmail.app, emails auto-expire - this is a no-op.
@@ -257,8 +394,10 @@ function createDisabledHelper(): EmailHelper {
isAvailable: false,
createInbox: notAvailable,
waitForEmail: notAvailable,
waitForCompletionEmail: notAvailable,
extractMagicLink: () => null,
extractVerificationCode: () => null,
isCompletionEmail: () => false,
deleteInbox: notAvailable,
};
}
@@ -154,6 +154,23 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
// Only create email helper if we're doing the full signup flow
const emailHelper = usePreConfiguredToken ? null : await createEmailHelper();
// Intercept API requests to disable YouTube upload for E2E tests
// This prevents test jobs from uploading to YouTube when server default is true
await page.route('**/api/audio-search/search', async (route) => {
const request = route.request();
if (request.method() === 'POST') {
const postData = request.postDataJSON();
// Explicitly disable YouTube upload for E2E test jobs
postData.enable_youtube_upload = false;
console.log(' [Request Intercept] Disabled YouTube upload for audio search');
await route.continue({
postData: JSON.stringify(postData),
});
} else {
await route.continue();
}
});
try {
// =========================================================================
// STEP 1: Landing Page
@@ -815,6 +832,62 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
await page.screenshot({ path: 'test-results/09-job-complete.png' });
console.log('STEP 9 COMPLETE: Job finished');
// =========================================================================
// STEP 9.5: Verify Completion Email (if email testing available)
// =========================================================================
if (emailHelper && inboxId) {
console.log('\n========================================');
console.log('STEP 9.5: Verify Completion Email');
console.log('========================================');
try {
console.log(' Waiting for completion email to arrive...');
// Wait up to 3 minutes for the completion email
// Email is sent asynchronously after job completion, may take a moment
const completionEmail = await emailHelper.waitForCompletionEmail(
inboxId,
TEST_SONG.artist,
TEST_SONG.title,
180_000 // 3 minutes
);
console.log(` ✓ Completion email received: "${completionEmail.subject}"`);
// Verify email content
const emailBody = completionEmail.body || '';
const emailSubject = completionEmail.subject || '';
// Check for expected content indicators
const hasVideoReadyText = emailBody.toLowerCase().includes('ready') ||
emailSubject.toLowerCase().includes('ready');
const hasSongInfo = emailBody.toLowerCase().includes(TEST_SONG.artist.toLowerCase()) ||
emailSubject.toLowerCase().includes(TEST_SONG.artist.toLowerCase());
console.log(` Email verification:`);
console.log(` - "ready" indicator: ${hasVideoReadyText ? '✓' : '✗'}`);
console.log(` - Artist name: ${hasSongInfo ? '✓' : '✗'}`);
// Log email details for debugging
console.log(` Email from: ${completionEmail.from}`);
console.log(` Subject: ${emailSubject}`);
console.log(` Body preview: ${emailBody.substring(0, 200)}...`);
await page.screenshot({ path: 'test-results/09.5-completion-email-received.png' });
console.log('STEP 9.5 COMPLETE: Completion email verified');
} catch (e) {
// Don't fail the test if email verification times out - it's an enhancement
console.log(` ⚠ WARNING: Completion email verification failed: ${e}`);
console.log(' This may indicate email sending is disabled or delayed.');
console.log(' Continuing with remaining steps...');
}
} else {
console.log('\n========================================');
console.log('STEP 9.5: Skip Completion Email Verification');
console.log('========================================');
console.log(' Skipping - using pre-configured token (no testmail inbox)');
}
// =========================================================================
// STEP 10: Verify Downloads
// =========================================================================
@@ -171,6 +171,71 @@ test.describe('Authentication - Beta Enrollment', () => {
await page.waitForURL(/\/app/, { timeout: 10000 });
expect(page.url()).toContain('/app');
});
test('beta enrollment shows credits after redirect to app', async ({ page }) => {
// This test verifies that after beta enrollment, the /app page
// correctly fetches and displays user data including credits.
// This catches the bug where client-side navigation didn't trigger fetchUser().
await setupApiFixtures(page, {
mocks: [
{
method: 'POST',
path: '/api/users/beta/enroll',
response: {
status: 200,
body: {
status: 'success',
message: 'Enrolled in beta',
credits_granted: 1,
session_token: 'mock-session-token-456',
},
},
},
{
method: 'GET',
path: '/api/users/me',
response: {
status: 200,
body: {
user: {
email: 'beta-tester@example.com',
credits: 1,
role: 'user',
display_name: null,
total_jobs_created: 0,
total_jobs_completed: 0,
},
has_session: true,
},
},
},
{
method: 'GET',
path: '/api/jobs',
response: { body: [] },
},
],
});
await page.goto('/');
await page.waitForLoadState('networkidle');
// Complete beta enrollment
await page.getByRole('button', { name: /join beta program/i }).click();
await page.locator('#beta-email').fill('beta-tester@example.com');
await page.locator('textarea').fill(
'I want to create a karaoke video for my favorite song!'
);
await page.locator('input[type="checkbox"]').check();
await page.getByRole('button', { name: /get my free credit/i }).click();
// Wait for redirect to /app
await page.waitForURL(/\/app/, { timeout: 10000 });
// Verify credits are displayed in the UI
// The AuthStatus component should show "1 credit" after fetchUser() completes
await expect(page.getByText(/1 credit/i)).toBeVisible({ timeout: 10000 });
});
});
test.describe('Authentication - Token Persistence', () => {
+69 -1
View File
@@ -138,6 +138,15 @@ export interface JobLog {
details?: Record<string, any>;
}
export interface EncodingWorkerHealth {
available: boolean;
status: string; // 'ok' | 'offline' | 'not_configured'
version?: string | null;
active_jobs?: number;
queue_length?: number;
error?: string;
}
class ApiError extends Error {
status: number;
data?: any;
@@ -418,6 +427,9 @@ export const api = {
theme_id?: string;
color_overrides?: ColorOverrides;
non_interactive?: boolean;
// Display overrides - if provided, these appear on title screens/filenames instead of search values
display_artist?: string;
display_title?: string;
}
): Promise<AudioSearchResponse> {
const body: Record<string, any> = { artist, title, auto_download: autoDownload };
@@ -428,6 +440,9 @@ export const api = {
if (options?.theme_id) body.theme_id = options.theme_id;
if (options?.color_overrides) body.color_overrides = options.color_overrides;
if (options?.non_interactive !== undefined) body.non_interactive = options.non_interactive;
// Display overrides
if (options?.display_artist) body.display_artist = options.display_artist;
if (options?.display_title) body.display_title = options.display_title;
const response = await fetch(`${API_BASE_URL}/api/audio-search/search`, {
method: 'POST',
@@ -583,7 +598,7 @@ export const api = {
},
// ==========================================================================
// Version API endpoint
// Version & Health API endpoints
// ==========================================================================
/**
@@ -596,6 +611,14 @@ export const api = {
return handleResponse(response);
},
/**
* Get encoding worker health status
*/
async getEncodingWorkerHealth(): Promise<EncodingWorkerHealth> {
const response = await fetch(`${API_BASE_URL}/health/encoding-worker`);
return handleResponse(response);
},
// ==========================================================================
// Credits/Payment API endpoints
// ==========================================================================
@@ -1038,7 +1061,52 @@ export const adminApi = {
);
return handleResponse(response);
},
/**
* Get rendered completion message for a job (for copy to clipboard)
*/
async getCompletionMessage(jobId: string): Promise<CompletionMessageResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/jobs/${jobId}/completion-message`,
{ headers: getAuthHeaders() }
);
return handleResponse(response);
},
/**
* Send completion email for a job
*/
async sendCompletionEmail(jobId: string, toEmail: string, ccAdmin: boolean = true): Promise<SendCompletionEmailResponse> {
const response = await fetch(
`${API_BASE_URL}/api/admin/jobs/${jobId}/send-completion-email`,
{
method: 'POST',
headers: {
...getAuthHeaders(),
'Content-Type': 'application/json',
},
body: JSON.stringify({ to_email: toEmail, cc_admin: ccAdmin }),
}
);
return handleResponse(response);
},
};
// Types for admin completion message API
export interface CompletionMessageResponse {
job_id: string;
message: string;
subject: string;
youtube_url?: string;
dropbox_url?: string;
}
export interface SendCompletionEmailResponse {
success: boolean;
job_id: string;
to_email: string;
message: string;
}
export { ApiError };
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "karaoke-gen-frontend",
"version": "0.79.0",
"version": "0.80.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "karaoke-gen-frontend",
"version": "0.79.0",
"version": "0.80.0",
"dependencies": {
"@hookform/resolvers": "^3.10.0",
"@radix-ui/react-accordion": "1.2.2",
+162 -269
View File
@@ -2,340 +2,233 @@
This directory contains Pulumi infrastructure code that manages all Google Cloud Platform resources for the karaoke generator backend.
## What This Replaces
## Directory Structure
This replaces the manual setup in:
- `backend/setup-gcp.sh` (shell script)
- `docs/GCP-SETUP.md` (manual instructions)
```
infrastructure/
├── __main__.py # Entry point - wires modules together (~340 lines)
├── config.py # Centralized configuration constants
├── Pulumi.yaml # Pulumi project config
├── Pulumi.prod.yaml # Production stack config
├── requirements.txt # Python dependencies
│
├── modules/ # Core GCP resources
│ ├── database.py # Firestore database + indexes
│ ├── storage.py # GCS bucket + lifecycle rules
│ ├── artifact_registry.py # Docker repository
│ ├── secrets.py # Secret Manager secrets
│ ├── cloud_tasks.py # Task queues
│ ├── cloud_run.py # Domain mapping, Cloud Run Jobs
│ ├── monitoring.py # Alert policies
│ └── iam/ # IAM by purpose
│ ├── backend_sa.py # karaoke-backend service account
│ ├── github_actions_sa.py # WIF + deployer permissions
│ ├── claude_automation_sa.py # Claude Code automation
│ └── worker_sas.py # VM service accounts
│
├── compute/ # VM-based resources
│ ├── flacfetch_vm.py # Audio download service
│ ├── encoding_worker_vm.py # Video encoding VM
│ ├── github_runners.py # Self-hosted CI runners
│ └── startup_scripts/ # VM startup scripts (.sh)
│ ├── flacfetch.sh
│ ├── encoding_worker.sh
│ └── github_runner.sh
│
├── functions/ # Cloud Functions
│ └── gdrive_validator/
│ ├── main.py
│ ├── requirements.txt
│ └── deploy.sh # Source upload script
│
└── monitoring/ # Monitoring configs
├── dashboard.json
└── job-inspector.html
```
## Resources Managed
This Pulumi program manages:
- **Firestore Database**: Native mode database for job state
- **Cloud Storage Bucket**: For uploads, outputs, and temp files
- **Artifact Registry**: Docker repository for backend images
- **Service Account**: For Cloud Run with appropriate IAM roles
- **Secrets**: Secret Manager secrets (you add the values)
- **IAM Bindings**: All necessary permissions
- **Cloud Function**: GDrive Validator for checking public share folder
- **Cloud Scheduler**: Daily trigger for GDrive validation
| Category | Resources |
|----------|-----------|
| **Database** | Firestore (Native mode), 10 indexes, TTL policy |
| **Storage** | GCS bucket with lifecycle rules |
| **Registry** | Artifact Registry for Docker images |
| **Secrets** | 18 Secret Manager secrets |
| **IAM** | 7 service accounts, WIF, 50+ IAM bindings |
| **Compute** | Encoding worker, Flacfetch, 10 GitHub runners |
| **Tasks** | 5 Cloud Tasks queues |
| **Monitoring** | 4 alert policies |
| **Functions** | GDrive validator + scheduler |
## Prerequisites
1. **Pulumi CLI**: Install with `brew install pulumi`
2. **Pulumi Account**: Sign up at https://app.pulumi.com
3. **Google Cloud SDK**: Already installed
4. **Python 3.10+**: For running Pulumi Python
## Setup
## Quick Start
```bash
cd infrastructure
# Login to Pulumi (interactive)
pulumi login
# Preview changes
pulumi preview --diff
# Create a new stack (e.g., "dev" or "prod")
pulumi stack init dev
# Set your GCP project
pulumi config set gcp:project nomadkaraoke
pulumi config set gcp:region us-central1
# Install Python dependencies
pip install -r requirements.txt
```
## Usage
### Preview Changes
```bash
pulumi preview
```
This shows you what will be created/updated/deleted without making changes.
### Apply Changes
```bash
# Apply changes
pulumi up
```
This prompts you to confirm, then applies the changes.
### View Current State
```bash
# View outputs
pulumi stack output
```
Shows all exported values (bucket name, service account email, etc.)
## Prerequisites
### Destroy Everything
1. **Pulumi CLI**: `brew install pulumi`
2. **Pulumi Account**: Sign up at https://app.pulumi.com
3. **Google Cloud SDK**: With authenticated account
4. **Python 3.10+**: For running Pulumi
```bash
pulumi destroy
```
## Module Guide
**Warning**: This deletes all infrastructure!
### config.py
Centralized configuration for the entire infrastructure:
- Project ID, region, zone
- Machine types and disk sizes
- Google Drive folder ID
- Number of GitHub runners
### modules/database.py
Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.
### modules/storage.py
GCS bucket with lifecycle rules:
- 7-day cleanup of staging data
- 24-hour cleanup of raw audio files
### modules/iam/
Service accounts organized by purpose:
- `backend_sa.py`: Cloud Run backend permissions
- `github_actions_sa.py`: CI/CD via Workload Identity Federation
- `worker_sas.py`: VM service accounts (flacfetch, encoding, runners)
### compute/
VM definitions with startup scripts as separate files. Each module creates:
- Static IP (where needed)
- VM instance
- Firewall rules
## Adding Secret Values
After running `pulumi up`, you need to add the actual secret values:
After `pulumi up`, add actual secret values:
```bash
# Add AudioShake API key
echo -n "your-audioshake-key" | \
gcloud secrets versions add audioshake-api-key --data-file=-
# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-
# Add Genius API key
echo -n "your-genius-key" | \
gcloud secrets versions add genius-api-key --data-file=-
# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-
# Add Audio Separator API URL
echo -n "https://your-modal-url" | \
gcloud secrets versions add audio-separator-api-url --data-file=-
# Add RED API key (for flacfetch audio search - private tracker)
echo -n "your-red-api-key" | \
gcloud secrets versions add red-api-key --data-file=-
# Add RED API URL (for flacfetch audio search - private tracker)
echo -n "https://your.red.url" | \
gcloud secrets versions add red-api-url --data-file=-
# Add OPS API key (for flacfetch audio search - private tracker)
echo -n "your-ops-api-key" | \
gcloud secrets versions add ops-api-key --data-file=-
# Add Pushbullet API key (for GDrive validator notifications)
echo -n "your-pushbullet-api-key" | \
gcloud secrets versions add pushbullet-api-key --data-file=-
# GitHub runner PAT (repo scope)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-
```
Or use Pulumi config secrets:
## GDrive Validator
The Cloud Function validates your public Google Drive folder daily:
- Checks for duplicate sequence numbers
- Validates filename formats
- Detects sequence gaps
- Sends Pushbullet notifications
**Deploy:**
```bash
pulumi config set --secret audioshake-api-key "your-key"
pulumi config set --secret genius-api-key "your-key"
pulumi config set audio-separator-api-url "https://your-modal-url"
cd functions/gdrive_validator
./deploy.sh
```
Then update `__main__.py` to create secret versions with these values.
Note: `deploy.sh` uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.
## Deploying Cloud Run
## Self-Hosted GitHub Runners
After infrastructure is set up, deploy Cloud Run separately:
20 GCP VMs serve as self-hosted runners with:
- 200GB SSD (vs 14GB on GitHub-hosted)
- Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
- Labels: `self-hosted`, `linux`, `x64`, `gcp`, `large-disk`
**Usage in workflows:**
```yaml
runs-on: [self-hosted, linux, gcp]
```
**Troubleshooting:**
```bash
# Build image
gcloud builds submit --config=../cloudbuild.yaml --timeout=20m
# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo cat /var/log/github-runner-startup.log"
# Deploy (using outputs from Pulumi)
SA_EMAIL=$(pulumi stack output service_account_email)
BUCKET_NAME=$(pulumi stack output bucket_name)
gcloud run deploy karaoke-backend \
--image us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:latest \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--service-account $SA_EMAIL \
--memory 2Gi \
--cpu 2 \
--timeout 600 \
--max-instances 10 \
--min-instances 1 \
--set-env-vars="GOOGLE_CLOUD_PROJECT=nomadkaraoke,GCS_BUCKET_NAME=$BUCKET_NAME,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,LOG_LEVEL=INFO" \
--set-secrets="AUDIOSHAKE_API_KEY=audioshake-api-key:latest,GENIUS_API_KEY=genius-api-key:latest,AUDIO_SEPARATOR_API_URL=audio-separator-api-url:latest"
# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo systemctl status actions.runner.*"
```
## Benefits
## Encoding Worker
✅ **Reproducible**: Run `pulumi up` to create identical infrastructure
✅ **Tracked**: All resources are tracked in Pulumi state
✅ **Versioned**: Infrastructure changes are in git
✅ **Safe**: Preview changes before applying
✅ **Documented**: Code is self-documenting
✅ **Collaborative**: Team can see and modify infrastructure
High-performance VM for video encoding:
- Machine: c4-standard-8 (8 vCPU, 32GB RAM)
- Disk: 100GB hyperdisk-balanced
- Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts
**API Endpoints:**
- `POST /encode` - Full encoding job
- `POST /encode-preview` - Quick 480p preview
- `GET /status/{job_id}` - Job status
- `GET /health` - Health check
**Troubleshooting:**
```bash
gcloud compute ssh encoding-worker --zone=us-central1-a -- \
"sudo systemctl status encoding-worker"
```
## State Management
Pulumi state is stored in Pulumi Cloud by default. This includes:
- Current infrastructure state
- Resource IDs and properties
- Stack outputs
- Configuration values
Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:
```bash
pulumi refresh
```
## Multi-Environment
To create separate environments:
```bash
# Create dev stack
pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up
# Create prod stack
pulumi stack init prod
pulumi config set gcp:project nomadkaraoke
pulumi up
# Switch between stacks
pulumi stack select dev
pulumi stack select prod
pulumi stack select prod # Switch stacks
```
## Troubleshooting
### "Resource already exists"
If resources were created manually, you can import them:
Import manually-created resources:
```bash
pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke
```
### Permissions errors
Ensure your gcloud account has:
- Owner or Editor role on the project
- Or specific roles: Cloud Build Editor, Storage Admin, etc.
### State conflicts
If multiple people are working:
```bash
pulumi refresh # Sync state with actual cloud resources
pulumi refresh # Sync state with cloud
```
## Deploying GDrive Validator
### Permissions errors
Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.
The GDrive Validator is a Cloud Function that runs daily to check for duplicate
sequence numbers, invalid filenames, and gaps in your public Google Drive folder.
## Future Improvements
### Initial Setup
### Phase 5: Custom Machine Images (Planned)
1. Add your Pushbullet API key to Secret Manager:
```bash
echo -n "your-pushbullet-api-key" | \
gcloud secrets versions add pushbullet-api-key --data-file=-
```
The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.
2. Deploy the function:
```bash
cd functions/gdrive_validator
./deploy.sh
```
3. Test the function manually:
```bash
gcloud functions call gdrive-validator --region=us-central1
```
### How It Works
- Runs daily at 9 PM EST (6 PM Pacific) via Cloud Scheduler
- Lists all files in your Google Drive folder (CDG, MP4, MP4-720p subfolders)
- Checks for:
- Duplicate sequence numbers (e.g., two files with NOMAD-1107)
- Invalid filename formats (should be `NOMAD-XXXX - Artist - Title.ext`)
- Sequence gaps (with configurable known gaps)
- Sends Pushbullet notification if issues are found
- Returns JSON with validation results
### Configuration
The function uses environment variables:
- `GDRIVE_FOLDER_ID`: The Google Drive folder ID to validate (default: your public share)
- `PUSHBULLET_API_KEY`: From Secret Manager, for sending notifications
Known gaps are configured in `main.py` and should match your local `validate_and_sync.py`.
## Self-Hosted GitHub Actions Runners
Multiple GCP VMs run self-hosted GitHub Actions runners to:
- Solve disk space issues on GitHub-hosted runners (14GB vs 200GB)
- Enable parallel job execution across multiple runners
### Specs
- **Instances**: 20x e2-standard-4 (4 vCPU, 16GB RAM each)
- **Disk**: 200GB SSD per runner
- **Pre-installed**: Docker, Python 3.13, Node.js 20, Java 21, Poetry, FFmpeg, gcloud CLI
- **Labels**: `self-hosted`, `linux`, `x64`, `gcp`, `large-disk`
### Setup
1. Create a GitHub Personal Access Token (PAT) with `repo` scope:
- Go to https://github.com/settings/tokens
- Generate new token (classic)
- Select `repo` scope (full control of private repositories)
- Copy the token
2. Add the PAT to Secret Manager:
```bash
echo -n "ghp_your_token_here" | \
gcloud secrets versions add github-runner-pat --data-file=-
```
3. Run `pulumi up` to create the runner VMs
4. Verify the runners registered:
- Go to https://github.com/nomadkaraoke/karaoke-gen/settings/actions/runners
- You should see `gcp-runner-github-runner-1` through `gcp-runner-github-runner-20`
### Usage in Workflows
Jobs that need more disk space use the self-hosted runners:
```yaml
jobs:
build:
runs-on: [self-hosted, linux, gcp]
steps:
- uses: actions/checkout@v4
# ... rest of job
```
GitHub automatically distributes jobs across available runners.
### Troubleshooting
**Runner not appearing in GitHub:**
```bash
# Check startup logs (replace N with runner number 1-10)
gcloud compute ssh github-runner-N --zone=us-central1-a -- \
"sudo cat /var/log/github-runner-startup.log"
```
**Re-register runner:**
```bash
# Restart the VM to re-run startup script
gcloud compute instances reset github-runner-N --zone=us-central1-a
```
**Check runner service status:**
```bash
gcloud compute ssh github-runner-N --zone=us-central1-a -- \
"sudo systemctl status actions.runner.*"
```
**List all runners:**
```bash
for i in {1..20}; do
echo "=== github-runner-$i ==="
gcloud compute instances describe github-runner-$i --zone=us-central1-a --format='get(status)'
done
```
## Next Steps
1. Run `pulumi up` to create infrastructure
2. Add secret values (including GitHub runner PAT)
3. Build and deploy Cloud Run service
4. Deploy GDrive validator function
5. Update documentation to reference Pulumi workflow
**See:** [`docs/PHASE5-PACKER-IMAGE-PLAN.md`](docs/PHASE5-PACKER-IMAGE-PLAN.md) for the implementation plan.
Benefits:
- Startup time: 10 min → 30 sec
- Pre-baked: Python 3.13, FFmpeg, fonts
- Hot updates: karaoke-gen wheel still downloaded at runtime
+210 -2144
View File
File diff suppressed because it is too large. Load diff
+26
View File
@@ -0,0 +1,26 @@
"""
Compute resources module.
Contains VM definitions for:
- Flacfetch service (audio download)
- Encoding worker (video encoding)
- GitHub self-hosted runners
"""
from .flacfetch_vm import create_flacfetch_vm, create_flacfetch_ip, create_flacfetch_firewall
from .encoding_worker_vm import (
create_encoding_worker_vm,
create_encoding_worker_ip,
create_encoding_worker_firewall,
)
from .github_runners import create_github_runners
__all__ = [
"create_flacfetch_vm",
"create_flacfetch_ip",
"create_flacfetch_firewall",
"create_encoding_worker_vm",
"create_encoding_worker_ip",
"create_encoding_worker_firewall",
"create_github_runners",
]
@@ -0,0 +1,109 @@
"""
Encoding Worker VM resources.
Manages the high-performance VM instance for video encoding,
including static IP and firewall rules.
"""
import pulumi
from pulumi_gcp import compute, serviceaccount
from config import REGION, ZONE, PROJECT_ID, MachineTypes, DiskSizes
from .startup_scripts import read_script
def create_encoding_worker_ip() -> compute.Address:
"""
Create static IP for the encoding worker.
Returns:
compute.Address: The static IP resource.
"""
return compute.Address(
"encoding-worker-ip",
name="encoding-worker-static-ip",
region=REGION,
address_type="EXTERNAL",
description="Static external IP for encoding worker service",
)
def create_encoding_worker_vm(
ip: compute.Address,
service_account: serviceaccount.Account,
) -> compute.Instance:
"""
Create the encoding worker VM instance.
This VM runs video encoding jobs with high CPU/memory resources.
Uses hyperdisk-balanced for fast I/O during encoding.
Uses a custom Packer-built image with Python 3.13, FFmpeg, and fonts
pre-installed to reduce startup time from ~10 minutes to ~30 seconds.
See infrastructure/packer/README.md for image build instructions.
Args:
ip: The static IP address for the VM.
service_account: The encoding worker service account.
Returns:
compute.Instance: The VM instance resource.
"""
startup_script = read_script("encoding_worker.sh")
# Use custom Packer image with dependencies pre-installed
# Image family returns latest image in the family
# Fallback: "debian-cloud/debian-12" if image doesn't exist yet
custom_image = f"projects/{PROJECT_ID}/global/images/family/encoding-worker"
return compute.Instance(
"encoding-worker",
name="encoding-worker",
machine_type=MachineTypes.ENCODING_WORKER,
zone=ZONE,
boot_disk=compute.InstanceBootDiskArgs(
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
image=custom_image,
size=DiskSizes.ENCODING_WORKER,
type="hyperdisk-balanced",
),
),
network_interfaces=[compute.InstanceNetworkInterfaceArgs(
network="default",
access_configs=[compute.InstanceNetworkInterfaceAccessConfigArgs(
nat_ip=ip.address,
)],
)],
service_account=compute.InstanceServiceAccountArgs(
email=service_account.email,
scopes=["cloud-platform"],
),
metadata_startup_script=startup_script,
tags=["encoding-worker"],
allow_stopping_for_update=True,
advanced_machine_features=compute.InstanceAdvancedMachineFeaturesArgs(
threads_per_core=2,
),
)
def create_encoding_worker_firewall() -> compute.Firewall:
"""
Create firewall rules for the encoding worker.
Opens port 8080 for the HTTP API (authentication required).
Returns:
compute.Firewall: The firewall resource.
"""
return compute.Firewall(
"encoding-worker-firewall",
name="encoding-worker-allow-http",
network="default",
allows=[
compute.FirewallAllowArgs(protocol="tcp", ports=["8080"]),
],
source_ranges=["0.0.0.0/0"],
target_tags=["encoding-worker"],
description="Allow HTTP access to encoding worker (auth required)",
)
+105
View File
@@ -0,0 +1,105 @@
"""
Flacfetch VM resources.
Manages the VM instance for the flacfetch audio download service,
including static IP and firewall rules.
"""
import pulumi
from pulumi_gcp import compute, storage, serviceaccount
from config import REGION, ZONE, MachineTypes, DiskSizes
from .startup_scripts import read_script
def create_flacfetch_ip() -> compute.Address:
"""
Create static IP for flacfetch.
This IP is whitelisted on private trackers, so it must remain stable.
Returns:
compute.Address: The static IP resource.
"""
return compute.Address(
"flacfetch-ip",
name="flacfetch-static-ip",
region=REGION,
address_type="EXTERNAL",
description="Static IP for flacfetch service (torrent downloads)",
)
def create_flacfetch_vm(
ip: compute.Address,
service_account: serviceaccount.Account,
bucket: storage.Bucket,
) -> compute.Instance:
"""
Create the flacfetch VM instance.
Args:
ip: The static IP address for the VM.
service_account: The flacfetch service account.
bucket: The GCS bucket for storing downloaded files.
Returns:
compute.Instance: The VM instance resource.
"""
startup_script = read_script("flacfetch.sh")
return compute.Instance(
"flacfetch-service",
name="flacfetch-service",
machine_type=MachineTypes.FLACFETCH,
zone=ZONE,
boot_disk=compute.InstanceBootDiskArgs(
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
image="debian-cloud/debian-12",
size=DiskSizes.FLACFETCH,
type="pd-ssd",
),
),
network_interfaces=[compute.InstanceNetworkInterfaceArgs(
network="default",
access_configs=[compute.InstanceNetworkInterfaceAccessConfigArgs(
nat_ip=ip.address,
)],
)],
service_account=compute.InstanceServiceAccountArgs(
email=service_account.email,
scopes=["cloud-platform"],
),
metadata_startup_script=startup_script,
metadata={
"gcs-bucket": bucket.name,
},
tags=["flacfetch-service"],
allow_stopping_for_update=True,
)
def create_flacfetch_firewall() -> compute.Firewall:
"""
Create firewall rules for flacfetch.
Opens ports for:
- 51413 TCP/UDP: BitTorrent peer connections
- 8080 TCP: HTTP API
Returns:
compute.Firewall: The firewall resource.
"""
return compute.Firewall(
"flacfetch-firewall",
name="flacfetch-firewall",
network="default",
allows=[
compute.FirewallAllowArgs(protocol="tcp", ports=["51413"]),
compute.FirewallAllowArgs(protocol="udp", ports=["51413"]),
compute.FirewallAllowArgs(protocol="tcp", ports=["8080"]),
],
source_ranges=["0.0.0.0/0"],
target_tags=["flacfetch-service"],
description="Firewall rules for flacfetch torrent/API service",
)
+65
View File
@@ -0,0 +1,65 @@
"""
GitHub Actions self-hosted runner VMs.
Manages multiple VM instances that serve as self-hosted runners
for GitHub Actions CI/CD workflows.
"""
import pulumi
from pulumi_gcp import compute, serviceaccount, secretmanager
from config import ZONE, MachineTypes, DiskSizes, NUM_GITHUB_RUNNERS
from .startup_scripts import read_script
def create_github_runners(
service_account: serviceaccount.Account,
runner_pat_secret: secretmanager.Secret,
) -> list[compute.Instance]:
"""
Create GitHub Actions self-hosted runner VM instances.
Creates NUM_GITHUB_RUNNERS VMs, each configured as a GitHub Actions
self-hosted runner for the karaoke-gen repository.
Args:
service_account: The GitHub runner service account.
runner_pat_secret: The Secret Manager secret containing the GitHub PAT.
Returns:
list[compute.Instance]: List of runner VM instances.
"""
startup_script = read_script("github_runner.sh")
runner_vms = []
for i in range(1, NUM_GITHUB_RUNNERS + 1):
vm = compute.Instance(
f"github-runner-{i}",
name=f"github-runner-{i}",
machine_type=MachineTypes.GITHUB_RUNNER,
zone=ZONE,
boot_disk=compute.InstanceBootDiskArgs(
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
image="debian-cloud/debian-12",
size=DiskSizes.GITHUB_RUNNER,
type="pd-ssd",
),
),
network_interfaces=[compute.InstanceNetworkInterfaceArgs(
network="default",
access_configs=[compute.InstanceNetworkInterfaceAccessConfigArgs()],
)],
service_account=compute.InstanceServiceAccountArgs(
email=service_account.email,
scopes=["cloud-platform"],
),
metadata_startup_script=startup_script,
tags=["github-runner"],
allow_stopping_for_update=True,
opts=pulumi.ResourceOptions(
depends_on=[runner_pat_secret],
),
)
runner_vms.append(vm)
return runner_vms
@@ -0,0 +1,38 @@
"""
Startup scripts for VM instances.
Provides helper functions to load startup scripts from external files,
making them easier to edit, test, and maintain.
"""
from pathlib import Path
def read_script(name: str, **substitutions) -> str:
"""
Read a startup script file and apply substitutions.
Args:
name: Name of the script file (e.g., "flacfetch.sh")
**substitutions: Key-value pairs for template substitution.
Keys should match ${KEY} patterns in the script.
Returns:
str: The script content with substitutions applied.
Example:
script = read_script("github_runner.sh",
RUNNER_VERSION="2.321.0",
RUNNER_LABELS="self-hosted,linux,x64"
)
"""
script_path = Path(__file__).parent / name
content = script_path.read_text()
for key, value in substitutions.items():
content = content.replace(f"${{{key}}}", str(value))
return content
__all__ = ["read_script"]
@@ -0,0 +1,31 @@
#!/bin/bash
# Encoding worker startup script (for Packer-built custom image)
#
# Most dependencies are pre-installed in the custom GCE image:
# - Python 3.13 at /opt/python313
# - FFmpeg at /usr/local/bin/ffmpeg
# - Virtual environment at /opt/encoding-worker/venv
# - Systemd service: encoding-worker.service
#
# This startup script just starts the service.
# The service's ExecStartPre (/opt/encoding-worker/startup.sh) handles:
# - Fetching API key from Secret Manager
# - Installing latest karaoke-gen wheel from GCS
#
# See infrastructure/packer/README.md for image build instructions.
set -e
exec > >(tee /var/log/encoding-worker-startup.log) 2>&1
echo "=== Encoding worker VM startup at $(date) ==="
# Start Docker (in case it's not running)
systemctl start docker || echo "Docker already running or not available"
# Start the encoding worker service
# This triggers ExecStartPre which fetches API key and installs wheel
echo "Starting encoding-worker service..."
systemctl start encoding-worker
echo "=== Encoding worker started at $(date) ==="
echo "Service listening on port 8080"
@@ -0,0 +1,119 @@
#!/bin/bash
set -e
# Log to a file for debugging
exec > >(tee /var/log/flacfetch-startup.log) 2>&1
echo "Starting flacfetch setup at $(date)"
# Install dependencies
apt-get update
apt-get install -y python3-pip python3-venv transmission-daemon ffmpeg git curl
# Configure Transmission
echo "Configuring Transmission daemon..."
systemctl stop transmission-daemon || true
# Create transmission directories
mkdir -p /var/lib/transmission-daemon/downloads
mkdir -p /var/lib/transmission-daemon/.config/transmission-daemon
chown -R debian-transmission:debian-transmission /var/lib/transmission-daemon
# Configure transmission settings
cat > /var/lib/transmission-daemon/.config/transmission-daemon/settings.json << 'SETTINGS'
{
"download-dir": "/var/lib/transmission-daemon/downloads",
"incomplete-dir": "/var/lib/transmission-daemon/.incomplete",
"incomplete-dir-enabled": true,
"rpc-authentication-required": false,
"rpc-bind-address": "127.0.0.1",
"rpc-enabled": true,
"rpc-port": 9091,
"rpc-whitelist-enabled": false,
"peer-port": 51413,
"port-forwarding-enabled": false,
"speed-limit-down": 0,
"speed-limit-down-enabled": false,
"speed-limit-up": 0,
"speed-limit-up-enabled": false,
"ratio-limit-enabled": false,
"umask": 2,
"encryption": 1
}
SETTINGS
chown debian-transmission:debian-transmission /var/lib/transmission-daemon/.config/transmission-daemon/settings.json
systemctl start transmission-daemon
echo "Transmission daemon started"
# Install flacfetch
echo "Installing flacfetch..."
cd /opt
# Clone flacfetch (we'll use the repo from GitHub)
if [ ! -d "flacfetch" ]; then
git clone https://github.com/nomadkaraoke/flacfetch.git
fi
cd flacfetch
# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate
# Install flacfetch with API dependencies
pip install -e ".[api]"
# Get secrets from Secret Manager
echo "Fetching secrets from Secret Manager..."
FLACFETCH_API_KEY=$(gcloud secrets versions access latest --secret=flacfetch-api-key 2>/dev/null || echo "")
RED_API_KEY=$(gcloud secrets versions access latest --secret=red-api-key 2>/dev/null || echo "")
RED_API_URL=$(gcloud secrets versions access latest --secret=red-api-url 2>/dev/null || echo "")
OPS_API_KEY=$(gcloud secrets versions access latest --secret=ops-api-key 2>/dev/null || echo "")
OPS_API_URL=$(gcloud secrets versions access latest --secret=ops-api-url 2>/dev/null || echo "")
# Get bucket name from project metadata
GCS_BUCKET=$(gcloud compute project-info describe --format='value(commonInstanceMetadata.items.gcs-bucket)' 2>/dev/null || echo "")
if [ -z "$GCS_BUCKET" ]; then
# Fallback: construct from project ID
PROJECT_ID=$(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google")
GCS_BUCKET="karaoke-gen-storage-${PROJECT_ID}"
fi
echo "Using GCS bucket: $GCS_BUCKET"
# Create systemd service for flacfetch
echo "Creating systemd service..."
cat > /etc/systemd/system/flacfetch.service << SYSTEMD
[Unit]
Description=Flacfetch HTTP API Service
After=network.target transmission-daemon.service
Requires=transmission-daemon.service
[Service]
Type=simple
User=root
WorkingDirectory=/opt/flacfetch
Environment="FLACFETCH_API_KEY=${FLACFETCH_API_KEY}"
Environment="RED_API_KEY=${RED_API_KEY}"
Environment="RED_API_URL=${RED_API_URL}"
Environment="OPS_API_KEY=${OPS_API_KEY}"
Environment="OPS_API_URL=${OPS_API_URL}"
Environment="GCS_BUCKET=${GCS_BUCKET}"
Environment="FLACFETCH_KEEP_SEEDING=true"
Environment="FLACFETCH_MIN_FREE_GB=5"
Environment="FLACFETCH_DOWNLOAD_DIR=/var/lib/transmission-daemon/downloads"
Environment="TRANSMISSION_HOST=localhost"
Environment="TRANSMISSION_PORT=9091"
ExecStart=/opt/flacfetch/venv/bin/flacfetch serve --host 0.0.0.0 --port 8080
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
SYSTEMD
systemctl daemon-reload
systemctl enable flacfetch
systemctl start flacfetch
echo "Flacfetch service started at $(date)"
echo "Setup complete!"
@@ -0,0 +1,261 @@
#!/bin/bash
set -e
# Log to a file for debugging
exec > >(tee /var/log/github-runner-startup.log) 2>&1
echo "Starting GitHub Actions runner setup at $(date)"
# System updates and core dependencies
apt-get update
apt-get install -y \
curl \
git \
jq \
unzip \
build-essential \
libssl-dev \
libffi-dev \
software-properties-common \
apt-transport-https \
ca-certificates \
gnupg \
lsb-release
# ==================== Docker ====================
echo "Installing Docker..."
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --batch --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Configure Docker for the runner user
systemctl enable docker
systemctl start docker
# ==================== Python 3.13 via pyenv ====================
# setup-python action on self-hosted runners needs Python in the tool cache
echo "Installing Python 3.13 via pyenv..."
# Install pyenv dependencies
apt-get install -y make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
# Install pyenv system-wide
export PYENV_ROOT=/opt/pyenv
curl https://pyenv.run | bash
# Add to system profile for future sessions
cat > /etc/profile.d/pyenv.sh << 'PYENV_PROFILE'
export PYENV_ROOT=/opt/pyenv
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
PYENV_PROFILE
# Source pyenv for current session
export PATH="/opt/pyenv/bin:$PATH"
eval "$(/opt/pyenv/bin/pyenv init -)"
# Install Python 3.13
/opt/pyenv/bin/pyenv install 3.13.0
/opt/pyenv/bin/pyenv global 3.13.0
# Create symlinks for system-wide access
ln -sf /opt/pyenv/shims/python3.13 /usr/local/bin/python3.13
ln -sf /opt/pyenv/shims/python3 /usr/local/bin/python3
ln -sf /opt/pyenv/shims/pip3 /usr/local/bin/pip3
# ==================== Node.js 20 ====================
echo "Installing Node.js 20..."
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
# ==================== Java 21 (for Firestore emulator) ====================
echo "Installing Java 21 via Temurin..."
# Use Temurin (Adoptium) - more reliable than Debian repos
curl -fsSL https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --batch --dearmor -o /usr/share/keyrings/adoptium.gpg
echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb $(lsb_release -cs) main" > /etc/apt/sources.list.d/adoptium.list
apt-get update
apt-get install -y temurin-21-jdk
# ==================== FFmpeg ====================
echo "Installing FFmpeg..."
apt-get install -y ffmpeg
# ==================== Poetry ====================
echo "Installing Poetry..."
curl -sSL https://install.python-poetry.org | python3 -
ln -sf /root/.local/bin/poetry /usr/local/bin/poetry
# ==================== Google Cloud SDK ====================
echo "Installing Google Cloud SDK..."
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --batch --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update
apt-get install -y google-cloud-cli google-cloud-cli-firestore-emulator
# ==================== Create runner user ====================
echo "Creating runner user..."
useradd -m -s /bin/bash runner || true
usermod -aG docker runner
# ==================== GitHub Actions Runner ====================
echo "Setting up GitHub Actions Runner..."
RUNNER_VERSION="2.321.0"
RUNNER_DIR="/home/runner/actions-runner"
mkdir -p $RUNNER_DIR
cd $RUNNER_DIR
# Download runner
curl -o actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz -L \
https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
tar xzf actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
rm actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
chown -R runner:runner $RUNNER_DIR
# Get GitHub PAT from Secret Manager
echo "Fetching GitHub PAT from Secret Manager..."
GITHUB_PAT=$(gcloud secrets versions access latest --secret=github-runner-pat)
if [ -z "$GITHUB_PAT" ]; then
echo "ERROR: GitHub PAT not found in Secret Manager"
exit 1
fi
# Get runner registration token using PAT
echo "Getting runner registration token..."
REGISTRATION_TOKEN=$(curl -s -X POST \
-H "Authorization: token $GITHUB_PAT" \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/nomadkaraoke/karaoke-gen/actions/runners/registration-token | jq -r .token)
if [ -z "$REGISTRATION_TOKEN" ] || [ "$REGISTRATION_TOKEN" = "null" ]; then
echo "ERROR: Failed to get registration token"
exit 1
fi
# Configure runner (non-interactive)
echo "Configuring runner..."
cd $RUNNER_DIR
sudo -u runner ./config.sh \
--url https://github.com/nomadkaraoke/karaoke-gen \
--token "$REGISTRATION_TOKEN" \
--name "gcp-runner-$(hostname)" \
--labels "self-hosted,linux,x64,gcp,large-disk" \
--work "_work" \
--unattended \
--replace
# Install and start runner as a service
echo "Installing runner service..."
./svc.sh install runner
./svc.sh start
echo "GitHub Actions runner setup complete at $(date)"
echo "Runner registered with labels: self-hosted,linux,x64,gcp,large-disk"
# ==================== Setup Python in tool cache ====================
# setup-python action looks for Python in RUNNER_TOOL_CACHE/_tool/Python
echo "Setting up Python 3.13 in tool cache for setup-python action..."
TOOL_CACHE="/home/runner/actions-runner/_work/_tool"
mkdir -p $TOOL_CACHE/Python/3.13.0/x64
# Copy pyenv Python to tool cache
cp -r /opt/pyenv/versions/3.13.0/* $TOOL_CACHE/Python/3.13.0/x64/
# Create marker file that setup-python looks for
touch $TOOL_CACHE/Python/3.13.0/x64.complete
# Fix permissions - must own entire _work directory, not just _tool
chown -R runner:runner /home/runner/actions-runner/_work
echo "Python 3.13 added to tool cache"
# ==================== Docker cleanup cron ====================
# Prevent disk from filling up with old Docker images
# Run hourly instead of daily, with threshold-based cleanup
echo "Setting up Docker cleanup cron..."
cat > /etc/cron.hourly/docker-cleanup << 'CRON'
#!/bin/bash
# Docker disk space cleanup for GitHub Actions runners
# Runs hourly, with aggressive cleanup when disk usage exceeds threshold
# Log to syslog
exec 1> >(logger -t docker-cleanup) 2>&1
THRESHOLD=70 # Cleanup when disk usage exceeds this percentage
# Get current disk usage percentage (remove % sign)
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Current disk usage: ${USAGE}%"
if [ "$USAGE" -gt "$THRESHOLD" ]; then
echo "Disk usage ${USAGE}% exceeds threshold ${THRESHOLD}%, running cleanup..."
# Remove ALL unused images (not just old ones) - this is the key fix
# The previous 168h filter let recent images accumulate
docker system prune -af
# Also clean builder cache which can grow large
docker builder prune -af
# Report new usage
NEW_USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "Cleanup complete. Disk usage: ${NEW_USAGE}%"
else
echo "Disk usage ${USAGE}% is below threshold ${THRESHOLD}%, skipping cleanup"
fi
CRON
chmod +x /etc/cron.hourly/docker-cleanup
# Remove old daily cleanup if it exists (migrating to hourly)
rm -f /etc/cron.daily/docker-cleanup
# ==================== Docker pre-warm cron ====================
# Keep base images fresh by pulling hourly
echo "Setting up Docker pre-warm cron..."
cat > /etc/cron.hourly/docker-prewarm << 'CRON'
#!/bin/bash
# Pre-warm Docker images used by CI/CD
# This ensures runners always have the latest base images cached
# Log to syslog
exec 1> >(logger -t docker-prewarm) 2>&1
echo "Starting Docker image pre-warm..."
# Configure Docker auth (uses instance service account)
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet 2>/dev/null
# Pull latest images (|| true to not fail if network issues)
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest || true
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:latest || true
echo "Docker image pre-warm complete"
CRON
chmod +x /etc/cron.hourly/docker-prewarm
# ==================== Pre-warm Docker images ====================
# Pre-fetch commonly used Docker images to speed up first CI runs
echo "Pre-warming Docker images for faster CI builds..."
# Configure Docker to authenticate with Artifact Registry
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
# Pull base images used by deploy-backend job (largest layers, most benefit from caching)
echo "Pulling karaoke-backend-base image..."
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest || true
echo "Pulling karaoke-backend image..."
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:latest || true
# Pull GCS emulator image used by backend-emulator-tests job
echo "Pulling fake-gcs-server image..."
docker pull fsouza/fake-gcs-server:latest || true
echo "Docker images pre-warmed"
echo "Setup complete!"
+93
View File
@@ -0,0 +1,93 @@
"""
Centralized configuration for Pulumi infrastructure.
This module provides shared constants and configuration used across all infrastructure modules.
"""
import pulumi
import pulumi_gcp as gcp
# GCP project info - lazily loaded to work with Pulumi runtime
_project = None
def _get_project():
"""Get GCP project info (lazy-loaded for Pulumi runtime compatibility)."""
global _project
if _project is None:
_project = gcp.organizations.get_project()
return _project
# Project ID and number are accessed as properties
# These will resolve correctly during Pulumi execution
PROJECT_ID = "nomadkaraoke" # Hardcoded for now - matches existing behavior
PROJECT_NUMBER = None # Set lazily when needed
def get_project_number() -> str:
"""Get the GCP project number."""
return _get_project().number
# Region configuration
REGION = "us-central1"
ZONE = f"{REGION}-a"
# Pulumi config accessor
config = pulumi.Config()
class MachineTypes:
"""Machine type configurations for GCE instances."""
GITHUB_RUNNER = "e2-standard-4" # 4 vCPU, 16GB RAM
ENCODING_WORKER = "c4-standard-8" # 8 vCPU, Intel Granite Rapids 3.9 GHz
FLACFETCH = "e2-small" # 0.5 vCPU, 2GB RAM
class DiskSizes:
"""Disk size configurations in GB."""
GITHUB_RUNNER = 200 # Large for Docker builds/caches
ENCODING_WORKER = 100 # For temp video files during encoding
FLACFETCH = 30 # For torrent storage
class QueueConfigs:
"""Cloud Tasks queue rate limiting configurations."""
class Audio:
MAX_DISPATCHES_PER_SECOND = 10
MAX_CONCURRENT_DISPATCHES = 50
MAX_RETRY_DURATION = "1800s" # 30 min
class Lyrics:
MAX_DISPATCHES_PER_SECOND = 10
MAX_CONCURRENT_DISPATCHES = 50
MAX_RETRY_DURATION = "1200s" # 20 min
class Screens:
MAX_DISPATCHES_PER_SECOND = 50
MAX_CONCURRENT_DISPATCHES = 100
MAX_RETRY_DURATION = "300s" # 5 min
class Render:
MAX_DISPATCHES_PER_SECOND = 5
MAX_CONCURRENT_DISPATCHES = 20
MAX_RETRY_DURATION = "3600s" # 60 min
class Video:
MAX_DISPATCHES_PER_SECOND = 3
MAX_CONCURRENT_DISPATCHES = 10
MAX_RETRY_DURATION = "7200s" # 2 hours
# Number of self-hosted GitHub Action runners
NUM_GITHUB_RUNNERS = 20
# Google Drive folder ID for validator (public share folder)
GDRIVE_FOLDER_ID = "1laRKAyxo0v817SstfM5XkpbWiNKNAMSX"
# GitHub repository for runner registration
GITHUB_REPO_OWNER = "nomadkaraoke"
GITHUB_REPO_NAME = "karaoke-gen"
@@ -0,0 +1,370 @@
# Phase 5: Encoding Worker Custom Machine Image
## Problem Statement
The encoding worker VM currently takes **~10 minutes** to become ready after startup because the startup script:
1. Builds Python 3.13 from source (`./configure && make -j$(nproc)`) - ~7 minutes
2. Downloads and installs FFmpeg static build - ~1 minute
3. Installs system packages and fonts - ~1 minute
4. Creates Python venv and installs dependencies - ~1 minute
This slow startup affects:
- **VM restarts**: Any maintenance event or restart requires waiting 10 minutes
- **Scaling**: Can't quickly add capacity if needed
- **Development**: Testing infrastructure changes is slow
## Solution: Pre-baked Machine Image with Packer
Create a custom GCE machine image using [Packer](https://www.packer.io/) that pre-installs all dependencies. The startup script then only needs to:
1. Fetch API key from Secret Manager (~5 seconds)
2. Download latest karaoke-gen wheel from GCS (~10 seconds)
3. Start the systemd service (~5 seconds)
**Expected startup time: ~30 seconds** (vs current ~10 minutes)
## Implementation Plan
### 1. Create Packer Directory Structure
```
infrastructure/packer/
├── encoding-worker.pkr.hcl # Packer template
├── scripts/
│ └── provision.sh # Provisioning script
└── README.md # Usage documentation
```
### 2. Packer Template (`encoding-worker.pkr.hcl`)
```hcl
packer {
required_plugins {
googlecompute = {
source = "github.com/hashicorp/googlecompute"
version = "~> 1.1"
}
}
}
variable "project_id" {
type = string
default = "nomadkaraoke"
}
variable "zone" {
type = string
default = "us-central1-a"
}
variable "python_version" {
type = string
default = "3.13.1"
}
source "googlecompute" "encoding-worker" {
project_id = var.project_id
zone = var.zone
source_image_family = "debian-12"
source_image_project_id = "debian-cloud"
# Build on same machine type as production for consistency
machine_type = "c4-standard-8"
disk_size = 100
disk_type = "pd-ssd" # Use SSD for faster builds
image_name = "encoding-worker-{{timestamp}}"
image_family = "encoding-worker"
image_description = "Encoding worker with Python ${var.python_version}, FFmpeg, and fonts pre-installed"
ssh_username = "packer"
# Tags for firewall rules during build
tags = ["packer-build"]
}
build {
sources = ["source.googlecompute.encoding-worker"]
# Upload provisioning script
provisioner "file" {
source = "scripts/provision.sh"
destination = "/tmp/provision.sh"
}
# Run provisioning
provisioner "shell" {
inline = [
"chmod +x /tmp/provision.sh",
"sudo PYTHON_VERSION=${var.python_version} /tmp/provision.sh"
]
}
# Clean up for smaller image
provisioner "shell" {
inline = [
"sudo apt-get clean",
"sudo rm -rf /var/lib/apt/lists/*",
"sudo rm -rf /tmp/*",
"sudo rm -rf /var/tmp/*"
]
}
}
```
### 3. Provisioning Script (`scripts/provision.sh`)
```bash
#!/bin/bash
# Provision encoding worker image with all dependencies pre-installed
set -e
echo "=== Starting encoding worker image provisioning ==="
# Install system packages
apt-get update
apt-get install -y \
docker.io \
curl \
git \
xz-utils \
build-essential \
libssl-dev \
zlib1g-dev \
libbz2-dev \
libreadline-dev \
libsqlite3-dev \
libncursesw5-dev \
tk-dev \
libxml2-dev \
libxmlsec1-dev \
libffi-dev \
liblzma-dev
# Install fonts for ASS subtitle rendering
apt-get install -y fonts-noto fonts-noto-cjk fontconfig
fc-cache -fv
# Enable Docker
systemctl enable docker
# Install optimized static FFmpeg build (John Van Sickle)
echo "Installing FFmpeg..."
curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz
tar -xf /tmp/ffmpeg.tar.xz -C /tmp
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe
rm -rf /tmp/ffmpeg*
ffmpeg -version
# Build and install Python from source
echo "Building Python ${PYTHON_VERSION}..."
cd /tmp
curl -L "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tar.xz" -o python.tar.xz
tar -xf python.tar.xz
cd "Python-${PYTHON_VERSION}"
./configure --prefix=/opt/python313 --enable-optimizations --with-lto
make -j$(nproc)
make install
rm -rf /tmp/python.tar.xz /tmp/Python-${PYTHON_VERSION}
/opt/python313/bin/python3.13 --version
echo "Python ${PYTHON_VERSION} installed at /opt/python313"
# Create working directory and venv
mkdir -p /opt/encoding-worker
cd /opt/encoding-worker
/opt/python313/bin/python3.13 -m venv venv
# Install base Python dependencies (wheel will add more at runtime)
source venv/bin/activate
pip install --upgrade pip
pip install fastapi uvicorn google-cloud-storage aiofiles aiohttp
# Create systemd service template (API key filled at runtime)
cat > /etc/systemd/system/encoding-worker.service << 'SYSTEMD'
[Unit]
Description=Encoding Worker HTTP API Service
After=network.target
[Service]
Type=simple
User=root
WorkingDirectory=/opt/encoding-worker
ExecStartPre=/opt/encoding-worker/startup.sh
ExecStart=/opt/encoding-worker/venv/bin/uvicorn backend.services.gce_encoding.main:app --host 0.0.0.0 --port 8080
Restart=always
RestartSec=10
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
EnvironmentFile=/opt/encoding-worker/env
[Install]
WantedBy=multi-user.target
SYSTEMD
# Create startup helper script (runs before service starts)
cat > /opt/encoding-worker/startup.sh << 'STARTUP'
#!/bin/bash
set -e
# Fetch API key from Secret Manager
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > /opt/encoding-worker/env
# Download and install latest karaoke-gen wheel
gsutil cp "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl" /tmp/ 2>/dev/null || true
WHEEL_FILE=$(ls -t /tmp/karaoke_gen-*.whl 2>/dev/null | head -1)
if [ -n "$WHEEL_FILE" ]; then
/opt/encoding-worker/venv/bin/pip install --upgrade --quiet "$WHEEL_FILE"
fi
STARTUP
chmod +x /opt/encoding-worker/startup.sh
# Enable service (will start on boot)
systemctl daemon-reload
systemctl enable encoding-worker
echo "=== Provisioning complete ==="
```
### 4. Update Pulumi to Use Custom Image
In `infrastructure/compute/encoding_worker_vm.py`, change:
```python
# Before (current)
boot_disk=compute.InstanceBootDiskArgs(
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
image="debian-cloud/debian-12",
size=DiskSizes.ENCODING_WORKER,
type="hyperdisk-balanced",
),
),
# After (with custom image)
boot_disk=compute.InstanceBootDiskArgs(
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
image=f"projects/{PROJECT_ID}/global/images/family/encoding-worker",
size=DiskSizes.ENCODING_WORKER,
type="hyperdisk-balanced",
),
),
```
### 5. Simplified Startup Script
Replace the 700-line `encoding_worker.sh` with a minimal script:
```bash
#!/bin/bash
# Encoding worker startup script (for custom image)
# Most dependencies are pre-installed in the image.
# This script just starts the service.
set -e
exec > >(tee /var/log/encoding-worker-startup.log) 2>&1
echo "Starting encoding worker at $(date)"
# Start Docker (in case it's not running)
systemctl start docker
# Start the encoding worker service
# The service's ExecStartPre handles:
# - Fetching API key from Secret Manager
# - Installing latest karaoke-gen wheel
systemctl start encoding-worker
echo "Encoding worker started at $(date)"
```
### 6. CI Integration (Optional)
Add a GitHub Actions workflow to rebuild the image when Packer files change:
```yaml
# .github/workflows/build-encoding-worker-image.yml
name: Build Encoding Worker Image
on:
push:
branches: [main]
paths:
- 'infrastructure/packer/**'
workflow_dispatch: # Manual trigger
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- name: Authenticate to Google Cloud
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
service_account: github-actions-deployer@nomadkaraoke.iam.gserviceaccount.com
- name: Setup Packer
uses: hashicorp/setup-packer@main
with:
version: latest
- name: Build Image
working-directory: infrastructure/packer
run: |
packer init encoding-worker.pkr.hcl
packer build encoding-worker.pkr.hcl
```
## Execution Steps
1. **Create the Packer files** in `infrastructure/packer/`
2. **Build the image locally** (one-time, or via CI):
```bash
cd infrastructure/packer
packer init encoding-worker.pkr.hcl
packer build encoding-worker.pkr.hcl
```
3. **Update Pulumi** to use the new image family
4. **Simplify the startup script** to just start the service
5. **Run `pulumi up`** to deploy with the new image
6. **Verify** the VM starts in ~30 seconds instead of ~10 minutes
## Rollback Plan
If issues occur:
1. Change the image back to `debian-cloud/debian-12` in Pulumi
2. Restore the full startup script
3. Run `pulumi up`
The old startup script is preserved in git history and can be restored.
## Benefits
| Metric | Before | After |
|--------|--------|-------|
| Startup time | ~10 min | ~30 sec |
| Python build | Every start | Never (pre-baked) |
| FFmpeg install | Every start | Never (pre-baked) |
| Image rebuild needed | N/A | Only when deps change |
## Maintenance
Rebuild the image when:
- Python version needs updating
- FFmpeg version needs updating
- New system packages required
- Font packages change
The karaoke-gen wheel is still downloaded at runtime, so application code updates don't require image rebuilds.
+25
View File
@@ -0,0 +1,25 @@
"""
Infrastructure modules for Pulumi resources.
Each module is responsible for a specific category of cloud resources.
"""
from . import database
from . import storage
from . import artifact_registry
from . import secrets
from . import cloud_tasks
from . import cloud_run
from . import monitoring
from . import iam
__all__ = [
"database",
"storage",
"artifact_registry",
"secrets",
"cloud_tasks",
"cloud_run",
"monitoring",
"iam",
]
@@ -0,0 +1,42 @@
"""
Artifact Registry resources.
Manages the Docker repository for backend container images.
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import artifactregistry
from config import PROJECT_ID, REGION
def create_repository() -> artifactregistry.Repository:
"""
Create the Artifact Registry repository for Docker images.
Returns:
artifactregistry.Repository: The created repository resource.
"""
artifact_repo = artifactregistry.Repository(
"karaoke-artifact-repo",
repository_id="karaoke-repo",
location=REGION,
format="DOCKER",
description="Docker repository for karaoke backend images",
)
return artifact_repo
def get_repo_url(repo: artifactregistry.Repository) -> pulumi.Output[str]:
"""
Get the full URL for the repository.
Args:
repo: The artifact registry repository.
Returns:
pulumi.Output[str]: The repository URL (e.g., us-central1-docker.pkg.dev/project/repo)
"""
return repo.name.apply(lambda name: f"{REGION}-docker.pkg.dev/{PROJECT_ID}/karaoke-repo")
+88
View File
@@ -0,0 +1,88 @@
"""
Cloud Run resources.
Manages Cloud Run domain mapping and Cloud Run Jobs.
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import cloudrun, cloudrunv2
from config import PROJECT_ID, REGION
def create_domain_mapping() -> cloudrun.DomainMapping:
"""
Create the Cloud Run domain mapping for api.nomadkaraoke.com.
Returns:
cloudrun.DomainMapping: The domain mapping resource.
"""
domain_mapping = cloudrun.DomainMapping(
"karaoke-backend-domain",
location=REGION,
name="api.nomadkaraoke.com",
metadata=cloudrun.DomainMappingMetadataArgs(
namespace=PROJECT_ID,
),
spec=cloudrun.DomainMappingSpecArgs(
route_name="karaoke-backend",
),
)
return domain_mapping
def create_video_encoding_job(
bucket: gcp.storage.Bucket,
service_account: gcp.serviceaccount.Account,
) -> cloudrunv2.Job:
"""
Create the Cloud Run Job for video encoding.
Cloud Run Jobs provide up to 24-hour execution time for long-running video encoding.
This is used when video encoding might exceed the 30-minute Cloud Tasks timeout.
Args:
bucket: The GCS bucket for job artifacts.
service_account: The service account to run the job.
Returns:
cloudrunv2.Job: The Cloud Run Job resource.
"""
video_encoding_job = cloudrunv2.Job(
"video-encoding-job",
name="video-encoding-job",
location=REGION,
template=cloudrunv2.JobTemplateArgs(
template=cloudrunv2.JobTemplateTemplateArgs(
containers=[
cloudrunv2.JobTemplateTemplateContainerArgs(
image=f"{REGION}-docker.pkg.dev/{PROJECT_ID}/karaoke-repo/karaoke-backend:latest",
args=["python", "-m", "backend.workers.video_worker"],
resources=cloudrunv2.JobTemplateTemplateContainerResourcesArgs(
limits={
"cpu": "4",
"memory": "8Gi",
},
),
envs=[
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
name="GOOGLE_CLOUD_PROJECT",
value=PROJECT_ID,
),
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
name="GCS_BUCKET_NAME",
value=bucket.name,
),
],
)
],
service_account=service_account.email,
timeout="3600s", # 1 hour max per task
max_retries=2,
),
),
)
return video_encoding_job
+133
View File
@@ -0,0 +1,133 @@
"""
Cloud Tasks resources.
Manages task queues for asynchronous worker processing.
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import cloudtasks
from config import REGION, QueueConfigs
def create_queues() -> dict[str, cloudtasks.Queue]:
"""
Create all Cloud Tasks queues.
These queues provide guaranteed delivery and horizontal scaling for worker tasks.
Workers are triggered via Cloud Tasks instead of BackgroundTasks, ensuring:
- Tasks survive container restarts
- Each task gets dedicated resources
- Automatic retries on failure
- Rate limiting to protect external APIs
Returns:
dict: Dictionary mapping queue names to Queue resources.
"""
queues = {}
# Audio worker queue - calls Modal API for audio separation
queues["audio-worker-queue"] = cloudtasks.Queue(
"audio-worker-queue",
name="audio-worker-queue",
location=REGION,
rate_limits=cloudtasks.QueueRateLimitsArgs(
max_dispatches_per_second=QueueConfigs.Audio.MAX_DISPATCHES_PER_SECOND,
max_concurrent_dispatches=QueueConfigs.Audio.MAX_CONCURRENT_DISPATCHES,
),
retry_config=cloudtasks.QueueRetryConfigArgs(
max_attempts=3,
min_backoff="10s",
max_backoff="300s",
max_retry_duration=QueueConfigs.Audio.MAX_RETRY_DURATION,
),
)
# Lyrics worker queue - calls AudioShake API for transcription
queues["lyrics-worker-queue"] = cloudtasks.Queue(
"lyrics-worker-queue",
name="lyrics-worker-queue",
location=REGION,
rate_limits=cloudtasks.QueueRateLimitsArgs(
max_dispatches_per_second=QueueConfigs.Lyrics.MAX_DISPATCHES_PER_SECOND,
max_concurrent_dispatches=QueueConfigs.Lyrics.MAX_CONCURRENT_DISPATCHES,
),
retry_config=cloudtasks.QueueRetryConfigArgs(
max_attempts=3,
min_backoff="10s",
max_backoff="300s",
max_retry_duration=QueueConfigs.Lyrics.MAX_RETRY_DURATION,
),
)
# Screens worker queue - generates title/end screens (fast, CPU-light)
queues["screens-worker-queue"] = cloudtasks.Queue(
"screens-worker-queue",
name="screens-worker-queue",
location=REGION,
rate_limits=cloudtasks.QueueRateLimitsArgs(
max_dispatches_per_second=QueueConfigs.Screens.MAX_DISPATCHES_PER_SECOND,
max_concurrent_dispatches=QueueConfigs.Screens.MAX_CONCURRENT_DISPATCHES,
),
retry_config=cloudtasks.QueueRetryConfigArgs(
max_attempts=3,
min_backoff="5s",
max_backoff="60s",
max_retry_duration=QueueConfigs.Screens.MAX_RETRY_DURATION,
),
)
# Render worker queue - LyricsTranscriber + FFmpeg (CPU-intensive)
queues["render-worker-queue"] = cloudtasks.Queue(
"render-worker-queue",
name="render-worker-queue",
location=REGION,
rate_limits=cloudtasks.QueueRateLimitsArgs(
max_dispatches_per_second=QueueConfigs.Render.MAX_DISPATCHES_PER_SECOND,
max_concurrent_dispatches=QueueConfigs.Render.MAX_CONCURRENT_DISPATCHES,
),
retry_config=cloudtasks.QueueRetryConfigArgs(
max_attempts=2,
min_backoff="30s",
max_backoff="300s",
max_retry_duration=QueueConfigs.Render.MAX_RETRY_DURATION,
),
)
# Video worker queue - final encoding (very CPU-intensive, longest running)
queues["video-worker-queue"] = cloudtasks.Queue(
"video-worker-queue",
name="video-worker-queue",
location=REGION,
rate_limits=cloudtasks.QueueRateLimitsArgs(
max_dispatches_per_second=QueueConfigs.Video.MAX_DISPATCHES_PER_SECOND,
max_concurrent_dispatches=QueueConfigs.Video.MAX_CONCURRENT_DISPATCHES,
),
retry_config=cloudtasks.QueueRetryConfigArgs(
max_attempts=2,
min_backoff="60s",
max_backoff="600s",
max_retry_duration=QueueConfigs.Video.MAX_RETRY_DURATION,
),
)
# Idle reminder queue - delayed tasks for sending reminder emails
# Tasks are scheduled with a 5-minute delay when jobs enter blocking states
queues["idle-reminder-queue"] = cloudtasks.Queue(
"idle-reminder-queue",
name="idle-reminder-queue",
location=REGION,
rate_limits=cloudtasks.QueueRateLimitsArgs(
max_dispatches_per_second=10, # Email sending is fast
max_concurrent_dispatches=50,
),
retry_config=cloudtasks.QueueRetryConfigArgs(
max_attempts=3,
min_backoff="10s",
max_backoff="60s",
max_retry_duration="600s", # 10 min total
),
)
return queues
+173
View File
@@ -0,0 +1,173 @@
"""
Firestore database resources.
Manages the Firestore database, composite indexes, and TTL configuration.
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import firestore
from config import PROJECT_ID
def create_database() -> dict:
"""
Create Firestore database and all composite indexes.
Returns:
dict: Dictionary containing all created resources for export.
"""
resources = {}
# ==================== Firestore Database ====================
firestore_db = firestore.Database(
"karaoke-firestore",
project=PROJECT_ID,
name="(default)",
location_id="us-central1",
type="FIRESTORE_NATIVE",
concurrency_mode="PESSIMISTIC",
app_engine_integration_mode="DISABLED",
)
resources["firestore_db"] = firestore_db
# ==================== Firestore Composite Indexes ====================
# These are required for queries that filter/sort on multiple fields.
# Jobs: query by user_email, order by created_at (for user job lists)
resources["firestore_index_jobs_user_email"] = firestore.Index(
"firestore-index-jobs-user-email",
project=PROJECT_ID,
database=firestore_db.name,
collection="jobs",
fields=[
firestore.IndexFieldArgs(field_path="user_email", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Jobs: query by status, order by created_at (for admin/system job lists)
resources["firestore_index_jobs_status"] = firestore.Index(
"firestore-index-jobs-status",
project=PROJECT_ID,
database=firestore_db.name,
collection="jobs",
fields=[
firestore.IndexFieldArgs(field_path="status", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Sessions: query by user_email + is_active, order by created_at
resources["firestore_index_sessions_active"] = firestore.Index(
"firestore-index-sessions-active",
project=PROJECT_ID,
database=firestore_db.name,
collection="sessions",
fields=[
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="user_email", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Magic links: query by email, order by created_at
resources["firestore_index_magic_links"] = firestore.Index(
"firestore-index-magic-links",
project=PROJECT_ID,
database=firestore_db.name,
collection="magic_links",
fields=[
firestore.IndexFieldArgs(field_path="email", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Gen Users: query by is_active, order by created_at
resources["firestore_index_gen_users_active_created"] = firestore.Index(
"firestore-index-gen-users-active-created",
project=PROJECT_ID,
database=firestore_db.name,
collection="gen_users",
fields=[
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Gen Users: query by is_active, order by last_login_at
resources["firestore_index_gen_users_active_login"] = firestore.Index(
"firestore-index-gen-users-active-login",
project=PROJECT_ID,
database=firestore_db.name,
collection="gen_users",
fields=[
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="last_login_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Gen Users: query by is_active, order by credits
resources["firestore_index_gen_users_active_credits"] = firestore.Index(
"firestore-index-gen-users-active-credits",
project=PROJECT_ID,
database=firestore_db.name,
collection="gen_users",
fields=[
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="credits", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Gen Users: query by is_active, order by email
resources["firestore_index_gen_users_active_email"] = firestore.Index(
"firestore-index-gen-users-active-email",
project=PROJECT_ID,
database=firestore_db.name,
collection="gen_users",
fields=[
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="email", order="ASCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# ==================== Firestore TTL Field Configuration ====================
# TTL policies automatically delete documents after expiration.
# Worker logs TTL: logs subcollection documents expire after 30 days
resources["firestore_field_logs_ttl"] = firestore.Field(
"firestore-field-logs-ttl",
project=PROJECT_ID,
database=firestore_db.name,
collection="logs", # Subcollection name (applies to all jobs/{job_id}/logs)
field="ttl_expiry",
ttl_config={}, # Empty block enables TTL based on ttl_expiry field
index_config={}, # Disable indexing on TTL field
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# ==================== Firestore Indexes for Logs Subcollection ====================
# Logs: query by worker, order by timestamp
resources["firestore_index_logs_worker_timestamp"] = firestore.Index(
"firestore-index-logs-worker-timestamp",
project=PROJECT_ID,
database=firestore_db.name,
collection="logs", # Subcollection name
fields=[
firestore.IndexFieldArgs(field_path="worker", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="timestamp", order="ASCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
return resources
+21
View File
@@ -0,0 +1,21 @@
"""
IAM modules for service accounts and role bindings.
Organized by purpose:
- backend_sa: Main karaoke-backend service account
- github_actions_sa: GitHub Actions deployer with Workload Identity Federation
- claude_automation_sa: Claude Code automation service account
- worker_sas: Service accounts for VMs (flacfetch, encoding, gdrive, runners)
"""
from . import backend_sa
from . import github_actions_sa
from . import claude_automation_sa
from . import worker_sas
__all__ = [
"backend_sa",
"github_actions_sa",
"claude_automation_sa",
"worker_sas",
]
+132
View File
@@ -0,0 +1,132 @@
"""
Backend service account and IAM bindings.
Manages the karaoke-backend service account used by Cloud Run and its associated
IAM roles for accessing Firestore, GCS, Secret Manager, and other services.
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import serviceaccount
from config import PROJECT_ID, get_project_number
def create_backend_service_account() -> serviceaccount.Account:
"""
Create the main backend service account.
Returns:
serviceaccount.Account: The service account resource.
"""
return serviceaccount.Account(
"karaoke-backend-sa",
account_id="karaoke-backend",
display_name="Karaoke Backend Service Account",
)
def grant_backend_permissions(
service_account: serviceaccount.Account,
) -> dict:
"""
Grant all necessary permissions to the backend service account.
Args:
service_account: The backend service account.
Returns:
dict: Dictionary of IAM binding resources.
"""
bindings = {}
# Grant Firestore permissions
bindings["firestore_iam"] = gcp.projects.IAMMember(
"karaoke-backend-firestore-access",
project=PROJECT_ID,
role="roles/datastore.user",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Storage permissions
bindings["storage_iam"] = gcp.projects.IAMMember(
"karaoke-backend-storage-access",
project=PROJECT_ID,
role="roles/storage.objectAdmin",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Secret Manager read permissions
bindings["secrets_read_iam"] = gcp.projects.IAMMember(
"karaoke-backend-secrets-access",
project=PROJECT_ID,
role="roles/secretmanager.secretAccessor",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Secret Manager write permissions (for OAuth token refresh/device auth)
bindings["secrets_write_iam"] = gcp.projects.IAMMember(
"karaoke-backend-secrets-write",
project=PROJECT_ID,
role="roles/secretmanager.secretVersionManager",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Cloud Trace write permissions (for OpenTelemetry tracing)
bindings["cloud_trace_iam"] = gcp.projects.IAMMember(
"karaoke-backend-cloudtrace-agent",
project=PROJECT_ID,
role="roles/cloudtrace.agent",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Vertex AI User permissions (for Gemini agentic AI correction)
bindings["vertex_ai_iam"] = gcp.projects.IAMMember(
"karaoke-backend-vertexai-user",
project=PROJECT_ID,
role="roles/aiplatform.user",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant backend service account permission to enqueue Cloud Tasks
bindings["cloud_tasks_enqueuer"] = gcp.projects.IAMMember(
"karaoke-backend-cloudtasks-enqueuer",
project=PROJECT_ID,
role="roles/cloudtasks.enqueuer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Service Account User - to act as itself for Cloud Tasks OIDC
bindings["act_as_self"] = serviceaccount.IAMBinding(
"karaoke-backend-act-as-self",
service_account_id=service_account.name,
role="roles/iam.serviceAccountUser",
members=[service_account.email.apply(lambda email: f"serviceAccount:{email}")],
)
# Cloud Run Jobs Developer - run encoding jobs
bindings["run_jobs_developer"] = gcp.projects.IAMMember(
"karaoke-backend-run-jobs-developer",
project=PROJECT_ID,
role="roles/run.developer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
return bindings
def grant_cloud_tasks_invoker_permission() -> gcp.projects.IAMMember:
"""
Grant Cloud Tasks service agent permission to invoke Cloud Run.
This allows Cloud Tasks to authenticate when calling Cloud Run endpoints.
Returns:
gcp.projects.IAMMember: The IAM binding resource.
"""
return gcp.projects.IAMMember(
"cloud-tasks-invoker",
project=PROJECT_ID,
role="roles/run.invoker",
member=f"serviceAccount:service-{get_project_number()}@gcp-sa-cloudtasks.iam.gserviceaccount.com",
)
@@ -0,0 +1,131 @@
"""
Claude Code automation service account.
Manages the service account used by Claude Code for read-only access
to production resources for debugging and monitoring.
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import serviceaccount
from config import PROJECT_ID
def create_claude_automation_service_account() -> serviceaccount.Account:
"""
Create the Claude Code automation service account.
This service account provides read-only access to production resources
for debugging and monitoring purposes.
Returns:
serviceaccount.Account: The service account resource.
"""
return serviceaccount.Account(
"claude-automation-sa",
account_id="claude-automation",
display_name="Claude Code Automation",
description="Service account for Claude Code CLI with read-only GCP access",
)
def grant_claude_automation_permissions(
service_account: serviceaccount.Account,
) -> dict:
"""
Grant read-only permissions to the Claude automation service account.
All permissions are read-only/viewer level to prevent accidental modifications.
Args:
service_account: The Claude automation service account.
Returns:
dict: Dictionary of IAM binding resources.
"""
bindings = {}
# Grant Cloud Run viewer permissions (view services, revisions, status)
bindings["run_viewer"] = gcp.projects.IAMMember(
"claude-automation-run-viewer",
project=PROJECT_ID,
role="roles/run.viewer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Logging viewer permissions (read logs)
bindings["logging_viewer"] = gcp.projects.IAMMember(
"claude-automation-logging-viewer",
project=PROJECT_ID,
role="roles/logging.viewer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Monitoring viewer permissions (read metrics, dashboards)
bindings["monitoring_viewer"] = gcp.projects.IAMMember(
"claude-automation-monitoring-viewer",
project=PROJECT_ID,
role="roles/monitoring.viewer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Cloud Build viewer permissions (view build status, logs)
bindings["cloudbuild_viewer"] = gcp.projects.IAMMember(
"claude-automation-cloudbuild-viewer",
project=PROJECT_ID,
role="roles/cloudbuild.builds.viewer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Cloud Trace viewer permissions (view distributed traces)
bindings["cloudtrace_viewer"] = gcp.projects.IAMMember(
"claude-automation-cloudtrace-viewer",
project=PROJECT_ID,
role="roles/cloudtrace.user",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Cloud Tasks viewer permissions (view queue status)
bindings["cloudtasks_viewer"] = gcp.projects.IAMMember(
"claude-automation-cloudtasks-viewer",
project=PROJECT_ID,
role="roles/cloudtasks.viewer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Storage viewer permissions (list buckets, view metadata - not object content)
bindings["storage_viewer"] = gcp.projects.IAMMember(
"claude-automation-storage-viewer",
project=PROJECT_ID,
role="roles/storage.objectViewer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
return bindings
def grant_impersonation_permission(
service_account: serviceaccount.Account,
) -> serviceaccount.IAMBinding:
"""
Allow users to impersonate the Claude automation service account.
This enables using `gcloud auth application-default login --impersonate-service-account`
for long-lived credentials without downloadable keys.
Args:
service_account: The Claude automation service account.
Returns:
serviceaccount.IAMBinding: The IAM binding resource.
"""
return serviceaccount.IAMBinding(
"claude-automation-impersonation",
service_account_id=service_account.name,
role="roles/iam.serviceAccountTokenCreator",
members=[
"user:andrew@beveridge.uk",
"user:admin@nomadkaraoke.com",
],
)
@@ -0,0 +1,223 @@
"""
GitHub Actions service account and Workload Identity Federation.
Manages the service account used by GitHub Actions for CI/CD deployments,
including keyless authentication via Workload Identity Federation.
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import serviceaccount, iam
from config import PROJECT_ID
def create_github_actions_service_account() -> serviceaccount.Account:
"""
Create the GitHub Actions service account for CI/CD.
Returns:
serviceaccount.Account: The service account resource.
"""
return serviceaccount.Account(
"github-actions-deployer-sa",
account_id="github-actions-deployer",
display_name="GitHub Actions Deployer",
description="Service account for GitHub Actions CD via Workload Identity",
)
def create_workload_identity_pool() -> iam.WorkloadIdentityPool:
"""
Create the Workload Identity Pool for GitHub OIDC.
Returns:
iam.WorkloadIdentityPool: The pool resource.
"""
return iam.WorkloadIdentityPool(
"github-actions-pool",
workload_identity_pool_id="github-actions-pool",
display_name="GitHub Actions Pool",
description="Workload Identity Pool for GitHub Actions",
disabled=False,
)
def create_workload_identity_provider(
pool: iam.WorkloadIdentityPool,
) -> iam.WorkloadIdentityPoolProvider:
"""
Create the Workload Identity Provider for GitHub OIDC.
Args:
pool: The Workload Identity Pool.
Returns:
iam.WorkloadIdentityPoolProvider: The provider resource.
"""
return iam.WorkloadIdentityPoolProvider(
"github-actions-provider",
workload_identity_pool_id=pool.workload_identity_pool_id,
workload_identity_pool_provider_id="github-actions-provider",
display_name="GitHub Actions Provider",
attribute_mapping={
"google.subject": "assertion.sub",
"attribute.actor": "assertion.actor",
"attribute.repository": "assertion.repository",
"attribute.repository_owner": "assertion.repository_owner",
},
attribute_condition="assertion.repository_owner == 'nomadkaraoke'",
oidc=iam.WorkloadIdentityPoolProviderOidcArgs(
issuer_uri="https://token.actions.githubusercontent.com",
),
)
def grant_github_actions_permissions(
service_account: serviceaccount.Account,
pool: iam.WorkloadIdentityPool,
) -> dict:
"""
Grant all necessary permissions to the GitHub Actions service account.
Args:
service_account: The GitHub Actions service account.
pool: The Workload Identity Pool.
Returns:
dict: Dictionary of IAM binding resources.
"""
bindings = {}
# Grant Cloud Run admin permissions
bindings["run_admin"] = gcp.projects.IAMMember(
"github-actions-run-admin",
project=PROJECT_ID,
role="roles/run.admin",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant service account user permissions (to deploy Cloud Run with other SA)
bindings["sa_user"] = gcp.projects.IAMMember(
"github-actions-sa-user",
project=PROJECT_ID,
role="roles/iam.serviceAccountUser",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Artifact Registry writer permissions
bindings["artifact_writer"] = gcp.projects.IAMMember(
"github-actions-artifact-writer",
project=PROJECT_ID,
role="roles/artifactregistry.writer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Cloud Build permissions - REQUIRED for gcloud builds submit
bindings["cloudbuild"] = gcp.projects.IAMMember(
"github-actions-cloudbuild",
project=PROJECT_ID,
role="roles/cloudbuild.builds.editor",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Storage Admin for Cloud Build bucket uploads
bindings["storage_admin"] = gcp.projects.IAMMember(
"github-actions-storage-admin",
project=PROJECT_ID,
role="roles/storage.admin",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Logging write permissions for Cloud Build logs
bindings["logging"] = gcp.projects.IAMMember(
"github-actions-logging",
project=PROJECT_ID,
role="roles/logging.logWriter",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Cloud Build logs viewer - needed for `gcloud builds log` to read build output
bindings["logging_viewer"] = gcp.projects.IAMMember(
"github-actions-logging-viewer",
project=PROJECT_ID,
role="roles/logging.viewer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Service Usage Consumer - required for serviceusage.services.use permission
bindings["service_usage"] = gcp.projects.IAMMember(
"github-actions-service-usage",
project=PROJECT_ID,
role="roles/serviceusage.serviceUsageConsumer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Project IAM Admin - required for Pulumi to refresh IAM binding state
bindings["iam_admin"] = gcp.projects.IAMMember(
"github-actions-iam-admin",
project=PROJECT_ID,
role="roles/resourcemanager.projectIamAdmin",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Secret Manager Admin - required for Pulumi to refresh secret state
bindings["secretmanager"] = gcp.projects.IAMMember(
"github-actions-secretmanager",
project=PROJECT_ID,
role="roles/secretmanager.admin",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Allow GitHub Actions to impersonate the service account
# All nomadkaraoke repos that deploy to GCP need access
bindings["wif_binding"] = serviceaccount.IAMBinding(
"github-actions-wif-binding",
service_account_id=service_account.name,
role="roles/iam.workloadIdentityUser",
members=[
pool.name.apply(
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/karaoke-gen"
),
pool.name.apply(
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/flacfetch"
),
pool.name.apply(
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/karaoke-decide"
),
],
)
# Grant Compute Engine permissions for flacfetch VM deployment
bindings["compute_admin"] = gcp.projects.IAMMember(
"github-actions-compute-admin",
project=PROJECT_ID,
role="roles/compute.instanceAdmin.v1",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant OS Login permissions for SSH access
bindings["os_login"] = gcp.projects.IAMMember(
"github-actions-os-login",
project=PROJECT_ID,
role="roles/compute.osLogin",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant IAP tunnel user for secure SSH (used by gcloud compute ssh)
bindings["iap_tunnel"] = gcp.projects.IAMMember(
"github-actions-iap-tunnel",
project=PROJECT_ID,
role="roles/iap.tunnelResourceAccessor",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant broad Editor access to avoid permission issues when adding new resources
bindings["editor"] = gcp.projects.IAMMember(
"github-actions-editor",
project=PROJECT_ID,
role="roles/editor",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
return bindings
+197
View File
@@ -0,0 +1,197 @@
"""
Worker service accounts for VM-based services.
Manages service accounts for:
- Flacfetch torrent/download VM
- Encoding worker VM
- GDrive validator Cloud Function
- GitHub Actions self-hosted runners
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import serviceaccount, storage
from config import PROJECT_ID
# ==================== Flacfetch Service Account ====================
def create_flacfetch_service_account() -> serviceaccount.Account:
"""Create the flacfetch service account for torrent downloads."""
return serviceaccount.Account(
"flacfetch-sa",
account_id="flacfetch-service",
display_name="Flacfetch Service Account",
description="Service account for flacfetch torrent/audio download VM",
)
def grant_flacfetch_permissions(
service_account: serviceaccount.Account,
bucket: storage.Bucket,
) -> dict:
"""Grant permissions to the flacfetch service account."""
bindings = {}
# Storage Object Creator - write downloaded files to GCS
bindings["storage_writer"] = storage.BucketIAMMember(
"flacfetch-storage-writer",
bucket=bucket.name,
role="roles/storage.objectCreator",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Storage Object Viewer - read uploads folder
bindings["storage_reader"] = storage.BucketIAMMember(
"flacfetch-storage-reader",
bucket=bucket.name,
role="roles/storage.objectViewer",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Secret Manager Accessor - read API keys
bindings["secrets_access"] = gcp.projects.IAMMember(
"flacfetch-secrets-access",
project=PROJECT_ID,
role="roles/secretmanager.secretAccessor",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
return bindings
# ==================== Encoding Worker Service Account ====================
def create_encoding_worker_service_account() -> serviceaccount.Account:
"""Create the encoding worker service account for video encoding."""
return serviceaccount.Account(
"encoding-worker-sa",
account_id="encoding-worker",
display_name="Encoding Worker Service Account",
description="Service account for high-performance video encoding VM",
)
def grant_encoding_worker_permissions(
service_account: serviceaccount.Account,
bucket: storage.Bucket,
) -> dict:
"""Grant permissions to the encoding worker service account."""
bindings = {}
# Storage Object Admin - read/write video files
bindings["storage_admin"] = storage.BucketIAMMember(
"encoding-worker-storage-admin",
bucket=bucket.name,
role="roles/storage.objectAdmin",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Secret Manager Accessor - read API key
bindings["secrets_access"] = gcp.projects.IAMMember(
"encoding-worker-secrets-access",
project=PROJECT_ID,
role="roles/secretmanager.secretAccessor",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Logging Writer - write structured logs
bindings["logging_access"] = gcp.projects.IAMMember(
"encoding-worker-logging-access",
project=PROJECT_ID,
role="roles/logging.logWriter",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Monitoring Metric Writer - write metrics for Ops Agent
bindings["monitoring_access"] = gcp.projects.IAMMember(
"encoding-worker-monitoring-access",
project=PROJECT_ID,
role="roles/monitoring.metricWriter",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
return bindings
# ==================== GDrive Validator Service Account ====================
def create_gdrive_validator_service_account() -> serviceaccount.Account:
"""Create the GDrive validator service account for Cloud Function."""
return serviceaccount.Account(
"gdrive-validator-sa",
account_id="gdrive-validator",
display_name="Google Drive Validator Function",
description="Service account for the GDrive validation Cloud Function",
)
def grant_gdrive_validator_permissions(
service_account: serviceaccount.Account,
) -> dict:
"""Grant permissions to the GDrive validator service account."""
bindings = {}
# Secret Manager Accessor - read Pushbullet API key
bindings["secrets_access"] = gcp.projects.IAMMember(
"gdrive-validator-secrets-access",
project=PROJECT_ID,
role="roles/secretmanager.secretAccessor",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
return bindings
# ==================== GitHub Runner Service Account ====================
def create_github_runner_service_account() -> serviceaccount.Account:
"""Create the GitHub runner service account for self-hosted runners."""
return serviceaccount.Account(
"github-runner-sa",
account_id="github-runner",
display_name="GitHub Actions Runner",
description="Service account for self-hosted GitHub Actions runner VM",
)
def grant_github_runner_permissions(
service_account: serviceaccount.Account,
) -> dict:
"""Grant permissions to the GitHub runner service account."""
bindings = {}
# Artifact Registry Reader - pull base images
bindings["artifact_reader"] = gcp.projects.IAMMember(
"github-runner-artifact-reader",
project=PROJECT_ID,
role="roles/artifactregistry.reader",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Secret Manager Accessor - read GitHub PAT
bindings["secrets_access"] = gcp.projects.IAMMember(
"github-runner-secrets-access",
project=PROJECT_ID,
role="roles/secretmanager.secretAccessor",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Logging Writer - write logs
bindings["logging_access"] = gcp.projects.IAMMember(
"github-runner-logging-access",
project=PROJECT_ID,
role="roles/logging.logWriter",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Storage Object Admin - upload/overwrite wheels in GCS
bindings["storage_access"] = gcp.projects.IAMMember(
"github-runner-storage-write",
project=PROJECT_ID,
role="roles/storage.objectAdmin",
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
)
return bindings
+159
View File
@@ -0,0 +1,159 @@
"""
Cloud Monitoring resources.
Manages alert policies for proactive monitoring and incident response.
"""
import pulumi
import pulumi_gcp as gcp
def create_alert_policies() -> dict[str, gcp.monitoring.AlertPolicy]:
"""
Create all Cloud Monitoring alert policies.
Returns:
dict: Dictionary mapping alert names to AlertPolicy resources.
"""
alerts = {}
# Alert: High Error Rate (>10% of requests returning errors)
alerts["error_rate"] = gcp.monitoring.AlertPolicy(
"high-error-rate-alert",
display_name="Karaoke Backend - High Error Rate",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Error rate > 10%",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/request_count" AND metric.labels.response_code_class!="2xx"',
comparison="COMPARISON_GT",
threshold_value=0.1,
duration="300s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_RATE",
cross_series_reducer="REDUCE_SUM",
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="3600s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="High error rate detected on karaoke-backend. Check Cloud Run logs for details.\n\nDashboard: https://console.cloud.google.com/run/detail/us-central1/karaoke-backend/logs?project=nomadkaraoke",
mime_type="text/markdown",
),
enabled=True,
)
# Alert: Cloud Tasks Queue Backlog (tasks piling up)
alerts["queue_backlog"] = gcp.monitoring.AlertPolicy(
"queue-backlog-alert",
display_name="Karaoke Backend - Queue Backlog",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Queue depth > 50 tasks",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='resource.type="cloud_tasks_queue" AND metric.type="cloudtasks.googleapis.com/queue/depth"',
comparison="COMPARISON_GT",
threshold_value=50,
duration="600s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_MEAN",
cross_series_reducer="REDUCE_MAX",
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="1800s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="Cloud Tasks queue backlog detected. Tasks are piling up faster than workers can process.\n\nPossible causes:\n- Worker errors (check logs)\n- External API issues (Modal, AudioShake)\n- Cloud Run scaling limits\n\nDashboard: https://console.cloud.google.com/cloudtasks?project=nomadkaraoke",
mime_type="text/markdown",
),
enabled=True,
)
# Alert: High Memory Utilization (workers may be crashing)
alerts["memory"] = gcp.monitoring.AlertPolicy(
"high-memory-alert",
display_name="Karaoke Backend - High Memory Usage",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Memory > 85%",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/container/memory/utilizations"',
comparison="COMPARISON_GT",
threshold_value=0.85,
duration="300s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_PERCENTILE_95",
cross_series_reducer="REDUCE_MAX",
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="1800s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="High memory utilization on karaoke-backend. Workers may be running out of memory.\n\nActions:\n- Check for memory leaks in worker code\n- Consider increasing Cloud Run memory limits\n- Review temp file cleanup",
mime_type="text/markdown",
),
enabled=True,
)
# Alert: Cloud Run Service Unavailable
alerts["service_unavailable"] = gcp.monitoring.AlertPolicy(
"service-unavailable-alert",
display_name="Karaoke Backend - Service Unavailable",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="No healthy instances",
condition_absent=gcp.monitoring.AlertPolicyConditionConditionAbsentArgs(
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/container/instance_count"',
duration="300s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionAbsentAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_MEAN",
cross_series_reducer="REDUCE_SUM",
),
],
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="1800s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="Karaoke backend service appears to be down - no healthy Cloud Run instances detected.\n\nImmediate actions:\n1. Check Cloud Run logs for crash reasons\n2. Verify recent deployments\n3. Check external dependencies (Modal, AudioShake)\n\nService URL: https://api.nomadkaraoke.com/api/health",
mime_type="text/markdown",
),
enabled=True,
)
return alerts
+65
View File
@@ -0,0 +1,65 @@
"""
Secret Manager resources.
Manages all secrets for the application. Secret values are added manually
via gcloud or the Cloud Console after resources are created.
"""
import pulumi
from pulumi_gcp import secretmanager
def create_secrets() -> dict[str, secretmanager.Secret]:
"""
Create all Secret Manager secrets.
Returns:
dict: Dictionary mapping secret names to Secret resources.
"""
# Define all secret names (matching original __main__.py exactly)
secret_names = [
# Audio processing
"audioshake-api-key",
"genius-api-key",
"audio-separator-api-url",
# Private tracker APIs (for flacfetch audio search)
"red-api-key",
"red-api-url",
"ops-api-key",
"ops-api-url",
# Payment and email
"stripe-secret-key",
"stripe-webhook-secret",
"sendgrid-api-key",
# Observability (Langfuse for LLM tracing)
"langfuse-public-key",
"langfuse-secret-key",
"langfuse-host",
# Service authentication
"flacfetch-api-key",
"flacfetch-api-url",
"encoding-worker-api-key",
# Notifications
"pushbullet-api-key",
# GitHub
"github-runner-pat",
]
secrets = {}
for secret_id in secret_names:
secrets[secret_id] = secretmanager.Secret(
secret_id,
secret_id=secret_id,
replication=secretmanager.SecretReplicationArgs(
auto=secretmanager.SecretReplicationAutoArgs(),
),
)
return secrets
+38
View File
@@ -0,0 +1,38 @@
"""
Cloud Storage resources.
Manages the GCS bucket for uploads, outputs, and temporary files.
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import storage
from config import PROJECT_ID
def create_bucket() -> storage.Bucket:
"""
Create the main GCS bucket with lifecycle rules.
Returns:
storage.Bucket: The created bucket resource.
"""
bucket = storage.Bucket(
"karaoke-storage",
name=f"karaoke-gen-storage-{PROJECT_ID}",
location="US-CENTRAL1",
force_destroy=False, # Prevent accidental deletion
uniform_bucket_level_access=True,
lifecycle_rules=[
storage.BucketLifecycleRuleArgs(
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
condition=storage.BucketLifecycleRuleConditionArgs(
age=7,
matches_prefixes=["temp/", "uploads/"]
),
),
],
)
return bucket
+122
View File
@@ -0,0 +1,122 @@
# Encoding Worker Packer Image
Custom GCE machine image for the encoding worker VM with all dependencies pre-installed.
## Why?
The encoding worker previously took **~10 minutes** to start because it built Python 3.13 from source on every boot. With this pre-baked image, startup time is reduced to **~30 seconds**.
| Component | Before (startup script) | After (Packer image) |
|-----------|------------------------|---------------------|
| Python 3.13 build | ~7 min (every boot) | Pre-installed |
| FFmpeg install | ~1 min (every boot) | Pre-installed |
| System packages | ~1 min (every boot) | Pre-installed |
| Font packages | ~30 sec (every boot) | Pre-installed |
| API key + wheel | ~30 sec | ~30 sec |
| **Total** | **~10 min** | **~30 sec** |
## Prerequisites
1. **Packer installed**: `brew install packer` or [download from HashiCorp](https://www.packer.io/downloads)
2. **GCP authentication**: `gcloud auth application-default login`
3. **Required permissions**: `roles/compute.instanceAdmin.v1` and `roles/iam.serviceAccountUser`
## Building the Image
```bash
cd infrastructure/packer
# Initialize Packer plugins (first time only)
packer init encoding-worker.pkr.hcl
# Build the image
packer build encoding-worker.pkr.hcl
```
Build takes approximately **15-20 minutes** (mostly Python compilation with optimizations).
### Build Variables
Override defaults with `-var`:
```bash
packer build \
-var 'python_version=3.13.2' \
-var 'zone=us-west1-a' \
encoding-worker.pkr.hcl
```
| Variable | Default | Description |
|----------|---------|-------------|
| `project_id` | `nomadkaraoke` | GCP project ID |
| `zone` | `us-central1-a` | Build zone |
| `python_version` | `3.13.1` | Python version to install |
## What's Included
The image contains:
- **Python 3.13** at `/opt/python313` (built with optimizations)
- **FFmpeg 7.x** at `/usr/local/bin/ffmpeg` (John Van Sickle static build)
- **Noto fonts** including CJK support for subtitle rendering
- **Virtual environment** at `/opt/encoding-worker/venv`
- **Systemd service** `encoding-worker.service` (enabled, not started)
- **Startup script** `/opt/encoding-worker/startup.sh`
## Runtime Behavior
When the VM boots with this image:
1. **Startup script** (`/opt/encoding-worker/startup.sh`) runs via `ExecStartPre`:
- Fetches API key from Secret Manager
- Downloads latest `karaoke-gen` wheel from GCS
- Installs wheel into venv
2. **Encoding worker service** starts via `ExecStart`:
- Runs `uvicorn backend.services.gce_encoding.main:app`
- Listens on port 8080
## When to Rebuild
Rebuild the image when:
- Python version needs updating
- FFmpeg version needs updating
- New system packages required
- Font packages change
**No rebuild needed** for application code changes - the wheel is downloaded at runtime.
## Image Family
Images are created with family `encoding-worker`. Pulumi uses the family to always get the latest image:
```python
image=f"projects/nomadkaraoke/global/images/family/encoding-worker"
```
Old images are automatically replaced in the family, but retained for rollback.
## Rollback
If the new image has issues:
1. Find previous image: `gcloud compute images list --filter="family:encoding-worker"`
2. Update Pulumi to use specific image instead of family
3. Or rebuild from a previous git commit
## CI Integration
For automated rebuilds, add a GitHub Actions workflow that triggers on changes to `infrastructure/packer/`:
```yaml
name: Build Encoding Worker Image
on:
push:
branches: [main]
paths:
- 'infrastructure/packer/**'
workflow_dispatch:
```
See `PHASE5-PACKER-IMAGE-PLAN.md` for full workflow example.
@@ -0,0 +1,94 @@
# Packer template for encoding worker GCE image
#
# This image pre-installs all dependencies to reduce VM startup time from
# ~10 minutes (building Python from source) to ~30 seconds (just start service).
#
# Usage:
# cd infrastructure/packer
# packer init encoding-worker.pkr.hcl
# packer build encoding-worker.pkr.hcl
packer {
required_plugins {
googlecompute = {
source = "github.com/hashicorp/googlecompute"
version = "~> 1.1"
}
}
}
variable "project_id" {
type = string
default = "nomadkaraoke"
description = "GCP project ID"
}
variable "zone" {
type = string
default = "us-central1-a"
description = "GCE zone for building the image"
}
variable "python_version" {
type = string
default = "3.13.1"
description = "Python version to install from source"
}
source "googlecompute" "encoding-worker" {
project_id = var.project_id
zone = var.zone
source_image_family = "debian-12"
source_image_project_id = ["debian-cloud"]
# Use n2 machine type for building (c4 requires hyperdisk which isn't supported by Packer)
# The resulting image will work on c4-standard-8 in production
machine_type = "n2-standard-8"
disk_size = 100
disk_type = "pd-ssd" # Use SSD for faster builds
# Image naming - uses timestamp for versioning, family for latest
image_name = "encoding-worker-{{timestamp}}"
image_family = "encoding-worker"
image_description = "Encoding worker with Python ${var.python_version}, FFmpeg 7.x, and fonts pre-installed"
ssh_username = "packer"
# Tags for firewall rules during build (if needed)
tags = ["packer-build"]
# Metadata for build tracking
image_labels = {
"managed-by" = "packer"
"python" = replace(var.python_version, ".", "-")
}
}
build {
sources = ["source.googlecompute.encoding-worker"]
# Upload provisioning script
provisioner "file" {
source = "scripts/provision.sh"
destination = "/tmp/provision.sh"
}
# Run provisioning
provisioner "shell" {
inline = [
"chmod +x /tmp/provision.sh",
"sudo PYTHON_VERSION=${var.python_version} /tmp/provision.sh"
]
}
# Clean up for smaller image
provisioner "shell" {
inline = [
"sudo apt-get clean",
"sudo rm -rf /var/lib/apt/lists/*",
"sudo rm -rf /tmp/*",
"sudo rm -rf /var/tmp/*",
"sudo journalctl --vacuum-time=1d"
]
}
}
+177
View File
@@ -0,0 +1,177 @@
#!/bin/bash
# Provision encoding worker image with all dependencies pre-installed
#
# This script runs during Packer image build. After it completes:
# - Python 3.13 is installed at /opt/python313
# - FFmpeg 7.x is installed at /usr/local/bin/ffmpeg
# - All fonts (including CJK) are installed
# - Virtual environment exists at /opt/encoding-worker/venv
# - Systemd service is configured (but not started - no API key yet)
#
# At runtime, the startup script only needs to:
# 1. Fetch API key from Secret Manager
# 2. Download latest karaoke-gen wheel
# 3. Start the systemd service
set -e
echo "=== Starting encoding worker image provisioning ==="
echo "Python version: ${PYTHON_VERSION}"
# Install system packages
echo "Installing system packages..."
apt-get update
apt-get install -y \
docker.io \
curl \
git \
xz-utils \
build-essential \
libssl-dev \
zlib1g-dev \
libbz2-dev \
libreadline-dev \
libsqlite3-dev \
libncursesw5-dev \
tk-dev \
libxml2-dev \
libxmlsec1-dev \
libffi-dev \
liblzma-dev
# Install fonts for ASS subtitle rendering (libass uses fontconfig)
# - fonts-noto: Noto Sans (default karaoke font) + musical symbols
# - fonts-noto-cjk: Chinese, Japanese, Korean character support
# - fontconfig: Font configuration system
echo "Installing fonts..."
apt-get install -y fonts-noto fonts-noto-cjk fontconfig
fc-cache -fv
# Enable Docker (will start on boot)
systemctl enable docker
# Install optimized static FFmpeg build (John Van Sickle)
# This is faster than Debian's package and has more codecs enabled
echo "Installing FFmpeg..."
curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz
tar -xf /tmp/ffmpeg.tar.xz -C /tmp
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe
rm -rf /tmp/ffmpeg*
echo "FFmpeg version:"
ffmpeg -version | head -1
# Build and install Python from source
# This is the slow part (~7 min) - pre-baking it saves startup time
echo "Building Python ${PYTHON_VERSION} from source..."
cd /tmp
curl -L "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tar.xz" -o python.tar.xz
tar -xf python.tar.xz
cd "Python-${PYTHON_VERSION}"
# Configure with optimizations
# --enable-optimizations: Profile-guided optimization (slower build, faster runtime)
# --with-lto: Link-time optimization
./configure --prefix=/opt/python313 --enable-optimizations --with-lto
make -j$(nproc)
make install
# Clean up Python build artifacts
rm -rf /tmp/python.tar.xz /tmp/Python-${PYTHON_VERSION}
echo "Python ${PYTHON_VERSION} installed:"
/opt/python313/bin/python3.13 --version
# Create working directory and venv
mkdir -p /opt/encoding-worker
cd /opt/encoding-worker
echo "Creating Python virtual environment..."
/opt/python313/bin/python3.13 -m venv venv
# Install base Python dependencies (wheel will add more at runtime)
source venv/bin/activate
pip install --upgrade pip
pip install fastapi uvicorn google-cloud-storage aiofiles aiohttp
# Create startup helper script (runs via ExecStartPre before service starts)
# This fetches API key and installs latest wheel at boot time
cat > /opt/encoding-worker/startup.sh << 'STARTUP'
#!/bin/bash
set -e
echo "=== Encoding worker startup at $(date) ==="
# Fetch API key from Secret Manager
echo "Fetching API key from Secret Manager..."
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
if [ -z "$ENCODING_API_KEY" ]; then
echo "WARNING: No API key found in Secret Manager"
fi
# Write environment file for systemd
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > /opt/encoding-worker/env
# Download and install latest karaoke-gen wheel from GCS
# This enables hot code updates without rebuilding the image
echo "Installing latest karaoke-gen wheel from GCS..."
gsutil cp "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl" /tmp/ 2>/dev/null || true
WHEEL_FILE=$(ls -t /tmp/karaoke_gen-*.whl 2>/dev/null | head -1)
if [ -n "$WHEEL_FILE" ]; then
echo "Installing wheel: $WHEEL_FILE"
/opt/encoding-worker/venv/bin/pip install --upgrade --quiet "$WHEEL_FILE" || echo "WARNING: Failed to install wheel"
else
echo "No wheel found in GCS"
fi
echo "=== Startup script complete ==="
STARTUP
chmod +x /opt/encoding-worker/startup.sh
# Create systemd service
# ExecStartPre runs the startup script to fetch API key and install wheel
# ExecStart runs uvicorn with the encoding worker from the installed wheel
cat > /etc/systemd/system/encoding-worker.service << 'SYSTEMD'
[Unit]
Description=Encoding Worker HTTP API Service
After=network.target docker.service
Wants=docker.service
[Service]
Type=simple
User=root
WorkingDirectory=/opt/encoding-worker
ExecStartPre=/opt/encoding-worker/startup.sh
ExecStart=/opt/encoding-worker/venv/bin/uvicorn backend.services.gce_encoding.main:app --host 0.0.0.0 --port 8080
Restart=always
RestartSec=10
TimeoutStartSec=300
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
EnvironmentFile=/opt/encoding-worker/env
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=encoding-worker
[Install]
WantedBy=multi-user.target
SYSTEMD
# Create empty env file (will be populated at runtime)
touch /opt/encoding-worker/env
# Reload systemd and enable service (will start on boot)
systemctl daemon-reload
systemctl enable encoding-worker
echo "=== Provisioning complete ==="
echo "Image includes:"
echo " - Python ${PYTHON_VERSION} at /opt/python313"
echo " - FFmpeg at /usr/local/bin/ffmpeg"
echo " - Noto fonts (including CJK)"
echo " - Virtual environment at /opt/encoding-worker/venv"
echo " - Systemd service: encoding-worker.service"
+9 -4
View File
@@ -156,15 +156,20 @@ class VideoBackgroundProcessor:
"""
Escape a file path for use in ffmpeg filter expressions.
FFmpeg filter syntax uses single quotes to protect special characters
like spaces, colons, and semicolons. Single quotes within the path
are escaped using the '\\'' pattern (end quote, literal quote, start quote).
Args:
path: File path to escape
Returns:
str: Escaped path
str: Escaped path (wrapped in single quotes)
"""
# Escape backslashes and colons for ffmpeg filter syntax
escaped = path.replace("\\", "\\\\").replace(":", "\\:")
return escaped
# Escape single quotes: ' becomes '\'' (end quote, \', start quote)
escaped = path.replace("'", "'\\''")
# Wrap entire path in single quotes
return f"'{escaped}'"
def build_video_filter(self, ass_subtitles_path, darkness_percent, fonts_dir=None):
"""
@@ -35,10 +35,14 @@ class ProviderConfig:
circuit_breaker_failure_threshold: int = 3
circuit_breaker_open_seconds: int = 60
# Initialization timeouts - fail fast instead of hanging forever
# These are separate from request_timeout to catch connection establishment issues
initialization_timeout_seconds: float = 30.0 # Model creation + warm-up
warmup_timeout_seconds: float = 15.0 # Just the warm-up call
# Initialization timeout - fail fast instead of hanging forever
# This is separate from request_timeout to catch connection establishment issues
initialization_timeout_seconds: float = 30.0 # Model creation
# Parallel processing settings
# Process multiple gaps concurrently to reduce total correction time
# Set to 1 to disable parallelism, higher values increase throughput but may hit rate limits
max_parallel_gaps: int = 5
@staticmethod
def from_env(cache_dir: Optional[str] = None) -> "ProviderConfig":
@@ -70,7 +74,7 @@ class ProviderConfig:
circuit_breaker_failure_threshold=int(os.getenv("AGENTIC_CIRCUIT_THRESHOLD", "3")),
circuit_breaker_open_seconds=int(os.getenv("AGENTIC_CIRCUIT_OPEN_SECONDS", "60")),
initialization_timeout_seconds=float(os.getenv("AGENTIC_INIT_TIMEOUT_SECONDS", "30.0")),
warmup_timeout_seconds=float(os.getenv("AGENTIC_WARMUP_TIMEOUT_SECONDS", "15.0")),
max_parallel_gaps=int(os.getenv("AGENTIC_MAX_PARALLEL_GAPS", "5")),
)
def validate_environment(self, logger: Optional[object] = None) -> None:
@@ -97,7 +97,6 @@ class LangChainBridge(BaseAIProvider):
# Lazy-initialized chat model
self._chat_model: Optional[Any] = None
self._warmed_up: bool = False
def name(self) -> str:
"""Return provider name for logging."""
@@ -164,13 +163,7 @@ class LangChainBridge(BaseAIProvider):
) from None
init_elapsed = time.time() - init_start
logger.info(f"🤖 Model created in {init_elapsed:.2f}s, starting warm-up...")
# Warm up the model to establish connection before real work
self._warm_up_model()
total_elapsed = time.time() - init_start
logger.info(f"🤖 Model initialization complete in {total_elapsed:.2f}s")
logger.info(f"🤖 Model initialized in {init_elapsed:.2f}s")
except InitializationTimeoutError as e:
self._circuit_breaker.record_failure(self._model)
@@ -271,47 +264,4 @@ class LangChainBridge(BaseAIProvider):
return content
def _warm_up_model(self) -> None:
"""Send a lightweight request to warm up the model connection.
This helps establish the REST connection and potentially warm up any
server-side resources before processing real correction requests.
The warm-up uses a timeout to fail fast if the service is unresponsive.
"""
if self._warmed_up:
return
timeout = self._config.warmup_timeout_seconds
# Use print with flush=True for visibility when output is redirected
print(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...", flush=True)
logger.info(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...")
warmup_start = time.time()
try:
from langchain_core.messages import HumanMessage
# Minimal prompt that requires almost no processing
warm_up_prompt = 'Respond with exactly: {"status":"ready"}'
# Use ThreadPoolExecutor for timeout on warm-up call
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(
self._chat_model.invoke,
[HumanMessage(content=warm_up_prompt)]
)
try:
future.result(timeout=timeout)
except FuturesTimeoutError:
raise TimeoutError(f"Warm-up timed out after {timeout}s") from None
elapsed = time.time() - warmup_start
self._warmed_up = True
print(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s", flush=True)
logger.info(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s")
except Exception as e:
elapsed = time.time() - warmup_start
# Don't fail the actual request if warm-up fails
# Just log and continue - the real request might still work
print(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)", flush=True)
logger.warning(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)")
@@ -268,12 +268,26 @@ class LyricsCorrector:
_adapt = None
_ModelRouter = None
# Pre-initialized agentic corrector (created once, reused for all gaps)
_agentic_agent = None
if use_agentic_env:
try:
from lyrics_transcriber.correction.agentic.agent import AgenticCorrector as _AgenticCorrector
from lyrics_transcriber.correction.agentic.adapter import adapt_proposals_to_word_corrections as _adapt
from lyrics_transcriber.correction.agentic.router import ModelRouter as _ModelRouter
self.logger.info("🤖 Agentic modules imported successfully - running in AGENTIC-ONLY mode")
# Create agent ONCE and reuse for all gaps (avoids repeated model initialization)
_router = _ModelRouter()
model_id = _router.choose_model("gap", uncertainty=0.5) # Use default uncertainty
self.logger.info(f"🤖 Creating single AgenticCorrector with model: {model_id}")
_agentic_agent = _AgenticCorrector.from_model(
model=model_id,
session_id=session_id,
cache_dir=str(self._cache_dir)
)
self.logger.info("🤖 AgenticCorrector initialized and ready for all gaps")
except Exception as e:
self.logger.error(f"🤖 Failed to import agentic modules but USE_AGENTIC_AI=1: {e}")
raise RuntimeError(f"Agentic AI correction is enabled but required modules could not be imported: {e}") from e
@@ -443,16 +457,186 @@ class LyricsCorrector:
sys.exit(0)
# === END TEMPORARY CODE ===
# AGENTIC-ONLY MODE: Process all gaps in parallel for better performance
if use_agentic_env:
from concurrent.futures import ThreadPoolExecutor, as_completed
from lyrics_transcriber.correction.agentic.providers.config import ProviderConfig
# Get parallel processing config
_config = ProviderConfig.from_env()
max_workers = _config.max_parallel_gaps
self.logger.info(f"🤖 Processing {len(gap_sequences)} gaps in parallel (max_workers={max_workers})")
# Pre-compute shared data structures once (not per-gap)
all_transcribed_words = []
for seg in segments:
all_transcribed_words.extend(seg.words)
word_position = {w.id: idx for idx, w in enumerate(all_transcribed_words)}
# Build reference contexts once (same for all gaps)
reference_contexts = {}
for source, lyrics_data in self.reference_lyrics.items():
if lyrics_data and lyrics_data.segments:
ref_words = []
for seg in lyrics_data.segments:
ref_words.extend([w.text for w in seg.words])
reference_contexts[source] = " ".join(ref_words)
# Get artist and title once
artist = metadata.get("artist") if metadata else None
title = metadata.get("title") if metadata else None
# Prepare all gap inputs upfront
gap_inputs = []
for i, gap in enumerate(gap_sequences, 1):
# Prepare gap words data
gap_words_data = []
for word_id in gap.transcribed_word_ids:
if word_id in word_map:
word = word_map[word_id]
gap_words_data.append({
"id": word_id,
"text": word.text,
"start_time": getattr(word, 'start_time', 0),
"end_time": getattr(word, 'end_time', 0)
})
# Compute context words
gap_positions = [word_position[wid] for wid in gap.transcribed_word_ids if wid in word_position]
preceding_words = ""
following_words = ""
if gap_positions:
first_gap_pos = min(gap_positions)
last_gap_pos = max(gap_positions)
# Get 10 words before
start_pos = max(0, first_gap_pos - 10)
preceding_list = [all_transcribed_words[idx].text for idx in range(start_pos, first_gap_pos) if idx < len(all_transcribed_words)]
preceding_words = " ".join(preceding_list)
# Get 10 words after
end_pos = min(len(all_transcribed_words), last_gap_pos + 11)
following_list = [all_transcribed_words[idx].text for idx in range(last_gap_pos + 1, end_pos) if idx < len(all_transcribed_words)]
following_words = " ".join(following_list)
gap_inputs.append({
'index': i,
'gap': gap,
'gap_id': f"gap_{i}",
'gap_words': gap_words_data,
'preceding_words': preceding_words,
'following_words': following_words,
'reference_contexts': reference_contexts,
'artist': artist,
'title': title
})
# Function to process a single gap (runs in thread pool)
def process_single_gap(gap_input):
"""Process a single gap and return proposals. Thread-safe."""
idx = gap_input['index']
try:
proposals = _agentic_agent.propose_for_gap(
gap_id=gap_input['gap_id'],
gap_words=gap_input['gap_words'],
preceding_words=gap_input['preceding_words'],
following_words=gap_input['following_words'],
reference_contexts=gap_input['reference_contexts'],
artist=gap_input['artist'],
title=gap_input['title']
)
return {'index': idx, 'gap': gap_input['gap'], 'proposals': proposals, 'error': None}
except Exception as e:
return {'index': idx, 'gap': gap_input['gap'], 'proposals': None, 'error': str(e)}
# Process gaps in parallel
results = [None] * len(gap_inputs)
completed_count = 0
errors = []
with ThreadPoolExecutor(max_workers=max_workers) as executor:
# Submit all tasks
future_to_input = {executor.submit(process_single_gap, g): g for g in gap_inputs}
# Collect results as they complete
for future in as_completed(future_to_input):
# Check deadline
if deadline and time.time() > deadline:
self.logger.warning(
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {completed_count}/{len(gap_sequences)} gaps. "
"Cancelling remaining gaps - human review will correct any issues."
)
# Cancel remaining futures (use list() to avoid mutating dict during iteration)
for f in list(future_to_input.keys()):
f.cancel()
break
result = future.result()
idx = result['index'] - 1 # Convert 1-based to 0-based
results[idx] = result
completed_count += 1
if result['error']:
errors.append(f"Gap {result['index']}: {result['error']}")
self.logger.error(f"🤖 Gap {result['index']} failed: {result['error']}")
else:
proposal_count = len(result['proposals']) if result['proposals'] else 0
self.logger.info(f"🤖 Gap {result['index']}/{len(gap_sequences)} completed ({proposal_count} proposals)")
self.logger.info(f"🤖 Parallel processing complete: {completed_count}/{len(gap_sequences)} gaps processed")
# If any errors occurred, fail fast
if errors:
raise RuntimeError(f"Agentic AI correction failed for {len(errors)} gaps: {'; '.join(errors)}")
# Apply corrections sequentially (must be in order due to segment modifications)
for result in results:
if result is None:
continue # Skipped due to deadline
i = result['index']
gap = result['gap']
_proposals = result['proposals']
_agentic_corrections = _adapt(_proposals, word_map, linear_position_map) if _proposals else []
if _agentic_corrections:
self.logger.info(f"🤖 Applying {len(_agentic_corrections)} agentic corrections for gap {i}")
affected_word_ids = [w.id for w in self._get_affected_words(gap, segments)]
affected_segment_ids = [s.id for s in self._get_affected_segments(gap, segments)]
updated_segments = self._apply_corrections_to_segments(self._get_affected_segments(gap, segments), _agentic_corrections)
for correction in _agentic_corrections:
if correction.word_id and correction.corrected_word_id:
word_id_map[correction.word_id] = correction.corrected_word_id
for old_seg, new_seg in zip(self._get_affected_segments(gap, segments), updated_segments):
segment_id_map[old_seg.id] = new_seg.id
step = CorrectionStep(
handler_name="AgenticCorrector",
affected_word_ids=affected_word_ids,
affected_segment_ids=affected_segment_ids,
corrections=_agentic_corrections,
segments_before=self._get_affected_segments(gap, segments),
segments_after=updated_segments,
created_word_ids=[w.id for w in self._get_new_words(updated_segments, affected_word_ids)],
deleted_word_ids=[id for id in affected_word_ids if not self._word_exists(id, updated_segments)],
)
correction_steps.append(step)
all_corrections.extend(_agentic_corrections)
# Log corrections made
for correction in _agentic_corrections:
self.logger.info(
f"Made correction: '{correction.original_word}' -> '{correction.corrected_word}' "
f"(confidence: {correction.confidence:.2f}, reason: {correction.reason})"
)
else:
self.logger.debug(f"🤖 No agentic corrections needed for gap {i}")
# RULE-BASED MODE: Process gaps sequentially
for i, gap in enumerate(gap_sequences, 1):
# Check deadline before processing each gap (agentic mode only)
# This allows us to abort early and return uncorrected results for human review
if deadline and use_agentic_env and time.time() > deadline:
self.logger.warning(
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {i-1}/{len(gap_sequences)} gaps. "
"Skipping remaining gaps - human review will correct any issues."
)
# Break out of loop - continue with whatever corrections we have (likely none)
break
# Skip if we already processed in agentic mode
if use_agentic_env:
continue
self.logger.info(f"Processing gap {i}/{len(gap_sequences)} at position {gap.transcription_position}")
@@ -460,128 +644,6 @@ class LyricsCorrector:
gap_words = [word_map[word_id] for word_id in gap.transcribed_word_ids]
self.logger.debug(f"Gap text: '{' '.join(w.text for w in gap_words)}'")
# AGENTIC-ONLY MODE: Use agentic correction exclusively
if use_agentic_env:
self.logger.info(f"🤖 Attempting agentic correction for gap {i}/{len(gap_sequences)}")
try:
# Prepare gap data for classification-first workflow
gap_words_data = []
for word_id in gap.transcribed_word_ids:
if word_id in word_map:
word = word_map[word_id]
gap_words_data.append({
"id": word_id,
"text": word.text,
"start_time": getattr(word, 'start_time', 0),
"end_time": getattr(word, 'end_time', 0)
})
# Get context words
all_transcribed_words = []
for seg in segments:
all_transcribed_words.extend(seg.words)
word_position = {w.id: idx for idx, w in enumerate(all_transcribed_words)}
gap_positions = [word_position[wid] for wid in gap.transcribed_word_ids if wid in word_position]
preceding_words = ""
following_words = ""
if gap_positions:
first_gap_pos = min(gap_positions)
last_gap_pos = max(gap_positions)
# Get 10 words before
start_pos = max(0, first_gap_pos - 10)
preceding_list = [all_transcribed_words[idx].text for idx in range(start_pos, first_gap_pos) if idx < len(all_transcribed_words)]
preceding_words = " ".join(preceding_list)
# Get 10 words after
end_pos = min(len(all_transcribed_words), last_gap_pos + 11)
following_list = [all_transcribed_words[idx].text for idx in range(last_gap_pos + 1, end_pos) if idx < len(all_transcribed_words)]
following_words = " ".join(following_list)
# Get reference contexts from all sources
reference_contexts = {}
for source, lyrics_data in self.reference_lyrics.items():
if lyrics_data and lyrics_data.segments:
ref_words = []
for seg in lyrics_data.segments:
ref_words.extend([w.text for w in seg.words])
# For now, use full text (handlers will extract relevant portions)
reference_contexts[source] = " ".join(ref_words)
# Get artist and title from metadata
artist = metadata.get("artist") if metadata else None
title = metadata.get("title") if metadata else None
# Choose model via router
_router = _ModelRouter()
uncertainty = 0.3 if len(gap_words_data) <= 2 else 0.7
model_id = _router.choose_model("gap", uncertainty)
self.logger.debug(f"🤖 Router selected model: {model_id}")
# Create agent and use new classification-first workflow
self.logger.debug(f"🤖 Creating AgenticCorrector with model: {model_id}")
_agent = _AgenticCorrector.from_model(
model=model_id,
session_id=session_id,
cache_dir=str(self._cache_dir)
)
# Use new propose_for_gap method
self.logger.debug(f"🤖 Calling agent.propose_for_gap() for gap {i}")
_proposals = _agent.propose_for_gap(
gap_id=f"gap_{i}",
gap_words=gap_words_data,
preceding_words=preceding_words,
following_words=following_words,
reference_contexts=reference_contexts,
artist=artist,
title=title
)
self.logger.debug(f"🤖 Agent returned {len(_proposals) if _proposals else 0} proposals")
_agentic_corrections = _adapt(_proposals, word_map, linear_position_map) if _proposals else []
self.logger.debug(f"🤖 Adapter returned {len(_agentic_corrections)} corrections")
if _agentic_corrections:
self.logger.info(f"🤖 Applying {len(_agentic_corrections)} agentic corrections for gap {i}")
affected_word_ids = [w.id for w in self._get_affected_words(gap, segments)]
affected_segment_ids = [s.id for s in self._get_affected_segments(gap, segments)]
updated_segments = self._apply_corrections_to_segments(self._get_affected_segments(gap, segments), _agentic_corrections)
for correction in _agentic_corrections:
if correction.word_id and correction.corrected_word_id:
word_id_map[correction.word_id] = correction.corrected_word_id
for old_seg, new_seg in zip(self._get_affected_segments(gap, segments), updated_segments):
segment_id_map[old_seg.id] = new_seg.id
step = CorrectionStep(
handler_name="AgenticCorrector",
affected_word_ids=affected_word_ids,
affected_segment_ids=affected_segment_ids,
corrections=_agentic_corrections,
segments_before=self._get_affected_segments(gap, segments),
segments_after=updated_segments,
created_word_ids=[w.id for w in self._get_new_words(updated_segments, affected_word_ids)],
deleted_word_ids=[id for id in affected_word_ids if not self._word_exists(id, updated_segments)],
)
correction_steps.append(step)
all_corrections.extend(_agentic_corrections)
# Log corrections made
for correction in _agentic_corrections:
self.logger.info(
f"Made correction: '{correction.original_word}' -> '{correction.corrected_word}' "
f"(confidence: {correction.confidence:.2f}, reason: {correction.reason})"
)
else:
self.logger.info(f"🤖 No agentic corrections needed for gap {i}")
except Exception as e:
# In agentic-only mode, fail fast instead of falling back
self.logger.error(f"🤖 Agentic correction failed for gap {i}: {e}", exc_info=True)
raise RuntimeError(f"Agentic AI correction failed for gap {i}: {e}") from e
# Skip rule-based handlers completely in agentic mode
continue
# RULE-BASED MODE: Try each handler in order
for handler in self.handlers:
handler_name = handler.__class__.__name__
@@ -274,11 +274,12 @@ class CorrectionOperations:
audio_filepath: str,
artist: Optional[str] = None,
title: Optional[str] = None,
logger: Optional[logging.Logger] = None
logger: Optional[logging.Logger] = None,
ass_only: bool = False,
) -> Dict[str, Any]:
"""
Generate a preview video with current corrections.
Args:
correction_result: Current correction result
updated_data: Updated correction data for preview
@@ -287,10 +288,12 @@ class CorrectionOperations:
artist: Optional artist name
title: Optional title
logger: Optional logger instance
ass_only: If True, generate only ASS subtitles without video encoding.
Useful when video encoding is offloaded to external service.
Returns:
Dict with status, preview_hash, and video_path
Dict with status, preview_hash, and video_path (or ass_path if ass_only)
Raises:
ValueError: If preview video generation fails
"""
@@ -338,15 +341,27 @@ class CorrectionOperations:
audio_filepath=audio_filepath,
artist=artist,
title=title,
ass_only=ass_only,
)
# When ass_only, we only need the ASS file (video encoding done externally)
if ass_only:
if not preview_outputs.ass:
raise ValueError("Preview ASS generation failed")
logger.info(f"Generated preview ASS: {preview_outputs.ass}")
return {
"status": "success",
"preview_hash": preview_hash,
"ass_path": preview_outputs.ass,
}
if not preview_outputs.video:
raise ValueError("Preview video generation failed")
logger.info(f"Generated preview video: {preview_outputs.video}")
return {
"status": "success",
"preview_hash": preview_hash,
"video_path": preview_outputs.video
"video_path": preview_outputs.video,
}
@@ -2,7 +2,7 @@
"name": "lyrics-transcriber-frontend",
"private": true,
"homepage": "https://nomadkaraoke.github.io/lyrics-transcriber-frontend",
"version": "0.83.0",
"version": "0.84.0",
"type": "module",
"scripts": {
"dev": "vite",
@@ -1 +1 @@
{"root":["./src/app.tsx","./src/api.ts","./src/main.tsx","./src/theme.ts","./src/types.ts","./src/validation.ts","./src/vite-env.d.ts","./src/components/aifeedbackmodal.tsx","./src/components/addlyricsmodal.tsx","./src/components/agenticcorrectionmetrics.tsx","./src/components/audioplayer.tsx","./src/components/correctedwordwithactions.tsx","./src/components/correctionannotationmodal.tsx","./src/components/correctiondetailcard.tsx","./src/components/correctionmetrics.tsx","./src/components/durationtimelineview.tsx","./src/components/editactionbar.tsx","./src/components/editmodal.tsx","./src/components/edittimelinesection.tsx","./src/components/editwordlist.tsx","./src/components/fileupload.tsx","./src/components/findreplacemodal.tsx","./src/components/header.tsx","./src/components/lyricsanalyzer.tsx","./src/components/metricsdashboard.tsx","./src/components/modeselectionmodal.tsx","./src/components/modeselector.tsx","./src/components/modelselector.tsx","./src/components/previewvideosection.tsx","./src/components/referenceview.tsx","./src/components/replacealllyricsmodal.tsx","./src/components/reviewchangesmodal.tsx","./src/components/segmentdetailsmodal.tsx","./src/components/timelineeditor.tsx","./src/components/timingoffsetmodal.tsx","./src/components/transcriptionview.tsx","./src/components/worddivider.tsx","./src/components/lyricssynchronizer/synccontrols.tsx","./src/components/lyricssynchronizer/timelinecanvas.tsx","./src/components/lyricssynchronizer/upcomingwordsbar.tsx","./src/components/lyricssynchronizer/index.tsx","./src/components/shared/constants.ts","./src/components/shared/styles.ts","./src/components/shared/types.ts","./src/components/shared/components/highlightedtext.tsx","./src/components/shared/components/sourceselector.tsx","./src/components/shared/components/word.tsx","./src/components/shared/hooks/usewordclick.ts","./src/components/shared/utils/keyboardhandlers.ts","./src/components/shared/utils/localstorage.ts","./src/components/shared/utils/referencelinecalculator.ts","./src/components/shared/utils/segmentoperations.ts","./src/components/shared/utils/timingutils.ts","./src/components/shared/utils/wordutils.ts","./src/hooks/usemanualsync.ts","./src/types/global.d.ts"],"version":"5.6.3"}
{"root":["./src/app.tsx","./src/api.ts","./src/main.tsx","./src/theme.ts","./src/types.ts","./src/validation.ts","./src/vite-env.d.ts","./src/components/aifeedbackmodal.tsx","./src/components/addlyricsmodal.tsx","./src/components/agenticcorrectionmetrics.tsx","./src/components/appheader.tsx","./src/components/audioplayer.tsx","./src/components/correctedwordwithactions.tsx","./src/components/correctionannotationmodal.tsx","./src/components/correctiondetailcard.tsx","./src/components/correctionmetrics.tsx","./src/components/durationtimelineview.tsx","./src/components/editactionbar.tsx","./src/components/editmodal.tsx","./src/components/edittimelinesection.tsx","./src/components/editwordlist.tsx","./src/components/fileupload.tsx","./src/components/findreplacemodal.tsx","./src/components/header.tsx","./src/components/lyricsanalyzer.tsx","./src/components/metricsdashboard.tsx","./src/components/modeselectionmodal.tsx","./src/components/modeselector.tsx","./src/components/modelselector.tsx","./src/components/previewvideosection.tsx","./src/components/referenceview.tsx","./src/components/replacealllyricsmodal.tsx","./src/components/reviewchangesmodal.tsx","./src/components/segmentdetailsmodal.tsx","./src/components/timelineeditor.tsx","./src/components/timingoffsetmodal.tsx","./src/components/transcriptionview.tsx","./src/components/worddivider.tsx","./src/components/lyricssynchronizer/synccontrols.tsx","./src/components/lyricssynchronizer/timelinecanvas.tsx","./src/components/lyricssynchronizer/upcomingwordsbar.tsx","./src/components/lyricssynchronizer/index.tsx","./src/components/shared/constants.ts","./src/components/shared/styles.ts","./src/components/shared/types.ts","./src/components/shared/components/highlightedtext.tsx","./src/components/shared/components/sourceselector.tsx","./src/components/shared/components/word.tsx","./src/components/shared/hooks/usewordclick.ts","./src/components/shared/utils/keyboardhandlers.ts","./src/components/shared/utils/localstorage.ts","./src/components/shared/utils/referencelinecalculator.ts","./src/components/shared/utils/segmentoperations.ts","./src/components/shared/utils/timingutils.ts","./src/components/shared/utils/wordutils.ts","./src/hooks/usemanualsync.ts","./src/types/global.d.ts"],"version":"5.6.3"}
@@ -3,6 +3,11 @@ import react from '@vitejs/plugin-react';
// https://vite.dev/config/
export default defineConfig({
plugins: [react()],
server: {
port: 5173,
// Enable CORS for local development
cors: true,
},
build: {
minify: false,
sourcemap: true,
@@ -1 +0,0 @@
{"version":3,"file":"index-BECn1o8Q.js","sources":["../../node_modules/react/cjs/react.production.min.js","../../node_modules/react/index.js","../../node_modules/react/cjs/react-jsx-runtime.production.min.js","../../node_modules/react/jsx-runtime.js","../../node_modules/scheduler/cjs/scheduler.production.min.js","../../node_modules/scheduler/index.js","../../node_modules/react-dom/cjs/react-dom.production.min.js","../../node_modules/react-dom/index.js","../../node_modules/react-dom/client.js","../../node_modules/@mui/utils/esm/formatMuiErrorMessage/formatMuiErrorMessage.js","../../node_modules/@mui/material/styles/identifier.js","../../node_modules/@babel/runtime/helpers/esm/extends.js","../../node_modules/@emotion/sheet/dist/emotion-sheet.esm.js","../../node_modules/stylis/src/Enum.js","../../node_modules/stylis/src/Utility.js","../../node_modules/stylis/src/Tokenizer.js","../../node_modules/stylis/src/Parser.js","../../node_modules/stylis/src/Serializer.js","../../node_modules/stylis/src/Middleware.js","../../node_modules/@emotion/memoize/dist/emotion-memoize.esm.js","../../node_modules/@emotion/cache/dist/emotion-cache.browser.esm.js","../../node_modules/react-is/cjs/react-is.production.min.js","../../node_modules/react-is/index.js","../../node_modules/hoist-non-react-statics/dist/hoist-non-react-statics.cjs.js","../../node_modules/@emotion/utils/dist/emotion-utils.browser.esm.js","../../node_modules/@emotion/hash/dist/emotion-hash.esm.js","../../node_modules/@emotion/unitless/dist/emotion-unitless.esm.js","../../node_modules/@emotion/serialize/dist/emotion-serialize.esm.js","../../node_modules/@emotion/use-insertion-effect-with-fallbacks/dist/emotion-use-insertion-effect-with-fallbacks.browser.esm.js","../../node_modules/@emotion/react/dist/emotion-element-f0de968e.browser.esm.js","../../node_modules/@emotion/react/dist/emotion-react.browser.esm.js","../../node_modules/@emotion/is-prop-valid/dist/emotion-is-prop-valid.esm.js","../../node_modules/@emotion/styled/base/dist/emotion-styled-base.browser.esm.js","../../node_modules/@emotion/styled/dist/emotion-styled.browser.esm.js","../../node_modules/@mui/styled-engine/GlobalStyles/GlobalStyles.js","../../node_modules/@mui/styled-engine/index.js","../../node_modules/@mui/utils/node_modules/react-is/cjs/react-is.production.js","../../node_modules/@mui/utils/node_modules/react-is/index.js","../../node_modules/@mui/utils/esm/deepmerge/deepmerge.js","../../node_modules/@mui/system/esm/createBreakpoints/createBreakpoints.js","../../node_modules/@mui/system/esm/cssContainerQueries/cssContainerQueries.js","../../node_modules/@mui/system/esm/createTheme/shape.js","../../node_modules/@mui/system/esm/merge/merge.js","../../node_modules/@mui/system/esm/breakpoints/breakpoints.js","../../node_modules/@mui/utils/esm/capitalize/capitalize.js","../../node_modules/@mui/system/esm/style/style.js","../../node_modules/@mui/system/esm/memoize/memoize.js","../../node_modules/@mui/system/esm/spacing/spacing.js","../../node_modules/@mui/system/esm/createTheme/createSpacing.js","../../node_modules/@mui/system/esm/compose/compose.js","../../node_modules/@mui/system/esm/borders/borders.js","../../node_modules/@mui/system/esm/cssGrid/cssGrid.js","../../node_modules/@mui/system/esm/palette/palette.js","../../node_modules/@mui/system/esm/sizing/sizing.js","../../node_modules/@mui/system/esm/styleFunctionSx/defaultSxConfig.js","../../node_modules/@mui/system/esm/styleFunctionSx/styleFunctionSx.js","../../node_modules/@mui/system/esm/createTheme/applyStyles.js","../../node_modules/@mui/system/esm/createTheme/createTheme.js","../../node_modules/@mui/system/esm/useThemeWithoutDefault/useThemeWithoutDefault.js","../../node_modules/@mui/system/esm/useTheme/useTheme.js","../../node_modules/@mui/system/esm/GlobalStyles/GlobalStyles.js","../../node_modules/@mui/system/esm/styleFunctionSx/extendSxProp.js","../../node_modules/@mui/utils/esm/ClassNameGenerator/ClassNameGenerator.js","../../node_modules/clsx/dist/clsx.mjs","../../node_modules/@mui/system/esm/createBox/createBox.js","../../node_modules/@mui/utils/esm/generateUtilityClass/generateUtilityClass.js","../../node_modules/@mui/utils/esm/generateUtilityClasses/generateUtilityClasses.js","../../node_modules/@mui/system/esm/preprocessStyles.js","../../node_modules/@mui/system/esm/createStyled/createStyled.js","../../node_modules/@mui/system/esm/styled/styled.js","../../node_modules/@mui/utils/esm/resolveProps/resolveProps.js","../../node_modules/@mui/system/esm/useThemeProps/getThemeProps.js","../../node_modules/@mui/system/esm/useThemeProps/useThemeProps.js","../../node_modules/@mui/utils/esm/useEnhancedEffect/useEnhancedEffect.js","../../node_modules/@mui/system/esm/useMediaQuery/useMediaQuery.js","../../node_modules/@mui/utils/esm/clamp/clamp.js","../../node_modules/@mui/system/esm/colorManipulator/colorManipulator.js","../../node_modules/@mui/utils/esm/createChainedFunction/createChainedFunction.js","../../node_modules/@mui/utils/esm/debLine truncated
@@ -0,0 +1 @@
{"version":3,"file":"index-V91xgc4E.js","sources":["../../node_modules/react/cjs/react.production.min.js","../../node_modules/react/index.js","../../node_modules/react/cjs/react-jsx-runtime.production.min.js","../../node_modules/react/jsx-runtime.js","../../node_modules/scheduler/cjs/scheduler.production.min.js","../../node_modules/scheduler/index.js","../../node_modules/react-dom/cjs/react-dom.production.min.js","../../node_modules/react-dom/index.js","../../node_modules/react-dom/client.js","../../node_modules/@mui/utils/node_modules/react-is/cjs/react-is.production.js","../../node_modules/@mui/utils/node_modules/react-is/index.js","../../node_modules/@mui/utils/esm/deepmerge/deepmerge.js","../../node_modules/@mui/utils/esm/formatMuiErrorMessage/formatMuiErrorMessage.js","../../node_modules/@mui/utils/esm/capitalize/capitalize.js","../../node_modules/@mui/utils/esm/createChainedFunction/createChainedFunction.js","../../node_modules/@mui/utils/esm/debounce/debounce.js","../../node_modules/@mui/utils/esm/isMuiElement/isMuiElement.js","../../node_modules/@mui/utils/esm/ownerDocument/ownerDocument.js","../../node_modules/@mui/utils/esm/ownerWindow/ownerWindow.js","../../node_modules/@mui/utils/esm/setRef/setRef.js","../../node_modules/@mui/utils/esm/useEnhancedEffect/useEnhancedEffect.js","../../node_modules/@mui/utils/esm/useId/useId.js","../../node_modules/@mui/utils/esm/useControlled/useControlled.js","../../node_modules/@mui/utils/esm/useEventCallback/useEventCallback.js","../../node_modules/@mui/utils/esm/useForkRef/useForkRef.js","../../node_modules/@mui/utils/esm/useLazyRef/useLazyRef.js","../../node_modules/@mui/utils/esm/useOnMount/useOnMount.js","../../node_modules/@mui/utils/esm/useTimeout/useTimeout.js","../../node_modules/@mui/utils/esm/isFocusVisible/isFocusVisible.js","../../node_modules/@mui/utils/esm/getScrollbarSize/getScrollbarSize.js","../../node_modules/@mui/utils/esm/getValidReactChildren/getValidReactChildren.js","../../node_modules/@mui/utils/esm/visuallyHidden/visuallyHidden.js","../../node_modules/@mui/utils/esm/resolveProps/resolveProps.js","../../node_modules/@mui/utils/esm/composeClasses/composeClasses.js","../../node_modules/@mui/utils/esm/ClassNameGenerator/ClassNameGenerator.js","../../node_modules/@mui/utils/esm/generateUtilityClass/generateUtilityClass.js","../../node_modules/@mui/utils/esm/generateUtilityClasses/generateUtilityClasses.js","../../node_modules/@mui/utils/esm/clamp/clamp.js","../../node_modules/@mui/utils/esm/isHostComponent/isHostComponent.js","../../node_modules/@mui/utils/esm/appendOwnerState/appendOwnerState.js","../../node_modules/clsx/dist/clsx.mjs","../../node_modules/@mui/utils/esm/extractEventHandlers/extractEventHandlers.js","../../node_modules/@mui/utils/esm/omitEventHandlers/omitEventHandlers.js","../../node_modules/@mui/utils/esm/mergeSlotProps/mergeSlotProps.js","../../node_modules/@mui/utils/esm/resolveComponentProps/resolveComponentProps.js","../../node_modules/@mui/utils/esm/useSlotProps/useSlotProps.js","../../node_modules/@mui/utils/esm/getReactElementRef/getReactElementRef.js","../../node_modules/@mui/system/esm/merge/merge.js","../../node_modules/@mui/system/esm/cssContainerQueries/cssContainerQueries.js","../../node_modules/@mui/system/esm/breakpoints/breakpoints.js","../../node_modules/@mui/system/esm/style/style.js","../../node_modules/@mui/system/esm/memoize/memoize.js","../../node_modules/@mui/system/esm/spacing/spacing.js","../../node_modules/@mui/system/esm/compose/compose.js","../../node_modules/@mui/system/esm/borders/borders.js","../../node_modules/@mui/system/esm/cssGrid/cssGrid.js","../../node_modules/@mui/system/esm/palette/palette.js","../../node_modules/@mui/system/esm/sizing/sizing.js","../../node_modules/@mui/system/esm/styleFunctionSx/defaultSxConfig.js","../../node_modules/@mui/system/esm/styleFunctionSx/styleFunctionSx.js","../../node_modules/@mui/system/esm/styleFunctionSx/extendSxProp.js","../../node_modules/@babel/runtime/helpers/esm/extends.js","../../node_modules/@emotion/sheet/dist/emotion-sheet.esm.js","../../node_modules/stylis/src/Enum.js","../../node_modules/stylis/src/Utility.js","../../node_modules/stylis/src/Tokenizer.js","../../node_modules/stylis/src/Parser.js","../../node_modules/stylis/src/Serializer.js","../../node_modules/stylis/src/Middleware.js","../../node_modules/@emotion/memoize/dist/emotion-memoize.esm.js","../../node_modules/@emotion/cache/dist/emotion-cache.browser.esm.js","../../node_modules/react-is/cjs/react-is.production.min.js","../../node_modules/react-is/index.js","../../node_modules/hoist-non-react-statics/dist/hoist-non-react-statics.cjs.js","../../node_modules/@emotion/utils/dist/emotion-utils.browser.esm.js","../../node_modules/@emotion/hash/dist/emotion-hash.esm.js","../../node_modules/@emotion/unitless/dist/emotion-unitless.esm.js","../../node_modules/@emotion/serialize/dist/emotion-serialize.esm.js","../../node_modules/@emotion/use-insertion-effect-with-fallbacks/dist/emotion-useLine truncated
@@ -2,6 +2,10 @@
<html lang="en">
<head>
<meta charset="UTF-8" />
<!-- Load Inter font to match karaoke-gen styling -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&display=swap" rel="stylesheet">
<link rel="icon" type="image/x-icon" href="/favicon.ico" />
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
@@ -9,8 +13,8 @@
<link rel="icon" type="image/png" sizes="192x192" href="/android-chrome-192x192.png" />
<link rel="icon" type="image/png" sizes="512x512" href="/android-chrome-512x512.png" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Nomad Karaoke: Lyrics Review</title>
<script type="module" crossorigin src="/assets/index-BECn1o8Q.js"></script>
<title>Nomad Karaoke: Lyrics Transcription Review</title>
<script type="module" crossorigin src="/assets/index-V91xgc4E.js"></script>
</head>
<body>
<div id="root"></div>
@@ -0,0 +1,5 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" version="1.1" viewBox="45 60 255 135">
<g data-v-70b83f88="" fill="#ff7acc" class="basesvg" transform="translate(48.368003845214844,62.62239074707031)"><g fill-rule="" class="tp-name" transform="translate(0,0)"><g transform="scale(1.4000000000000004)"><g stroke="#ff7acc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" paint-order="stroke" data-gra-attr="stroke" fill-opacity="0"><path d="M1.4-33.44L1.4 0C31.48 0 3.96 0 34.04 0L34.04-32.64 17.81-32.64 17.81-20.61 1.4-33.44ZM52.21-33.76C42.6-33.76 34.81-25.97 34.81-16.37 34.81-6.76 42.6 1.03 52.21 1.03 61.81 1.03 69.6-6.76 69.6-16.37 69.6-25.97 61.81-33.76 52.21-33.76ZM70.7 0L103.34 0 103.25-33.9 86.93-21.54 70.61-33.9 70.7 0ZM102.34 0C138.2 0 107.52 0 143.56 0L122.95-34.93 102.34 0ZM158.84 0.05C167.84 0.05 175.16-7.27 175.16-16.27 175.16-25.27 167.84-32.64 158.84-32.64L142.52-32.64C142.52 0.75 142.52-33.34 142.52 0.05L158.84 0.05Z" transform="translate(-1.399999976158142, 34.93000030517578)"></path></g><g transform="translate(0,38.959999084472656)"><g stroke="#ff7acc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" paint-order="stroke" data-gra-attr="stroke" fill="#ff7acc" fill-opacity="0" transform="scale(1.02719)"><path d="M24.43 0L17.67 0 7.98-11.87 7.98 0 2.63 0 2.63-26.64 7.98-26.64 7.98-14.69 17.67-26.64 24.12-26.64 13.13-13.44 24.43 0ZM45.03 0L43.27-5.08 32.66-5.08 30.91 0 25.3 0 34.88-26.68 41.1-26.68 50.68 0 45.03 0ZM34.11-9.35L41.82-9.35 37.97-20.5 34.11-9.35ZM73.19 0L67.01 0 61.13-10.38 58.61-10.38 58.61 0 53.27 0 53.27-26.64 63.27-26.64Q66.36-26.64 68.53-25.55 70.71-24.47 71.8-22.61 72.89-20.76 72.89-18.47L72.89-18.47Q72.89-15.84 71.36-13.72 69.83-11.6 66.82-10.8L66.82-10.8 73.19 0ZM58.61-22.21L58.61-14.39 63.08-14.39Q65.25-14.39 66.32-15.44 67.39-16.49 67.39-18.36L67.39-18.36Q67.39-20.19 66.32-21.2 65.25-22.21 63.08-22.21L63.08-22.21 58.61-22.21ZM94.82 0L93.07-5.08 82.46-5.08 80.7 0 75.09 0 84.67-26.68 90.89-26.68 100.47 0 94.82 0ZM83.91-9.35L91.62-9.35 87.76-20.5 83.91-9.35ZM115.43 0.27Q111.69 0.27 108.56-1.49 105.43-3.24 103.6-6.35 101.76-9.47 101.76-13.4L101.76-13.4Q101.76-17.29 103.6-20.4 105.43-23.51 108.56-25.27 111.69-27.02 115.43-27.02L115.43-27.02Q119.21-27.02 122.32-25.27 125.43-23.51 127.24-20.4 129.05-17.29 129.05-13.4L129.05-13.4Q129.05-9.47 127.24-6.35 125.43-3.24 122.3-1.49 119.17 0.27 115.43 0.27L115.43 0.27ZM115.43-4.5Q117.83-4.5 119.66-5.59 121.5-6.68 122.53-8.7 123.56-10.73 123.56-13.4L123.56-13.4Q123.56-16.07 122.53-18.07 121.5-20.08 119.66-21.15 117.83-22.21 115.43-22.21L115.43-22.21Q113.02-22.21 111.17-21.15 109.32-20.08 108.29-18.07 107.26-16.07 107.26-13.4L107.26-13.4Q107.26-10.73 108.29-8.7 109.32-6.68 111.17-5.59 113.02-4.5 115.43-4.5L115.43-4.5ZM153.82 0L147.06 0 137.37-11.87 137.37 0 132.02 0 132.02-26.64 137.37-26.64 137.37-14.69 147.06-26.64 153.51-26.64 142.52-13.44 153.82 0ZM171.79-22.33L161.67-22.33 161.67-15.65 170.64-15.65 170.64-11.41 161.67-11.41 161.67-4.35 171.79-4.35 171.79 0 156.33 0 156.33-26.68 171.79-26.68 171.79-22.33Z" transform="translate(-2.630000114440918, 27.020000457763672)"></path></g></g></g></g> <g data-gra="path-slogan" fill-rule="" class="tp-slogan" fill="#ffdf6b" transform="translate(5,111.788818359375)"><rect x="0" height="1" y="5.9832000732421875" width="14.278396606445312"></rect> <rect height="1" y="5.9832000732421875" width="14.278396606445312" x="218.985595703125"></rect> <g transform="translate(17.278396606445312,0)"><g transform="scale(1.2800000000000002)"><path d="M10.21-8.38L12.02-8.38L9.68 0L7.70 0L6.13-5.96L4.49 0L2.52 0.01L0.26-8.38L2.06-8.38L3.54-1.87L5.24-8.38L7.12-8.38L8.72-1.91L10.21-8.38ZM18.38-8.38L20.06-8.38L20.06 0L18.38 0L18.38-3.56L14.80-3.56L14.80 0L13.12 0L13.12-8.38L14.80-8.38L14.80-4.93L18.38-4.93L18.38-8.38ZM26.58-7.02L23.40-7.02L23.40-4.92L26.22-4.92L26.22-3.59L23.40-3.59L23.40-1.37L26.58-1.37L26.58 0L21.72 0L21.72-8.39L26.58-8.39L26.58-7.02ZM34.37 0L32.42 0L30.58-3.26L29.78-3.26L29.78 0L28.10 0L28.10-8.38L31.25-8.38Q32.22-8.38 32.90-8.03Q33.59-7.69 33.93-7.11Q34.27-6.53 34.27-5.81L34.27-5.81Q34.27-4.98 33.79-4.31Q33.31-3.65 32.36-3.40L32.36-3.40L34.37 0ZM29.78-6.98L29.78-4.52L31.19-4.52Q31.87-4.52 32.21-4.85Q32.54-5.18 32.54-5.77L32.54-5.77Q32.54-6.35 32.21-6.67Q31.87-6.98 31.19-6.98L31.19-6.98L29.78-6.98ZM40.66-7.02L37.48-7.02L37.48-4.92L40.30-4.92L40.30-3.59L37.48-3.59L37.48-1.37L40.66-1.37L40.66 0L35.80 0L35.80-8.39L40.66-8.39L40.66-7.02ZM47.92-8.38L49.70-8.38L46.63 0L44.59 0L41.52-8.38L43.32-8.38L45.62-1.72L47.92-8.38ZM55.57-7.02L52.39-7.02L52.39-4.92L55.21-4.92L55.21-3.59L52.39-3.59L52.39-1.37L55.57-1.37L55.57 0L50.71 0L50.71-8.39L55.57-8.39L55.57-7.02ZM63.36 0L61.42 0L59.57-3.26L58.78-3.26L58.78 0L57.10 0L57.10-8.38L60.24-8.38Q61.21-8.38 61.90-8.03Q62.58-7.69 62.92-7.11Q63.26-6.53 63.26-5.81L63.26-5.81Q63.26-4.98 62.78-4.31Q62.30-3.65 61.36-3.40L61.36-3.40L63.36 0ZM58.78-6.98L58.78-4.52L60.18-4.52Q60.86-4.52 61.20-4.85Q61.54-5.18 61.54-5.77L61.54-5.77Q61.54-6.35 61.20-6.67Q60.86-6.98 60.18-6.98L60.18-6.98L58.78-6.98ZM72.43-8.38L74.30-8.38L71.47-2.92L71.47 0L69.Line truncated
</svg>

After

Width:  |  Height:  |  Size: 9.0 KiB

@@ -148,8 +148,21 @@ class OutputGenerator:
audio_filepath: str,
artist: Optional[str] = None,
title: Optional[str] = None,
ass_only: bool = False,
) -> OutputPaths:
"""Generate all requested output formats."""
"""Generate all requested output formats.
Args:
transcription_corrected: Corrected transcription data
lyrics_results: Lyrics data from various providers
output_prefix: Prefix for output filenames
audio_filepath: Path to audio file
artist: Optional artist name
title: Optional title
ass_only: If True (only in preview_mode), generate only ASS subtitles
without video encoding. Useful when video encoding is offloaded
to an external service.
"""
outputs = OutputPaths()
try:
@@ -165,8 +178,9 @@ class OutputGenerator:
# Generate ASS subtitles for preview
outputs.ass = self.subtitle.generate_ass(transcription_corrected.resized_segments, output_prefix, audio_filepath)
# Generate preview video
outputs.video = self.video.generate_preview_video(outputs.ass, audio_filepath, output_prefix)
# Generate preview video (unless ass_only mode for GCE encoding)
if not ass_only:
outputs.video = self.video.generate_preview_video(outputs.ass, audio_filepath, output_prefix)
return outputs
@@ -249,6 +249,9 @@ class VideoGenerator:
def generate_preview_video(self, ass_path: str, audio_path: str, output_prefix: str) -> str:
"""Generate lower resolution MP4 preview video with lyrics overlay.
Delegates to LocalPreviewEncodingService for the actual encoding to ensure
consistent preview generation across local CLI, Cloud Run, and GCE worker.
Args:
ass_path: Path to ASS subtitles file
audio_path: Path to audio file
@@ -257,6 +260,11 @@ class VideoGenerator:
Returns:
Path to generated preview video file
"""
from backend.services.local_preview_encoding_service import (
LocalPreviewEncodingService,
PreviewEncodingConfig,
)
self.logger.info("Generating preview video with lyrics overlay")
output_path = os.path.join(self.cache_dir, f"{output_prefix}_preview.mp4")
@@ -269,17 +277,36 @@ class VideoGenerator:
try:
# Create a temporary copy of the ASS file with a unique filename
import time
import shutil
safe_prefix = "".join(c if c.isalnum() else "_" for c in output_prefix)
timestamp = int(time.time() * 1000)
temp_ass_path = os.path.join(self.cache_dir, f"temp_preview_subtitles_{safe_prefix}_{timestamp}.ass")
import shutil
shutil.copy2(ass_path, temp_ass_path)
self.logger.debug(f"Created temporary ASS file: {temp_ass_path}")
cmd = self._build_preview_ffmpeg_command(temp_ass_path, audio_path, output_path)
self._run_ffmpeg_command(cmd)
# Get font path from styles configuration
karaoke_styles = self.styles.get("karaoke", {})
font_path = karaoke_styles.get("font_path")
# Build config for LocalPreviewEncodingService
config = PreviewEncodingConfig(
ass_path=temp_ass_path,
audio_path=audio_path,
output_path=output_path,
background_image_path=self.background_image,
background_color=self.background_color,
font_path=font_path if font_path and os.path.isfile(font_path) else None,
)
# Use LocalPreviewEncodingService for consistent encoding
service = LocalPreviewEncodingService(logger=self.logger)
result = service.encode_preview(config)
if not result.success:
raise RuntimeError(f"Preview encoding failed: {result.error}")
self.logger.info(f"Preview video generated: {output_path}")
# Clean up temporary file
@@ -352,19 +379,44 @@ class VideoGenerator:
self.logger.error(f"Failed to resize background image: {e.output}")
raise
def _escape_ffmpeg_filter_path(self, path: str) -> str:
"""Escape a path for FFmpeg filter expressions (for subprocess without shell).
When using subprocess with a command list (no shell), FFmpeg receives the
filter string directly. FFmpeg's filter parser requires escaping:
- Backslashes: double them (\ -> \\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\')
- Spaces: escape with backslash ( -> \ )
Note: This is different from shell escaping. The '\\'\\''' pattern used for
shell escaping does NOT work when subprocess passes args directly to FFmpeg.
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
"""
# First escape existing backslashes (\ -> \\)
escaped = path.replace("\\", "\\\\")
# Escape single quotes (' -> \\')
# In the actual string we need 3 backslashes before the quote
escaped = escaped.replace("'", "\\\\\\'")
# Escape spaces
escaped = escaped.replace(" ", "\\ ")
return escaped
def _build_ass_filter(self, ass_path: str) -> str:
"""Build ASS filter with font directory support."""
ass_filter = f"ass={ass_path}"
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
ass_filter = f"ass={escaped_ass_path}"
# Get font path from styles configuration
karaoke_styles = self.styles.get("karaoke", {})
font_path = karaoke_styles.get("font_path")
if font_path and os.path.isfile(font_path):
font_dir = os.path.dirname(font_path)
ass_filter += f":fontsdir={font_dir}"
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
ass_filter += f":fontsdir={escaped_font_dir}"
self.logger.info(f"Returning ASS filter with fonts dir: {ass_filter}")
return ass_filter
def _build_ffmpeg_command(self, ass_path: str, audio_path: str, output_path: str) -> List[str]:
@@ -439,93 +491,6 @@ class VideoGenerator:
return cmd
def _build_preview_ffmpeg_command(self, ass_path: str, audio_path: str, output_path: str) -> List[str]:
"""Build FFmpeg command for preview video generation with hardware acceleration when available."""
# Use even lower resolution for preview (480x270 instead of 640x360 for faster encoding)
width, height = 480, 270
cmd = [
"ffmpeg",
"-hide_banner",
"-loglevel", "error",
"-r", "24", # Reduced frame rate to 24 fps for faster encoding
]
# Add hardware acceleration flags if available
cmd.extend(self.hwaccel_flags)
# Input source (background) - simplified for preview
if self.background_image:
# For preview, use the original image without resizing to save time
self.logger.debug(f"Using original background image for preview: {self.background_image}")
cmd.extend([
"-loop", "1", # Loop the image
"-i", self.background_image,
])
# Build video filter with scaling and ASS subtitles
video_filter = f"scale={width}:{height}:force_original_aspect_ratio=decrease,pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,{self._build_ass_filter(ass_path)}"
else:
self.logger.debug(
f"Using solid {self.background_color} background "
f"with resolution: {width}x{height}"
)
cmd.extend([
"-f", "lavfi",
"-i", f"color=c={self.background_color}:s={width}x{height}:r=24",
])
# Build video filter with just ASS subtitles (no scaling needed)
video_filter = self._build_ass_filter(ass_path)
cmd.extend([
"-i", audio_path,
"-vf", video_filter, # Apply the video filter
"-c:a", "aac", # Use AAC for audio compatibility
"-b:a", "96k", # Reduced audio bitrate for faster encoding
"-c:v", self.video_encoder,
])
# Add encoder-specific settings for preview with maximum speed priority
if self.nvenc_available:
# NVENC settings optimized for maximum speed
cmd.extend([
"-preset", "p1", # Fastest NVENC preset
"-tune", "ll", # Low latency
"-rc", "cbr", # Constant bitrate for speed
"-b:v", "800k", # Lower bitrate for speed
"-profile:v", "baseline", # Most compatible profile
"-level", "3.1", # Lower level for speed
])
self.logger.debug("Using NVENC encoding with maximum speed settings for preview video generation")
else:
# Software encoding with maximum speed priority
cmd.extend([
"-profile:v", "baseline", # Most compatible H.264 profile
"-level", "3.0", # Compatibility level
"-preset", "superfast", # Even faster than ultrafast for preview
"-tune", "fastdecode", # Optimize for fast decoding
"-b:v", "600k", # Lower base bitrate for speed
"-maxrate", "800k", # Lower max bitrate
"-bufsize", "1200k", # Smaller buffer size
"-crf", "28", # Higher CRF for faster encoding (lower quality but faster)
])
self.logger.debug("Using software encoding with maximum speed settings for preview video generation")
cmd.extend([
"-pix_fmt", "yuv420p", # Required for browser compatibility
"-movflags", "+faststart+frag_keyframe+empty_moov+dash", # Enhanced streaming with dash for faster start
"-g", "48", # Keyframe every 48 frames (2 seconds at 24fps) - fewer keyframes for speed
"-keyint_min", "48", # Minimum keyframe interval
"-sc_threshold", "0", # Disable scene change detection for speed
"-threads", "0", # Use all available CPU threads
"-shortest", # End encoding after shortest stream
"-y" # Overwrite output without asking
])
# Add output path
cmd.append(output_path)
return cmd
def _get_video_codec(self) -> str:
"""Determine the best available video codec (legacy method - use video_encoder instead)."""
# This method is kept for backwards compatibility but is deprecated
@@ -210,6 +210,7 @@ def test_build_ass_filter_no_font_path(video_generator):
"""Test _build_ass_filter with no font path configured."""
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping needed
assert result == "ass=test.ass"
@@ -220,13 +221,16 @@ def test_build_ass_filter_with_valid_font_path(video_generator, tmp_path):
font_dir.mkdir()
font_file = font_dir / "test.ttf"
font_file.touch()
# Update styles to include font path
video_generator.styles["karaoke"]["font_path"] = str(font_file)
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
expected = f"ass=test.ass:fontsdir={str(font_dir)}"
# Paths with no special chars don't need escaping
# But font_dir path from tmp_path may have spaces on some systems
escaped_font_dir = str(font_dir).replace("\\", "\\\\").replace("'", "\\\\'").replace(" ", "\\ ")
expected = f"ass=test.ass:fontsdir={escaped_font_dir}"
assert result == expected
@@ -234,9 +238,10 @@ def test_build_ass_filter_with_invalid_font_path(video_generator):
"""Test _build_ass_filter with invalid font path (file doesn't exist)."""
# Set non-existent font path
video_generator.styles["karaoke"]["font_path"] = "/nonexistent/font.ttf"
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -244,9 +249,10 @@ def test_build_ass_filter_with_empty_font_path(video_generator):
"""Test _build_ass_filter with empty font path."""
# Set empty font path
video_generator.styles["karaoke"]["font_path"] = ""
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -254,9 +260,10 @@ def test_build_ass_filter_with_none_font_path(video_generator):
"""Test _build_ass_filter with None font path."""
# Set None font path
video_generator.styles["karaoke"]["font_path"] = None
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -269,9 +276,10 @@ def test_build_ass_filter_no_karaoke_section(tmp_path):
video_resolution=(1920, 1080),
styles=styles,
)
ass_path = "test.ass"
result = generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -279,19 +287,73 @@ def test_build_ass_filter_logging(video_generator, tmp_path, caplog):
"""Test _build_ass_filter logging behavior."""
# Set log level to INFO to capture the logs
caplog.set_level(logging.INFO)
# Create a test font file
font_dir = tmp_path / "fonts"
font_dir.mkdir()
font_file = font_dir / "test.ttf"
font_file.touch()
# Update styles to include font path
video_generator.styles["karaoke"]["font_path"] = str(font_file)
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# Check that info log was generated
assert "Returning ASS filter with fonts dir:" in caplog.text
assert str(font_dir) in caplog.text
def test_escape_ffmpeg_filter_path(video_generator):
"""Test _escape_ffmpeg_filter_path uses FFmpeg filter escaping (not shell escaping).
FFmpeg filter escaping rules (for subprocess without shell):
- Backslashes: double them (\ -> \\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\')
- Spaces: escape with backslash ( -> \ )
"""
# Test simple path without special chars (no escaping needed)
assert video_generator._escape_ffmpeg_filter_path("/path/to/file.ass") == "/path/to/file.ass"
# Test apostrophe escaping (fixes paths like "I'm With You")
# Apostrophe becomes \\\' (three backslashes + quote) in the actual string
result = video_generator._escape_ffmpeg_filter_path("./Avril Lavigne - I'm With You/file.ass")
assert result == "./Avril\\ Lavigne\\ -\\ I\\\\\\'m\\ With\\ You/file.ass"
# Test path with spaces (escaped with backslash)
assert video_generator._escape_ffmpeg_filter_path("/path/with spaces/file.ass") == "/path/with\\ spaces/file.ass"
# Test path with colons (no escaping needed - colon is only special in filter options)
assert video_generator._escape_ffmpeg_filter_path("C:/path/file.ass") == "C:/path/file.ass"
# Test multiple apostrophes
result = video_generator._escape_ffmpeg_filter_path("It's Bob's file.ass")
assert result == "It\\\\\\'s\\ Bob\\\\\\'s\\ file.ass"
def test_build_ass_filter_escapes_path_with_apostrophe(video_generator):
"""Test _build_ass_filter properly escapes paths containing apostrophes."""
ass_path = "./Avril Lavigne - I'm With You/cache/temp.ass"
result = video_generator._build_ass_filter(ass_path)
# Spaces escaped with backslash, apostrophe escaped with \\\'
assert result == "ass=./Avril\\ Lavigne\\ -\\ I\\\\\\'m\\ With\\ You/cache/temp.ass"
def test_build_ass_filter_escapes_font_dir_with_special_chars(video_generator, tmp_path):
"""Test _build_ass_filter properly escapes font directory with special characters."""
# Create a font directory with special characters
font_dir = tmp_path / "It's a Font Dir"
font_dir.mkdir()
font_file = font_dir / "test.ttf"
font_file.touch()
# Update styles to include font path
video_generator.styles["karaoke"]["font_path"] = str(font_file)
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# Should have fontsdir with escaped path
assert "fontsdir=" in result
# Apostrophe should be escaped using \\\' pattern (3 backslashes + quote)
assert "\\\\\\'s" in result
Generated
+467 -454
View File
File diff suppressed because it is too large. Load diff
+3 -2
View File
@@ -1,13 +1,14 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.90.0"
version = "0.95.2"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
readme = "README.md"
packages = [
{ include = "karaoke_gen" },
{ include = "lyrics_transcriber", from = "lyrics_transcriber_temp" }
{ include = "lyrics_transcriber", from = "lyrics_transcriber_temp" },
{ include = "backend" }
]
include = [
"karaoke_gen/instrumental_review/static/*",
+41
View File
@@ -0,0 +1,41 @@
#!/bin/bash
# Admin API helper script
# Usage: ./scripts/admin_api.sh <endpoint> [method] [data]
#
# Examples:
# ./scripts/admin_api.sh /api/jobs # List all jobs (admin sees all)
# ./scripts/admin_api.sh /api/jobs?limit=10 # List 10 jobs
# ./scripts/admin_api.sh /api/jobs/abc123 # Get specific job
# ./scripts/admin_api.sh /api/jobs/abc123/retry POST # Retry a job
# ./scripts/admin_api.sh /api/jobs/abc123/select-instrumental POST '{"selection":"clean"}'
set -e
API_URL="${KARAOKE_API_URL:-https://api.nomadkaraoke.com}"
# Get admin token from Secret Manager if not set
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke 2>/dev/null)
fi
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
echo "Error: Could not get admin token. Run 'gcloud auth login' first." >&2
exit 1
fi
ENDPOINT="${1:-/api/admin/jobs}"
METHOD="${2:-GET}"
DATA="$3"
if [ "$METHOD" = "GET" ]; then
curl -s "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
elif [ -n "$DATA" ]; then
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d "$DATA" | jq .
else
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
fi
+34 -24
View File
@@ -121,16 +121,25 @@ class TestVideoBackgroundProcessor:
processor.get_audio_duration("/path/to/audio.wav")
def test_escape_filter_path(self, processor):
"""Test path escaping for ffmpeg filters."""
# Test backslash escaping
assert processor.escape_filter_path("C:\\Users\\test\\file.ass") == "C\\:\\\\Users\\\\test\\\\file.ass"
# Test colon escaping
assert processor.escape_filter_path("/path/to/file.ass") == "/path/to/file.ass"
assert processor.escape_filter_path("C:/path/file.ass") == "C\\:/path/file.ass"
# Test combined escaping
assert processor.escape_filter_path("C:\\path:with:colons\\file.ass") == "C\\:\\\\path\\:with\\:colons\\\\file.ass"
"""Test path escaping for ffmpeg filters using single-quote wrapping."""
# Test simple path (wrapped in single quotes)
assert processor.escape_filter_path("/path/to/file.ass") == "'/path/to/file.ass'"
# Test path with spaces (protected by single quotes)
assert processor.escape_filter_path("/path/with spaces/file.ass") == "'/path/with spaces/file.ass'"
# Test apostrophe escaping (fixes paths like "I'm With You")
# Single quote becomes '\'' (end quote, escaped quote, start quote)
assert processor.escape_filter_path("./Avril Lavigne - I'm With You/file.ass") == "'./Avril Lavigne - I'\\''m With You/file.ass'"
# Test path with colons (protected by single quotes)
assert processor.escape_filter_path("C:/path/file.ass") == "'C:/path/file.ass'"
# Test Windows-style path (backslashes preserved, wrapped in quotes)
assert processor.escape_filter_path("C:\\Users\\test\\file.ass") == "'C:\\Users\\test\\file.ass'"
# Test multiple apostrophes
assert processor.escape_filter_path("It's Bob's file.ass") == "'It'\\''s Bob'\\''s file.ass'"
def test_build_video_filter_no_darkness(self, processor):
"""Test video filter building without darkness overlay."""
@@ -139,13 +148,13 @@ class TestVideoBackgroundProcessor:
darkness_percent=0,
fonts_dir=None
)
# Should contain scale and crop
assert "scale=w=3840:h=2160:force_original_aspect_ratio=increase,crop=3840:2160" in vf_filter
# Should contain ASS filter
assert "ass=/path/to/subs.ass" in vf_filter
# Should contain ASS filter with single-quoted path
assert "ass='/path/to/subs.ass'" in vf_filter
# Should NOT contain darkening
assert "drawbox" not in vf_filter
@@ -156,10 +165,10 @@ class TestVideoBackgroundProcessor:
darkness_percent=50,
fonts_dir=None
)
# Should contain darkening with correct alpha
assert "drawbox=x=0:y=0:w=iw:h=ih:color=black@0.50:t=fill" in vf_filter
# Should contain ASS filter after darkening
assert vf_filter.index("drawbox") < vf_filter.index("ass=")
@@ -171,20 +180,21 @@ class TestVideoBackgroundProcessor:
darkness_percent=0,
fonts_dir="/path/to/fonts"
)
# Should contain fonts directory in ASS filter
assert "fontsdir=/path/to/fonts" in vf_filter
# Should contain fonts directory in ASS filter (single-quoted)
assert "fontsdir='/path/to/fonts'" in vf_filter
def test_build_video_filter_path_escaping(self, processor):
"""Test video filter building with special characters in paths."""
# Test with apostrophe in path
vf_filter = processor.build_video_filter(
ass_subtitles_path="C:\\Users\\test\\subs.ass",
ass_subtitles_path="./Avril Lavigne - I'm With You/subs.ass",
darkness_percent=0,
fonts_dir=None
)
# Paths should be escaped
assert "C\\:\\\\Users\\\\test\\\\subs.ass" in vf_filter
# Paths should be single-quoted with apostrophes escaped as '\''
assert "'./Avril Lavigne - I'\\''m With You/subs.ass'" in vf_filter
@patch('subprocess.run')
def test_execute_command_with_fallback_gpu_success(self, mock_run, processor_with_nvenc, mock_logger):