mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-21 21:38:07 +09:00
Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
84f73ac099 | ||
|
|
f3d4214566 | ||
|
|
d7e856acbd | ||
|
|
4b3166d75f |
No files matched your search
@@ -169,6 +169,43 @@ karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses
|
||||
|
||||
**Note**: The `users` collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
|
||||
|
||||
## Video Worker Orchestrator
|
||||
|
||||
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ VideoWorkerOrchestrator │
|
||||
│ │
|
||||
│ Stage 1: PACKAGING Stage 2: ENCODING │
|
||||
│ ├─ CDG file generation ├─ Local (FFmpeg) │
|
||||
│ └─ TXT lyrics export └─ GCE (remote VM) │
|
||||
│ │
|
||||
│ Stage 3: ORGANIZATION Stage 4: DISTRIBUTION │
|
||||
│ ├─ Final path setup ├─ Dropbox upload │
|
||||
│ └─ File organization ├─ Google Drive upload │
|
||||
│ └─ YouTube upload │
|
||||
│ │
|
||||
│ Stage 5: NOTIFICATIONS │
|
||||
│ └─ Discord webhooks │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Key Design Decision**: The orchestrator coordinates stages rather than embedding logic in monolithic functions. This:
|
||||
- Ensures all features work with both local and GCE encoding
|
||||
- Makes each stage independently testable (139 new tests)
|
||||
- Allows feature flag rollback via `USE_NEW_ORCHESTRATOR` env var
|
||||
|
||||
**Service Modules**:
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `youtube_upload_service.py` | YouTube OAuth flow and upload |
|
||||
| `discord_service.py` | Discord webhook notifications |
|
||||
| `packaging_service.py` | CDG/TXT package generation |
|
||||
| `local_encoding_service.py` | FFmpeg encoding (Cloud Run) |
|
||||
| `encoding_interface.py` | Abstract interface + GCE backend |
|
||||
| `video_worker_orchestrator.py` | Stage coordination |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Backend**: FastAPI, Python 3.12, Cloud Run
|
||||
|
||||
@@ -1002,6 +1002,42 @@ async def upload(auth_result: AuthResult = Depends(require_auth)):
|
||||
|
||||
See `docs/archive/2024-12-31-job-failure-investigation.md` for full details.
|
||||
|
||||
### Alternative Code Paths Must Implement All Features
|
||||
|
||||
**Problem**: Adding a GCE encoding option created a code path that completely bypassed the existing `KaraokeFinalise` class, which contained YouTube upload, Discord notifications, and CDG/TXT packaging. Jobs using GCE encoding silently missed these features.
|
||||
|
||||
```python
|
||||
# BAD - two divergent code paths
|
||||
if use_gce_encoding:
|
||||
await gce_encoding_worker.encode(job_id) # Returns, does nothing else
|
||||
return # YouTube upload, Discord, CDG never happen!
|
||||
else:
|
||||
finalise = KaraokeFinalise(...) # Has all the features
|
||||
await finalise.run()
|
||||
```
|
||||
|
||||
**Symptoms**:
|
||||
- Jobs complete successfully but YouTube URL is never set
|
||||
- No Discord notifications for GCE-encoded jobs
|
||||
- CDG/TXT files never generated for GCE jobs
|
||||
- No errors in logs (the code path simply doesn't include those features)
|
||||
|
||||
**Solution**: Create an orchestrator that runs all stages regardless of encoding backend:
|
||||
|
||||
```python
|
||||
# GOOD - unified pipeline with swappable encoding
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
result = await orchestrator.run()
|
||||
# Orchestrator always runs: packaging → encoding → organization → distribution → notifications
|
||||
# The encoding step uses either local or GCE backend based on config
|
||||
```
|
||||
|
||||
**Key insight**: When adding an alternative implementation (like GCE encoding), audit ALL side effects of the original code path. The new path must replicate them or delegate to shared code.
|
||||
|
||||
**Architecture pattern**: The orchestrator pattern separates "what stages run" from "how each stage runs". Stages are always executed; only the implementation varies.
|
||||
|
||||
See `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` for full details.
|
||||
|
||||
## What We'd Do Differently
|
||||
|
||||
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
|
||||
|
||||
@@ -36,6 +36,8 @@
|
||||
|
||||
## Recent Changes
|
||||
|
||||
- **Video Worker Orchestrator** (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag `USE_NEW_ORCHESTRATOR` (default: true) enables rollback. 139 new tests across 6 new service modules. See [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator) and [archive/2026-01-04-video-worker-orchestrator-refactor.md](archive/2026-01-04-video-worker-orchestrator-refactor.md).
|
||||
|
||||
- **Full Unicode Font Support** (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fonts-in-docker-for-video-rendering).
|
||||
|
||||
- **Flacfetch Cache API Integration** (2026-01-03): Enhanced audio search cache management to also clear flacfetch's GCS cache. When admins clear a job's cache, both Firestore (job state) and flacfetch (GCS search results) are cleared simultaneously. Added "Clear All Cache" button and cache stats display to admin UI. New endpoints: `DELETE /api/admin/cache`, `GET /api/admin/cache/stats`. See [API.md](API.md#audio-search-management-admin).
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
# Video Worker Orchestrator Refactor - 2026-01-04
|
||||
|
||||
## Summary
|
||||
|
||||
Major refactor to unify video generation pipeline with a new VideoWorkerOrchestrator. This resolves the GCE vs KaraokeFinalise code path divergence that caused features like YouTube upload, Discord notifications, and CDG/TXT packaging to be missing from the GCE encoding path.
|
||||
|
||||
## Problem
|
||||
|
||||
Job 5b78ad46 completed successfully via GCE encoding but had no YouTube URL. Investigation revealed:
|
||||
|
||||
1. The GCE encoding path was a completely separate code branch
|
||||
2. When `USE_GCE_ENCODING=true`, the worker triggered GCE encoding and returned immediately
|
||||
3. This bypassed the entire `KaraokeFinalise` class which contained:
|
||||
- YouTube upload logic
|
||||
- Discord webhook notifications
|
||||
- CDG/TXT package generation
|
||||
- Dropbox/Google Drive uploads
|
||||
|
||||
The root cause wasn't a bug in any individual feature - it was an architectural issue where alternative implementations didn't share common post-processing stages.
|
||||
|
||||
## Solution
|
||||
|
||||
Created a stage-based orchestrator pattern:
|
||||
|
||||
```
|
||||
VideoWorkerOrchestrator
|
||||
├── Stage 1: Packaging (CDG, TXT)
|
||||
├── Stage 2: Encoding (Local or GCE - swappable)
|
||||
├── Stage 3: Organization (file paths, structure)
|
||||
├── Stage 4: Distribution (Dropbox, GDrive, YouTube)
|
||||
└── Stage 5: Notifications (Discord)
|
||||
```
|
||||
|
||||
The orchestrator always runs all stages. Only the encoding implementation varies based on backend selection.
|
||||
|
||||
## Changes
|
||||
|
||||
### New Service Modules
|
||||
|
||||
| File | Lines | Purpose |
|
||||
|------|-------|---------|
|
||||
| `youtube_upload_service.py` | ~445 | YouTube OAuth and upload |
|
||||
| `discord_service.py` | ~172 | Discord webhook notifications |
|
||||
| `packaging_service.py` | ~287 | CDG/TXT package generation |
|
||||
| `local_encoding_service.py` | ~590 | FFmpeg encoding on Cloud Run |
|
||||
| `encoding_interface.py` | ~418 | Abstract interface + GCE backend |
|
||||
| `video_worker_orchestrator.py` | ~628 | Stage coordination |
|
||||
|
||||
### Test Coverage
|
||||
|
||||
| Test File | Tests | Coverage |
|
||||
|-----------|-------|----------|
|
||||
| `test_youtube_upload_service.py` | 25 | OAuth, upload, error handling |
|
||||
| `test_discord_service.py` | 18 | Webhook formatting, delivery |
|
||||
| `test_packaging_service.py` | 22 | CDG/TXT generation |
|
||||
| `test_local_encoding_service.py` | 24 | FFmpeg encoding variants |
|
||||
| `test_encoding_interface.py` | 23 | Interface + GCE backend |
|
||||
| `test_video_worker_orchestrator.py` | 27 | Full pipeline, stages |
|
||||
| **Total** | **139** | |
|
||||
|
||||
### video_worker.py Updates
|
||||
|
||||
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: `true`)
|
||||
- `generate_video()` routes to orchestrator or legacy path
|
||||
- Legacy code preserved as `generate_video_legacy()` for rollback
|
||||
- New `generate_video_orchestrated()` builds config and runs orchestrator
|
||||
|
||||
## Feature Flag
|
||||
|
||||
```bash
|
||||
# Enable new orchestrator (default)
|
||||
USE_NEW_ORCHESTRATOR=true
|
||||
|
||||
# Rollback to legacy behavior
|
||||
USE_NEW_ORCHESTRATOR=false
|
||||
```
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. **Stage-based orchestration**: Each stage is independent and testable
|
||||
2. **Strategy pattern for encoding**: `EncodingBackend` interface with local and GCE implementations
|
||||
3. **Lazy service loading**: Services instantiated only when needed via `_get_*()` methods
|
||||
4. **Dataclass configs**: `OrchestratorConfig` and `OrchestratorResult` for clear contracts
|
||||
5. **Feature flag default true**: New code active immediately since tests are comprehensive
|
||||
|
||||
## Testing Notes
|
||||
|
||||
- All 976 existing tests continue to pass
|
||||
- 139 new tests added across 6 test files
|
||||
- E2E verification: Jobs 501258e1 and d21e6ef0 successfully processed through new pipeline
|
||||
- Both jobs reached `awaiting_instrumental_selection` as expected (requires user input to complete)
|
||||
|
||||
## PR
|
||||
|
||||
- PR #182: https://github.com/nomadkaraoke/karaoke-gen/pull/182
|
||||
- Merged: 2026-01-04
|
||||
- Version: 0.89.0
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
See [LESSONS-LEARNED.md](../LESSONS-LEARNED.md#alternative-code-paths-must-implement-all-features) for the key architectural lesson about alternative code paths.
|
||||
@@ -45,7 +45,7 @@ describe('AuthStatus', () => {
|
||||
|
||||
// Should show user email and credits
|
||||
expect(screen.getByText('test@example.com')).toBeInTheDocument()
|
||||
expect(screen.getByText('5')).toBeInTheDocument()
|
||||
expect(screen.getByText('5 credits')).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('opens auth dialog when Login is clicked', () => {
|
||||
|
||||
@@ -70,7 +70,7 @@ export function AuthStatus({ onAuthChange }: AuthStatusProps) {
|
||||
</span>
|
||||
<span className="flex items-center gap-1 text-amber-400 font-medium">
|
||||
<Coins className="w-3 h-3" />
|
||||
{user.credits}
|
||||
{user.credits} {user.credits === 1 ? 'credit' : 'credits'}
|
||||
</span>
|
||||
</Button>
|
||||
</DropdownMenuTrigger>
|
||||
|
||||
@@ -453,8 +453,8 @@ test.describe('Production E2E - Full User Journey', () => {
|
||||
jobId,
|
||||
['awaiting_instrumental_selection', 'complete'],
|
||||
accessToken!,
|
||||
TIMEOUTS.jobProcessing // Video rendering can take several minutes
|
||||
).catch(() => ({ status: 'unknown', job: null }));
|
||||
TIMEOUTS.download // Video rendering can take 10-15 minutes with orchestrator
|
||||
);
|
||||
|
||||
if (statusBeforeInstrumental === 'awaiting_instrumental_selection') {
|
||||
console.log(' Selecting clean instrumental via API...');
|
||||
|
||||
Reference in new issue
Block a user