mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.
Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.
## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
- Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.
## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
- the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
- backfill dry run / apply / error isolation;
- the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.
## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GCE Encoding Worker
HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.
Overview
This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the encoding-worker GCE VM instance.
Endpoints
| Endpoint | Method | Description |
|---|---|---|
/encode |
POST | Submit a full encoding job |
/encode-preview |
POST | Submit a preview encoding job (480x270, fast) |
/status/{job_id} |
GET | Get job status and progress |
/health |
GET | Health check |
Authentication
All endpoints (except /health) require an API key via the X-API-Key header. The key is stored in Secret Manager (encoding-worker-api-key).
Encoding Process
Full Encoding (/encode)
- Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
- Uses
LocalEncodingServicefrom the karaoke-gen wheel to produce:- Lossless 4K MP4
- Lossy 4K MP4
- Lossless MKV
- 720p MP4
- Uploads results to GCS
- Reports progress via status endpoint
Preview Encoding (/encode-preview)
- Downloads ASS subtitle and audio files from GCS
- Runs FFmpeg to create a quick 480x270 preview video
- Uploads result to GCS
- Used for real-time preview in the web UI
Dependencies
- FFmpeg: Static build from John Van Sickle
- Python 3.13: Built from source (required for karaoke-gen)
- karaoke-gen wheel: Downloaded from GCS at job start (enables hot updates)
Deployment
This service is deployed via the VM's startup script. The script:
- Installs system dependencies (FFmpeg, fonts, Python build deps)
- Builds Python 3.13 from source
- Creates venv and installs dependencies
- Writes
main.pyto/opt/encoding-worker/ - Creates systemd service
- Starts the service on port 8080
Local Development
This code is extracted from the infrastructure for maintainability. To test locally:
cd backend/services/gce_encoding
pip install -r requirements.txt
pip install ../../../ # Install karaoke-gen wheel
ENCODING_API_KEY=test python main.py
Architecture Notes
- Hot code updates: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
- In-memory job tracking: Job state is stored in memory. Restart clears queue.
- Parallel encoding: ThreadPoolExecutor with 4 workers
- GCS integration: Direct download/upload via google-cloud-storage client