mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Problem Follow-up to #908. During the OOM-fix verification, a fresh n2 fallback booted and three renders hit it at once; one (`374dec26`) failed with `karaoke-gen wheel/dependencies not ready on this worker after retries; job should be retried on a healthy worker`. **Root cause:** `ensure_latest_wheel()` runs on the **light** executor (>1 thread) at the start of every job — the heavy-lane serialization (`ENCODING_HEAVY_CONCURRENCY=1`) does **not** cover it. So concurrent jobs on a fresh boot raced on the shared `/tmp/karaoke_gen-<v>.whl` path (`gsutil cp` clobbered mid-copy) and the venv (two `pip install`s at once). It also re-downloaded + re-installed the wheel on **every** job even when the installed version already matched. ## Fix (`backend/services/gce_encoding/main.py`) - **Process-wide lock** (`threading.Lock`) around the whole check/download/install/verify so only one caller does the work; the rest wait. - **"Already current" fast path:** compare the installed `karaoke-gen` version to the latest GCS wheel; if they match (the common case — `startup.sh` installs it at boot), skip download/install entirely. A per-process verified-version cache makes repeat calls a no-op. Only the *first* job after a new wheel actually installs. Net effect: on a fresh boot, the first job installs (under the lock) and the rest take the fast path — no clobbering, no redundant installs, no spurious "wheel not ready" failures. ## Testing - New tests: fast path (already-verified / installed-matches → zero `cp`/`pip`), full install when the version differs, and **3 concurrent callers install exactly once** (`cp==1`, `pip==1`). - Existing wheel-verification tests get an autouse cache reset so the new module global can't leak between tests. - Encoding suite: 163 passed. - Local CodeRabbit review completed; feedback addressed. ## Deploy note Same as #908: the worker picks this up on a fresh boot / service restart (it ships in the wheel). No infra changes. @coderabbitai ignore
GCE Encoding Worker
HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.
Overview
This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the encoding-worker GCE VM instance.
Endpoints
| Endpoint | Method | Description |
|---|---|---|
/encode |
POST | Submit a full encoding job |
/encode-preview |
POST | Submit a preview encoding job (480x270, fast) |
/status/{job_id} |
GET | Get job status and progress |
/health |
GET | Health check |
Authentication
All endpoints (except /health) require an API key via the X-API-Key header. The key is stored in Secret Manager (encoding-worker-api-key).
Encoding Process
Full Encoding (/encode)
- Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
- Uses
LocalEncodingServicefrom the karaoke-gen wheel to produce:- Lossless 4K MP4
- Lossy 4K MP4
- Lossless MKV
- 720p MP4
- Uploads results to GCS
- Reports progress via status endpoint
Preview Encoding (/encode-preview)
- Downloads ASS subtitle and audio files from GCS
- Runs FFmpeg to create a quick 480x270 preview video
- Uploads result to GCS
- Used for real-time preview in the web UI
Dependencies
- FFmpeg: Static build from John Van Sickle
- Python 3.13: Built from source (required for karaoke-gen)
- karaoke-gen wheel: Downloaded from GCS at job start (enables hot updates)
Deployment
This service is deployed via the VM's startup script. The script:
- Installs system dependencies (FFmpeg, fonts, Python build deps)
- Builds Python 3.13 from source
- Creates venv and installs dependencies
- Writes
main.pyto/opt/encoding-worker/ - Creates systemd service
- Starts the service on port 8080
Local Development
This code is extracted from the infrastructure for maintainability. To test locally:
cd backend/services/gce_encoding
pip install -r requirements.txt
pip install ../../../ # Install karaoke-gen wheel
ENCODING_API_KEY=test python main.py
Architecture Notes
- Hot code updates: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
- In-memory job tracking: Job state is stored in memory. Restart clears queue.
- Parallel encoding: ThreadPoolExecutor with 4 workers
- GCS integration: Direct download/upload via google-cloud-storage client