Files
karaoke-gen/backend/services/gce_encoding
Andrew Beveridge ☄️andClaude Opus 5.5 983f6ecdcd fix(storage): keep user-uploaded inputs forever; uploads/ expires after 180d not 7d (v0.268.2) (#1137)
## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.

Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.

## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
  - Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.

## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
  - the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
  - backfill dry run / apply / error isolation;
  - the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.

## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:33:06 -04:00
..

GCE Encoding Worker

HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.

Overview

This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the encoding-worker GCE VM instance.

Endpoints

Endpoint Method Description
/encode POST Submit a full encoding job
/encode-preview POST Submit a preview encoding job (480x270, fast)
/status/{job_id} GET Get job status and progress
/health GET Health check

Authentication

All endpoints (except /health) require an API key via the X-API-Key header. The key is stored in Secret Manager (encoding-worker-api-key).

Encoding Process

Full Encoding (/encode)

  1. Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
  2. Uses LocalEncodingService from the karaoke-gen wheel to produce:
    • Lossless 4K MP4
    • Lossy 4K MP4
    • Lossless MKV
    • 720p MP4
  3. Uploads results to GCS
  4. Reports progress via status endpoint

Preview Encoding (/encode-preview)

  1. Downloads ASS subtitle and audio files from GCS
  2. Runs FFmpeg to create a quick 480x270 preview video
  3. Uploads result to GCS
  4. Used for real-time preview in the web UI

Dependencies

  • FFmpeg: Static build from John Van Sickle
  • Python 3.13: Built from source (required for karaoke-gen)
  • karaoke-gen wheel: Downloaded from GCS at job start (enables hot updates)

Deployment

This service is deployed via the VM's startup script. The script:

  1. Installs system dependencies (FFmpeg, fonts, Python build deps)
  2. Builds Python 3.13 from source
  3. Creates venv and installs dependencies
  4. Writes main.py to /opt/encoding-worker/
  5. Creates systemd service
  6. Starts the service on port 8080

Local Development

This code is extracted from the infrastructure for maintainability. To test locally:

cd backend/services/gce_encoding
pip install -r requirements.txt
pip install ../../../  # Install karaoke-gen wheel
ENCODING_API_KEY=test python main.py

Architecture Notes

  • Hot code updates: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
  • In-memory job tracking: Job state is stored in memory. Restart clears queue.
  • Parallel encoding: ThreadPoolExecutor with 4 workers
  • GCS integration: Direct download/upload via google-cloud-storage client