mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Phase 2 of moving encoding to AWS; Phase 1 was #1236 and #1239. Design and rationale are in `docs/archive/2026-10-29-aws-encoding-worker-plan.md` § Phase 2. When Firestore `config/encoding-worker.aws_video_worker` is true **and** the AWS worker ranks first, `trigger_video_worker` runs the video worker **on the AWS box** instead of starting the Cloud Run job `video-encoding-job`. - **Uploads move to AWS:** the Dropbox/YouTube/Drive uploads (~187 GB/mo, ~$18/mo of Cloud Run internet egress) leave from AWS, on credits. - **No Cloud Run CPU** for the job (~$11/mo). - **Off by default.** Any failure falls back to the Cloud Run job, so this path can't drop a job. ## Changes - **`POST /run-worker` (worker):** - Spawns `python -m backend.workers.video_worker --job-id …` with the box's video-worker env. - Status is reported via `/status/{id}`. - A run counts in `/health` `active_jobs` from the moment it's accepted, so the idle watchdog won't stop the box mid-run. - Returns 503 unless the env file exists. - `job_id` is pattern-validated. - Any startup failure marks the run failed. - **`backend/services/local_encoded_outputs.py`:** a same-box handoff. - When the co-located video worker asks (`stash_local_outputs`, set only where `LOCAL_ENCODED_OUTPUTS_DIR` exists), the encoder keeps local copies of the finals, keyed by blob path. - The video worker takes those instead of downloading them back out of GCS. Without this, the move would *add* ~400 MB/job of GCS egress. - Copies are made off the event loop and purged after 6 h. - No change on GCE or Cloud Run. - **`video-worker-env.py` (box, every boot):** - Mirrors the **live** `video-encoding-job` env, with secrets resolved, so nothing is hand-copied and nothing drifts. - Runs as the job's identity, `karaoke-backend@`, via a second WIF binding (Pulumi `aws-encoding-worker-wif-backend-user`). - On failure it writes no env file, so `/run-worker` answers 503 and the Cloud Run job is used. - **`WorkerService._trigger_video_worker_on_aws`:** checks the flag and that the AWS worker ranks first, ensures it's running, waits for health, then POSTs `/run-worker`. Returns False on any doubt. - **Also:** `startup-aws.sh` creates the backend credential config and runs the env helper; CI uploads `video-worker-env.py`; `worker_code_paths.txt` now gates `local_encoded_outputs.py`. ## Rollout (needs `/gcp-admin`) 1. **Before merge:** targeted `pulumi up` of the backend WIF binding. It's additive. 2. **After stage B of Phase 1** (AWS worker preferred, timings verified): stop/start the box once so the new `startup-aws.sh` writes the env, then set `aws_video_worker: true`. Set it to `false` to roll back. ## Testing - **27 new tests:** - handoff stash/take/purge, path-escape guard, and the per-request stash flag; - `/run-worker`: disabled, bad module, dedupe, active immediately, spawn args/env, failure log tail, wheel-not-ready, safe start, unsafe ids; - env builder; - the trigger's AWS branch: off, not first, happy path, didn't start, refusals, errors. - **Full backend + unit suites:** no new failures (the 13 local failures are identical on main). - **Local CodeRabbit:** 5 findings, all fixed. The code-paths gate test also caught a missing worker path, now fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>