## Summary Two bugs, both hit by job `82f48738` (a private → public visibility change) on 2026-10-12. ### 1. Encoding worker VMs boot into an unrecoverable restart loop If gcloud's first call (the API key fetch in `startup.sh`) happens before DHCP, gcloud caches "not on GCE" in `/root/.config/gcloud/gce`. Every later gcloud call in that boot then has no credentials, so the API key comes back empty, the v0.266.4 self-heal refuses to serve, and systemd re-runs `startup.sh` against the same poisoned cache until the VM idle-stops. worker-b was up 20:30–20:42 and never answered /health, so the render failed with `Cannot connect to host 34.10.189.118:8080`. The same thing happened on 10-09 (`f4310aea`): it was fixed by hand and the durable fix was noted but never shipped. - `startup.sh`: `rm -f /root/.config/gcloud/gce`, wait up to 60s for the metadata server, retry the secret fetch 6× (clearing the cache each time). CI uploads this script on every deploy and bootstrap downloads it on every service start, so existing VM disks pick it up. - Packer `startup-fallback.sh`: clears the cache too (new images). ### 2. Retry after a failed change-to-public 404s forever `change_to_public` deletes screens, `videos/with_vocals.mkv` and the finals but left their `file_urls`. `/retry` then saw `videos.with_vocals` + `instrumental_selection`, skipped the render and 404'd on the deleted MKV. It now drops those refs, as `ThemeRerenderService` already does. ## Testing - New `backend/tests/test_encoding_worker_startup_script.py` and `test_change_visibility.py::test_drops_file_urls_for_deleted_artifacts` - `pytest backend/tests/test_encoding_worker_startup_script.py backend/tests/test_encoding_worker_self_heal.py backend/tests/test_encoding_worker_code_paths.py backend/tests/test_change_visibility.py`: all pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JUFVxMZME6ydzkmLvyuXRo <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * When a private job is made public, references to deleted screen and video files are cleared before processing resumes, preventing stale files from appearing. * Improved encoding worker startup reliability by retrying API-key retrieval when the metadata service or secret store is temporarily unavailable. <!-- end of auto-generated comment: release notes by coderabbit.ai --> @coderabbitai ignore --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Encoding Worker Packer Image
Custom GCE machine image for the encoding worker VM with all dependencies pre-installed.
Why?
The encoding worker previously took ~10 minutes to start because it built Python 3.13 from source on every boot. With this pre-baked image, startup time is reduced to ~30 seconds.
| Component | Before (startup script) | After (Packer image) |
|---|---|---|
| Python 3.13 build | ~7 min (every boot) | Pre-installed |
| FFmpeg install | ~1 min (every boot) | Pre-installed |
| System packages | ~1 min (every boot) | Pre-installed |
| Font packages | ~30 sec (every boot) | Pre-installed |
| API key + wheel | ~30 sec | ~30 sec |
| Total | ~10 min | ~30 sec |
Prerequisites
- Packer installed:
brew install packeror download from HashiCorp - GCP authentication:
gcloud auth application-default login - Required permissions:
roles/compute.instanceAdmin.v1androles/iam.serviceAccountUser
Building the Image
cd infrastructure/packer
# Initialize Packer plugins (first time only)
packer init encoding-worker.pkr.hcl
# Build the image
packer build encoding-worker.pkr.hcl
Build takes approximately 15-20 minutes (mostly Python compilation with optimizations).
Build Variables
Override defaults with -var:
packer build \
-var 'python_version=3.13.2' \
-var 'zone=us-west1-a' \
encoding-worker.pkr.hcl
| Variable | Default | Description |
|---|---|---|
project_id |
nomadkaraoke |
GCP project ID |
zone |
us-central1-a |
Build zone |
python_version |
3.13.1 |
Python version to install |
What's Included
The image contains:
- Python 3.13 at
/opt/python313(built with optimizations) - FFmpeg 7.x at
/usr/local/bin/ffmpeg(John Van Sickle static build) - Noto fonts including CJK support for subtitle rendering
- Virtual environment at
/opt/encoding-worker/venv - Systemd service
encoding-worker.service(enabled, not started) - Startup script
/opt/encoding-worker/startup.sh
Runtime Behavior (Immutable Deployment Pattern)
When the VM boots with this image:
-
bootstrap.sh runs via
ExecStartPre:- Downloads
startup.shfrom GCS (CI-managed, always current) - Executes the downloaded startup script
- Downloads
-
startup.sh (downloaded from GCS):
- Fetches API key from Secret Manager
- Downloads
karaoke_gen-current.whlfrom GCS (fixed path, no sorting!) - Installs wheel into venv
- Verifies installed version matches
version.txtfrom GCS - Fails fast if version mismatch
-
Encoding worker service starts via
ExecStart:- Runs
uvicorn backend.services.gce_encoding.main:app - Listens on port 8080
- Runs
See infrastructure/encoding-worker/README.md for details on this pattern.
When to Rebuild
Rebuild the image when:
- Python version needs updating
- FFmpeg version needs updating
- New system packages required
- Font packages change
- bootstrap.sh changes (extremely rare)
Image disk size: disk_size in the template (30 GB) is the minimum boot-disk
size of any VM created from the image — GCE cannot create a disk smaller than its
source image. Keep it ≤ DiskSizes.ENCODING_WORKER in infrastructure/config.py
(50 GB); infrastructure/test_encoding_worker_config.py enforces this.
After a rebuild, existing VMs keep their old disks (Pulumi only uses the new
image when a VM/disk is recreated). To roll a new image onto the stopped workers
without re-allocating VMs (stockout-safe), use the in-place boot-disk swap in
docs/archive/2026-09-26-encoding-worker-disk-cost.md.
No rebuild needed for:
- Application code changes (wheel downloaded at runtime)
- Startup logic changes (startup.sh downloaded from GCS)
Image Family
Images are created with family encoding-worker. Pulumi uses the family to always get the latest image:
image=f"projects/nomadkaraoke/global/images/family/encoding-worker"
Old images are automatically replaced in the family, but retained for rollback.
Rollback
If the new image has issues:
- Find previous image:
gcloud compute images list --filter="family:encoding-worker" - Update Pulumi to use specific image instead of family
- Or rebuild from a previous git commit
CI Integration
For automated rebuilds, add a GitHub Actions workflow that triggers on changes to infrastructure/packer/:
name: Build Encoding Worker Image
on:
push:
branches: [main]
paths:
- 'infrastructure/packer/**'
workflow_dispatch:
See PHASE5-PACKER-IMAGE-PLAN.md for full workflow example.