Files
Andrew Beveridge ☄️andClaude Opus 5.5 77bec3327c fix: encoding-worker boot loop (poisoned gcloud cache) + stale file_urls after change-to-public (v0.268.3) (#1138)
## Summary
Two bugs, both hit by job `82f48738` (a private → public visibility
change) on 2026-10-12.

### 1. Encoding worker VMs boot into an unrecoverable restart loop
If gcloud's first call (the API key fetch in `startup.sh`) happens
before DHCP, gcloud caches "not on GCE" in `/root/.config/gcloud/gce`.
Every later gcloud call in that boot then has no credentials, so the API
key comes back empty, the v0.266.4 self-heal refuses to serve, and
systemd re-runs `startup.sh` against the same poisoned cache until the
VM idle-stops. worker-b was up 20:30–20:42 and never answered /health,
so the render failed with `Cannot connect to host 34.10.189.118:8080`.
The same thing happened on 10-09 (`f4310aea`): it was fixed by hand and
the durable fix was noted but never shipped.

- `startup.sh`: `rm -f /root/.config/gcloud/gce`, wait up to 60s for the
metadata server, retry the secret fetch 6× (clearing the cache each
time). CI uploads this script on every deploy and bootstrap downloads it
on every service start, so existing VM disks pick it up.
- Packer `startup-fallback.sh`: clears the cache too (new images).

### 2. Retry after a failed change-to-public 404s forever
`change_to_public` deletes screens, `videos/with_vocals.mkv` and the
finals but left their `file_urls`. `/retry` then saw
`videos.with_vocals` + `instrumental_selection`, skipped the render and
404'd on the deleted MKV. It now drops those refs, as
`ThemeRerenderService` already does.

## Testing
- New `backend/tests/test_encoding_worker_startup_script.py` and
`test_change_visibility.py::test_drops_file_urls_for_deleted_artifacts`
- `pytest backend/tests/test_encoding_worker_startup_script.py
backend/tests/test_encoding_worker_self_heal.py
backend/tests/test_encoding_worker_code_paths.py
backend/tests/test_change_visibility.py`: all pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01JUFVxMZME6ydzkmLvyuXRo

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* When a private job is made public, references to deleted screen and
video files are cleared before processing resumes, preventing stale
files from appearing.
* Improved encoding worker startup reliability by retrying API-key
retrieval when the metadata service or secret store is temporarily
unavailable.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:29:43 -04:00
..

Encoding Worker Packer Image

Custom GCE machine image for the encoding worker VM with all dependencies pre-installed.

Why?

The encoding worker previously took ~10 minutes to start because it built Python 3.13 from source on every boot. With this pre-baked image, startup time is reduced to ~30 seconds.

Component Before (startup script) After (Packer image)
Python 3.13 build ~7 min (every boot) Pre-installed
FFmpeg install ~1 min (every boot) Pre-installed
System packages ~1 min (every boot) Pre-installed
Font packages ~30 sec (every boot) Pre-installed
API key + wheel ~30 sec ~30 sec
Total ~10 min ~30 sec

Prerequisites

  1. Packer installed: brew install packer or download from HashiCorp
  2. GCP authentication: gcloud auth application-default login
  3. Required permissions: roles/compute.instanceAdmin.v1 and roles/iam.serviceAccountUser

Building the Image

cd infrastructure/packer

# Initialize Packer plugins (first time only)
packer init encoding-worker.pkr.hcl

# Build the image
packer build encoding-worker.pkr.hcl

Build takes approximately 15-20 minutes (mostly Python compilation with optimizations).

Build Variables

Override defaults with -var:

packer build \
  -var 'python_version=3.13.2' \
  -var 'zone=us-west1-a' \
  encoding-worker.pkr.hcl
Variable Default Description
project_id nomadkaraoke GCP project ID
zone us-central1-a Build zone
python_version 3.13.1 Python version to install

What's Included

The image contains:

  • Python 3.13 at /opt/python313 (built with optimizations)
  • FFmpeg 7.x at /usr/local/bin/ffmpeg (John Van Sickle static build)
  • Noto fonts including CJK support for subtitle rendering
  • Virtual environment at /opt/encoding-worker/venv
  • Systemd service encoding-worker.service (enabled, not started)
  • Startup script /opt/encoding-worker/startup.sh

Runtime Behavior (Immutable Deployment Pattern)

When the VM boots with this image:

  1. bootstrap.sh runs via ExecStartPre:

    • Downloads startup.sh from GCS (CI-managed, always current)
    • Executes the downloaded startup script
  2. startup.sh (downloaded from GCS):

    • Fetches API key from Secret Manager
    • Downloads karaoke_gen-current.whl from GCS (fixed path, no sorting!)
    • Installs wheel into venv
    • Verifies installed version matches version.txt from GCS
    • Fails fast if version mismatch
  3. Encoding worker service starts via ExecStart:

    • Runs uvicorn backend.services.gce_encoding.main:app
    • Listens on port 8080

See infrastructure/encoding-worker/README.md for details on this pattern.

When to Rebuild

Rebuild the image when:

  • Python version needs updating
  • FFmpeg version needs updating
  • New system packages required
  • Font packages change
  • bootstrap.sh changes (extremely rare)

Image disk size: disk_size in the template (30 GB) is the minimum boot-disk size of any VM created from the image — GCE cannot create a disk smaller than its source image. Keep it ≤ DiskSizes.ENCODING_WORKER in infrastructure/config.py (50 GB); infrastructure/test_encoding_worker_config.py enforces this.

After a rebuild, existing VMs keep their old disks (Pulumi only uses the new image when a VM/disk is recreated). To roll a new image onto the stopped workers without re-allocating VMs (stockout-safe), use the in-place boot-disk swap in docs/archive/2026-09-26-encoding-worker-disk-cost.md.

No rebuild needed for:

  • Application code changes (wheel downloaded at runtime)
  • Startup logic changes (startup.sh downloaded from GCS)

Image Family

Images are created with family encoding-worker. Pulumi uses the family to always get the latest image:

image=f"projects/nomadkaraoke/global/images/family/encoding-worker"

Old images are automatically replaced in the family, but retained for rollback.

Rollback

If the new image has issues:

  1. Find previous image: gcloud compute images list --filter="family:encoding-worker"
  2. Update Pulumi to use specific image instead of family
  3. Or rebuild from a previous git commit

CI Integration

For automated rebuilds, add a GitHub Actions workflow that triggers on changes to infrastructure/packer/:

name: Build Encoding Worker Image
on:
  push:
    branches: [main]
    paths:
      - 'infrastructure/packer/**'
  workflow_dispatch:

See PHASE5-PACKER-IMAGE-PLAN.md for full workflow example.