Files
karaoke-gen/infrastructure/encoding-worker
Andrew Beveridge ☄️andClaude Opus 5.5 77bec3327c fix: encoding-worker boot loop (poisoned gcloud cache) + stale file_urls after change-to-public (v0.268.3) (#1138)
## Summary
Two bugs, both hit by job `82f48738` (a private → public visibility
change) on 2026-10-12.

### 1. Encoding worker VMs boot into an unrecoverable restart loop
If gcloud's first call (the API key fetch in `startup.sh`) happens
before DHCP, gcloud caches "not on GCE" in `/root/.config/gcloud/gce`.
Every later gcloud call in that boot then has no credentials, so the API
key comes back empty, the v0.266.4 self-heal refuses to serve, and
systemd re-runs `startup.sh` against the same poisoned cache until the
VM idle-stops. worker-b was up 20:30–20:42 and never answered /health,
so the render failed with `Cannot connect to host 34.10.189.118:8080`.
The same thing happened on 10-09 (`f4310aea`): it was fixed by hand and
the durable fix was noted but never shipped.

- `startup.sh`: `rm -f /root/.config/gcloud/gce`, wait up to 60s for the
metadata server, retry the secret fetch 6× (clearing the cache each
time). CI uploads this script on every deploy and bootstrap downloads it
on every service start, so existing VM disks pick it up.
- Packer `startup-fallback.sh`: clears the cache too (new images).

### 2. Retry after a failed change-to-public 404s forever
`change_to_public` deletes screens, `videos/with_vocals.mkv` and the
finals but left their `file_urls`. `/retry` then saw
`videos.with_vocals` + `instrumental_selection`, skipped the render and
404'd on the deleted MKV. It now drops those refs, as
`ThemeRerenderService` already does.

## Testing
- New `backend/tests/test_encoding_worker_startup_script.py` and
`test_change_visibility.py::test_drops_file_urls_for_deleted_artifacts`
- `pytest backend/tests/test_encoding_worker_startup_script.py
backend/tests/test_encoding_worker_self_heal.py
backend/tests/test_encoding_worker_code_paths.py
backend/tests/test_change_visibility.py`: all pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01JUFVxMZME6ydzkmLvyuXRo

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* When a private job is made public, references to deleted screen and
video files are cleared before processing resumes, preventing stale
files from appearing.
* Improved encoding worker startup reliability by retrying API-key
retrieval when the metadata service or secret store is temporarily
unavailable.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:29:43 -04:00
..

Encoding Worker Deployment

This directory contains the scripts for the GCE encoding worker's immutable deployment pattern.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    PACKER IMAGE (rarely changes)                │
├─────────────────────────────────────────────────────────────────┤
│ - Python 3.13, FFmpeg, fonts (static dependencies)             │
│ - bootstrap.sh (downloads startup.sh from GCS)                 │
│ - systemd service definition                                    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ downloads on every service start
┌─────────────────────────────────────────────────────────────────┐
│                    GCS (CI-managed, always current)             │
├─────────────────────────────────────────────────────────────────┤
│ gs://karaoke-gen-storage-nomadkaraoke/                         │
│ ├── encoding-worker/                                            │
│ │   ├── startup.sh      ← Main logic (this file)               │
│ │   └── version.txt     ← Expected version number              │
│ └── wheels/                                                     │
│     └── karaoke_gen-current.whl  ← Latest wheel (fixed path)   │
└─────────────────────────────────────────────────────────────────┘

Files

File Location Purpose
bootstrap.sh Baked in Packer image Downloads and runs startup.sh
startup.sh Deployed to GCS by CI Main startup logic
version.txt Written to GCS by CI Expected version for verification

Deployment Flow

  1. CI builds wheel → uploads to wheels/karaoke_gen-current.whl
  2. CI uploads startup.sh → encoding-worker/startup.sh
  3. CI writes version → encoding-worker/version.txt
  4. CI restarts service → triggers bootstrap.sh
  5. bootstrap.sh downloads fresh startup.sh from GCS
  6. startup.sh downloads wheel, installs, verifies version
  7. CI verifies health endpoint returns correct version
  8. CI FAILS if version mismatch

Why This Pattern?

Before (problematic)

  • startup.sh baked into Packer image
  • Required image rebuild + VM recreation for logic changes
  • 51 versioned wheels, sorting could fail
  • CI warned on version mismatch but continued

After (robust)

  • startup.sh downloaded from GCS on every start
  • Logic changes via CI, no image rebuild needed
  • Fixed path karaoke_gen-current.whl, no sorting
  • CI FAILS on version mismatch

Modifying Startup Logic

  1. Edit infrastructure/encoding-worker/startup.sh
  2. Push to main
  3. CI automatically deploys to GCS
  4. Service restart picks up new script

No Packer image rebuild needed!

When to Rebuild Packer Image

Only rebuild when changing:

  • Python version
  • FFmpeg version
  • System packages/fonts
  • bootstrap.sh (should be extremely rare)

Manual Operations

# Check current version on VM
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
  --tunnel-through-iap \
  --command="curl -s http://localhost:8080/health | jq '.wheel_version'"

# View startup logs
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
  --tunnel-through-iap \
  --command="sudo journalctl -u encoding-worker -n 50"

# Force service restart (triggers fresh startup.sh download)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
  --tunnel-through-iap \
  --command="sudo systemctl restart encoding-worker"

Troubleshooting

Version mismatch after deployment

  1. Check CI logs for GCS upload success
  2. Check /var/log/encoding-worker-startup.log on VM
  3. Verify version.txt in GCS matches expected

Service won't start

  1. Check bootstrap.sh can reach GCS: gsutil ls gs://karaoke-gen-storage-nomadkaraoke/
  2. Check Secret Manager access for API key
  3. Check startup.sh logs: cat /var/log/encoding-worker-startup.log