Files
karaoke-gen/infrastructure/encoding-worker/bootstrap.sh
T
Andrew Beveridge ☄️andClaude Opus 4.5 b68a46d174 feat(infra): Implement immutable deployment pattern for encoding worker (#333)
## Summary
- Implements robust deployment pattern for GCE encoding worker to
eliminate recurring version mismatch issues
- Fixed wheel path (`karaoke_gen-current.whl`) eliminates version
sorting bugs
- Self-updating startup script downloaded from GCS on every service
start
- Version manifest with strict CI verification (deployment fails on
mismatch)

## Problem
The encoding worker had recurring version issues because:
- `startup.sh` was baked into the Packer image (required rebuild + VM
recreation)
- Wheel selection used version sorting across 51 files
- CI only warned on version mismatch, didn't fail
- Multiple deployment vectors (CI, Pulumi, manual SSH) caused drift

## Solution
1. **Fixed wheel path**: CI uploads to `karaoke_gen-current.whl` (no
sorting needed)
2. **Self-updating startup**: `bootstrap.sh` downloads `startup.sh` from
GCS on every start
3. **Version manifest**: CI writes expected version, service verifies
before starting
4. **Strict CI verification**: Deployment fails if version doesn't match
(was just warning)

## Changes
- **New**: `infrastructure/encoding-worker/` with `bootstrap.sh`,
`startup.sh`, README
- **Modified**: CI workflow for fixed paths and strict verification
- **Modified**: Packer `provision.sh` for bootstrap pattern
- **Updated**: `LESSONS-LEARNED.md` with immutable deployment pattern
- **Updated**: `docs/README.md` and `docs/ARCHITECTURE.md`

## Test plan
- [x] Shell scripts validated (`bash -n`)
- [x] Backend tests pass (1703 passed)
- [x] CodeRabbit review passed
- [ ] After merge: Rebuild Packer image
- [ ] After merge: Recreate VM to use new image
- [ ] After merge: Verify first deployment uses new pattern

## Post-Merge Steps Required
1. Rebuild Packer image: `packer build
infrastructure/packer/encoding-worker.pkr.hcl`
2. Recreate VM: Delete encoding-worker, run `pulumi up`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 23:56:53 -08:00

43 lines
1.4 KiB
Bash

#!/bin/bash
# Encoding Worker Bootstrap Script
#
# This is the ONLY script baked into the Packer image.
# It downloads and executes the real startup script from GCS.
#
# Why this pattern?
# - startup.sh logic can be updated without rebuilding the image
# - All code changes go through CI → GCS → service restart
# - No version sorting bugs (fixed path for wheel)
# - Strict version verification before starting
#
# This script should NEVER need modification. All logic lives in
# the GCS-hosted startup.sh which is managed by CI.
set -e
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
WORKER_DIR="/opt/encoding-worker"
LOG_FILE="/var/log/encoding-worker-bootstrap.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "=== Bootstrap: $(date) ==="
echo "Downloading latest startup script from GCS..."
# Download the CI-managed startup script
if ! gsutil cp "${BUCKET}/encoding-worker/startup.sh" "${WORKER_DIR}/startup-latest.sh" 2>&1; then
echo "ERROR: Failed to download startup.sh from GCS"
echo "Falling back to local startup.sh if available..."
if [ -f "${WORKER_DIR}/startup-fallback.sh" ]; then
cp "${WORKER_DIR}/startup-fallback.sh" "${WORKER_DIR}/startup-latest.sh"
else
echo "FATAL: No fallback startup script available"
exit 1
fi
fi
chmod +x "${WORKER_DIR}/startup-latest.sh"
echo "=== Bootstrap: Executing downloaded startup script ==="
exec "${WORKER_DIR}/startup-latest.sh"