feat(infra): Implement immutable deployment pattern for encoding worker (#333)

## Summary
- Implements robust deployment pattern for GCE encoding worker to
eliminate recurring version mismatch issues
- Fixed wheel path (`karaoke_gen-current.whl`) eliminates version
sorting bugs
- Self-updating startup script downloaded from GCS on every service
start
- Version manifest with strict CI verification (deployment fails on
mismatch)

## Problem
The encoding worker had recurring version issues because:
- `startup.sh` was baked into the Packer image (required rebuild + VM
recreation)
- Wheel selection used version sorting across 51 files
- CI only warned on version mismatch, didn't fail
- Multiple deployment vectors (CI, Pulumi, manual SSH) caused drift

## Solution
1. **Fixed wheel path**: CI uploads to `karaoke_gen-current.whl` (no
sorting needed)
2. **Self-updating startup**: `bootstrap.sh` downloads `startup.sh` from
GCS on every start
3. **Version manifest**: CI writes expected version, service verifies
before starting
4. **Strict CI verification**: Deployment fails if version doesn't match
(was just warning)

## Changes
- **New**: `infrastructure/encoding-worker/` with `bootstrap.sh`,
`startup.sh`, README
- **Modified**: CI workflow for fixed paths and strict verification
- **Modified**: Packer `provision.sh` for bootstrap pattern
- **Updated**: `LESSONS-LEARNED.md` with immutable deployment pattern
- **Updated**: `docs/README.md` and `docs/ARCHITECTURE.md`

## Test plan
- [x] Shell scripts validated (`bash -n`)
- [x] Backend tests pass (1703 passed)
- [x] CodeRabbit review passed
- [ ] After merge: Rebuild Packer image
- [ ] After merge: Recreate VM to use new image
- [ ] After merge: Verify first deployment uses new pattern

## Post-Merge Steps Required
1. Rebuild Packer image: `packer build
infrastructure/packer/encoding-worker.pkr.hcl`
2. Recreate VM: Delete encoding-worker, run `pulumi up`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.5 authored and GitHub committed 2026-01-22 23:56:53 -08:00
1 parent e2fc6b616a
commit b68a46d174
11 files changed
+584 -72

No files matched your search

@@ -0,0 +1,99 @@
# Plan: Encoding Worker Immutable Deployment Pattern
## Status: In Progress
## Problem Statement
The GCE encoding worker has recurring version mismatch issues because:
1. `startup.sh` is baked into the Packer image (requires image rebuild + VM recreation)
2. Wheel selection uses `sort -V` which can fail if image is outdated
3. CI only restarts service - doesn't verify version strictly
4. Multiple actors (CI, Pulumi, manual SSH) can cause drift
5. 51 old wheel files accumulate in GCS bucket
## Solution: Immutable Deployment Pattern
### Core Principles
1. **Single Source of Truth**: All code and config from GCS, nothing baked in image
2. **Fixed Wheel Path**: `karaoke_gen-current.whl` eliminates sorting
3. **Version Manifest**: CI writes expected version, service MUST match
4. **Strict Verification**: CI FAILS if deployed version doesn't match
5. **Self-Updating Startup**: Download startup.sh from GCS on every boot
## Implementation Steps
### Phase 1: GCS Structure & Scripts
- [x] **Step 1.1**: Create `infrastructure/encoding-worker/` directory
- [x] **Step 1.2**: Create `bootstrap.sh` (minimal, baked in image)
- [x] **Step 1.3**: Create `startup.sh` (CI-managed, deployed to GCS)
- [x] **Step 1.4**: Create README.md with documentation
### Phase 2: CI Workflow Updates
- [x] **Step 2.1**: Update CI to upload wheel to fixed path
- [x] **Step 2.2**: Update CI to upload startup.sh to GCS
- [x] **Step 2.3**: Update CI to write version.txt manifest
- [x] **Step 2.4**: Update CI to strictly verify deployed version (fail on mismatch)
### Phase 3: Packer Image Updates
- [x] **Step 3.1**: Update provision.sh to use bootstrap.sh
- [ ] **Step 3.2**: Rebuild Packer image (manual step after merge)
- [ ] **Step 3.3**: Recreate VM to use new image (manual step after merge)
### Phase 4: Documentation & Cleanup
- [x] **Step 4.1**: Update LESSONS-LEARNED.md with new pattern
- [x] **Step 4.2**: Update infrastructure/packer/README.md
- [ ] **Step 4.3**: Add lifecycle rule for old wheels (optional, later)
## New GCS Structure
```
gs://karaoke-gen-storage-nomadkaraoke/
├── wheels/
│ ├── karaoke_gen-current.whl # Latest (CI overwrites)
│ └── karaoke_gen-{version}.whl # Versioned for rollback
│
└── encoding-worker/
├── bootstrap.sh # Reference copy
├── startup.sh # CI-managed, downloaded on boot
└── version.txt # Expected version number
```
## Files to Create/Modify
| File | Action | Purpose |
|------|--------|---------|
| `infrastructure/encoding-worker/bootstrap.sh` | Create | Minimal loader (baked in image) |
| `infrastructure/encoding-worker/startup.sh` | Create | Main logic (deployed to GCS) |
| `infrastructure/packer/scripts/provision.sh` | Modify | Use bootstrap.sh |
| `.github/workflows/ci.yml` | Modify | New deployment flow |
| `docs/LESSONS-LEARNED.md` | Modify | Document pattern |
## Testing Strategy
1. **Unit test**: Verify startup.sh logic locally
2. **Integration test**: After merge, verify:
- CI uploads files to correct GCS paths
- Service starts with correct version
- Health endpoint returns expected version
3. **Manual verification**: Check VM logs after first deployment
## Rollback Plan
If issues occur:
1. CI can re-upload previous wheel to `karaoke_gen-current.whl`
2. Previous versioned wheels remain in GCS for manual rollback
3. Old Packer images remain for VM rollback
## Open Questions
None - design is complete.
## Notes
- Bootstrap.sh is ~10 lines and should never need changes
- All logic changes go through startup.sh in GCS
- Version verification is strict - CI fails on mismatch
+96 -26
View File
@@ -967,8 +967,16 @@ jobs:
version: "latest"
# ============================================
# Upload wheel to GCS for GCE encoding worker
# The GCE worker installs this wheel at job start for hot code updates
# Deploy to GCE Encoding Worker (Immutable Deployment Pattern)
#
# This pattern ensures version consistency:
# 1. Upload wheel to FIXED path (no version sorting)
# 2. Upload startup.sh (logic can change without image rebuild)
# 3. Write version manifest for verification
# 4. Restart service (triggers bootstrap.sh -> startup.sh)
# 5. STRICTLY verify deployed version (fail CI on mismatch)
#
# See infrastructure/encoding-worker/README.md for details
# ============================================
- name: Download wheel artifact
@@ -977,52 +985,114 @@ jobs:
name: package-dist
path: dist/
- name: Upload wheel to GCS for GCE worker
- name: Deploy to GCE Encoding Worker
run: |
echo "📦 Uploading wheel to GCS for GCE encoding worker..."
VERSION="${{ steps.version_check.outputs.version }}"
BUCKET="gs://karaoke-gen-storage-${{ env.PROJECT_ID }}"
echo "🚀 Deploying encoding worker v${VERSION}..."
# 1. Find the wheel file
WHEEL_FILE=$(ls dist/*.whl | head -1)
if [ -z "$WHEEL_FILE" ]; then
echo "::error::No wheel file found in dist/"
exit 1
fi
echo "Found wheel: $WHEEL_FILE"
echo "📦 Found wheel: $WHEEL_FILE"
# Upload to GCS (GCE worker downloads latest at job start)
gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/
# 2. Upload wheel to FIXED path (atomic via temp file)
# This eliminates version sorting - always use karaoke_gen-current.whl
echo "📤 Uploading wheel to fixed path..."
gsutil cp "$WHEEL_FILE" "${BUCKET}/wheels/karaoke_gen-current.whl.tmp"
gsutil mv "${BUCKET}/wheels/karaoke_gen-current.whl.tmp" \
"${BUCKET}/wheels/karaoke_gen-current.whl"
echo "✅ Wheel uploaded to GCS"
# Also keep versioned copy for rollback/debugging
gsutil cp "$WHEEL_FILE" "${BUCKET}/wheels/"
# 3. Upload startup.sh (allows logic changes without image rebuild)
echo "📤 Uploading startup.sh..."
gsutil cp infrastructure/encoding-worker/startup.sh \
"${BUCKET}/encoding-worker/startup.sh"
# 4. Write version manifest for verification
echo "📝 Writing version manifest..."
echo "${VERSION}" | gsutil cp - "${BUCKET}/encoding-worker/version.txt"
echo "✅ GCS deployment complete"
- name: Restart GCE Encoding Worker
run: |
echo "🔄 Restarting encoding worker to pick up new wheel..."
VERSION="${{ steps.version_check.outputs.version }}"
echo "🔄 Restarting encoding worker..."
# Use IAP tunneling - works directly with service account credentials
# (avoids SSH key propagation delays that cause silent failures)
gcloud compute ssh encoding-worker \
--zone=us-central1-c \
--project=nomadkaraoke \
--tunnel-through-iap \
--command="sudo systemctl restart encoding-worker"
echo "⏳ Waiting for service to start..."
sleep 15
echo "⏳ Waiting for service to start (up to 60s)..."
sleep 20
# Verify the restart worked by checking the wheel version
NEW_VERSION=$(gcloud compute ssh encoding-worker \
--zone=us-central1-c \
--project=nomadkaraoke \
--tunnel-through-iap \
--command="curl -s http://localhost:8080/health | grep -o '\"wheel_version\":\"[^\"]*\"' | cut -d'\"' -f4")
- name: Verify Encoding Worker Version
run: |
VERSION="${{ steps.version_check.outputs.version }}"
EXPECTED_VERSION=$(cat pyproject.toml | grep 'version = ' | head -1 | cut -d'"' -f2)
echo "🔍 Verifying encoding worker version..."
if [ "$NEW_VERSION" = "$EXPECTED_VERSION" ]; then
echo "✅ Encoding worker restarted successfully (v$NEW_VERSION)"
else
echo "⚠️ Version mismatch: encoder has v$NEW_VERSION, expected v$EXPECTED_VERSION"
echo " This may indicate the restart failed or the wheel wasn't downloaded"
# Don't fail the build - the encoder will still work, just with old version
fi
# Retry loop - service may take time to fully start
MAX_RETRIES=6
RETRY_DELAY=10
for i in $(seq 1 $MAX_RETRIES); do
echo "Attempt $i/$MAX_RETRIES..."
NEW_VERSION=$(gcloud compute ssh encoding-worker \
--zone=us-central1-c \
--project=nomadkaraoke \
--tunnel-through-iap \
--command="curl -s http://localhost:8080/health 2>/dev/null | grep -o '\"wheel_version\":\"[^\"]*\"' | cut -d'\"' -f4" 2>/dev/null || echo "")
if [ "$NEW_VERSION" = "$VERSION" ]; then
echo "✅ Encoding worker version verified: v$NEW_VERSION"
exit 0
fi
if [ -n "$NEW_VERSION" ]; then
echo "⚠️ Version: $NEW_VERSION (expected: $VERSION)"
else
echo "⚠️ Health endpoint not responding yet"
fi
if [ $i -lt $MAX_RETRIES ]; then
echo " Retrying in ${RETRY_DELAY}s..."
sleep $RETRY_DELAY
fi
done
# STRICT VERIFICATION: Fail CI on version mismatch
echo ""
echo "=============================================="
echo "❌ ENCODING WORKER DEPLOYMENT FAILED"
echo "=============================================="
echo "Expected version: $VERSION"
echo "Deployed version: $NEW_VERSION"
echo ""
echo "Possible causes:"
echo " - GCS upload failed (check 'Deploy to GCE' step)"
echo " - startup.sh download failed (check VM logs)"
echo " - Wheel installation failed (check VM logs)"
echo ""
echo "To debug, SSH to VM and check logs:"
echo " gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke --tunnel-through-iap"
echo " sudo journalctl -u encoding-worker -n 100"
echo " cat /var/log/encoding-worker-startup.log"
echo "=============================================="
exit 1
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
+1 -1
View File
@@ -33,7 +33,7 @@
(*) GCE Encoding Worker: c4d-highcpu-32 VM with AMD EPYC 9B45 (Turin) for
high-performance FFmpeg encoding (4.92x faster than previous c4-standard-8).
Used for both final video encoding and preview video generation.
Uses LocalEncodingService via GCS wheel deployment.
Uses immutable deployment pattern - see infrastructure/encoding-worker/README.md.
```
## Processing Pipeline
+27 -2
View File
@@ -177,8 +177,33 @@ Use `pulumi up --skip-preview --yes` when CI service account can't run preview.
### Docker Disk Management
Self-hosted runners need aggressive disk cleanup. Use threshold-based (70%) not age-based cleanup.
### Version Sort for Artifact Selection
When downloading multiple versioned files (wheels, tarballs), use `sort -V | tail -1` not `ls -t | head -1`. Files downloaded simultaneously have the same timestamp, so time-based sorting picks arbitrarily. Version sorting (`sort -V`) correctly identifies the highest semantic version.
### Version Sort for Artifact Selection (SUPERSEDED)
**Note:** This lesson is superseded by the "Immutable Deployment Pattern" below, which eliminates version sorting entirely.
Original lesson: When downloading multiple versioned files (wheels, tarballs), use `sort -V | tail -1` not `ls -t | head -1`. Files downloaded simultaneously have the same timestamp, so time-based sorting picks arbitrarily.
### Immutable Deployment Pattern for GCE Workers
When deploying to long-running VMs (like the encoding worker), avoid baking deployment logic into Packer images:
**Problem:** The encoding worker had recurring version mismatch issues because:
1. `startup.sh` was baked into the Packer image
2. Fixing logic required image rebuild + VM recreation
3. CI only warned on version mismatch, didn't fail
4. Multiple deployment vectors (CI, Pulumi, manual SSH) caused drift
**Solution:** Self-updating startup script from GCS:
1. Packer image contains minimal `bootstrap.sh` (~15 lines, rarely changes)
2. `bootstrap.sh` downloads real `startup.sh` from GCS on every service start
3. CI uploads `startup.sh`, wheel, and version manifest to GCS
4. CI **strictly verifies** deployed version - fails on mismatch
**Benefits:**
- Logic changes via CI, no Packer rebuild needed
- Fixed wheel path (`karaoke_gen-current.whl`) eliminates sorting bugs
- Version verification is mandatory, not optional
- Single source of truth (GCS), no drift from manual changes
See `infrastructure/encoding-worker/README.md` for implementation details.
---
+2
View File
@@ -42,6 +42,8 @@
## Recent Changes
- **Encoding Worker Immutable Deployment** (2026-01-22): Implemented robust deployment pattern for GCE encoding worker to eliminate version mismatch issues. Changes: (1) Fixed wheel path `karaoke_gen-current.whl` eliminates version sorting bugs, (2) Self-updating startup script downloaded from GCS on every service start (no Packer rebuild needed for logic changes), (3) Version manifest with strict CI verification (deployment fails on mismatch). Root cause: startup.sh was baked into Packer image, requiring image rebuild + VM recreation for fixes. New pattern allows CI to update deployment logic without infrastructure changes. See [infrastructure/encoding-worker/README.md](../infrastructure/encoding-worker/README.md) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#immutable-deployment-pattern-for-gce-workers).
- **Theme Validation & Default Removal** (2026-01-22): **BREAKING CHANGE in v0.109.0** - Removed all default style fallback logic to ensure cloud jobs always use complete themes. Style getter functions now raise `ValueError` if themes are incomplete instead of merging with defaults. Added `--list-themes` and `--validate-theme` CLI flags. All cloud jobs guaranteed to use complete nomad theme with no silent fallbacks. See [archive/2026-01-22-theme-validation-remove-defaults.md](archive/2026-01-22-theme-validation-remove-defaults.md).
- **Admin Job Reprocess** (2026-01-21): Added "Reprocess" button to admin job reset toolbar. Allows re-encoding and re-distributing a job with existing settings after deleting outputs. Resets job to `instrumental_selected` state and automatically triggers video worker. Clears video/encoding/distribution state while preserving instrumental selection and lyrics metadata. Use case: Fix distribution issues, re-upload to YouTube with different settings, or re-run encoding after infrastructure changes. See [API.md](API.md#admin-job-reset).
+108
View File
@@ -0,0 +1,108 @@
# Encoding Worker Deployment
This directory contains the scripts for the GCE encoding worker's immutable deployment pattern.
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ PACKER IMAGE (rarely changes) │
├─────────────────────────────────────────────────────────────────┤
│ - Python 3.13, FFmpeg, fonts (static dependencies) │
│ - bootstrap.sh (downloads startup.sh from GCS) │
│ - systemd service definition │
└─────────────────────────────────────────────────────────────────┘
│
▼ downloads on every service start
┌─────────────────────────────────────────────────────────────────┐
│ GCS (CI-managed, always current) │
├─────────────────────────────────────────────────────────────────┤
│ gs://karaoke-gen-storage-nomadkaraoke/ │
│ ├── encoding-worker/ │
│ │ ├── startup.sh ← Main logic (this file) │
│ │ └── version.txt ← Expected version number │
│ └── wheels/ │
│ └── karaoke_gen-current.whl ← Latest wheel (fixed path) │
└─────────────────────────────────────────────────────────────────┘
```
## Files
| File | Location | Purpose |
|------|----------|---------|
| `bootstrap.sh` | Baked in Packer image | Downloads and runs startup.sh |
| `startup.sh` | Deployed to GCS by CI | Main startup logic |
| `version.txt` | Written to GCS by CI | Expected version for verification |
## Deployment Flow
1. **CI builds wheel** → uploads to `wheels/karaoke_gen-current.whl`
2. **CI uploads startup.sh** → `encoding-worker/startup.sh`
3. **CI writes version** → `encoding-worker/version.txt`
4. **CI restarts service** → triggers bootstrap.sh
5. **bootstrap.sh** downloads fresh startup.sh from GCS
6. **startup.sh** downloads wheel, installs, verifies version
7. **CI verifies** health endpoint returns correct version
8. **CI FAILS** if version mismatch
## Why This Pattern?
### Before (problematic)
- startup.sh baked into Packer image
- Required image rebuild + VM recreation for logic changes
- 51 versioned wheels, sorting could fail
- CI warned on version mismatch but continued
### After (robust)
- startup.sh downloaded from GCS on every start
- Logic changes via CI, no image rebuild needed
- Fixed path `karaoke_gen-current.whl`, no sorting
- CI FAILS on version mismatch
## Modifying Startup Logic
1. Edit `infrastructure/encoding-worker/startup.sh`
2. Push to main
3. CI automatically deploys to GCS
4. Service restart picks up new script
No Packer image rebuild needed!
## When to Rebuild Packer Image
Only rebuild when changing:
- Python version
- FFmpeg version
- System packages/fonts
- bootstrap.sh (should be extremely rare)
## Manual Operations
```bash
# Check current version on VM
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--tunnel-through-iap \
--command="curl -s http://localhost:8080/health | jq '.wheel_version'"
# View startup logs
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--tunnel-through-iap \
--command="sudo journalctl -u encoding-worker -n 50"
# Force service restart (triggers fresh startup.sh download)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--tunnel-through-iap \
--command="sudo systemctl restart encoding-worker"
```
## Troubleshooting
### Version mismatch after deployment
1. Check CI logs for GCS upload success
2. Check `/var/log/encoding-worker-startup.log` on VM
3. Verify `version.txt` in GCS matches expected
### Service won't start
1. Check bootstrap.sh can reach GCS: `gsutil ls gs://karaoke-gen-storage-nomadkaraoke/`
2. Check Secret Manager access for API key
3. Check startup.sh logs: `cat /var/log/encoding-worker-startup.log`
@@ -0,0 +1,42 @@
#!/bin/bash
# Encoding Worker Bootstrap Script
#
# This is the ONLY script baked into the Packer image.
# It downloads and executes the real startup script from GCS.
#
# Why this pattern?
# - startup.sh logic can be updated without rebuilding the image
# - All code changes go through CI → GCS → service restart
# - No version sorting bugs (fixed path for wheel)
# - Strict version verification before starting
#
# This script should NEVER need modification. All logic lives in
# the GCS-hosted startup.sh which is managed by CI.
set -e
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
WORKER_DIR="/opt/encoding-worker"
LOG_FILE="/var/log/encoding-worker-bootstrap.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "=== Bootstrap: $(date) ==="
echo "Downloading latest startup script from GCS..."
# Download the CI-managed startup script
if ! gsutil cp "${BUCKET}/encoding-worker/startup.sh" "${WORKER_DIR}/startup-latest.sh" 2>&1; then
echo "ERROR: Failed to download startup.sh from GCS"
echo "Falling back to local startup.sh if available..."
if [ -f "${WORKER_DIR}/startup-fallback.sh" ]; then
cp "${WORKER_DIR}/startup-fallback.sh" "${WORKER_DIR}/startup-latest.sh"
else
echo "FATAL: No fallback startup script available"
exit 1
fi
fi
chmod +x "${WORKER_DIR}/startup-latest.sh"
echo "=== Bootstrap: Executing downloaded startup script ==="
exec "${WORKER_DIR}/startup-latest.sh"
+108
View File
@@ -0,0 +1,108 @@
#!/bin/bash
# Encoding Worker Startup Script
#
# This script is deployed to GCS and downloaded by bootstrap.sh on every service start.
# It handles:
# 1. Fetching API key from Secret Manager
# 2. Downloading the current wheel (fixed path, no version sorting!)
# 3. Installing the wheel
# 4. Verifying installed version matches expected version
# 5. Failing fast if version mismatch (don't start with wrong code)
#
# CI uploads this script to: gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/startup.sh
# CI also uploads:
# - gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-current.whl (latest wheel)
# - gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/version.txt (expected version)
set -e
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
WORKER_DIR="/opt/encoding-worker"
VENV="${WORKER_DIR}/venv"
TMP_DIR="/tmp/encoding-worker-startup"
LOG_FILE="/var/log/encoding-worker-startup.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "=============================================="
echo "Encoding Worker Startup: $(date)"
echo "=============================================="
# Create temp directory for downloads
rm -rf "${TMP_DIR}"
mkdir -p "${TMP_DIR}"
# 1. Fetch API key from Secret Manager
echo "[1/5] Fetching API key from Secret Manager..."
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
if [ -z "$ENCODING_API_KEY" ]; then
echo "WARNING: No API key found in Secret Manager"
echo "Service will start but may reject requests"
fi
# Write environment file for systemd
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > "${WORKER_DIR}/env"
chmod 600 "${WORKER_DIR}/env"
# 2. Download expected version manifest
echo "[2/5] Downloading version manifest..."
if ! gsutil cp "${BUCKET}/encoding-worker/version.txt" "${TMP_DIR}/version.txt" 2>&1; then
echo "WARNING: Could not download version.txt - skipping version verification"
EXPECTED_VERSION=""
else
EXPECTED_VERSION=$(cat "${TMP_DIR}/version.txt" | tr -d '[:space:]')
echo "Expected version: ${EXPECTED_VERSION}"
fi
# 3. Download current wheel (FIXED PATH - no sorting!)
echo "[3/5] Downloading current wheel..."
WHEEL_PATH="${TMP_DIR}/karaoke_gen-current.whl"
if ! gsutil cp "${BUCKET}/wheels/karaoke_gen-current.whl" "${WHEEL_PATH}" 2>&1; then
echo "ERROR: Failed to download karaoke_gen-current.whl"
echo "Checking for any available wheel as fallback..."
# Fallback: try to get the latest versioned wheel (with proper version sort)
LATEST_WHEEL=$(gsutil ls "${BUCKET}/wheels/karaoke_gen-*.whl" 2>/dev/null | grep -v 'current' | sort -V | tail -1 || echo "")
if [ -n "$LATEST_WHEEL" ]; then
echo "Fallback: Using ${LATEST_WHEEL}"
gsutil cp "${LATEST_WHEEL}" "${WHEEL_PATH}"
else
echo "FATAL: No wheel available in GCS"
exit 1
fi
fi
# 4. Install the wheel
echo "[4/5] Installing wheel..."
"${VENV}/bin/pip" install --upgrade --quiet --force-reinstall "${WHEEL_PATH}"
# 5. Verify installed version
echo "[5/5] Verifying installed version..."
INSTALLED_VERSION=$("${VENV}/bin/pip" show karaoke-gen 2>/dev/null | grep "^Version:" | cut -d' ' -f2 | tr -d '[:space:]')
echo "Installed version: ${INSTALLED_VERSION}"
if [ -n "$EXPECTED_VERSION" ]; then
if [ "${INSTALLED_VERSION}" != "${EXPECTED_VERSION}" ]; then
echo "=============================================="
echo "FATAL: VERSION MISMATCH!"
echo " Expected: ${EXPECTED_VERSION}"
echo " Installed: ${INSTALLED_VERSION}"
echo "=============================================="
echo "Refusing to start with incorrect version."
echo "This indicates a deployment issue - check CI logs."
exit 1
fi
echo "Version verified successfully!"
else
echo "WARNING: Version verification skipped (no version.txt)"
fi
# Cleanup
rm -rf "${TMP_DIR}"
echo "=============================================="
echo "Startup complete: karaoke-gen v${INSTALLED_VERSION}"
echo "=============================================="
+18 -7
View File
@@ -63,19 +63,27 @@ The image contains:
- **Systemd service** `encoding-worker.service` (enabled, not started)
- **Startup script** `/opt/encoding-worker/startup.sh`
## Runtime Behavior
## Runtime Behavior (Immutable Deployment Pattern)
When the VM boots with this image:
1. **Startup script** (`/opt/encoding-worker/startup.sh`) runs via `ExecStartPre`:
- Fetches API key from Secret Manager
- Downloads latest `karaoke-gen` wheel from GCS
- Installs wheel into venv
1. **bootstrap.sh** runs via `ExecStartPre`:
- Downloads `startup.sh` from GCS (CI-managed, always current)
- Executes the downloaded startup script
2. **Encoding worker service** starts via `ExecStart`:
2. **startup.sh** (downloaded from GCS):
- Fetches API key from Secret Manager
- Downloads `karaoke_gen-current.whl` from GCS (fixed path, no sorting!)
- Installs wheel into venv
- **Verifies** installed version matches `version.txt` from GCS
- Fails fast if version mismatch
3. **Encoding worker service** starts via `ExecStart`:
- Runs `uvicorn backend.services.gce_encoding.main:app`
- Listens on port 8080
See `infrastructure/encoding-worker/README.md` for details on this pattern.
## When to Rebuild
Rebuild the image when:
@@ -84,8 +92,11 @@ Rebuild the image when:
- FFmpeg version needs updating
- New system packages required
- Font packages change
- bootstrap.sh changes (extremely rare)
**No rebuild needed** for application code changes - the wheel is downloaded at runtime.
**No rebuild needed** for:
- Application code changes (wheel downloaded at runtime)
- Startup logic changes (startup.sh downloaded from GCS)
## Image Family
+82 -35
View File
@@ -8,10 +8,13 @@
# - Virtual environment exists at /opt/encoding-worker/venv
# - Systemd service is configured (but not started - no API key yet)
#
# At runtime, the startup script only needs to:
# 1. Fetch API key from Secret Manager
# 2. Download latest karaoke-gen wheel
# 3. Start the systemd service
# IMPORTANT: This image uses the "immutable deployment" pattern:
# - bootstrap.sh is baked into the image (minimal, rarely changes)
# - bootstrap.sh downloads startup.sh from GCS on every service start
# - startup.sh contains all logic and is CI-managed
# - This allows logic updates without rebuilding the image
#
# See infrastructure/encoding-worker/README.md for details.
set -e
@@ -95,46 +98,83 @@ source venv/bin/activate
pip install --upgrade pip
pip install fastapi uvicorn google-cloud-storage aiofiles aiohttp
# Create startup helper script (runs via ExecStartPre before service starts)
# This fetches API key and installs latest wheel at boot time
cat > /opt/encoding-worker/startup.sh << 'STARTUP'
# Create bootstrap script (runs via ExecStartPre before service starts)
# This minimal script downloads the REAL startup script from GCS.
# All actual logic lives in the GCS-hosted startup.sh, managed by CI.
#
# Why this pattern?
# - startup.sh can be updated without rebuilding the Packer image
# - All code changes go through CI -> GCS -> service restart
# - No version sorting bugs (fixed wheel path)
# - Strict version verification before starting
cat > /opt/encoding-worker/bootstrap.sh << 'BOOTSTRAP'
#!/bin/bash
# Encoding Worker Bootstrap Script
#
# This is the ONLY script baked into the Packer image.
# It downloads and executes the real startup script from GCS.
set -e
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
WORKER_DIR="/opt/encoding-worker"
LOG_FILE="/var/log/encoding-worker-bootstrap.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "=== Bootstrap: $(date) ==="
echo "Downloading latest startup script from GCS..."
# Download the CI-managed startup script
if ! gsutil cp "${BUCKET}/encoding-worker/startup.sh" "${WORKER_DIR}/startup-latest.sh" 2>&1; then
echo "ERROR: Failed to download startup.sh from GCS"
echo "Falling back to local startup.sh if available..."
if [ -f "${WORKER_DIR}/startup-fallback.sh" ]; then
cp "${WORKER_DIR}/startup-fallback.sh" "${WORKER_DIR}/startup-latest.sh"
else
echo "FATAL: No fallback startup script available"
exit 1
fi
fi
chmod +x "${WORKER_DIR}/startup-latest.sh"
echo "=== Bootstrap: Executing downloaded startup script ==="
exec "${WORKER_DIR}/startup-latest.sh"
BOOTSTRAP
chmod +x /opt/encoding-worker/bootstrap.sh
# Create fallback startup script (used if GCS is unreachable)
# This is a simplified version that at least tries to start the service
cat > /opt/encoding-worker/startup-fallback.sh << 'FALLBACK'
#!/bin/bash
set -e
echo "=== Encoding worker startup at $(date) ==="
echo "=== FALLBACK startup at $(date) ==="
echo "WARNING: Using fallback startup - GCS was unreachable"
# Fetch API key from Secret Manager
echo "Fetching API key from Secret Manager..."
WORKER_DIR="/opt/encoding-worker"
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
# Fetch API key
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > "${WORKER_DIR}/env"
chmod 600 "${WORKER_DIR}/env"
if [ -z "$ENCODING_API_KEY" ]; then
echo "WARNING: No API key found in Secret Manager"
# Try to install wheel from fixed path
echo "Attempting to download wheel..."
if gsutil cp "${BUCKET}/wheels/karaoke_gen-current.whl" /tmp/karaoke_gen.whl 2>/dev/null; then
"${WORKER_DIR}/venv/bin/pip" install --upgrade --quiet /tmp/karaoke_gen.whl || true
fi
# Write environment file for systemd
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > /opt/encoding-worker/env
echo "=== Fallback startup complete ==="
FALLBACK
# Download and install latest karaoke-gen wheel from GCS
# This enables hot code updates without rebuilding the image
echo "Installing latest karaoke-gen wheel from GCS..."
gsutil cp "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl" /tmp/ 2>/dev/null || true
# Use version sort (-V) to get the highest version, not time sort (-t)
# All wheels download at the same time, so -t picks arbitrarily
WHEEL_FILE=$(ls /tmp/karaoke_gen-*.whl 2>/dev/null | sort -V | tail -1)
if [ -n "$WHEEL_FILE" ]; then
echo "Installing wheel: $WHEEL_FILE"
/opt/encoding-worker/venv/bin/pip install --upgrade --quiet "$WHEEL_FILE" || echo "WARNING: Failed to install wheel"
else
echo "No wheel found in GCS"
fi
echo "=== Startup script complete ==="
STARTUP
chmod +x /opt/encoding-worker/startup.sh
chmod +x /opt/encoding-worker/startup-fallback.sh
# Create systemd service
# ExecStartPre runs the startup script to fetch API key and install wheel
# ExecStartPre runs bootstrap.sh which downloads and runs the real startup.sh from GCS
# ExecStart runs uvicorn with the encoding worker from the installed wheel
cat > /etc/systemd/system/encoding-worker.service << 'SYSTEMD'
[Unit]
@@ -146,11 +186,13 @@ Wants=docker.service
Type=simple
User=root
WorkingDirectory=/opt/encoding-worker
ExecStartPre=/opt/encoding-worker/startup.sh
# bootstrap.sh downloads startup.sh from GCS, then runs it
ExecStartPre=/opt/encoding-worker/bootstrap.sh
ExecStart=/opt/encoding-worker/venv/bin/uvicorn backend.services.gce_encoding.main:app --host 0.0.0.0 --port 8080
Restart=always
RestartSec=10
TimeoutStartSec=300
# Increased timeout to allow for GCS downloads
TimeoutStartSec=600
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
EnvironmentFile=/opt/encoding-worker/env
@@ -177,3 +219,8 @@ echo " - FFmpeg at /usr/local/bin/ffmpeg"
echo " - Noto fonts (including CJK)"
echo " - Virtual environment at /opt/encoding-worker/venv"
echo " - Systemd service: encoding-worker.service"
echo ""
echo "Immutable deployment pattern:"
echo " - bootstrap.sh downloads startup.sh from GCS on every start"
echo " - startup.sh is CI-managed, no image rebuild needed for logic changes"
echo " - See infrastructure/encoding-worker/README.md"
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.109.0"
version = "0.110.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"