From b68a46d174e3d683cbf73d8f7e808a507bcc0b5a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andrew=20Beveridge=20=E2=98=84=EF=B8=8F?= Date: Thu, 22 Jan 2026 23:56:53 -0800 Subject: [PATCH] feat(infra): Implement immutable deployment pattern for encoding worker (#333) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Implements robust deployment pattern for GCE encoding worker to eliminate recurring version mismatch issues - Fixed wheel path (`karaoke_gen-current.whl`) eliminates version sorting bugs - Self-updating startup script downloaded from GCS on every service start - Version manifest with strict CI verification (deployment fails on mismatch) ## Problem The encoding worker had recurring version issues because: - `startup.sh` was baked into the Packer image (required rebuild + VM recreation) - Wheel selection used version sorting across 51 files - CI only warned on version mismatch, didn't fail - Multiple deployment vectors (CI, Pulumi, manual SSH) caused drift ## Solution 1. **Fixed wheel path**: CI uploads to `karaoke_gen-current.whl` (no sorting needed) 2. **Self-updating startup**: `bootstrap.sh` downloads `startup.sh` from GCS on every start 3. **Version manifest**: CI writes expected version, service verifies before starting 4. **Strict CI verification**: Deployment fails if version doesn't match (was just warning) ## Changes - **New**: `infrastructure/encoding-worker/` with `bootstrap.sh`, `startup.sh`, README - **Modified**: CI workflow for fixed paths and strict verification - **Modified**: Packer `provision.sh` for bootstrap pattern - **Updated**: `LESSONS-LEARNED.md` with immutable deployment pattern - **Updated**: `docs/README.md` and `docs/ARCHITECTURE.md` ## Test plan - [x] Shell scripts validated (`bash -n`) - [x] Backend tests pass (1703 passed) - [x] CodeRabbit review passed - [ ] After merge: Rebuild Packer image - [ ] After merge: Recreate VM to use new image - [ ] After merge: Verify first deployment uses new pattern ## Post-Merge Steps Required 1. Rebuild Packer image: `packer build infrastructure/packer/encoding-worker.pkr.hcl` 2. Recreate VM: Delete encoding-worker, run `pulumi up` 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 4.5 --- .../encoding-worker-immutable-deploy.plan.md | 99 ++++++++++++++ .github/workflows/ci.yml | 122 ++++++++++++++---- docs/ARCHITECTURE.md | 2 +- docs/LESSONS-LEARNED.md | 29 ++++- docs/README.md | 2 + infrastructure/encoding-worker/README.md | 108 ++++++++++++++++ infrastructure/encoding-worker/bootstrap.sh | 42 ++++++ infrastructure/encoding-worker/startup.sh | 108 ++++++++++++++++ infrastructure/packer/README.md | 25 +++- infrastructure/packer/scripts/provision.sh | 117 ++++++++++++----- pyproject.toml | 2 +- 11 files changed, 584 insertions(+), 72 deletions(-) create mode 100644 .claude/plans/encoding-worker-immutable-deploy.plan.md create mode 100644 infrastructure/encoding-worker/README.md create mode 100644 infrastructure/encoding-worker/bootstrap.sh create mode 100644 infrastructure/encoding-worker/startup.sh diff --git a/.claude/plans/encoding-worker-immutable-deploy.plan.md b/.claude/plans/encoding-worker-immutable-deploy.plan.md new file mode 100644 index 00000000..3a8ab0a0 --- /dev/null +++ b/.claude/plans/encoding-worker-immutable-deploy.plan.md @@ -0,0 +1,99 @@ +# Plan: Encoding Worker Immutable Deployment Pattern + +## Status: In Progress + +## Problem Statement + +The GCE encoding worker has recurring version mismatch issues because: +1. `startup.sh` is baked into the Packer image (requires image rebuild + VM recreation) +2. Wheel selection uses `sort -V` which can fail if image is outdated +3. CI only restarts service - doesn't verify version strictly +4. Multiple actors (CI, Pulumi, manual SSH) can cause drift +5. 51 old wheel files accumulate in GCS bucket + +## Solution: Immutable Deployment Pattern + +### Core Principles +1. **Single Source of Truth**: All code and config from GCS, nothing baked in image +2. **Fixed Wheel Path**: `karaoke_gen-current.whl` eliminates sorting +3. **Version Manifest**: CI writes expected version, service MUST match +4. **Strict Verification**: CI FAILS if deployed version doesn't match +5. **Self-Updating Startup**: Download startup.sh from GCS on every boot + +## Implementation Steps + +### Phase 1: GCS Structure & Scripts + +- [x] **Step 1.1**: Create `infrastructure/encoding-worker/` directory +- [x] **Step 1.2**: Create `bootstrap.sh` (minimal, baked in image) +- [x] **Step 1.3**: Create `startup.sh` (CI-managed, deployed to GCS) +- [x] **Step 1.4**: Create README.md with documentation + +### Phase 2: CI Workflow Updates + +- [x] **Step 2.1**: Update CI to upload wheel to fixed path +- [x] **Step 2.2**: Update CI to upload startup.sh to GCS +- [x] **Step 2.3**: Update CI to write version.txt manifest +- [x] **Step 2.4**: Update CI to strictly verify deployed version (fail on mismatch) + +### Phase 3: Packer Image Updates + +- [x] **Step 3.1**: Update provision.sh to use bootstrap.sh +- [ ] **Step 3.2**: Rebuild Packer image (manual step after merge) +- [ ] **Step 3.3**: Recreate VM to use new image (manual step after merge) + +### Phase 4: Documentation & Cleanup + +- [x] **Step 4.1**: Update LESSONS-LEARNED.md with new pattern +- [x] **Step 4.2**: Update infrastructure/packer/README.md +- [ ] **Step 4.3**: Add lifecycle rule for old wheels (optional, later) + +## New GCS Structure + +``` +gs://karaoke-gen-storage-nomadkaraoke/ +├── wheels/ +│ ├── karaoke_gen-current.whl # Latest (CI overwrites) +│ └── karaoke_gen-{version}.whl # Versioned for rollback +│ +└── encoding-worker/ + ├── bootstrap.sh # Reference copy + ├── startup.sh # CI-managed, downloaded on boot + └── version.txt # Expected version number +``` + +## Files to Create/Modify + +| File | Action | Purpose | +|------|--------|---------| +| `infrastructure/encoding-worker/bootstrap.sh` | Create | Minimal loader (baked in image) | +| `infrastructure/encoding-worker/startup.sh` | Create | Main logic (deployed to GCS) | +| `infrastructure/packer/scripts/provision.sh` | Modify | Use bootstrap.sh | +| `.github/workflows/ci.yml` | Modify | New deployment flow | +| `docs/LESSONS-LEARNED.md` | Modify | Document pattern | + +## Testing Strategy + +1. **Unit test**: Verify startup.sh logic locally +2. **Integration test**: After merge, verify: + - CI uploads files to correct GCS paths + - Service starts with correct version + - Health endpoint returns expected version +3. **Manual verification**: Check VM logs after first deployment + +## Rollback Plan + +If issues occur: +1. CI can re-upload previous wheel to `karaoke_gen-current.whl` +2. Previous versioned wheels remain in GCS for manual rollback +3. Old Packer images remain for VM rollback + +## Open Questions + +None - design is complete. + +## Notes + +- Bootstrap.sh is ~10 lines and should never need changes +- All logic changes go through startup.sh in GCS +- Version verification is strict - CI fails on mismatch diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5638dc48..5e8d7f08 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -967,8 +967,16 @@ jobs: version: "latest" # ============================================ - # Upload wheel to GCS for GCE encoding worker - # The GCE worker installs this wheel at job start for hot code updates + # Deploy to GCE Encoding Worker (Immutable Deployment Pattern) + # + # This pattern ensures version consistency: + # 1. Upload wheel to FIXED path (no version sorting) + # 2. Upload startup.sh (logic can change without image rebuild) + # 3. Write version manifest for verification + # 4. Restart service (triggers bootstrap.sh -> startup.sh) + # 5. STRICTLY verify deployed version (fail CI on mismatch) + # + # See infrastructure/encoding-worker/README.md for details # ============================================ - name: Download wheel artifact @@ -977,52 +985,114 @@ jobs: name: package-dist path: dist/ - - name: Upload wheel to GCS for GCE worker + - name: Deploy to GCE Encoding Worker run: | - echo "📦 Uploading wheel to GCS for GCE encoding worker..." + VERSION="${{ steps.version_check.outputs.version }}" + BUCKET="gs://karaoke-gen-storage-${{ env.PROJECT_ID }}" + + echo "🚀 Deploying encoding worker v${VERSION}..." + + # 1. Find the wheel file WHEEL_FILE=$(ls dist/*.whl | head -1) if [ -z "$WHEEL_FILE" ]; then echo "::error::No wheel file found in dist/" exit 1 fi - echo "Found wheel: $WHEEL_FILE" + echo "📦 Found wheel: $WHEEL_FILE" - # Upload to GCS (GCE worker downloads latest at job start) - gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/ + # 2. Upload wheel to FIXED path (atomic via temp file) + # This eliminates version sorting - always use karaoke_gen-current.whl + echo "📤 Uploading wheel to fixed path..." + gsutil cp "$WHEEL_FILE" "${BUCKET}/wheels/karaoke_gen-current.whl.tmp" + gsutil mv "${BUCKET}/wheels/karaoke_gen-current.whl.tmp" \ + "${BUCKET}/wheels/karaoke_gen-current.whl" - echo "✅ Wheel uploaded to GCS" + # Also keep versioned copy for rollback/debugging + gsutil cp "$WHEEL_FILE" "${BUCKET}/wheels/" + + # 3. Upload startup.sh (allows logic changes without image rebuild) + echo "📤 Uploading startup.sh..." + gsutil cp infrastructure/encoding-worker/startup.sh \ + "${BUCKET}/encoding-worker/startup.sh" + + # 4. Write version manifest for verification + echo "📝 Writing version manifest..." + echo "${VERSION}" | gsutil cp - "${BUCKET}/encoding-worker/version.txt" + + echo "✅ GCS deployment complete" - name: Restart GCE Encoding Worker run: | - echo "🔄 Restarting encoding worker to pick up new wheel..." + VERSION="${{ steps.version_check.outputs.version }}" + + echo "🔄 Restarting encoding worker..." # Use IAP tunneling - works directly with service account credentials - # (avoids SSH key propagation delays that cause silent failures) gcloud compute ssh encoding-worker \ --zone=us-central1-c \ --project=nomadkaraoke \ --tunnel-through-iap \ --command="sudo systemctl restart encoding-worker" - echo "⏳ Waiting for service to start..." - sleep 15 + echo "⏳ Waiting for service to start (up to 60s)..." + sleep 20 - # Verify the restart worked by checking the wheel version - NEW_VERSION=$(gcloud compute ssh encoding-worker \ - --zone=us-central1-c \ - --project=nomadkaraoke \ - --tunnel-through-iap \ - --command="curl -s http://localhost:8080/health | grep -o '\"wheel_version\":\"[^\"]*\"' | cut -d'\"' -f4") + - name: Verify Encoding Worker Version + run: | + VERSION="${{ steps.version_check.outputs.version }}" - EXPECTED_VERSION=$(cat pyproject.toml | grep 'version = ' | head -1 | cut -d'"' -f2) + echo "🔍 Verifying encoding worker version..." - if [ "$NEW_VERSION" = "$EXPECTED_VERSION" ]; then - echo "✅ Encoding worker restarted successfully (v$NEW_VERSION)" - else - echo "⚠️ Version mismatch: encoder has v$NEW_VERSION, expected v$EXPECTED_VERSION" - echo " This may indicate the restart failed or the wheel wasn't downloaded" - # Don't fail the build - the encoder will still work, just with old version - fi + # Retry loop - service may take time to fully start + MAX_RETRIES=6 + RETRY_DELAY=10 + + for i in $(seq 1 $MAX_RETRIES); do + echo "Attempt $i/$MAX_RETRIES..." + + NEW_VERSION=$(gcloud compute ssh encoding-worker \ + --zone=us-central1-c \ + --project=nomadkaraoke \ + --tunnel-through-iap \ + --command="curl -s http://localhost:8080/health 2>/dev/null | grep -o '\"wheel_version\":\"[^\"]*\"' | cut -d'\"' -f4" 2>/dev/null || echo "") + + if [ "$NEW_VERSION" = "$VERSION" ]; then + echo "✅ Encoding worker version verified: v$NEW_VERSION" + exit 0 + fi + + if [ -n "$NEW_VERSION" ]; then + echo "⚠️ Version: $NEW_VERSION (expected: $VERSION)" + else + echo "⚠️ Health endpoint not responding yet" + fi + + if [ $i -lt $MAX_RETRIES ]; then + echo " Retrying in ${RETRY_DELAY}s..." + sleep $RETRY_DELAY + fi + done + + # STRICT VERIFICATION: Fail CI on version mismatch + echo "" + echo "==============================================" + echo "❌ ENCODING WORKER DEPLOYMENT FAILED" + echo "==============================================" + echo "Expected version: $VERSION" + echo "Deployed version: $NEW_VERSION" + echo "" + echo "Possible causes:" + echo " - GCS upload failed (check 'Deploy to GCE' step)" + echo " - startup.sh download failed (check VM logs)" + echo " - Wheel installation failed (check VM logs)" + echo "" + echo "To debug, SSH to VM and check logs:" + echo " gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke --tunnel-through-iap" + echo " sudo journalctl -u encoding-worker -n 100" + echo " cat /var/log/encoding-worker-startup.log" + echo "==============================================" + + exit 1 - name: Configure Docker for Artifact Registry run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index d69202ad..4a96add3 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -33,7 +33,7 @@ (*) GCE Encoding Worker: c4d-highcpu-32 VM with AMD EPYC 9B45 (Turin) for high-performance FFmpeg encoding (4.92x faster than previous c4-standard-8). Used for both final video encoding and preview video generation. - Uses LocalEncodingService via GCS wheel deployment. + Uses immutable deployment pattern - see infrastructure/encoding-worker/README.md. ``` ## Processing Pipeline diff --git a/docs/LESSONS-LEARNED.md b/docs/LESSONS-LEARNED.md index 9b3875c5..5a8fac14 100644 --- a/docs/LESSONS-LEARNED.md +++ b/docs/LESSONS-LEARNED.md @@ -177,8 +177,33 @@ Use `pulumi up --skip-preview --yes` when CI service account can't run preview. ### Docker Disk Management Self-hosted runners need aggressive disk cleanup. Use threshold-based (70%) not age-based cleanup. -### Version Sort for Artifact Selection -When downloading multiple versioned files (wheels, tarballs), use `sort -V | tail -1` not `ls -t | head -1`. Files downloaded simultaneously have the same timestamp, so time-based sorting picks arbitrarily. Version sorting (`sort -V`) correctly identifies the highest semantic version. +### Version Sort for Artifact Selection (SUPERSEDED) +**Note:** This lesson is superseded by the "Immutable Deployment Pattern" below, which eliminates version sorting entirely. + +Original lesson: When downloading multiple versioned files (wheels, tarballs), use `sort -V | tail -1` not `ls -t | head -1`. Files downloaded simultaneously have the same timestamp, so time-based sorting picks arbitrarily. + +### Immutable Deployment Pattern for GCE Workers +When deploying to long-running VMs (like the encoding worker), avoid baking deployment logic into Packer images: + +**Problem:** The encoding worker had recurring version mismatch issues because: +1. `startup.sh` was baked into the Packer image +2. Fixing logic required image rebuild + VM recreation +3. CI only warned on version mismatch, didn't fail +4. Multiple deployment vectors (CI, Pulumi, manual SSH) caused drift + +**Solution:** Self-updating startup script from GCS: +1. Packer image contains minimal `bootstrap.sh` (~15 lines, rarely changes) +2. `bootstrap.sh` downloads real `startup.sh` from GCS on every service start +3. CI uploads `startup.sh`, wheel, and version manifest to GCS +4. CI **strictly verifies** deployed version - fails on mismatch + +**Benefits:** +- Logic changes via CI, no Packer rebuild needed +- Fixed wheel path (`karaoke_gen-current.whl`) eliminates sorting bugs +- Version verification is mandatory, not optional +- Single source of truth (GCS), no drift from manual changes + +See `infrastructure/encoding-worker/README.md` for implementation details. --- diff --git a/docs/README.md b/docs/README.md index 0dbdaf28..16f7897a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -42,6 +42,8 @@ ## Recent Changes +- **Encoding Worker Immutable Deployment** (2026-01-22): Implemented robust deployment pattern for GCE encoding worker to eliminate version mismatch issues. Changes: (1) Fixed wheel path `karaoke_gen-current.whl` eliminates version sorting bugs, (2) Self-updating startup script downloaded from GCS on every service start (no Packer rebuild needed for logic changes), (3) Version manifest with strict CI verification (deployment fails on mismatch). Root cause: startup.sh was baked into Packer image, requiring image rebuild + VM recreation for fixes. New pattern allows CI to update deployment logic without infrastructure changes. See [infrastructure/encoding-worker/README.md](../infrastructure/encoding-worker/README.md) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#immutable-deployment-pattern-for-gce-workers). + - **Theme Validation & Default Removal** (2026-01-22): **BREAKING CHANGE in v0.109.0** - Removed all default style fallback logic to ensure cloud jobs always use complete themes. Style getter functions now raise `ValueError` if themes are incomplete instead of merging with defaults. Added `--list-themes` and `--validate-theme` CLI flags. All cloud jobs guaranteed to use complete nomad theme with no silent fallbacks. See [archive/2026-01-22-theme-validation-remove-defaults.md](archive/2026-01-22-theme-validation-remove-defaults.md). - **Admin Job Reprocess** (2026-01-21): Added "Reprocess" button to admin job reset toolbar. Allows re-encoding and re-distributing a job with existing settings after deleting outputs. Resets job to `instrumental_selected` state and automatically triggers video worker. Clears video/encoding/distribution state while preserving instrumental selection and lyrics metadata. Use case: Fix distribution issues, re-upload to YouTube with different settings, or re-run encoding after infrastructure changes. See [API.md](API.md#admin-job-reset). diff --git a/infrastructure/encoding-worker/README.md b/infrastructure/encoding-worker/README.md new file mode 100644 index 00000000..552acea1 --- /dev/null +++ b/infrastructure/encoding-worker/README.md @@ -0,0 +1,108 @@ +# Encoding Worker Deployment + +This directory contains the scripts for the GCE encoding worker's immutable deployment pattern. + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ PACKER IMAGE (rarely changes) │ +├─────────────────────────────────────────────────────────────────┤ +│ - Python 3.13, FFmpeg, fonts (static dependencies) │ +│ - bootstrap.sh (downloads startup.sh from GCS) │ +│ - systemd service definition │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ downloads on every service start +┌─────────────────────────────────────────────────────────────────┐ +│ GCS (CI-managed, always current) │ +├─────────────────────────────────────────────────────────────────┤ +│ gs://karaoke-gen-storage-nomadkaraoke/ │ +│ ├── encoding-worker/ │ +│ │ ├── startup.sh ← Main logic (this file) │ +│ │ └── version.txt ← Expected version number │ +│ └── wheels/ │ +│ └── karaoke_gen-current.whl ← Latest wheel (fixed path) │ +└─────────────────────────────────────────────────────────────────┘ +``` + +## Files + +| File | Location | Purpose | +|------|----------|---------| +| `bootstrap.sh` | Baked in Packer image | Downloads and runs startup.sh | +| `startup.sh` | Deployed to GCS by CI | Main startup logic | +| `version.txt` | Written to GCS by CI | Expected version for verification | + +## Deployment Flow + +1. **CI builds wheel** → uploads to `wheels/karaoke_gen-current.whl` +2. **CI uploads startup.sh** → `encoding-worker/startup.sh` +3. **CI writes version** → `encoding-worker/version.txt` +4. **CI restarts service** → triggers bootstrap.sh +5. **bootstrap.sh** downloads fresh startup.sh from GCS +6. **startup.sh** downloads wheel, installs, verifies version +7. **CI verifies** health endpoint returns correct version +8. **CI FAILS** if version mismatch + +## Why This Pattern? + +### Before (problematic) +- startup.sh baked into Packer image +- Required image rebuild + VM recreation for logic changes +- 51 versioned wheels, sorting could fail +- CI warned on version mismatch but continued + +### After (robust) +- startup.sh downloaded from GCS on every start +- Logic changes via CI, no image rebuild needed +- Fixed path `karaoke_gen-current.whl`, no sorting +- CI FAILS on version mismatch + +## Modifying Startup Logic + +1. Edit `infrastructure/encoding-worker/startup.sh` +2. Push to main +3. CI automatically deploys to GCS +4. Service restart picks up new script + +No Packer image rebuild needed! + +## When to Rebuild Packer Image + +Only rebuild when changing: +- Python version +- FFmpeg version +- System packages/fonts +- bootstrap.sh (should be extremely rare) + +## Manual Operations + +```bash +# Check current version on VM +gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \ + --tunnel-through-iap \ + --command="curl -s http://localhost:8080/health | jq '.wheel_version'" + +# View startup logs +gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \ + --tunnel-through-iap \ + --command="sudo journalctl -u encoding-worker -n 50" + +# Force service restart (triggers fresh startup.sh download) +gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \ + --tunnel-through-iap \ + --command="sudo systemctl restart encoding-worker" +``` + +## Troubleshooting + +### Version mismatch after deployment +1. Check CI logs for GCS upload success +2. Check `/var/log/encoding-worker-startup.log` on VM +3. Verify `version.txt` in GCS matches expected + +### Service won't start +1. Check bootstrap.sh can reach GCS: `gsutil ls gs://karaoke-gen-storage-nomadkaraoke/` +2. Check Secret Manager access for API key +3. Check startup.sh logs: `cat /var/log/encoding-worker-startup.log` diff --git a/infrastructure/encoding-worker/bootstrap.sh b/infrastructure/encoding-worker/bootstrap.sh new file mode 100644 index 00000000..c2db6ef5 --- /dev/null +++ b/infrastructure/encoding-worker/bootstrap.sh @@ -0,0 +1,42 @@ +#!/bin/bash +# Encoding Worker Bootstrap Script +# +# This is the ONLY script baked into the Packer image. +# It downloads and executes the real startup script from GCS. +# +# Why this pattern? +# - startup.sh logic can be updated without rebuilding the image +# - All code changes go through CI → GCS → service restart +# - No version sorting bugs (fixed path for wheel) +# - Strict version verification before starting +# +# This script should NEVER need modification. All logic lives in +# the GCS-hosted startup.sh which is managed by CI. + +set -e + +BUCKET="gs://karaoke-gen-storage-nomadkaraoke" +WORKER_DIR="/opt/encoding-worker" +LOG_FILE="/var/log/encoding-worker-bootstrap.log" + +exec > >(tee -a "$LOG_FILE") 2>&1 + +echo "=== Bootstrap: $(date) ===" +echo "Downloading latest startup script from GCS..." + +# Download the CI-managed startup script +if ! gsutil cp "${BUCKET}/encoding-worker/startup.sh" "${WORKER_DIR}/startup-latest.sh" 2>&1; then + echo "ERROR: Failed to download startup.sh from GCS" + echo "Falling back to local startup.sh if available..." + if [ -f "${WORKER_DIR}/startup-fallback.sh" ]; then + cp "${WORKER_DIR}/startup-fallback.sh" "${WORKER_DIR}/startup-latest.sh" + else + echo "FATAL: No fallback startup script available" + exit 1 + fi +fi + +chmod +x "${WORKER_DIR}/startup-latest.sh" + +echo "=== Bootstrap: Executing downloaded startup script ===" +exec "${WORKER_DIR}/startup-latest.sh" diff --git a/infrastructure/encoding-worker/startup.sh b/infrastructure/encoding-worker/startup.sh new file mode 100644 index 00000000..9d14c373 --- /dev/null +++ b/infrastructure/encoding-worker/startup.sh @@ -0,0 +1,108 @@ +#!/bin/bash +# Encoding Worker Startup Script +# +# This script is deployed to GCS and downloaded by bootstrap.sh on every service start. +# It handles: +# 1. Fetching API key from Secret Manager +# 2. Downloading the current wheel (fixed path, no version sorting!) +# 3. Installing the wheel +# 4. Verifying installed version matches expected version +# 5. Failing fast if version mismatch (don't start with wrong code) +# +# CI uploads this script to: gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/startup.sh +# CI also uploads: +# - gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-current.whl (latest wheel) +# - gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/version.txt (expected version) + +set -e + +BUCKET="gs://karaoke-gen-storage-nomadkaraoke" +WORKER_DIR="/opt/encoding-worker" +VENV="${WORKER_DIR}/venv" +TMP_DIR="/tmp/encoding-worker-startup" +LOG_FILE="/var/log/encoding-worker-startup.log" + +exec > >(tee -a "$LOG_FILE") 2>&1 + +echo "==============================================" +echo "Encoding Worker Startup: $(date)" +echo "==============================================" + +# Create temp directory for downloads +rm -rf "${TMP_DIR}" +mkdir -p "${TMP_DIR}" + +# 1. Fetch API key from Secret Manager +echo "[1/5] Fetching API key from Secret Manager..." +ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "") + +if [ -z "$ENCODING_API_KEY" ]; then + echo "WARNING: No API key found in Secret Manager" + echo "Service will start but may reject requests" +fi + +# Write environment file for systemd +echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > "${WORKER_DIR}/env" +chmod 600 "${WORKER_DIR}/env" + +# 2. Download expected version manifest +echo "[2/5] Downloading version manifest..." +if ! gsutil cp "${BUCKET}/encoding-worker/version.txt" "${TMP_DIR}/version.txt" 2>&1; then + echo "WARNING: Could not download version.txt - skipping version verification" + EXPECTED_VERSION="" +else + EXPECTED_VERSION=$(cat "${TMP_DIR}/version.txt" | tr -d '[:space:]') + echo "Expected version: ${EXPECTED_VERSION}" +fi + +# 3. Download current wheel (FIXED PATH - no sorting!) +echo "[3/5] Downloading current wheel..." +WHEEL_PATH="${TMP_DIR}/karaoke_gen-current.whl" + +if ! gsutil cp "${BUCKET}/wheels/karaoke_gen-current.whl" "${WHEEL_PATH}" 2>&1; then + echo "ERROR: Failed to download karaoke_gen-current.whl" + echo "Checking for any available wheel as fallback..." + + # Fallback: try to get the latest versioned wheel (with proper version sort) + LATEST_WHEEL=$(gsutil ls "${BUCKET}/wheels/karaoke_gen-*.whl" 2>/dev/null | grep -v 'current' | sort -V | tail -1 || echo "") + + if [ -n "$LATEST_WHEEL" ]; then + echo "Fallback: Using ${LATEST_WHEEL}" + gsutil cp "${LATEST_WHEEL}" "${WHEEL_PATH}" + else + echo "FATAL: No wheel available in GCS" + exit 1 + fi +fi + +# 4. Install the wheel +echo "[4/5] Installing wheel..." +"${VENV}/bin/pip" install --upgrade --quiet --force-reinstall "${WHEEL_PATH}" + +# 5. Verify installed version +echo "[5/5] Verifying installed version..." +INSTALLED_VERSION=$("${VENV}/bin/pip" show karaoke-gen 2>/dev/null | grep "^Version:" | cut -d' ' -f2 | tr -d '[:space:]') +echo "Installed version: ${INSTALLED_VERSION}" + +if [ -n "$EXPECTED_VERSION" ]; then + if [ "${INSTALLED_VERSION}" != "${EXPECTED_VERSION}" ]; then + echo "==============================================" + echo "FATAL: VERSION MISMATCH!" + echo " Expected: ${EXPECTED_VERSION}" + echo " Installed: ${INSTALLED_VERSION}" + echo "==============================================" + echo "Refusing to start with incorrect version." + echo "This indicates a deployment issue - check CI logs." + exit 1 + fi + echo "Version verified successfully!" +else + echo "WARNING: Version verification skipped (no version.txt)" +fi + +# Cleanup +rm -rf "${TMP_DIR}" + +echo "==============================================" +echo "Startup complete: karaoke-gen v${INSTALLED_VERSION}" +echo "==============================================" diff --git a/infrastructure/packer/README.md b/infrastructure/packer/README.md index 9789483f..6784a28c 100644 --- a/infrastructure/packer/README.md +++ b/infrastructure/packer/README.md @@ -63,19 +63,27 @@ The image contains: - **Systemd service** `encoding-worker.service` (enabled, not started) - **Startup script** `/opt/encoding-worker/startup.sh` -## Runtime Behavior +## Runtime Behavior (Immutable Deployment Pattern) When the VM boots with this image: -1. **Startup script** (`/opt/encoding-worker/startup.sh`) runs via `ExecStartPre`: - - Fetches API key from Secret Manager - - Downloads latest `karaoke-gen` wheel from GCS - - Installs wheel into venv +1. **bootstrap.sh** runs via `ExecStartPre`: + - Downloads `startup.sh` from GCS (CI-managed, always current) + - Executes the downloaded startup script -2. **Encoding worker service** starts via `ExecStart`: +2. **startup.sh** (downloaded from GCS): + - Fetches API key from Secret Manager + - Downloads `karaoke_gen-current.whl` from GCS (fixed path, no sorting!) + - Installs wheel into venv + - **Verifies** installed version matches `version.txt` from GCS + - Fails fast if version mismatch + +3. **Encoding worker service** starts via `ExecStart`: - Runs `uvicorn backend.services.gce_encoding.main:app` - Listens on port 8080 +See `infrastructure/encoding-worker/README.md` for details on this pattern. + ## When to Rebuild Rebuild the image when: @@ -84,8 +92,11 @@ Rebuild the image when: - FFmpeg version needs updating - New system packages required - Font packages change +- bootstrap.sh changes (extremely rare) -**No rebuild needed** for application code changes - the wheel is downloaded at runtime. +**No rebuild needed** for: +- Application code changes (wheel downloaded at runtime) +- Startup logic changes (startup.sh downloaded from GCS) ## Image Family diff --git a/infrastructure/packer/scripts/provision.sh b/infrastructure/packer/scripts/provision.sh index 46c549ae..624377b5 100644 --- a/infrastructure/packer/scripts/provision.sh +++ b/infrastructure/packer/scripts/provision.sh @@ -8,10 +8,13 @@ # - Virtual environment exists at /opt/encoding-worker/venv # - Systemd service is configured (but not started - no API key yet) # -# At runtime, the startup script only needs to: -# 1. Fetch API key from Secret Manager -# 2. Download latest karaoke-gen wheel -# 3. Start the systemd service +# IMPORTANT: This image uses the "immutable deployment" pattern: +# - bootstrap.sh is baked into the image (minimal, rarely changes) +# - bootstrap.sh downloads startup.sh from GCS on every service start +# - startup.sh contains all logic and is CI-managed +# - This allows logic updates without rebuilding the image +# +# See infrastructure/encoding-worker/README.md for details. set -e @@ -95,46 +98,83 @@ source venv/bin/activate pip install --upgrade pip pip install fastapi uvicorn google-cloud-storage aiofiles aiohttp -# Create startup helper script (runs via ExecStartPre before service starts) -# This fetches API key and installs latest wheel at boot time -cat > /opt/encoding-worker/startup.sh << 'STARTUP' +# Create bootstrap script (runs via ExecStartPre before service starts) +# This minimal script downloads the REAL startup script from GCS. +# All actual logic lives in the GCS-hosted startup.sh, managed by CI. +# +# Why this pattern? +# - startup.sh can be updated without rebuilding the Packer image +# - All code changes go through CI -> GCS -> service restart +# - No version sorting bugs (fixed wheel path) +# - Strict version verification before starting +cat > /opt/encoding-worker/bootstrap.sh << 'BOOTSTRAP' +#!/bin/bash +# Encoding Worker Bootstrap Script +# +# This is the ONLY script baked into the Packer image. +# It downloads and executes the real startup script from GCS. + +set -e + +BUCKET="gs://karaoke-gen-storage-nomadkaraoke" +WORKER_DIR="/opt/encoding-worker" +LOG_FILE="/var/log/encoding-worker-bootstrap.log" + +exec > >(tee -a "$LOG_FILE") 2>&1 + +echo "=== Bootstrap: $(date) ===" +echo "Downloading latest startup script from GCS..." + +# Download the CI-managed startup script +if ! gsutil cp "${BUCKET}/encoding-worker/startup.sh" "${WORKER_DIR}/startup-latest.sh" 2>&1; then + echo "ERROR: Failed to download startup.sh from GCS" + echo "Falling back to local startup.sh if available..." + if [ -f "${WORKER_DIR}/startup-fallback.sh" ]; then + cp "${WORKER_DIR}/startup-fallback.sh" "${WORKER_DIR}/startup-latest.sh" + else + echo "FATAL: No fallback startup script available" + exit 1 + fi +fi + +chmod +x "${WORKER_DIR}/startup-latest.sh" + +echo "=== Bootstrap: Executing downloaded startup script ===" +exec "${WORKER_DIR}/startup-latest.sh" +BOOTSTRAP + +chmod +x /opt/encoding-worker/bootstrap.sh + +# Create fallback startup script (used if GCS is unreachable) +# This is a simplified version that at least tries to start the service +cat > /opt/encoding-worker/startup-fallback.sh << 'FALLBACK' #!/bin/bash set -e -echo "=== Encoding worker startup at $(date) ===" +echo "=== FALLBACK startup at $(date) ===" +echo "WARNING: Using fallback startup - GCS was unreachable" -# Fetch API key from Secret Manager -echo "Fetching API key from Secret Manager..." +WORKER_DIR="/opt/encoding-worker" +BUCKET="gs://karaoke-gen-storage-nomadkaraoke" + +# Fetch API key ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "") +echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > "${WORKER_DIR}/env" +chmod 600 "${WORKER_DIR}/env" -if [ -z "$ENCODING_API_KEY" ]; then - echo "WARNING: No API key found in Secret Manager" +# Try to install wheel from fixed path +echo "Attempting to download wheel..." +if gsutil cp "${BUCKET}/wheels/karaoke_gen-current.whl" /tmp/karaoke_gen.whl 2>/dev/null; then + "${WORKER_DIR}/venv/bin/pip" install --upgrade --quiet /tmp/karaoke_gen.whl || true fi -# Write environment file for systemd -echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > /opt/encoding-worker/env +echo "=== Fallback startup complete ===" +FALLBACK -# Download and install latest karaoke-gen wheel from GCS -# This enables hot code updates without rebuilding the image -echo "Installing latest karaoke-gen wheel from GCS..." -gsutil cp "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl" /tmp/ 2>/dev/null || true -# Use version sort (-V) to get the highest version, not time sort (-t) -# All wheels download at the same time, so -t picks arbitrarily -WHEEL_FILE=$(ls /tmp/karaoke_gen-*.whl 2>/dev/null | sort -V | tail -1) -if [ -n "$WHEEL_FILE" ]; then - echo "Installing wheel: $WHEEL_FILE" - /opt/encoding-worker/venv/bin/pip install --upgrade --quiet "$WHEEL_FILE" || echo "WARNING: Failed to install wheel" -else - echo "No wheel found in GCS" -fi - -echo "=== Startup script complete ===" -STARTUP - -chmod +x /opt/encoding-worker/startup.sh +chmod +x /opt/encoding-worker/startup-fallback.sh # Create systemd service -# ExecStartPre runs the startup script to fetch API key and install wheel +# ExecStartPre runs bootstrap.sh which downloads and runs the real startup.sh from GCS # ExecStart runs uvicorn with the encoding worker from the installed wheel cat > /etc/systemd/system/encoding-worker.service << 'SYSTEMD' [Unit] @@ -146,11 +186,13 @@ Wants=docker.service Type=simple User=root WorkingDirectory=/opt/encoding-worker -ExecStartPre=/opt/encoding-worker/startup.sh +# bootstrap.sh downloads startup.sh from GCS, then runs it +ExecStartPre=/opt/encoding-worker/bootstrap.sh ExecStart=/opt/encoding-worker/venv/bin/uvicorn backend.services.gce_encoding.main:app --host 0.0.0.0 --port 8080 Restart=always RestartSec=10 -TimeoutStartSec=300 +# Increased timeout to allow for GCS downloads +TimeoutStartSec=600 Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke" EnvironmentFile=/opt/encoding-worker/env @@ -177,3 +219,8 @@ echo " - FFmpeg at /usr/local/bin/ffmpeg" echo " - Noto fonts (including CJK)" echo " - Virtual environment at /opt/encoding-worker/venv" echo " - Systemd service: encoding-worker.service" +echo "" +echo "Immutable deployment pattern:" +echo " - bootstrap.sh downloads startup.sh from GCS on every start" +echo " - startup.sh is CI-managed, no image rebuild needed for logic changes" +echo " - See infrastructure/encoding-worker/README.md" diff --git a/pyproject.toml b/pyproject.toml index 92cd6ded..293c6464 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "karaoke-gen" -version = "0.109.0" +version = "0.110.0" description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens." authors = ["Andrew Beveridge "] license = "MIT"