mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
feat(infra): Implement immutable deployment pattern for encoding worker (#333)
## Summary - Implements robust deployment pattern for GCE encoding worker to eliminate recurring version mismatch issues - Fixed wheel path (`karaoke_gen-current.whl`) eliminates version sorting bugs - Self-updating startup script downloaded from GCS on every service start - Version manifest with strict CI verification (deployment fails on mismatch) ## Problem The encoding worker had recurring version issues because: - `startup.sh` was baked into the Packer image (required rebuild + VM recreation) - Wheel selection used version sorting across 51 files - CI only warned on version mismatch, didn't fail - Multiple deployment vectors (CI, Pulumi, manual SSH) caused drift ## Solution 1. **Fixed wheel path**: CI uploads to `karaoke_gen-current.whl` (no sorting needed) 2. **Self-updating startup**: `bootstrap.sh` downloads `startup.sh` from GCS on every start 3. **Version manifest**: CI writes expected version, service verifies before starting 4. **Strict CI verification**: Deployment fails if version doesn't match (was just warning) ## Changes - **New**: `infrastructure/encoding-worker/` with `bootstrap.sh`, `startup.sh`, README - **Modified**: CI workflow for fixed paths and strict verification - **Modified**: Packer `provision.sh` for bootstrap pattern - **Updated**: `LESSONS-LEARNED.md` with immutable deployment pattern - **Updated**: `docs/README.md` and `docs/ARCHITECTURE.md` ## Test plan - [x] Shell scripts validated (`bash -n`) - [x] Backend tests pass (1703 passed) - [x] CodeRabbit review passed - [ ] After merge: Rebuild Packer image - [ ] After merge: Recreate VM to use new image - [ ] After merge: Verify first deployment uses new pattern ## Post-Merge Steps Required 1. Rebuild Packer image: `packer build infrastructure/packer/encoding-worker.pkr.hcl` 2. Recreate VM: Delete encoding-worker, run `pulumi up` 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e2fc6b616a
commit
b68a46d174
11 files changed
+584
-72
No files matched your search
@@ -0,0 +1,99 @@
|
||||
# Plan: Encoding Worker Immutable Deployment Pattern
|
||||
|
||||
## Status: In Progress
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The GCE encoding worker has recurring version mismatch issues because:
|
||||
1. `startup.sh` is baked into the Packer image (requires image rebuild + VM recreation)
|
||||
2. Wheel selection uses `sort -V` which can fail if image is outdated
|
||||
3. CI only restarts service - doesn't verify version strictly
|
||||
4. Multiple actors (CI, Pulumi, manual SSH) can cause drift
|
||||
5. 51 old wheel files accumulate in GCS bucket
|
||||
|
||||
## Solution: Immutable Deployment Pattern
|
||||
|
||||
### Core Principles
|
||||
1. **Single Source of Truth**: All code and config from GCS, nothing baked in image
|
||||
2. **Fixed Wheel Path**: `karaoke_gen-current.whl` eliminates sorting
|
||||
3. **Version Manifest**: CI writes expected version, service MUST match
|
||||
4. **Strict Verification**: CI FAILS if deployed version doesn't match
|
||||
5. **Self-Updating Startup**: Download startup.sh from GCS on every boot
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: GCS Structure & Scripts
|
||||
|
||||
- [x] **Step 1.1**: Create `infrastructure/encoding-worker/` directory
|
||||
- [x] **Step 1.2**: Create `bootstrap.sh` (minimal, baked in image)
|
||||
- [x] **Step 1.3**: Create `startup.sh` (CI-managed, deployed to GCS)
|
||||
- [x] **Step 1.4**: Create README.md with documentation
|
||||
|
||||
### Phase 2: CI Workflow Updates
|
||||
|
||||
- [x] **Step 2.1**: Update CI to upload wheel to fixed path
|
||||
- [x] **Step 2.2**: Update CI to upload startup.sh to GCS
|
||||
- [x] **Step 2.3**: Update CI to write version.txt manifest
|
||||
- [x] **Step 2.4**: Update CI to strictly verify deployed version (fail on mismatch)
|
||||
|
||||
### Phase 3: Packer Image Updates
|
||||
|
||||
- [x] **Step 3.1**: Update provision.sh to use bootstrap.sh
|
||||
- [ ] **Step 3.2**: Rebuild Packer image (manual step after merge)
|
||||
- [ ] **Step 3.3**: Recreate VM to use new image (manual step after merge)
|
||||
|
||||
### Phase 4: Documentation & Cleanup
|
||||
|
||||
- [x] **Step 4.1**: Update LESSONS-LEARNED.md with new pattern
|
||||
- [x] **Step 4.2**: Update infrastructure/packer/README.md
|
||||
- [ ] **Step 4.3**: Add lifecycle rule for old wheels (optional, later)
|
||||
|
||||
## New GCS Structure
|
||||
|
||||
```
|
||||
gs://karaoke-gen-storage-nomadkaraoke/
|
||||
├── wheels/
|
||||
│ ├── karaoke_gen-current.whl # Latest (CI overwrites)
|
||||
│ └── karaoke_gen-{version}.whl # Versioned for rollback
|
||||
│
|
||||
└── encoding-worker/
|
||||
├── bootstrap.sh # Reference copy
|
||||
├── startup.sh # CI-managed, downloaded on boot
|
||||
└── version.txt # Expected version number
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Purpose |
|
||||
|------|--------|---------|
|
||||
| `infrastructure/encoding-worker/bootstrap.sh` | Create | Minimal loader (baked in image) |
|
||||
| `infrastructure/encoding-worker/startup.sh` | Create | Main logic (deployed to GCS) |
|
||||
| `infrastructure/packer/scripts/provision.sh` | Modify | Use bootstrap.sh |
|
||||
| `.github/workflows/ci.yml` | Modify | New deployment flow |
|
||||
| `docs/LESSONS-LEARNED.md` | Modify | Document pattern |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. **Unit test**: Verify startup.sh logic locally
|
||||
2. **Integration test**: After merge, verify:
|
||||
- CI uploads files to correct GCS paths
|
||||
- Service starts with correct version
|
||||
- Health endpoint returns expected version
|
||||
3. **Manual verification**: Check VM logs after first deployment
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues occur:
|
||||
1. CI can re-upload previous wheel to `karaoke_gen-current.whl`
|
||||
2. Previous versioned wheels remain in GCS for manual rollback
|
||||
3. Old Packer images remain for VM rollback
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - design is complete.
|
||||
|
||||
## Notes
|
||||
|
||||
- Bootstrap.sh is ~10 lines and should never need changes
|
||||
- All logic changes go through startup.sh in GCS
|
||||
- Version verification is strict - CI fails on mismatch
|
||||
+96
-26
@@ -967,8 +967,16 @@ jobs:
|
||||
version: "latest"
|
||||
|
||||
# ============================================
|
||||
# Upload wheel to GCS for GCE encoding worker
|
||||
# The GCE worker installs this wheel at job start for hot code updates
|
||||
# Deploy to GCE Encoding Worker (Immutable Deployment Pattern)
|
||||
#
|
||||
# This pattern ensures version consistency:
|
||||
# 1. Upload wheel to FIXED path (no version sorting)
|
||||
# 2. Upload startup.sh (logic can change without image rebuild)
|
||||
# 3. Write version manifest for verification
|
||||
# 4. Restart service (triggers bootstrap.sh -> startup.sh)
|
||||
# 5. STRICTLY verify deployed version (fail CI on mismatch)
|
||||
#
|
||||
# See infrastructure/encoding-worker/README.md for details
|
||||
# ============================================
|
||||
|
||||
- name: Download wheel artifact
|
||||
@@ -977,52 +985,114 @@ jobs:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
|
||||
- name: Upload wheel to GCS for GCE worker
|
||||
- name: Deploy to GCE Encoding Worker
|
||||
run: |
|
||||
echo "📦 Uploading wheel to GCS for GCE encoding worker..."
|
||||
VERSION="${{ steps.version_check.outputs.version }}"
|
||||
BUCKET="gs://karaoke-gen-storage-${{ env.PROJECT_ID }}"
|
||||
|
||||
echo "🚀 Deploying encoding worker v${VERSION}..."
|
||||
|
||||
# 1. Find the wheel file
|
||||
WHEEL_FILE=$(ls dist/*.whl | head -1)
|
||||
if [ -z "$WHEEL_FILE" ]; then
|
||||
echo "::error::No wheel file found in dist/"
|
||||
exit 1
|
||||
fi
|
||||
echo "Found wheel: $WHEEL_FILE"
|
||||
echo "📦 Found wheel: $WHEEL_FILE"
|
||||
|
||||
# Upload to GCS (GCE worker downloads latest at job start)
|
||||
gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/
|
||||
# 2. Upload wheel to FIXED path (atomic via temp file)
|
||||
# This eliminates version sorting - always use karaoke_gen-current.whl
|
||||
echo "📤 Uploading wheel to fixed path..."
|
||||
gsutil cp "$WHEEL_FILE" "${BUCKET}/wheels/karaoke_gen-current.whl.tmp"
|
||||
gsutil mv "${BUCKET}/wheels/karaoke_gen-current.whl.tmp" \
|
||||
"${BUCKET}/wheels/karaoke_gen-current.whl"
|
||||
|
||||
echo "✅ Wheel uploaded to GCS"
|
||||
# Also keep versioned copy for rollback/debugging
|
||||
gsutil cp "$WHEEL_FILE" "${BUCKET}/wheels/"
|
||||
|
||||
# 3. Upload startup.sh (allows logic changes without image rebuild)
|
||||
echo "📤 Uploading startup.sh..."
|
||||
gsutil cp infrastructure/encoding-worker/startup.sh \
|
||||
"${BUCKET}/encoding-worker/startup.sh"
|
||||
|
||||
# 4. Write version manifest for verification
|
||||
echo "📝 Writing version manifest..."
|
||||
echo "${VERSION}" | gsutil cp - "${BUCKET}/encoding-worker/version.txt"
|
||||
|
||||
echo "✅ GCS deployment complete"
|
||||
|
||||
- name: Restart GCE Encoding Worker
|
||||
run: |
|
||||
echo "🔄 Restarting encoding worker to pick up new wheel..."
|
||||
VERSION="${{ steps.version_check.outputs.version }}"
|
||||
|
||||
echo "🔄 Restarting encoding worker..."
|
||||
|
||||
# Use IAP tunneling - works directly with service account credentials
|
||||
# (avoids SSH key propagation delays that cause silent failures)
|
||||
gcloud compute ssh encoding-worker \
|
||||
--zone=us-central1-c \
|
||||
--project=nomadkaraoke \
|
||||
--tunnel-through-iap \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
|
||||
echo "⏳ Waiting for service to start..."
|
||||
sleep 15
|
||||
echo "⏳ Waiting for service to start (up to 60s)..."
|
||||
sleep 20
|
||||
|
||||
# Verify the restart worked by checking the wheel version
|
||||
NEW_VERSION=$(gcloud compute ssh encoding-worker \
|
||||
--zone=us-central1-c \
|
||||
--project=nomadkaraoke \
|
||||
--tunnel-through-iap \
|
||||
--command="curl -s http://localhost:8080/health | grep -o '\"wheel_version\":\"[^\"]*\"' | cut -d'\"' -f4")
|
||||
- name: Verify Encoding Worker Version
|
||||
run: |
|
||||
VERSION="${{ steps.version_check.outputs.version }}"
|
||||
|
||||
EXPECTED_VERSION=$(cat pyproject.toml | grep 'version = ' | head -1 | cut -d'"' -f2)
|
||||
echo "🔍 Verifying encoding worker version..."
|
||||
|
||||
if [ "$NEW_VERSION" = "$EXPECTED_VERSION" ]; then
|
||||
echo "✅ Encoding worker restarted successfully (v$NEW_VERSION)"
|
||||
else
|
||||
echo "⚠️ Version mismatch: encoder has v$NEW_VERSION, expected v$EXPECTED_VERSION"
|
||||
echo " This may indicate the restart failed or the wheel wasn't downloaded"
|
||||
# Don't fail the build - the encoder will still work, just with old version
|
||||
fi
|
||||
# Retry loop - service may take time to fully start
|
||||
MAX_RETRIES=6
|
||||
RETRY_DELAY=10
|
||||
|
||||
for i in $(seq 1 $MAX_RETRIES); do
|
||||
echo "Attempt $i/$MAX_RETRIES..."
|
||||
|
||||
NEW_VERSION=$(gcloud compute ssh encoding-worker \
|
||||
--zone=us-central1-c \
|
||||
--project=nomadkaraoke \
|
||||
--tunnel-through-iap \
|
||||
--command="curl -s http://localhost:8080/health 2>/dev/null | grep -o '\"wheel_version\":\"[^\"]*\"' | cut -d'\"' -f4" 2>/dev/null || echo "")
|
||||
|
||||
if [ "$NEW_VERSION" = "$VERSION" ]; then
|
||||
echo "✅ Encoding worker version verified: v$NEW_VERSION"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -n "$NEW_VERSION" ]; then
|
||||
echo "⚠️ Version: $NEW_VERSION (expected: $VERSION)"
|
||||
else
|
||||
echo "⚠️ Health endpoint not responding yet"
|
||||
fi
|
||||
|
||||
if [ $i -lt $MAX_RETRIES ]; then
|
||||
echo " Retrying in ${RETRY_DELAY}s..."
|
||||
sleep $RETRY_DELAY
|
||||
fi
|
||||
done
|
||||
|
||||
# STRICT VERIFICATION: Fail CI on version mismatch
|
||||
echo ""
|
||||
echo "=============================================="
|
||||
echo "❌ ENCODING WORKER DEPLOYMENT FAILED"
|
||||
echo "=============================================="
|
||||
echo "Expected version: $VERSION"
|
||||
echo "Deployed version: $NEW_VERSION"
|
||||
echo ""
|
||||
echo "Possible causes:"
|
||||
echo " - GCS upload failed (check 'Deploy to GCE' step)"
|
||||
echo " - startup.sh download failed (check VM logs)"
|
||||
echo " - Wheel installation failed (check VM logs)"
|
||||
echo ""
|
||||
echo "To debug, SSH to VM and check logs:"
|
||||
echo " gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke --tunnel-through-iap"
|
||||
echo " sudo journalctl -u encoding-worker -n 100"
|
||||
echo " cat /var/log/encoding-worker-startup.log"
|
||||
echo "=============================================="
|
||||
|
||||
exit 1
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
@@ -33,7 +33,7 @@
|
||||
(*) GCE Encoding Worker: c4d-highcpu-32 VM with AMD EPYC 9B45 (Turin) for
|
||||
high-performance FFmpeg encoding (4.92x faster than previous c4-standard-8).
|
||||
Used for both final video encoding and preview video generation.
|
||||
Uses LocalEncodingService via GCS wheel deployment.
|
||||
Uses immutable deployment pattern - see infrastructure/encoding-worker/README.md.
|
||||
```
|
||||
|
||||
## Processing Pipeline
|
||||
|
||||
+27
-2
@@ -177,8 +177,33 @@ Use `pulumi up --skip-preview --yes` when CI service account can't run preview.
|
||||
### Docker Disk Management
|
||||
Self-hosted runners need aggressive disk cleanup. Use threshold-based (70%) not age-based cleanup.
|
||||
|
||||
### Version Sort for Artifact Selection
|
||||
When downloading multiple versioned files (wheels, tarballs), use `sort -V | tail -1` not `ls -t | head -1`. Files downloaded simultaneously have the same timestamp, so time-based sorting picks arbitrarily. Version sorting (`sort -V`) correctly identifies the highest semantic version.
|
||||
### Version Sort for Artifact Selection (SUPERSEDED)
|
||||
**Note:** This lesson is superseded by the "Immutable Deployment Pattern" below, which eliminates version sorting entirely.
|
||||
|
||||
Original lesson: When downloading multiple versioned files (wheels, tarballs), use `sort -V | tail -1` not `ls -t | head -1`. Files downloaded simultaneously have the same timestamp, so time-based sorting picks arbitrarily.
|
||||
|
||||
### Immutable Deployment Pattern for GCE Workers
|
||||
When deploying to long-running VMs (like the encoding worker), avoid baking deployment logic into Packer images:
|
||||
|
||||
**Problem:** The encoding worker had recurring version mismatch issues because:
|
||||
1. `startup.sh` was baked into the Packer image
|
||||
2. Fixing logic required image rebuild + VM recreation
|
||||
3. CI only warned on version mismatch, didn't fail
|
||||
4. Multiple deployment vectors (CI, Pulumi, manual SSH) caused drift
|
||||
|
||||
**Solution:** Self-updating startup script from GCS:
|
||||
1. Packer image contains minimal `bootstrap.sh` (~15 lines, rarely changes)
|
||||
2. `bootstrap.sh` downloads real `startup.sh` from GCS on every service start
|
||||
3. CI uploads `startup.sh`, wheel, and version manifest to GCS
|
||||
4. CI **strictly verifies** deployed version - fails on mismatch
|
||||
|
||||
**Benefits:**
|
||||
- Logic changes via CI, no Packer rebuild needed
|
||||
- Fixed wheel path (`karaoke_gen-current.whl`) eliminates sorting bugs
|
||||
- Version verification is mandatory, not optional
|
||||
- Single source of truth (GCS), no drift from manual changes
|
||||
|
||||
See `infrastructure/encoding-worker/README.md` for implementation details.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -42,6 +42,8 @@
|
||||
|
||||
## Recent Changes
|
||||
|
||||
- **Encoding Worker Immutable Deployment** (2026-01-22): Implemented robust deployment pattern for GCE encoding worker to eliminate version mismatch issues. Changes: (1) Fixed wheel path `karaoke_gen-current.whl` eliminates version sorting bugs, (2) Self-updating startup script downloaded from GCS on every service start (no Packer rebuild needed for logic changes), (3) Version manifest with strict CI verification (deployment fails on mismatch). Root cause: startup.sh was baked into Packer image, requiring image rebuild + VM recreation for fixes. New pattern allows CI to update deployment logic without infrastructure changes. See [infrastructure/encoding-worker/README.md](../infrastructure/encoding-worker/README.md) and [LESSONS-LEARNED.md](LESSONS-LEARNED.md#immutable-deployment-pattern-for-gce-workers).
|
||||
|
||||
- **Theme Validation & Default Removal** (2026-01-22): **BREAKING CHANGE in v0.109.0** - Removed all default style fallback logic to ensure cloud jobs always use complete themes. Style getter functions now raise `ValueError` if themes are incomplete instead of merging with defaults. Added `--list-themes` and `--validate-theme` CLI flags. All cloud jobs guaranteed to use complete nomad theme with no silent fallbacks. See [archive/2026-01-22-theme-validation-remove-defaults.md](archive/2026-01-22-theme-validation-remove-defaults.md).
|
||||
|
||||
- **Admin Job Reprocess** (2026-01-21): Added "Reprocess" button to admin job reset toolbar. Allows re-encoding and re-distributing a job with existing settings after deleting outputs. Resets job to `instrumental_selected` state and automatically triggers video worker. Clears video/encoding/distribution state while preserving instrumental selection and lyrics metadata. Use case: Fix distribution issues, re-upload to YouTube with different settings, or re-run encoding after infrastructure changes. See [API.md](API.md#admin-job-reset).
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
# Encoding Worker Deployment
|
||||
|
||||
This directory contains the scripts for the GCE encoding worker's immutable deployment pattern.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ PACKER IMAGE (rarely changes) │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ - Python 3.13, FFmpeg, fonts (static dependencies) │
|
||||
│ - bootstrap.sh (downloads startup.sh from GCS) │
|
||||
│ - systemd service definition │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼ downloads on every service start
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ GCS (CI-managed, always current) │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ gs://karaoke-gen-storage-nomadkaraoke/ │
|
||||
│ ├── encoding-worker/ │
|
||||
│ │ ├── startup.sh ← Main logic (this file) │
|
||||
│ │ └── version.txt ← Expected version number │
|
||||
│ └── wheels/ │
|
||||
│ └── karaoke_gen-current.whl ← Latest wheel (fixed path) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
| File | Location | Purpose |
|
||||
|------|----------|---------|
|
||||
| `bootstrap.sh` | Baked in Packer image | Downloads and runs startup.sh |
|
||||
| `startup.sh` | Deployed to GCS by CI | Main startup logic |
|
||||
| `version.txt` | Written to GCS by CI | Expected version for verification |
|
||||
|
||||
## Deployment Flow
|
||||
|
||||
1. **CI builds wheel** → uploads to `wheels/karaoke_gen-current.whl`
|
||||
2. **CI uploads startup.sh** → `encoding-worker/startup.sh`
|
||||
3. **CI writes version** → `encoding-worker/version.txt`
|
||||
4. **CI restarts service** → triggers bootstrap.sh
|
||||
5. **bootstrap.sh** downloads fresh startup.sh from GCS
|
||||
6. **startup.sh** downloads wheel, installs, verifies version
|
||||
7. **CI verifies** health endpoint returns correct version
|
||||
8. **CI FAILS** if version mismatch
|
||||
|
||||
## Why This Pattern?
|
||||
|
||||
### Before (problematic)
|
||||
- startup.sh baked into Packer image
|
||||
- Required image rebuild + VM recreation for logic changes
|
||||
- 51 versioned wheels, sorting could fail
|
||||
- CI warned on version mismatch but continued
|
||||
|
||||
### After (robust)
|
||||
- startup.sh downloaded from GCS on every start
|
||||
- Logic changes via CI, no image rebuild needed
|
||||
- Fixed path `karaoke_gen-current.whl`, no sorting
|
||||
- CI FAILS on version mismatch
|
||||
|
||||
## Modifying Startup Logic
|
||||
|
||||
1. Edit `infrastructure/encoding-worker/startup.sh`
|
||||
2. Push to main
|
||||
3. CI automatically deploys to GCS
|
||||
4. Service restart picks up new script
|
||||
|
||||
No Packer image rebuild needed!
|
||||
|
||||
## When to Rebuild Packer Image
|
||||
|
||||
Only rebuild when changing:
|
||||
- Python version
|
||||
- FFmpeg version
|
||||
- System packages/fonts
|
||||
- bootstrap.sh (should be extremely rare)
|
||||
|
||||
## Manual Operations
|
||||
|
||||
```bash
|
||||
# Check current version on VM
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--tunnel-through-iap \
|
||||
--command="curl -s http://localhost:8080/health | jq '.wheel_version'"
|
||||
|
||||
# View startup logs
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--tunnel-through-iap \
|
||||
--command="sudo journalctl -u encoding-worker -n 50"
|
||||
|
||||
# Force service restart (triggers fresh startup.sh download)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--tunnel-through-iap \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Version mismatch after deployment
|
||||
1. Check CI logs for GCS upload success
|
||||
2. Check `/var/log/encoding-worker-startup.log` on VM
|
||||
3. Verify `version.txt` in GCS matches expected
|
||||
|
||||
### Service won't start
|
||||
1. Check bootstrap.sh can reach GCS: `gsutil ls gs://karaoke-gen-storage-nomadkaraoke/`
|
||||
2. Check Secret Manager access for API key
|
||||
3. Check startup.sh logs: `cat /var/log/encoding-worker-startup.log`
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/bin/bash
|
||||
# Encoding Worker Bootstrap Script
|
||||
#
|
||||
# This is the ONLY script baked into the Packer image.
|
||||
# It downloads and executes the real startup script from GCS.
|
||||
#
|
||||
# Why this pattern?
|
||||
# - startup.sh logic can be updated without rebuilding the image
|
||||
# - All code changes go through CI → GCS → service restart
|
||||
# - No version sorting bugs (fixed path for wheel)
|
||||
# - Strict version verification before starting
|
||||
#
|
||||
# This script should NEVER need modification. All logic lives in
|
||||
# the GCS-hosted startup.sh which is managed by CI.
|
||||
|
||||
set -e
|
||||
|
||||
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
|
||||
WORKER_DIR="/opt/encoding-worker"
|
||||
LOG_FILE="/var/log/encoding-worker-bootstrap.log"
|
||||
|
||||
exec > >(tee -a "$LOG_FILE") 2>&1
|
||||
|
||||
echo "=== Bootstrap: $(date) ==="
|
||||
echo "Downloading latest startup script from GCS..."
|
||||
|
||||
# Download the CI-managed startup script
|
||||
if ! gsutil cp "${BUCKET}/encoding-worker/startup.sh" "${WORKER_DIR}/startup-latest.sh" 2>&1; then
|
||||
echo "ERROR: Failed to download startup.sh from GCS"
|
||||
echo "Falling back to local startup.sh if available..."
|
||||
if [ -f "${WORKER_DIR}/startup-fallback.sh" ]; then
|
||||
cp "${WORKER_DIR}/startup-fallback.sh" "${WORKER_DIR}/startup-latest.sh"
|
||||
else
|
||||
echo "FATAL: No fallback startup script available"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
chmod +x "${WORKER_DIR}/startup-latest.sh"
|
||||
|
||||
echo "=== Bootstrap: Executing downloaded startup script ==="
|
||||
exec "${WORKER_DIR}/startup-latest.sh"
|
||||
@@ -0,0 +1,108 @@
|
||||
#!/bin/bash
|
||||
# Encoding Worker Startup Script
|
||||
#
|
||||
# This script is deployed to GCS and downloaded by bootstrap.sh on every service start.
|
||||
# It handles:
|
||||
# 1. Fetching API key from Secret Manager
|
||||
# 2. Downloading the current wheel (fixed path, no version sorting!)
|
||||
# 3. Installing the wheel
|
||||
# 4. Verifying installed version matches expected version
|
||||
# 5. Failing fast if version mismatch (don't start with wrong code)
|
||||
#
|
||||
# CI uploads this script to: gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/startup.sh
|
||||
# CI also uploads:
|
||||
# - gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-current.whl (latest wheel)
|
||||
# - gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/version.txt (expected version)
|
||||
|
||||
set -e
|
||||
|
||||
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
|
||||
WORKER_DIR="/opt/encoding-worker"
|
||||
VENV="${WORKER_DIR}/venv"
|
||||
TMP_DIR="/tmp/encoding-worker-startup"
|
||||
LOG_FILE="/var/log/encoding-worker-startup.log"
|
||||
|
||||
exec > >(tee -a "$LOG_FILE") 2>&1
|
||||
|
||||
echo "=============================================="
|
||||
echo "Encoding Worker Startup: $(date)"
|
||||
echo "=============================================="
|
||||
|
||||
# Create temp directory for downloads
|
||||
rm -rf "${TMP_DIR}"
|
||||
mkdir -p "${TMP_DIR}"
|
||||
|
||||
# 1. Fetch API key from Secret Manager
|
||||
echo "[1/5] Fetching API key from Secret Manager..."
|
||||
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
|
||||
|
||||
if [ -z "$ENCODING_API_KEY" ]; then
|
||||
echo "WARNING: No API key found in Secret Manager"
|
||||
echo "Service will start but may reject requests"
|
||||
fi
|
||||
|
||||
# Write environment file for systemd
|
||||
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > "${WORKER_DIR}/env"
|
||||
chmod 600 "${WORKER_DIR}/env"
|
||||
|
||||
# 2. Download expected version manifest
|
||||
echo "[2/5] Downloading version manifest..."
|
||||
if ! gsutil cp "${BUCKET}/encoding-worker/version.txt" "${TMP_DIR}/version.txt" 2>&1; then
|
||||
echo "WARNING: Could not download version.txt - skipping version verification"
|
||||
EXPECTED_VERSION=""
|
||||
else
|
||||
EXPECTED_VERSION=$(cat "${TMP_DIR}/version.txt" | tr -d '[:space:]')
|
||||
echo "Expected version: ${EXPECTED_VERSION}"
|
||||
fi
|
||||
|
||||
# 3. Download current wheel (FIXED PATH - no sorting!)
|
||||
echo "[3/5] Downloading current wheel..."
|
||||
WHEEL_PATH="${TMP_DIR}/karaoke_gen-current.whl"
|
||||
|
||||
if ! gsutil cp "${BUCKET}/wheels/karaoke_gen-current.whl" "${WHEEL_PATH}" 2>&1; then
|
||||
echo "ERROR: Failed to download karaoke_gen-current.whl"
|
||||
echo "Checking for any available wheel as fallback..."
|
||||
|
||||
# Fallback: try to get the latest versioned wheel (with proper version sort)
|
||||
LATEST_WHEEL=$(gsutil ls "${BUCKET}/wheels/karaoke_gen-*.whl" 2>/dev/null | grep -v 'current' | sort -V | tail -1 || echo "")
|
||||
|
||||
if [ -n "$LATEST_WHEEL" ]; then
|
||||
echo "Fallback: Using ${LATEST_WHEEL}"
|
||||
gsutil cp "${LATEST_WHEEL}" "${WHEEL_PATH}"
|
||||
else
|
||||
echo "FATAL: No wheel available in GCS"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# 4. Install the wheel
|
||||
echo "[4/5] Installing wheel..."
|
||||
"${VENV}/bin/pip" install --upgrade --quiet --force-reinstall "${WHEEL_PATH}"
|
||||
|
||||
# 5. Verify installed version
|
||||
echo "[5/5] Verifying installed version..."
|
||||
INSTALLED_VERSION=$("${VENV}/bin/pip" show karaoke-gen 2>/dev/null | grep "^Version:" | cut -d' ' -f2 | tr -d '[:space:]')
|
||||
echo "Installed version: ${INSTALLED_VERSION}"
|
||||
|
||||
if [ -n "$EXPECTED_VERSION" ]; then
|
||||
if [ "${INSTALLED_VERSION}" != "${EXPECTED_VERSION}" ]; then
|
||||
echo "=============================================="
|
||||
echo "FATAL: VERSION MISMATCH!"
|
||||
echo " Expected: ${EXPECTED_VERSION}"
|
||||
echo " Installed: ${INSTALLED_VERSION}"
|
||||
echo "=============================================="
|
||||
echo "Refusing to start with incorrect version."
|
||||
echo "This indicates a deployment issue - check CI logs."
|
||||
exit 1
|
||||
fi
|
||||
echo "Version verified successfully!"
|
||||
else
|
||||
echo "WARNING: Version verification skipped (no version.txt)"
|
||||
fi
|
||||
|
||||
# Cleanup
|
||||
rm -rf "${TMP_DIR}"
|
||||
|
||||
echo "=============================================="
|
||||
echo "Startup complete: karaoke-gen v${INSTALLED_VERSION}"
|
||||
echo "=============================================="
|
||||
@@ -63,19 +63,27 @@ The image contains:
|
||||
- **Systemd service** `encoding-worker.service` (enabled, not started)
|
||||
- **Startup script** `/opt/encoding-worker/startup.sh`
|
||||
|
||||
## Runtime Behavior
|
||||
## Runtime Behavior (Immutable Deployment Pattern)
|
||||
|
||||
When the VM boots with this image:
|
||||
|
||||
1. **Startup script** (`/opt/encoding-worker/startup.sh`) runs via `ExecStartPre`:
|
||||
- Fetches API key from Secret Manager
|
||||
- Downloads latest `karaoke-gen` wheel from GCS
|
||||
- Installs wheel into venv
|
||||
1. **bootstrap.sh** runs via `ExecStartPre`:
|
||||
- Downloads `startup.sh` from GCS (CI-managed, always current)
|
||||
- Executes the downloaded startup script
|
||||
|
||||
2. **Encoding worker service** starts via `ExecStart`:
|
||||
2. **startup.sh** (downloaded from GCS):
|
||||
- Fetches API key from Secret Manager
|
||||
- Downloads `karaoke_gen-current.whl` from GCS (fixed path, no sorting!)
|
||||
- Installs wheel into venv
|
||||
- **Verifies** installed version matches `version.txt` from GCS
|
||||
- Fails fast if version mismatch
|
||||
|
||||
3. **Encoding worker service** starts via `ExecStart`:
|
||||
- Runs `uvicorn backend.services.gce_encoding.main:app`
|
||||
- Listens on port 8080
|
||||
|
||||
See `infrastructure/encoding-worker/README.md` for details on this pattern.
|
||||
|
||||
## When to Rebuild
|
||||
|
||||
Rebuild the image when:
|
||||
@@ -84,8 +92,11 @@ Rebuild the image when:
|
||||
- FFmpeg version needs updating
|
||||
- New system packages required
|
||||
- Font packages change
|
||||
- bootstrap.sh changes (extremely rare)
|
||||
|
||||
**No rebuild needed** for application code changes - the wheel is downloaded at runtime.
|
||||
**No rebuild needed** for:
|
||||
- Application code changes (wheel downloaded at runtime)
|
||||
- Startup logic changes (startup.sh downloaded from GCS)
|
||||
|
||||
## Image Family
|
||||
|
||||
|
||||
@@ -8,10 +8,13 @@
|
||||
# - Virtual environment exists at /opt/encoding-worker/venv
|
||||
# - Systemd service is configured (but not started - no API key yet)
|
||||
#
|
||||
# At runtime, the startup script only needs to:
|
||||
# 1. Fetch API key from Secret Manager
|
||||
# 2. Download latest karaoke-gen wheel
|
||||
# 3. Start the systemd service
|
||||
# IMPORTANT: This image uses the "immutable deployment" pattern:
|
||||
# - bootstrap.sh is baked into the image (minimal, rarely changes)
|
||||
# - bootstrap.sh downloads startup.sh from GCS on every service start
|
||||
# - startup.sh contains all logic and is CI-managed
|
||||
# - This allows logic updates without rebuilding the image
|
||||
#
|
||||
# See infrastructure/encoding-worker/README.md for details.
|
||||
|
||||
set -e
|
||||
|
||||
@@ -95,46 +98,83 @@ source venv/bin/activate
|
||||
pip install --upgrade pip
|
||||
pip install fastapi uvicorn google-cloud-storage aiofiles aiohttp
|
||||
|
||||
# Create startup helper script (runs via ExecStartPre before service starts)
|
||||
# This fetches API key and installs latest wheel at boot time
|
||||
cat > /opt/encoding-worker/startup.sh << 'STARTUP'
|
||||
# Create bootstrap script (runs via ExecStartPre before service starts)
|
||||
# This minimal script downloads the REAL startup script from GCS.
|
||||
# All actual logic lives in the GCS-hosted startup.sh, managed by CI.
|
||||
#
|
||||
# Why this pattern?
|
||||
# - startup.sh can be updated without rebuilding the Packer image
|
||||
# - All code changes go through CI -> GCS -> service restart
|
||||
# - No version sorting bugs (fixed wheel path)
|
||||
# - Strict version verification before starting
|
||||
cat > /opt/encoding-worker/bootstrap.sh << 'BOOTSTRAP'
|
||||
#!/bin/bash
|
||||
# Encoding Worker Bootstrap Script
|
||||
#
|
||||
# This is the ONLY script baked into the Packer image.
|
||||
# It downloads and executes the real startup script from GCS.
|
||||
|
||||
set -e
|
||||
|
||||
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
|
||||
WORKER_DIR="/opt/encoding-worker"
|
||||
LOG_FILE="/var/log/encoding-worker-bootstrap.log"
|
||||
|
||||
exec > >(tee -a "$LOG_FILE") 2>&1
|
||||
|
||||
echo "=== Bootstrap: $(date) ==="
|
||||
echo "Downloading latest startup script from GCS..."
|
||||
|
||||
# Download the CI-managed startup script
|
||||
if ! gsutil cp "${BUCKET}/encoding-worker/startup.sh" "${WORKER_DIR}/startup-latest.sh" 2>&1; then
|
||||
echo "ERROR: Failed to download startup.sh from GCS"
|
||||
echo "Falling back to local startup.sh if available..."
|
||||
if [ -f "${WORKER_DIR}/startup-fallback.sh" ]; then
|
||||
cp "${WORKER_DIR}/startup-fallback.sh" "${WORKER_DIR}/startup-latest.sh"
|
||||
else
|
||||
echo "FATAL: No fallback startup script available"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
chmod +x "${WORKER_DIR}/startup-latest.sh"
|
||||
|
||||
echo "=== Bootstrap: Executing downloaded startup script ==="
|
||||
exec "${WORKER_DIR}/startup-latest.sh"
|
||||
BOOTSTRAP
|
||||
|
||||
chmod +x /opt/encoding-worker/bootstrap.sh
|
||||
|
||||
# Create fallback startup script (used if GCS is unreachable)
|
||||
# This is a simplified version that at least tries to start the service
|
||||
cat > /opt/encoding-worker/startup-fallback.sh << 'FALLBACK'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
echo "=== Encoding worker startup at $(date) ==="
|
||||
echo "=== FALLBACK startup at $(date) ==="
|
||||
echo "WARNING: Using fallback startup - GCS was unreachable"
|
||||
|
||||
# Fetch API key from Secret Manager
|
||||
echo "Fetching API key from Secret Manager..."
|
||||
WORKER_DIR="/opt/encoding-worker"
|
||||
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
|
||||
|
||||
# Fetch API key
|
||||
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
|
||||
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > "${WORKER_DIR}/env"
|
||||
chmod 600 "${WORKER_DIR}/env"
|
||||
|
||||
if [ -z "$ENCODING_API_KEY" ]; then
|
||||
echo "WARNING: No API key found in Secret Manager"
|
||||
# Try to install wheel from fixed path
|
||||
echo "Attempting to download wheel..."
|
||||
if gsutil cp "${BUCKET}/wheels/karaoke_gen-current.whl" /tmp/karaoke_gen.whl 2>/dev/null; then
|
||||
"${WORKER_DIR}/venv/bin/pip" install --upgrade --quiet /tmp/karaoke_gen.whl || true
|
||||
fi
|
||||
|
||||
# Write environment file for systemd
|
||||
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > /opt/encoding-worker/env
|
||||
echo "=== Fallback startup complete ==="
|
||||
FALLBACK
|
||||
|
||||
# Download and install latest karaoke-gen wheel from GCS
|
||||
# This enables hot code updates without rebuilding the image
|
||||
echo "Installing latest karaoke-gen wheel from GCS..."
|
||||
gsutil cp "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl" /tmp/ 2>/dev/null || true
|
||||
# Use version sort (-V) to get the highest version, not time sort (-t)
|
||||
# All wheels download at the same time, so -t picks arbitrarily
|
||||
WHEEL_FILE=$(ls /tmp/karaoke_gen-*.whl 2>/dev/null | sort -V | tail -1)
|
||||
if [ -n "$WHEEL_FILE" ]; then
|
||||
echo "Installing wheel: $WHEEL_FILE"
|
||||
/opt/encoding-worker/venv/bin/pip install --upgrade --quiet "$WHEEL_FILE" || echo "WARNING: Failed to install wheel"
|
||||
else
|
||||
echo "No wheel found in GCS"
|
||||
fi
|
||||
|
||||
echo "=== Startup script complete ==="
|
||||
STARTUP
|
||||
|
||||
chmod +x /opt/encoding-worker/startup.sh
|
||||
chmod +x /opt/encoding-worker/startup-fallback.sh
|
||||
|
||||
# Create systemd service
|
||||
# ExecStartPre runs the startup script to fetch API key and install wheel
|
||||
# ExecStartPre runs bootstrap.sh which downloads and runs the real startup.sh from GCS
|
||||
# ExecStart runs uvicorn with the encoding worker from the installed wheel
|
||||
cat > /etc/systemd/system/encoding-worker.service << 'SYSTEMD'
|
||||
[Unit]
|
||||
@@ -146,11 +186,13 @@ Wants=docker.service
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/encoding-worker
|
||||
ExecStartPre=/opt/encoding-worker/startup.sh
|
||||
# bootstrap.sh downloads startup.sh from GCS, then runs it
|
||||
ExecStartPre=/opt/encoding-worker/bootstrap.sh
|
||||
ExecStart=/opt/encoding-worker/venv/bin/uvicorn backend.services.gce_encoding.main:app --host 0.0.0.0 --port 8080
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
TimeoutStartSec=300
|
||||
# Increased timeout to allow for GCS downloads
|
||||
TimeoutStartSec=600
|
||||
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
|
||||
EnvironmentFile=/opt/encoding-worker/env
|
||||
|
||||
@@ -177,3 +219,8 @@ echo " - FFmpeg at /usr/local/bin/ffmpeg"
|
||||
echo " - Noto fonts (including CJK)"
|
||||
echo " - Virtual environment at /opt/encoding-worker/venv"
|
||||
echo " - Systemd service: encoding-worker.service"
|
||||
echo ""
|
||||
echo "Immutable deployment pattern:"
|
||||
echo " - bootstrap.sh downloads startup.sh from GCS on every start"
|
||||
echo " - startup.sh is CI-managed, no image rebuild needed for logic changes"
|
||||
echo " - See infrastructure/encoding-worker/README.md"
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.109.0"
|
||||
version = "0.110.0"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
Reference in new issue
Block a user