mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary - Implements robust deployment pattern for GCE encoding worker to eliminate recurring version mismatch issues - Fixed wheel path (`karaoke_gen-current.whl`) eliminates version sorting bugs - Self-updating startup script downloaded from GCS on every service start - Version manifest with strict CI verification (deployment fails on mismatch) ## Problem The encoding worker had recurring version issues because: - `startup.sh` was baked into the Packer image (required rebuild + VM recreation) - Wheel selection used version sorting across 51 files - CI only warned on version mismatch, didn't fail - Multiple deployment vectors (CI, Pulumi, manual SSH) caused drift ## Solution 1. **Fixed wheel path**: CI uploads to `karaoke_gen-current.whl` (no sorting needed) 2. **Self-updating startup**: `bootstrap.sh` downloads `startup.sh` from GCS on every start 3. **Version manifest**: CI writes expected version, service verifies before starting 4. **Strict CI verification**: Deployment fails if version doesn't match (was just warning) ## Changes - **New**: `infrastructure/encoding-worker/` with `bootstrap.sh`, `startup.sh`, README - **Modified**: CI workflow for fixed paths and strict verification - **Modified**: Packer `provision.sh` for bootstrap pattern - **Updated**: `LESSONS-LEARNED.md` with immutable deployment pattern - **Updated**: `docs/README.md` and `docs/ARCHITECTURE.md` ## Test plan - [x] Shell scripts validated (`bash -n`) - [x] Backend tests pass (1703 passed) - [x] CodeRabbit review passed - [ ] After merge: Rebuild Packer image - [ ] After merge: Recreate VM to use new image - [ ] After merge: Verify first deployment uses new pattern ## Post-Merge Steps Required 1. Rebuild Packer image: `packer build infrastructure/packer/encoding-worker.pkr.hcl` 2. Recreate VM: Delete encoding-worker, run `pulumi up` 🤖 Generated with [Claude Code](https://claude.com/claude-code) @coderabbitai ignore --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
4.8 KiB
4.8 KiB
Encoding Worker Deployment
This directory contains the scripts for the GCE encoding worker's immutable deployment pattern.
Architecture
┌─────────────────────────────────────────────────────────────────┐
│ PACKER IMAGE (rarely changes) │
├─────────────────────────────────────────────────────────────────┤
│ - Python 3.13, FFmpeg, fonts (static dependencies) │
│ - bootstrap.sh (downloads startup.sh from GCS) │
│ - systemd service definition │
└─────────────────────────────────────────────────────────────────┘
│
▼ downloads on every service start
┌─────────────────────────────────────────────────────────────────┐
│ GCS (CI-managed, always current) │
├─────────────────────────────────────────────────────────────────┤
│ gs://karaoke-gen-storage-nomadkaraoke/ │
│ ├── encoding-worker/ │
│ │ ├── startup.sh ← Main logic (this file) │
│ │ └── version.txt ← Expected version number │
│ └── wheels/ │
│ └── karaoke_gen-current.whl ← Latest wheel (fixed path) │
└─────────────────────────────────────────────────────────────────┘
Files
| File | Location | Purpose |
|---|---|---|
bootstrap.sh |
Baked in Packer image | Downloads and runs startup.sh |
startup.sh |
Deployed to GCS by CI | Main startup logic |
version.txt |
Written to GCS by CI | Expected version for verification |
Deployment Flow
- CI builds wheel → uploads to
wheels/karaoke_gen-current.whl - CI uploads startup.sh →
encoding-worker/startup.sh - CI writes version →
encoding-worker/version.txt - CI restarts service → triggers bootstrap.sh
- bootstrap.sh downloads fresh startup.sh from GCS
- startup.sh downloads wheel, installs, verifies version
- CI verifies health endpoint returns correct version
- CI FAILS if version mismatch
Why This Pattern?
Before (problematic)
- startup.sh baked into Packer image
- Required image rebuild + VM recreation for logic changes
- 51 versioned wheels, sorting could fail
- CI warned on version mismatch but continued
After (robust)
- startup.sh downloaded from GCS on every start
- Logic changes via CI, no image rebuild needed
- Fixed path
karaoke_gen-current.whl, no sorting - CI FAILS on version mismatch
Modifying Startup Logic
- Edit
infrastructure/encoding-worker/startup.sh - Push to main
- CI automatically deploys to GCS
- Service restart picks up new script
No Packer image rebuild needed!
When to Rebuild Packer Image
Only rebuild when changing:
- Python version
- FFmpeg version
- System packages/fonts
- bootstrap.sh (should be extremely rare)
Manual Operations
# Check current version on VM
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--tunnel-through-iap \
--command="curl -s http://localhost:8080/health | jq '.wheel_version'"
# View startup logs
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--tunnel-through-iap \
--command="sudo journalctl -u encoding-worker -n 50"
# Force service restart (triggers fresh startup.sh download)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--tunnel-through-iap \
--command="sudo systemctl restart encoding-worker"
Troubleshooting
Version mismatch after deployment
- Check CI logs for GCS upload success
- Check
/var/log/encoding-worker-startup.logon VM - Verify
version.txtin GCS matches expected
Service won't start
- Check bootstrap.sh can reach GCS:
gsutil ls gs://karaoke-gen-storage-nomadkaraoke/ - Check Secret Manager access for API key
- Check startup.sh logs:
cat /var/log/encoding-worker-startup.log