mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Why Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has handled every encode and distribution job. 27 jobs went to it, with no Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs ~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP fallback. ## What Deleting the pair needs code changes first, because several paths assumed it exists: - **`encoding_worker_manager`** - The primary is optional: an empty `primary_vm` means none. - `ranked_candidates` skips a missing primary and always sets `is_primary`, so with no primary a successful start records the AWS/c2df override. Previously the legacy index-0 rule would have cleared it. - `ensure_primary_running` and `ensure_preferred_running` start a ranked candidate. If there are no candidates they raise `NO_CANDIDATES`. - `demote_active_worker` looks up the override's `machine_type`, so the AWS box can now be cooled down. It then clears the override so the retry goes elsewhere. - **`encoding_service`** - When `active_url` is empty, it targets the top-ranked candidate. - It also invalidates its cached URL after a single-candidate fallback. - **CI deploy (`ci.yml` + `deploy_promote.py`)** - While AWS serves, a c2df green is validate-only: it writes `validated_version`, keeps the AWS override, and stops the green. Before this, it would have replaced the AWS override with c2df. - The worker-change gate now diffs against `validated_version`, because `primary_version` stops moving. - An in-place restart of a serving override first waits for it to drain (15-min cap). If it is still busy, the restart is skipped. - **Idle function:** a missing VM is logged as a WARNING returning `NOT_FOUND`, not an ERROR. - **Error monitor:** watches c2df instead of a/b. - **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows exactly 4 deletes. A separate, unrelated `youtube-comment-autoreply-scheduler` drift shows up too; it will not be applied. - **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING. ## Rollout (after merge + deploy) 1. Back up `config/encoding-worker`, then set `primary_vm`, `primary_ip`, `secondary_vm` and `secondary_ip` to `""`. 2. Apply `pulumi destroy --target` on the 4 a/b URNs only. 3. Redeploy the idle function with `deploy.sh`. 4. Verify: one prod encode on AWS, warmup, and the logs. **Rollback:** in Firestore, delete `speed_rank_overrides` and set `aws_video_worker=false`. Restoring the pair would need a revert plus `pulumi up`, which gives new static IPs. ## Testing - New `backend/tests/test_encoding_worker_no_primary.py` (13 tests), plus new deploy_promote, idle-function and error-monitor tests. - Full backend suite: 7188 passed. The 8 `test_emulator_integration` failures also fail on main, because of a local emulator from another session. - CodeRabbit CLI: no findings. An independent review found 2 issues, both fixed: the demote retry path, and the in-place restart drain. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe @coderabbitai ignore Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Encoding Worker Deployment
This directory contains the scripts for the GCE encoding worker's immutable deployment pattern.
Architecture
┌─────────────────────────────────────────────────────────────────┐
│ PACKER IMAGE (rarely changes) │
├─────────────────────────────────────────────────────────────────┤
│ - Python 3.13, FFmpeg, fonts (static dependencies) │
│ - bootstrap.sh (downloads startup.sh from GCS) │
│ - systemd service definition │
└─────────────────────────────────────────────────────────────────┘
│
▼ downloads on every service start
┌─────────────────────────────────────────────────────────────────┐
│ GCS (CI-managed, always current) │
├─────────────────────────────────────────────────────────────────┤
│ gs://karaoke-gen-storage-nomadkaraoke/ │
│ ├── encoding-worker/ │
│ │ ├── startup.sh ← Main logic (this file) │
│ │ └── version.txt ← Expected version number │
│ └── wheels/ │
│ └── karaoke_gen-current.whl ← Latest wheel (fixed path) │
└─────────────────────────────────────────────────────────────────┘
Files
| File | Location | Purpose |
|---|---|---|
bootstrap.sh |
Baked in Packer image | Downloads and runs startup.sh |
startup.sh |
Deployed to GCS by CI | Main startup logic |
version.txt |
Written to GCS by CI | Expected version for verification |
Deployment Flow
- CI builds wheel → uploads to
wheels/karaoke_gen-current.whl - CI uploads startup.sh →
encoding-worker/startup.sh - CI writes version →
encoding-worker/version.txt - CI restarts service → triggers bootstrap.sh
- bootstrap.sh downloads fresh startup.sh from GCS
- startup.sh downloads wheel, installs, verifies version
- CI verifies health endpoint returns correct version
- CI FAILS if version mismatch
Why This Pattern?
Before (problematic)
- startup.sh baked into Packer image
- Required image rebuild + VM recreation for logic changes
- 51 versioned wheels, sorting could fail
- CI warned on version mismatch but continued
After (robust)
- startup.sh downloaded from GCS on every start
- Logic changes via CI, no image rebuild needed
- Fixed path
karaoke_gen-current.whl, no sorting - CI FAILS on version mismatch
Modifying Startup Logic
- Edit
infrastructure/encoding-worker/startup.sh - Push to main
- CI automatically deploys to GCS
- Service restart picks up new script
No Packer image rebuild needed!
When to Rebuild Packer Image
Only rebuild when changing:
- Python version
- FFmpeg version
- System packages/fonts
- bootstrap.sh (should be extremely rare)
Manual Operations
# Check current version on VM
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--tunnel-through-iap \
--command="curl -s http://localhost:8080/health | jq '.wheel_version'"
# View startup logs
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--tunnel-through-iap \
--command="sudo journalctl -u encoding-worker -n 50"
# Force service restart (triggers fresh startup.sh download)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--tunnel-through-iap \
--command="sudo systemctl restart encoding-worker"
Troubleshooting
Version mismatch after deployment
- Check CI logs for GCS upload success
- Check
/var/log/encoding-worker-startup.logon VM - Verify
version.txtin GCS matches expected
Service won't start
- Check bootstrap.sh can reach GCS:
gsutil ls gs://karaoke-gen-storage-nomadkaraoke/ - Check Secret Manager access for API key
- Check startup.sh logs:
cat /var/log/encoding-worker-startup.log