Files
karaoke-gen/scripts/seed-encoding-worker-config.py
T
Andrew Beveridge ☄️andClaude Opus 4.6 19efa269a8 feat: blue-green encoding worker deployment with deep health checks (#595)
## Summary

- Replaces single encoding worker VM with a blue-green pair (a/b) for
zero-downtime deployments
- VMs auto-start on demand (JIT warmup when user opens lyrics review)
and auto-shutdown after 15 min idle
- CI deploys to secondary VM, runs a real encode test, then swaps the
Firestore pointer atomically
- Deep health check catches issues like PR #587's pip crash before they
reach users
- Cloud Function checks idle VMs every 5 minutes and stops them to
minimize cost (~$10-15/mo idle vs $800/mo always-on)

## Architecture

- **Firestore config doc** (`config/encoding-worker`) is single source
of truth for primary/secondary routing
- **Backend** reads primary IP from Firestore (30s TTL cache), starts VM
on demand via Compute API
- **CI workflow** handles blue-green swap: deploy → health check → real
encode test → atomic Firestore swap → drain & stop old
- **Cloud Function** auto-stops idle VMs based on active jobs, user
activity, and deploy status
- **Frontend** sends warmup signal on lyrics review page load +
heartbeat on interactions

## Changes

### New Files
- `backend/services/encoding_worker_manager.py` — VM lifecycle +
Firestore config management
- `backend/api/routes/encoding_worker.py` — Warmup and heartbeat API
endpoints
- `infrastructure/functions/encoding_worker_idle/` — Idle shutdown Cloud
Function
- `infrastructure/modules/encoding_worker_manager.py` — Pulumi module
for Cloud Function + Scheduler
- `scripts/seed-encoding-worker-config.py` — Firestore config seeding
script
- `infrastructure/encoding-worker/test-assets/test-preview.ass` — Test
file for deploy health check

### Modified Files
- `infrastructure/compute/encoding_worker_vm.py` — Single VM →
blue-green pair
- `infrastructure/__main__.py` — Wire up new VMs, Cloud Function, IAM
- `backend/services/encoding_service.py` — Dynamic URL from Firestore,
warmup-aware retry
- `.github/workflows/ci.yml` — Blue-green deploy flow
- `frontend/` — Warmup signal on lyrics review, heartbeat, warming-up UI
state
- `docs/` — Architecture, development, troubleshooting, lessons learned

## Test plan

- [x] 47 new backend tests pass (EncodingWorkerManager, routes, dynamic
URL resolution)
- [x] 9 Cloud Function unit tests pass
- [x] All 2808 existing backend tests pass (1 pre-existing failure
unrelated)
- [x] All 1116 existing unit tests pass (3 pre-existing failures
unrelated)
- [ ] Pulumi preview shows expected resource changes
- [ ] Pulumi up applied locally before merge
- [ ] Firestore config seeded with VM IPs
- [ ] End-to-end verification after deployment

## Migration Steps (post-merge)

1. `pulumi up` creates 2 VMs, removes old VM
2. Seed Firestore: `python scripts/seed-encoding-worker-config.py <ip-a>
<ip-b>`
3. Start VM A, verify health
4. First CI deploy exercises the blue-green flow

## Design Docs

- Spec:
`docs/superpowers/specs/2026-03-24-blue-green-encoding-worker-design.md`
- Plan:
`docs/superpowers/plans/2026-03-24-blue-green-encoding-worker.md`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 11:28:45 -04:00

70 lines
1.9 KiB
Python
Executable File

#!/usr/bin/env python3
"""
Seed the Firestore config/encoding-worker document.
Run once during migration to initialize the blue-green config.
Uses the Pulumi-assigned IPs for VMs A and B.
Usage:
python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>
Requires:
- GOOGLE_CLOUD_PROJECT=nomadkaraoke (or gcloud default project)
- Firestore write access
"""
import sys
from datetime import datetime, timezone
from google.cloud import firestore
PROJECT_ID = "nomadkaraoke"
def main():
if len(sys.argv) != 3:
print("Usage: python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>")
print(" ip-a: Static IP assigned to encoding-worker-a")
print(" ip-b: Static IP assigned to encoding-worker-b")
sys.exit(1)
ip_a, ip_b = sys.argv[1], sys.argv[2]
db = firestore.Client(project=PROJECT_ID)
now = datetime.now(timezone.utc).isoformat()
config = {
"primary_vm": "encoding-worker-a",
"primary_ip": ip_a,
"primary_version": "unknown",
"primary_deployed_at": now,
"secondary_vm": "encoding-worker-b",
"secondary_ip": ip_b,
"secondary_version": None,
"secondary_deployed_at": None,
"last_swap_at": None,
"last_activity_at": now,
"deploy_in_progress": False,
"deploy_in_progress_since": None,
}
doc_ref = db.collection("config").document("encoding-worker")
# Check if already exists
if doc_ref.get().exists:
print("WARNING: config/encoding-worker already exists!")
print("Current value:", doc_ref.get().to_dict())
response = input("Overwrite? (y/N): ")
if response.lower() != "y":
print("Aborted.")
sys.exit(0)
doc_ref.set(config)
print(f"Seeded config/encoding-worker:")
print(f" Primary: encoding-worker-a ({ip_a})")
print(f" Secondary: encoding-worker-b ({ip_b})")
print(f" Timestamp: {now}")
if __name__ == "__main__":
main()