mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary - Replaces single encoding worker VM with a blue-green pair (a/b) for zero-downtime deployments - VMs auto-start on demand (JIT warmup when user opens lyrics review) and auto-shutdown after 15 min idle - CI deploys to secondary VM, runs a real encode test, then swaps the Firestore pointer atomically - Deep health check catches issues like PR #587's pip crash before they reach users - Cloud Function checks idle VMs every 5 minutes and stops them to minimize cost (~$10-15/mo idle vs $800/mo always-on) ## Architecture - **Firestore config doc** (`config/encoding-worker`) is single source of truth for primary/secondary routing - **Backend** reads primary IP from Firestore (30s TTL cache), starts VM on demand via Compute API - **CI workflow** handles blue-green swap: deploy → health check → real encode test → atomic Firestore swap → drain & stop old - **Cloud Function** auto-stops idle VMs based on active jobs, user activity, and deploy status - **Frontend** sends warmup signal on lyrics review page load + heartbeat on interactions ## Changes ### New Files - `backend/services/encoding_worker_manager.py` — VM lifecycle + Firestore config management - `backend/api/routes/encoding_worker.py` — Warmup and heartbeat API endpoints - `infrastructure/functions/encoding_worker_idle/` — Idle shutdown Cloud Function - `infrastructure/modules/encoding_worker_manager.py` — Pulumi module for Cloud Function + Scheduler - `scripts/seed-encoding-worker-config.py` — Firestore config seeding script - `infrastructure/encoding-worker/test-assets/test-preview.ass` — Test file for deploy health check ### Modified Files - `infrastructure/compute/encoding_worker_vm.py` — Single VM → blue-green pair - `infrastructure/__main__.py` — Wire up new VMs, Cloud Function, IAM - `backend/services/encoding_service.py` — Dynamic URL from Firestore, warmup-aware retry - `.github/workflows/ci.yml` — Blue-green deploy flow - `frontend/` — Warmup signal on lyrics review, heartbeat, warming-up UI state - `docs/` — Architecture, development, troubleshooting, lessons learned ## Test plan - [x] 47 new backend tests pass (EncodingWorkerManager, routes, dynamic URL resolution) - [x] 9 Cloud Function unit tests pass - [x] All 2808 existing backend tests pass (1 pre-existing failure unrelated) - [x] All 1116 existing unit tests pass (3 pre-existing failures unrelated) - [ ] Pulumi preview shows expected resource changes - [ ] Pulumi up applied locally before merge - [ ] Firestore config seeded with VM IPs - [ ] End-to-end verification after deployment ## Migration Steps (post-merge) 1. `pulumi up` creates 2 VMs, removes old VM 2. Seed Firestore: `python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>` 3. Start VM A, verify health 4. First CI deploy exercises the blue-green flow ## Design Docs - Spec: `docs/superpowers/specs/2026-03-24-blue-green-encoding-worker-design.md` - Plan: `docs/superpowers/plans/2026-03-24-blue-green-encoding-worker.md` @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
70 lines
1.9 KiB
Python
Executable File
70 lines
1.9 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""
|
|
Seed the Firestore config/encoding-worker document.
|
|
|
|
Run once during migration to initialize the blue-green config.
|
|
Uses the Pulumi-assigned IPs for VMs A and B.
|
|
|
|
Usage:
|
|
python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>
|
|
|
|
Requires:
|
|
- GOOGLE_CLOUD_PROJECT=nomadkaraoke (or gcloud default project)
|
|
- Firestore write access
|
|
"""
|
|
|
|
import sys
|
|
from datetime import datetime, timezone
|
|
from google.cloud import firestore
|
|
|
|
PROJECT_ID = "nomadkaraoke"
|
|
|
|
|
|
def main():
|
|
if len(sys.argv) != 3:
|
|
print("Usage: python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>")
|
|
print(" ip-a: Static IP assigned to encoding-worker-a")
|
|
print(" ip-b: Static IP assigned to encoding-worker-b")
|
|
sys.exit(1)
|
|
|
|
ip_a, ip_b = sys.argv[1], sys.argv[2]
|
|
|
|
db = firestore.Client(project=PROJECT_ID)
|
|
now = datetime.now(timezone.utc).isoformat()
|
|
|
|
config = {
|
|
"primary_vm": "encoding-worker-a",
|
|
"primary_ip": ip_a,
|
|
"primary_version": "unknown",
|
|
"primary_deployed_at": now,
|
|
"secondary_vm": "encoding-worker-b",
|
|
"secondary_ip": ip_b,
|
|
"secondary_version": None,
|
|
"secondary_deployed_at": None,
|
|
"last_swap_at": None,
|
|
"last_activity_at": now,
|
|
"deploy_in_progress": False,
|
|
"deploy_in_progress_since": None,
|
|
}
|
|
|
|
doc_ref = db.collection("config").document("encoding-worker")
|
|
|
|
# Check if already exists
|
|
if doc_ref.get().exists:
|
|
print("WARNING: config/encoding-worker already exists!")
|
|
print("Current value:", doc_ref.get().to_dict())
|
|
response = input("Overwrite? (y/N): ")
|
|
if response.lower() != "y":
|
|
print("Aborted.")
|
|
sys.exit(0)
|
|
|
|
doc_ref.set(config)
|
|
print(f"Seeded config/encoding-worker:")
|
|
print(f" Primary: encoding-worker-a ({ip_a})")
|
|
print(f" Secondary: encoding-worker-b ({ip_b})")
|
|
print(f" Timestamp: {now}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|