mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Why Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has handled every encode and distribution job. 27 jobs went to it, with no Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs ~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP fallback. ## What Deleting the pair needs code changes first, because several paths assumed it exists: - **`encoding_worker_manager`** - The primary is optional: an empty `primary_vm` means none. - `ranked_candidates` skips a missing primary and always sets `is_primary`, so with no primary a successful start records the AWS/c2df override. Previously the legacy index-0 rule would have cleared it. - `ensure_primary_running` and `ensure_preferred_running` start a ranked candidate. If there are no candidates they raise `NO_CANDIDATES`. - `demote_active_worker` looks up the override's `machine_type`, so the AWS box can now be cooled down. It then clears the override so the retry goes elsewhere. - **`encoding_service`** - When `active_url` is empty, it targets the top-ranked candidate. - It also invalidates its cached URL after a single-candidate fallback. - **CI deploy (`ci.yml` + `deploy_promote.py`)** - While AWS serves, a c2df green is validate-only: it writes `validated_version`, keeps the AWS override, and stops the green. Before this, it would have replaced the AWS override with c2df. - The worker-change gate now diffs against `validated_version`, because `primary_version` stops moving. - An in-place restart of a serving override first waits for it to drain (15-min cap). If it is still busy, the restart is skipped. - **Idle function:** a missing VM is logged as a WARNING returning `NOT_FOUND`, not an ERROR. - **Error monitor:** watches c2df instead of a/b. - **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows exactly 4 deletes. A separate, unrelated `youtube-comment-autoreply-scheduler` drift shows up too; it will not be applied. - **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING. ## Rollout (after merge + deploy) 1. Back up `config/encoding-worker`, then set `primary_vm`, `primary_ip`, `secondary_vm` and `secondary_ip` to `""`. 2. Apply `pulumi destroy --target` on the 4 a/b URNs only. 3. Redeploy the idle function with `deploy.sh`. 4. Verify: one prod encode on AWS, warmup, and the logs. **Rollback:** in Firestore, delete `speed_rank_overrides` and set `aws_video_worker=false`. Restoring the pair would need a revert plus `pulumi up`, which gives new static IPs. ## Testing - New `backend/tests/test_encoding_worker_no_primary.py` (13 tests), plus new deploy_promote, idle-function and error-monitor tests. - Full backend suite: 7188 passed. The 8 `test_emulator_integration` failures also fail on main, because of a local emulator from another session. - CodeRabbit CLI: no findings. An independent review found 2 issues, both fixed: the demote retry path, and the in-place restart drain. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe @coderabbitai ignore Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
197 lines
7.6 KiB
Python
197 lines
7.6 KiB
Python
"""EncodingWorkerManager once the GCE c4d blue-green pair is retired (2026-11).
|
|
|
|
config/encoding-worker then has empty (or absent) primary_vm/secondary_vm, and the
|
|
pool is just the fallbacks: the AWS worker (preferred) + the GCE c2df VM. Every
|
|
path must route to a fallback and record it as the active override, never to a
|
|
blank primary.
|
|
"""
|
|
import json
|
|
from unittest.mock import AsyncMock, MagicMock
|
|
|
|
import pytest
|
|
|
|
from backend.services.encoding_errors import EncodingWorkerStartError
|
|
from backend.services.encoding_worker_manager import (
|
|
EncodingWorkerCandidate,
|
|
EncodingWorkerManager,
|
|
)
|
|
|
|
AWS_ENTRY = {
|
|
"vm": "aws-encoding-worker-1", "zone": "us-east-2b", "ip": "3.3.3.3",
|
|
"machine_type": "c7a-4xlarge", "provider": "aws", "instance_id": "i-0abc", "region": "us-east-2",
|
|
}
|
|
C2D_ENTRY = {"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "2.2.2.2",
|
|
"machine_type": "c2d-highcpu-16"}
|
|
|
|
NO_PAIR = {"primary_vm": "", "primary_ip": "", "primary_version": "0.291.0",
|
|
"secondary_vm": "", "secondary_ip": ""}
|
|
|
|
|
|
def _db(**config):
|
|
data = {"last_activity_at": None, "capacity_state": {}, "speed_rank_overrides": {"c7a": 5},
|
|
**config}
|
|
db = MagicMock()
|
|
snap = MagicMock(exists=True)
|
|
snap.to_dict.return_value = data
|
|
doc = MagicMock()
|
|
doc.get.return_value = snap
|
|
db.collection.return_value.document.return_value = doc
|
|
return db, doc
|
|
|
|
|
|
def _manager(db, entries=(C2D_ENTRY, AWS_ENTRY)):
|
|
gce = MagicMock(name="gce")
|
|
aws = MagicMock(name="aws")
|
|
aws.handles.side_effect = lambda vm: vm == AWS_ENTRY["vm"]
|
|
m = EncodingWorkerManager(db=db, compute_client=gce,
|
|
fallback_vms_raw=json.dumps(list(entries)), aws_compute=aws)
|
|
return m, gce, aws
|
|
|
|
|
|
def test_get_config_tolerates_absent_pair_keys():
|
|
db, _ = _db() # no primary/secondary keys at all
|
|
m, _, _ = _manager(db)
|
|
config = m.get_config()
|
|
assert config.primary_vm == "" and config.secondary_vm == ""
|
|
assert not config.has_primary
|
|
assert config.primary_url == ""
|
|
assert config.active_url == ""
|
|
|
|
|
|
def test_active_url_follows_override_without_primary():
|
|
db, _ = _db(**NO_PAIR, active_override_vm="aws-encoding-worker-1", active_override_ip="3.3.3.3")
|
|
m, _, _ = _manager(db)
|
|
assert m.get_config().active_url == "http://3.3.3.3:8080"
|
|
|
|
|
|
def test_ranked_candidates_skip_blank_primary():
|
|
db, _ = _db(**NO_PAIR)
|
|
m, _, _ = _manager(db)
|
|
ranked = m.ranked_candidates()
|
|
assert [c.vm_name for c in ranked] == ["aws-encoding-worker-1", "encoding-worker-fallback-c2df"]
|
|
assert all(c.is_primary is False for c in ranked)
|
|
|
|
|
|
def test_aws_landing_sets_override_not_clears_it():
|
|
"""With no primary, index 0 (AWS) is a fallback: starting it must SET the
|
|
override (the legacy 'index 0 is the primary' rule would clear it, leaving
|
|
active_url empty)."""
|
|
db, doc = _db(**NO_PAIR)
|
|
m, _, aws = _manager(db)
|
|
aws.get.return_value = MagicMock(status="RUNNING")
|
|
m.update_activity = MagicMock()
|
|
result = m.ensure_any_running(m.ranked_candidates())
|
|
assert result["vm_name"] == "aws-encoding-worker-1"
|
|
assert result["fell_back"] is True
|
|
update = doc.update.call_args[0][0]
|
|
assert update["active_override_vm"] == "aws-encoding-worker-1"
|
|
assert update["active_override_provider"] == "aws"
|
|
|
|
|
|
def test_hand_built_candidates_keep_legacy_index0_rule():
|
|
"""Candidates built without is_primary (None) keep the old position rule."""
|
|
db, doc = _db(**NO_PAIR)
|
|
m, gce, _ = _manager(db)
|
|
gce.get.return_value = MagicMock(status="RUNNING")
|
|
m.update_activity = MagicMock()
|
|
m._clear_active_override = MagicMock()
|
|
m._set_active_override = MagicMock()
|
|
result = m.ensure_any_running([EncodingWorkerCandidate("vm-x", "us-central1-f", "9.9.9.9")])
|
|
assert result["fell_back"] is False
|
|
m._clear_active_override.assert_called_once()
|
|
m._set_active_override.assert_not_called()
|
|
|
|
|
|
def test_ensure_primary_running_without_pair_starts_ranked_candidate():
|
|
db, _ = _db(**NO_PAIR)
|
|
m, gce, _ = _manager(db)
|
|
m.ensure_any_running = MagicMock(return_value={"started": True, "vm_name": "aws-encoding-worker-1",
|
|
"fell_back": True})
|
|
result = m.ensure_primary_running()
|
|
assert result["vm_name"] == "aws-encoding-worker-1"
|
|
(candidates,), _ = m.ensure_any_running.call_args
|
|
assert candidates[0].vm_name == "aws-encoding-worker-1"
|
|
gce.get.assert_not_called() # never looks up a blank primary on GCE
|
|
|
|
|
|
def test_ensure_preferred_running_without_pair_never_takes_primary_path():
|
|
# No rank override → c2df ranks first; still no primary path.
|
|
db, _ = _db(**NO_PAIR, speed_rank_overrides={})
|
|
m, _, _ = _manager(db)
|
|
m.ensure_any_running = MagicMock(return_value={"started": False})
|
|
m.ensure_preferred_running()
|
|
m.ensure_any_running.assert_called_once()
|
|
|
|
|
|
@pytest.mark.parametrize("method", ["ensure_preferred_running", "ensure_primary_running"])
|
|
def test_no_candidates_at_all_raises_start_error(method):
|
|
db, _ = _db(**NO_PAIR)
|
|
m, _, _ = _manager(db, entries=())
|
|
with pytest.raises(EncodingWorkerStartError) as ei:
|
|
getattr(m, method)()
|
|
assert ei.value.code == "NO_CANDIDATES"
|
|
|
|
|
|
def test_demote_aws_without_primary_retries_on_another_worker():
|
|
"""A broken AWS worker is cooled down (via its machine_type) and the override
|
|
cleared, so the retry's URL resolves to the GCE fallback instead."""
|
|
from backend.services.encoding_service import EncodingService
|
|
|
|
db, doc = _db(**NO_PAIR, active_override_vm="aws-encoding-worker-1",
|
|
active_override_zone="us-east-2b", active_override_ip="3.3.3.3")
|
|
m, _, _ = _manager(db)
|
|
m.demote_active_worker("metadata unreachable")
|
|
state = doc.set.call_args[0][0]["capacity_state"]
|
|
assert list(state) == ["c7a-4xlarge@us-east-2b"]
|
|
assert doc.update.call_args[0][0]["active_override_vm"] is None
|
|
|
|
# Next read: override gone, AWS in cooldown → URL is c2df.
|
|
db2, _ = _db(**NO_PAIR, capacity_state=state)
|
|
m2, _, _ = _manager(db2)
|
|
svc = EncodingService.__new__(EncodingService)
|
|
svc._cached_url = None
|
|
svc._url_cached_at = 0.0
|
|
svc._worker_manager = m2
|
|
assert svc._get_worker_url() == "http://2.2.2.2:8080"
|
|
|
|
|
|
def test_demote_with_nothing_serving_is_quiet(caplog):
|
|
db, _ = _db(**NO_PAIR)
|
|
m, _, _ = _manager(db)
|
|
m.demote_active_worker("x")
|
|
assert not [r for r in caplog.records if r.levelname == "ERROR"]
|
|
|
|
|
|
def test_encoding_service_resolves_top_candidate_when_no_url():
|
|
from backend.services.encoding_service import EncodingService
|
|
|
|
db, _ = _db(**NO_PAIR)
|
|
m, _, _ = _manager(db)
|
|
svc = EncodingService.__new__(EncodingService)
|
|
svc._cached_url = None
|
|
svc._url_cached_at = 0.0
|
|
svc._worker_manager = m
|
|
assert svc._get_worker_url() == "http://3.3.3.3:8080"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_single_candidate_warmup_invalidates_cache_on_fallback():
|
|
from backend.services.encoding_service import EncodingService
|
|
|
|
svc = EncodingService.__new__(EncodingService)
|
|
svc._cached_url = "http://stale:8080"
|
|
svc._url_cached_at = 1e12
|
|
manager = MagicMock()
|
|
manager.get_config.return_value = MagicMock()
|
|
manager.ranked_candidates.return_value = [
|
|
EncodingWorkerCandidate("aws-encoding-worker-1", "us-east-2b", "3.3.3.3", is_primary=False)
|
|
]
|
|
manager.ensure_primary_running.return_value = {
|
|
"started": False, "vm_name": "aws-encoding-worker-1", "zone": "us-east-2b",
|
|
"primary_url": "http://3.3.3.3:8080", "fell_back": True,
|
|
}
|
|
manager.wait_for_worker_ready = AsyncMock()
|
|
svc._worker_manager = manager
|
|
await svc._warmup_encoding_worker_fallback("job1")
|
|
assert svc._cached_url is None
|