mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
262 lines
13 KiB
Python
262 lines
13 KiB
Python
"""Unit tests for the encoding-worker deploy decision logic.
|
|
|
|
The module lives under infrastructure/ (not an importable package), so we load
|
|
it by path. Covers the two things the CI workflow can't test itself: which VMs
|
|
to try as a fresh green target, and how to promote a validated green to serving.
|
|
"""
|
|
import importlib.util
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
_MODULE_PATH = (
|
|
Path(__file__).resolve().parents[2]
|
|
/ "infrastructure" / "encoding-worker" / "deploy_promote.py"
|
|
)
|
|
_spec = importlib.util.spec_from_file_location("deploy_promote", _MODULE_PATH)
|
|
deploy_promote = importlib.util.module_from_spec(_spec)
|
|
_spec.loader.exec_module(deploy_promote)
|
|
|
|
select_green_candidates = deploy_promote.select_green_candidates
|
|
promote_plan = deploy_promote.promote_plan
|
|
|
|
|
|
FALLBACKS = [
|
|
{"vm": "encoding-worker-fallback-a", "zone": "us-central1-a", "ip": "34.55.231.54"},
|
|
{"vm": "encoding-worker-fallback-b", "zone": "us-central1-b", "ip": "35.193.127.24"},
|
|
{"vm": "encoding-worker-fallback-n2c", "zone": "us-central1-c", "ip": "35.226.119.227"},
|
|
{"vm": "encoding-worker-fallback-n2f", "zone": "us-central1-f", "ip": "35.254.152.14"},
|
|
]
|
|
|
|
|
|
def _config(**over):
|
|
base = {
|
|
"primary_vm": "encoding-worker-b", "primary_ip": "34.10.189.118",
|
|
"primary_version": "0.192.9",
|
|
"secondary_vm": "encoding-worker-a", "secondary_ip": "34.57.78.246",
|
|
"active_override_vm": "encoding-worker-fallback-n2f",
|
|
"active_override_ip": "35.254.152.14", "active_override_zone": "us-central1-f",
|
|
}
|
|
base.update(over)
|
|
return base
|
|
|
|
|
|
class TestSelectGreenCandidates:
|
|
def test_secondary_first_then_n2_before_c4d_excluding_override(self):
|
|
# LEGACY secret shape (no machine_type). The shared ranking infers n2 from
|
|
# the n2c/n2f names (rank 70) but CANNOT infer the bare c4d fallbacks a/b
|
|
# (no family token) → they get UNKNOWN_RANK and sort last. Net order here
|
|
# matches the pre-refactor "secondary, then n2, then c4d" result. Once the
|
|
# secret carries machine_type (post-rollout), a/b rank as c4d — see
|
|
# TestSelectGreenRankingSharedWithRuntime.
|
|
cands = select_green_candidates(_config(), FALLBACKS)
|
|
vms = [c["vm"] for c in cands]
|
|
# c4d secondary leads; current override (n2f) excluded; n2c (n2) before c4d fallbacks.
|
|
assert vms[0] == "encoding-worker-a"
|
|
assert cands[0]["kind"] == "secondary"
|
|
assert "encoding-worker-fallback-n2f" not in vms # the serving override is excluded
|
|
assert vms[1] == "encoding-worker-fallback-n2c" # remaining n2 preferred
|
|
assert vms[1:] == [
|
|
"encoding-worker-fallback-n2c",
|
|
"encoding-worker-fallback-a",
|
|
"encoding-worker-fallback-b",
|
|
]
|
|
assert all(c["kind"] == "fallback" for c in cands[1:])
|
|
|
|
def test_candidate_carries_zone_and_ip(self):
|
|
cands = select_green_candidates(_config(), FALLBACKS)
|
|
n2c = next(c for c in cands if c["vm"] == "encoding-worker-fallback-n2c")
|
|
assert n2c["zone"] == "us-central1-c"
|
|
assert n2c["ip"] == "35.226.119.227"
|
|
|
|
def test_no_secondary_still_lists_fallbacks(self):
|
|
cands = select_green_candidates(_config(secondary_vm=None), FALLBACKS)
|
|
assert all(c["kind"] == "fallback" for c in cands)
|
|
assert cands[0]["vm"] == "encoding-worker-fallback-n2c"
|
|
|
|
def test_malformed_fallback_entries_are_dropped(self):
|
|
fbs = [
|
|
{"vm": "encoding-worker-fallback-n2c", "zone": "us-central1-c", "ip": "35.226.119.227"},
|
|
{"vm": "encoding-worker-fallback-broken", "ip": "1.2.3.4"}, # no zone
|
|
{"zone": "us-central1-a", "ip": "5.6.7.8"}, # no vm
|
|
{"vm": "encoding-worker-fallback-noip", "zone": "us-central1-b"}, # no ip
|
|
]
|
|
cands = select_green_candidates(_config(active_override_vm=None), fbs)
|
|
vms = [c["vm"] for c in cands]
|
|
assert vms.count("encoding-worker-fallback-n2c") == 1
|
|
assert "encoding-worker-fallback-broken" not in vms
|
|
assert "encoding-worker-fallback-noip" not in vms
|
|
assert None not in vms # the entry with no "vm" is dropped, not emitted as vm=None
|
|
|
|
def test_no_override_keeps_all_fallbacks(self):
|
|
cands = select_green_candidates(_config(active_override_vm=None), FALLBACKS)
|
|
fb_vms = [c["vm"] for c in cands if c["kind"] == "fallback"]
|
|
# n2 fallbacks first (either order among the two n2s is fine — assert as a set head)
|
|
assert set(fb_vms[:2]) == {"encoding-worker-fallback-n2c", "encoding-worker-fallback-n2f"}
|
|
assert set(fb_vms) == {f["vm"] for f in FALLBACKS}
|
|
|
|
|
|
class TestPromotePlanFallbackBranch:
|
|
def test_fallback_green_becomes_override_and_drains_old(self):
|
|
green = {"vm": "encoding-worker-fallback-n2c", "ip": "35.226.119.227",
|
|
"zone": "us-central1-c", "kind": "fallback"}
|
|
plan = promote_plan(green, _config(), now="NOW", version="0.194.1")
|
|
u = plan["firestore_updates"]
|
|
assert u["active_override_vm"] == "encoding-worker-fallback-n2c"
|
|
assert u["active_override_ip"] == "35.226.119.227"
|
|
assert u["active_override_zone"] == "us-central1-c"
|
|
assert u["active_override_version"] == "0.194.1"
|
|
assert u["deploy_in_progress"] is False
|
|
# primary/secondary untouched (c4d resumes when capacity returns)
|
|
assert "primary_vm" not in u and "secondary_vm" not in u
|
|
# old override (n2f) drained + stopped
|
|
assert plan["drain_and_stop"] == [
|
|
{"vm": "encoding-worker-fallback-n2f", "ip": "35.254.152.14", "zone": "us-central1-f"}
|
|
]
|
|
|
|
def test_no_drain_when_no_previous_override(self):
|
|
green = {"vm": "encoding-worker-fallback-n2c", "ip": "35.226.119.227",
|
|
"zone": "us-central1-c", "kind": "fallback"}
|
|
plan = promote_plan(green, _config(active_override_vm=None), now="NOW", version="0.194.1")
|
|
assert plan["drain_and_stop"] == []
|
|
|
|
def test_no_self_drain_when_green_is_current_override(self):
|
|
# Degenerate (shouldn't happen since candidates exclude the override), but
|
|
# must never drain-stop the worker we just promoted.
|
|
green = {"vm": "encoding-worker-fallback-n2f", "ip": "35.254.152.14",
|
|
"zone": "us-central1-f", "kind": "fallback"}
|
|
plan = promote_plan(green, _config(), now="NOW", version="0.194.1")
|
|
assert plan["drain_and_stop"] == []
|
|
|
|
|
|
class TestPromotePlanSecondaryBranch:
|
|
def test_secondary_green_swaps_and_clears_stale_override(self):
|
|
green = {"vm": "encoding-worker-a", "ip": "34.57.78.246",
|
|
"zone": "us-central1-c", "kind": "secondary"}
|
|
plan = promote_plan(green, _config(), now="NOW", version="0.194.1")
|
|
u = plan["firestore_updates"]
|
|
# green -> primary, old primary -> secondary
|
|
assert u["primary_vm"] == "encoding-worker-a"
|
|
assert u["primary_version"] == "0.194.1"
|
|
assert u["secondary_vm"] == "encoding-worker-b"
|
|
assert u["secondary_version"] == "0.192.9"
|
|
# stale override cleared so active_url follows the fresh primary
|
|
assert u["active_override_vm"] is None
|
|
assert u["active_override_version"] is None
|
|
assert u["deploy_in_progress"] is False
|
|
# both the old primary and the old override get drained + stopped
|
|
drained = {d["vm"] for d in plan["drain_and_stop"]}
|
|
assert drained == {"encoding-worker-b", "encoding-worker-fallback-n2f"}
|
|
|
|
def test_secondary_never_drains_the_promoted_green(self):
|
|
# Pathological config where a stale override (or primary) names the green
|
|
# VM — the plan must never tell the workflow to stop what it just promoted.
|
|
green = {"vm": "encoding-worker-a", "ip": "34.57.78.246",
|
|
"zone": "us-central1-c", "kind": "secondary"}
|
|
cfg = _config(active_override_vm="encoding-worker-a",
|
|
active_override_ip="34.57.78.246",
|
|
active_override_zone="us-central1-c")
|
|
plan = promote_plan(green, cfg, now="NOW", version="0.194.2")
|
|
assert all(d["vm"] != "encoding-worker-a" for d in plan["drain_and_stop"])
|
|
# The override is still cleared in config even though we don't stop that VM.
|
|
assert plan["firestore_updates"]["active_override_vm"] is None
|
|
|
|
def test_secondary_swap_without_override(self):
|
|
green = {"vm": "encoding-worker-a", "ip": "34.57.78.246",
|
|
"zone": "us-central1-c", "kind": "secondary"}
|
|
plan = promote_plan(green, _config(active_override_vm=None), now="NOW", version="0.194.1")
|
|
u = plan["firestore_updates"]
|
|
assert "active_override_vm" not in u # nothing to clear
|
|
assert [d["vm"] for d in plan["drain_and_stop"]] == ["encoding-worker-b"]
|
|
|
|
|
|
class TestSelectGreenRankingSharedWithRuntime:
|
|
"""Green selection now ranks via the shared preference (machine_type + cooldown)."""
|
|
|
|
# 6-type pool with explicit machine_type (mirrors the post-rollout secret).
|
|
POOL = [
|
|
{"vm": "encoding-worker-fallback-c4a", "zone": "us-central1-a", "ip": "10.0.0.1",
|
|
"machine_type": "c4-highcpu-16"},
|
|
{"vm": "encoding-worker-fallback-n4db", "zone": "us-central1-b", "ip": "10.0.0.2",
|
|
"machine_type": "n4d-highcpu-16"},
|
|
{"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "10.0.0.3",
|
|
"machine_type": "c2d-highcpu-16"},
|
|
{"vm": "encoding-worker-fallback-n2da", "zone": "us-central1-a", "ip": "10.0.0.4",
|
|
"machine_type": "n2d-highcpu-16"},
|
|
{"vm": "encoding-worker-fallback-n2c", "zone": "us-central1-c", "ip": "10.0.0.5",
|
|
"machine_type": "n2-highcpu-16"},
|
|
]
|
|
|
|
def test_secondary_c4d_leads_then_fastest_fallbacks(self):
|
|
cfg = _config(active_override_vm=None)
|
|
vms = [c["vm"] for c in select_green_candidates(cfg, self.POOL)]
|
|
# c4d secondary (rank 10) first, then c4 > n4d > c2d > n2d > n2.
|
|
assert vms == [
|
|
"encoding-worker-a", # secondary, c4d
|
|
"encoding-worker-fallback-c4a", # c4
|
|
"encoding-worker-fallback-n4db", # n4d
|
|
"encoding-worker-fallback-c2df", # c2d
|
|
"encoding-worker-fallback-n2da", # n2d
|
|
"encoding-worker-fallback-n2c", # n2
|
|
]
|
|
|
|
def test_c4d_stockout_demotes_secondary_below_deep_pool(self):
|
|
# c4d@us-central1-c (the secondary's zone) recently stocked out.
|
|
from datetime import datetime, timezone
|
|
cfg = _config(
|
|
active_override_vm=None,
|
|
capacity_state={"c4d-highcpu-16@us-central1-c": datetime.now(timezone.utc).isoformat()},
|
|
)
|
|
cands = select_green_candidates(cfg, self.POOL)
|
|
vms = [c["vm"] for c in cands]
|
|
# The c4d secondary is demoted to the back; a fast fallback (c4) now leads.
|
|
assert vms[0] == "encoding-worker-fallback-c4a"
|
|
assert vms[-1] == "encoding-worker-a" # demoted secondary re-probed last
|
|
# kind survives the reordering so promote_plan still treats it correctly.
|
|
assert cands[-1]["kind"] == "secondary"
|
|
|
|
def test_machine_type_carried_through(self):
|
|
cands = select_green_candidates(_config(active_override_vm=None), self.POOL)
|
|
c4 = next(c for c in cands if c["vm"] == "encoding-worker-fallback-c4a")
|
|
assert c4["machine_type"] == "c4-highcpu-16"
|
|
|
|
|
|
# --- AWS encoding worker: never touched by the GCE blue-green deploy ----------
|
|
|
|
_AWS = {"vm": "aws-encoding-worker-1", "zone": "us-east-2b", "ip": "3.3.3.3",
|
|
"machine_type": "c7a-8xlarge", "provider": "aws", "instance_id": "i-0abc", "region": "us-east-2"}
|
|
|
|
|
|
def test_aws_worker_is_never_a_green_candidate():
|
|
config = {"secondary_vm": "encoding-worker-b", "secondary_ip": "1.1.1.2"}
|
|
fallbacks = [{"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "2.2.2.2"}, _AWS]
|
|
names = [c["vm"] for c in deploy_promote.select_green_candidates(config, fallbacks)]
|
|
assert "aws-encoding-worker-1" not in names
|
|
assert names == ["encoding-worker-b", "encoding-worker-fallback-c2df"]
|
|
|
|
|
|
def test_secondary_promotion_leaves_aws_override_serving():
|
|
config = {
|
|
"primary_vm": "encoding-worker-a", "primary_ip": "1.1.1.1", "primary_version": "1.0",
|
|
"active_override_vm": "aws-encoding-worker-1", "active_override_ip": "3.3.3.3",
|
|
"active_override_zone": "us-east-2b", "active_override_provider": "aws",
|
|
}
|
|
green = {"vm": "encoding-worker-b", "ip": "1.1.1.2", "zone": "us-central1-c", "kind": "secondary"}
|
|
plan = deploy_promote.promote_plan(green, config, now="2026-10-29T00:00:00Z", version="2.0")
|
|
assert "active_override_vm" not in plan["firestore_updates"]
|
|
assert [d["vm"] for d in plan["drain_and_stop"]] == ["encoding-worker-a"]
|
|
|
|
|
|
def test_fallback_promotion_does_not_drain_aws_override():
|
|
config = {"active_override_vm": "aws-encoding-worker-1", "active_override_zone": "us-east-2b",
|
|
"active_override_provider": "aws"}
|
|
green = {"vm": "encoding-worker-fallback-c2df", "ip": "2.2.2.2", "zone": "us-central1-f", "kind": "fallback"}
|
|
plan = deploy_promote.promote_plan(green, config, now="2026-10-29T00:00:00Z", version="2.0")
|
|
assert plan["drain_and_stop"] == []
|
|
|
|
|
|
def test_fallback_promotion_clears_provider_tag():
|
|
green = {"vm": "encoding-worker-fallback-c2df", "ip": "2.2.2.2", "zone": "us-central1-f", "kind": "fallback"}
|
|
plan = deploy_promote.promote_plan(green, {}, now="2026-10-29T00:00:00Z", version="2.0")
|
|
assert plan["firestore_updates"]["active_override_provider"] is None
|