Files
karaoke-gen/backend/tests/test_deploy_promote.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00

262 lines
13 KiB
Python

"""Unit tests for the encoding-worker deploy decision logic.
The module lives under infrastructure/ (not an importable package), so we load
it by path. Covers the two things the CI workflow can't test itself: which VMs
to try as a fresh green target, and how to promote a validated green to serving.
"""
import importlib.util
from pathlib import Path
import pytest
_MODULE_PATH = (
Path(__file__).resolve().parents[2]
/ "infrastructure" / "encoding-worker" / "deploy_promote.py"
)
_spec = importlib.util.spec_from_file_location("deploy_promote", _MODULE_PATH)
deploy_promote = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(deploy_promote)
select_green_candidates = deploy_promote.select_green_candidates
promote_plan = deploy_promote.promote_plan
FALLBACKS = [
{"vm": "encoding-worker-fallback-a", "zone": "us-central1-a", "ip": "34.55.231.54"},
{"vm": "encoding-worker-fallback-b", "zone": "us-central1-b", "ip": "35.193.127.24"},
{"vm": "encoding-worker-fallback-n2c", "zone": "us-central1-c", "ip": "35.226.119.227"},
{"vm": "encoding-worker-fallback-n2f", "zone": "us-central1-f", "ip": "35.254.152.14"},
]
def _config(**over):
base = {
"primary_vm": "encoding-worker-b", "primary_ip": "34.10.189.118",
"primary_version": "0.192.9",
"secondary_vm": "encoding-worker-a", "secondary_ip": "34.57.78.246",
"active_override_vm": "encoding-worker-fallback-n2f",
"active_override_ip": "35.254.152.14", "active_override_zone": "us-central1-f",
}
base.update(over)
return base
class TestSelectGreenCandidates:
def test_secondary_first_then_n2_before_c4d_excluding_override(self):
# LEGACY secret shape (no machine_type). The shared ranking infers n2 from
# the n2c/n2f names (rank 70) but CANNOT infer the bare c4d fallbacks a/b
# (no family token) → they get UNKNOWN_RANK and sort last. Net order here
# matches the pre-refactor "secondary, then n2, then c4d" result. Once the
# secret carries machine_type (post-rollout), a/b rank as c4d — see
# TestSelectGreenRankingSharedWithRuntime.
cands = select_green_candidates(_config(), FALLBACKS)
vms = [c["vm"] for c in cands]
# c4d secondary leads; current override (n2f) excluded; n2c (n2) before c4d fallbacks.
assert vms[0] == "encoding-worker-a"
assert cands[0]["kind"] == "secondary"
assert "encoding-worker-fallback-n2f" not in vms # the serving override is excluded
assert vms[1] == "encoding-worker-fallback-n2c" # remaining n2 preferred
assert vms[1:] == [
"encoding-worker-fallback-n2c",
"encoding-worker-fallback-a",
"encoding-worker-fallback-b",
]
assert all(c["kind"] == "fallback" for c in cands[1:])
def test_candidate_carries_zone_and_ip(self):
cands = select_green_candidates(_config(), FALLBACKS)
n2c = next(c for c in cands if c["vm"] == "encoding-worker-fallback-n2c")
assert n2c["zone"] == "us-central1-c"
assert n2c["ip"] == "35.226.119.227"
def test_no_secondary_still_lists_fallbacks(self):
cands = select_green_candidates(_config(secondary_vm=None), FALLBACKS)
assert all(c["kind"] == "fallback" for c in cands)
assert cands[0]["vm"] == "encoding-worker-fallback-n2c"
def test_malformed_fallback_entries_are_dropped(self):
fbs = [
{"vm": "encoding-worker-fallback-n2c", "zone": "us-central1-c", "ip": "35.226.119.227"},
{"vm": "encoding-worker-fallback-broken", "ip": "1.2.3.4"}, # no zone
{"zone": "us-central1-a", "ip": "5.6.7.8"}, # no vm
{"vm": "encoding-worker-fallback-noip", "zone": "us-central1-b"}, # no ip
]
cands = select_green_candidates(_config(active_override_vm=None), fbs)
vms = [c["vm"] for c in cands]
assert vms.count("encoding-worker-fallback-n2c") == 1
assert "encoding-worker-fallback-broken" not in vms
assert "encoding-worker-fallback-noip" not in vms
assert None not in vms # the entry with no "vm" is dropped, not emitted as vm=None
def test_no_override_keeps_all_fallbacks(self):
cands = select_green_candidates(_config(active_override_vm=None), FALLBACKS)
fb_vms = [c["vm"] for c in cands if c["kind"] == "fallback"]
# n2 fallbacks first (either order among the two n2s is fine — assert as a set head)
assert set(fb_vms[:2]) == {"encoding-worker-fallback-n2c", "encoding-worker-fallback-n2f"}
assert set(fb_vms) == {f["vm"] for f in FALLBACKS}
class TestPromotePlanFallbackBranch:
def test_fallback_green_becomes_override_and_drains_old(self):
green = {"vm": "encoding-worker-fallback-n2c", "ip": "35.226.119.227",
"zone": "us-central1-c", "kind": "fallback"}
plan = promote_plan(green, _config(), now="NOW", version="0.194.1")
u = plan["firestore_updates"]
assert u["active_override_vm"] == "encoding-worker-fallback-n2c"
assert u["active_override_ip"] == "35.226.119.227"
assert u["active_override_zone"] == "us-central1-c"
assert u["active_override_version"] == "0.194.1"
assert u["deploy_in_progress"] is False
# primary/secondary untouched (c4d resumes when capacity returns)
assert "primary_vm" not in u and "secondary_vm" not in u
# old override (n2f) drained + stopped
assert plan["drain_and_stop"] == [
{"vm": "encoding-worker-fallback-n2f", "ip": "35.254.152.14", "zone": "us-central1-f"}
]
def test_no_drain_when_no_previous_override(self):
green = {"vm": "encoding-worker-fallback-n2c", "ip": "35.226.119.227",
"zone": "us-central1-c", "kind": "fallback"}
plan = promote_plan(green, _config(active_override_vm=None), now="NOW", version="0.194.1")
assert plan["drain_and_stop"] == []
def test_no_self_drain_when_green_is_current_override(self):
# Degenerate (shouldn't happen since candidates exclude the override), but
# must never drain-stop the worker we just promoted.
green = {"vm": "encoding-worker-fallback-n2f", "ip": "35.254.152.14",
"zone": "us-central1-f", "kind": "fallback"}
plan = promote_plan(green, _config(), now="NOW", version="0.194.1")
assert plan["drain_and_stop"] == []
class TestPromotePlanSecondaryBranch:
def test_secondary_green_swaps_and_clears_stale_override(self):
green = {"vm": "encoding-worker-a", "ip": "34.57.78.246",
"zone": "us-central1-c", "kind": "secondary"}
plan = promote_plan(green, _config(), now="NOW", version="0.194.1")
u = plan["firestore_updates"]
# green -> primary, old primary -> secondary
assert u["primary_vm"] == "encoding-worker-a"
assert u["primary_version"] == "0.194.1"
assert u["secondary_vm"] == "encoding-worker-b"
assert u["secondary_version"] == "0.192.9"
# stale override cleared so active_url follows the fresh primary
assert u["active_override_vm"] is None
assert u["active_override_version"] is None
assert u["deploy_in_progress"] is False
# both the old primary and the old override get drained + stopped
drained = {d["vm"] for d in plan["drain_and_stop"]}
assert drained == {"encoding-worker-b", "encoding-worker-fallback-n2f"}
def test_secondary_never_drains_the_promoted_green(self):
# Pathological config where a stale override (or primary) names the green
# VM — the plan must never tell the workflow to stop what it just promoted.
green = {"vm": "encoding-worker-a", "ip": "34.57.78.246",
"zone": "us-central1-c", "kind": "secondary"}
cfg = _config(active_override_vm="encoding-worker-a",
active_override_ip="34.57.78.246",
active_override_zone="us-central1-c")
plan = promote_plan(green, cfg, now="NOW", version="0.194.2")
assert all(d["vm"] != "encoding-worker-a" for d in plan["drain_and_stop"])
# The override is still cleared in config even though we don't stop that VM.
assert plan["firestore_updates"]["active_override_vm"] is None
def test_secondary_swap_without_override(self):
green = {"vm": "encoding-worker-a", "ip": "34.57.78.246",
"zone": "us-central1-c", "kind": "secondary"}
plan = promote_plan(green, _config(active_override_vm=None), now="NOW", version="0.194.1")
u = plan["firestore_updates"]
assert "active_override_vm" not in u # nothing to clear
assert [d["vm"] for d in plan["drain_and_stop"]] == ["encoding-worker-b"]
class TestSelectGreenRankingSharedWithRuntime:
"""Green selection now ranks via the shared preference (machine_type + cooldown)."""
# 6-type pool with explicit machine_type (mirrors the post-rollout secret).
POOL = [
{"vm": "encoding-worker-fallback-c4a", "zone": "us-central1-a", "ip": "10.0.0.1",
"machine_type": "c4-highcpu-16"},
{"vm": "encoding-worker-fallback-n4db", "zone": "us-central1-b", "ip": "10.0.0.2",
"machine_type": "n4d-highcpu-16"},
{"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "10.0.0.3",
"machine_type": "c2d-highcpu-16"},
{"vm": "encoding-worker-fallback-n2da", "zone": "us-central1-a", "ip": "10.0.0.4",
"machine_type": "n2d-highcpu-16"},
{"vm": "encoding-worker-fallback-n2c", "zone": "us-central1-c", "ip": "10.0.0.5",
"machine_type": "n2-highcpu-16"},
]
def test_secondary_c4d_leads_then_fastest_fallbacks(self):
cfg = _config(active_override_vm=None)
vms = [c["vm"] for c in select_green_candidates(cfg, self.POOL)]
# c4d secondary (rank 10) first, then c4 > n4d > c2d > n2d > n2.
assert vms == [
"encoding-worker-a", # secondary, c4d
"encoding-worker-fallback-c4a", # c4
"encoding-worker-fallback-n4db", # n4d
"encoding-worker-fallback-c2df", # c2d
"encoding-worker-fallback-n2da", # n2d
"encoding-worker-fallback-n2c", # n2
]
def test_c4d_stockout_demotes_secondary_below_deep_pool(self):
# c4d@us-central1-c (the secondary's zone) recently stocked out.
from datetime import datetime, timezone
cfg = _config(
active_override_vm=None,
capacity_state={"c4d-highcpu-16@us-central1-c": datetime.now(timezone.utc).isoformat()},
)
cands = select_green_candidates(cfg, self.POOL)
vms = [c["vm"] for c in cands]
# The c4d secondary is demoted to the back; a fast fallback (c4) now leads.
assert vms[0] == "encoding-worker-fallback-c4a"
assert vms[-1] == "encoding-worker-a" # demoted secondary re-probed last
# kind survives the reordering so promote_plan still treats it correctly.
assert cands[-1]["kind"] == "secondary"
def test_machine_type_carried_through(self):
cands = select_green_candidates(_config(active_override_vm=None), self.POOL)
c4 = next(c for c in cands if c["vm"] == "encoding-worker-fallback-c4a")
assert c4["machine_type"] == "c4-highcpu-16"
# --- AWS encoding worker: never touched by the GCE blue-green deploy ----------
_AWS = {"vm": "aws-encoding-worker-1", "zone": "us-east-2b", "ip": "3.3.3.3",
"machine_type": "c7a-8xlarge", "provider": "aws", "instance_id": "i-0abc", "region": "us-east-2"}
def test_aws_worker_is_never_a_green_candidate():
config = {"secondary_vm": "encoding-worker-b", "secondary_ip": "1.1.1.2"}
fallbacks = [{"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "2.2.2.2"}, _AWS]
names = [c["vm"] for c in deploy_promote.select_green_candidates(config, fallbacks)]
assert "aws-encoding-worker-1" not in names
assert names == ["encoding-worker-b", "encoding-worker-fallback-c2df"]
def test_secondary_promotion_leaves_aws_override_serving():
config = {
"primary_vm": "encoding-worker-a", "primary_ip": "1.1.1.1", "primary_version": "1.0",
"active_override_vm": "aws-encoding-worker-1", "active_override_ip": "3.3.3.3",
"active_override_zone": "us-east-2b", "active_override_provider": "aws",
}
green = {"vm": "encoding-worker-b", "ip": "1.1.1.2", "zone": "us-central1-c", "kind": "secondary"}
plan = deploy_promote.promote_plan(green, config, now="2026-10-29T00:00:00Z", version="2.0")
assert "active_override_vm" not in plan["firestore_updates"]
assert [d["vm"] for d in plan["drain_and_stop"]] == ["encoding-worker-a"]
def test_fallback_promotion_does_not_drain_aws_override():
config = {"active_override_vm": "aws-encoding-worker-1", "active_override_zone": "us-east-2b",
"active_override_provider": "aws"}
green = {"vm": "encoding-worker-fallback-c2df", "ip": "2.2.2.2", "zone": "us-central1-f", "kind": "fallback"}
plan = deploy_promote.promote_plan(green, config, now="2026-10-29T00:00:00Z", version="2.0")
assert plan["drain_and_stop"] == []
def test_fallback_promotion_clears_provider_tag():
green = {"vm": "encoding-worker-fallback-c2df", "ip": "2.2.2.2", "zone": "us-central1-f", "kind": "fallback"}
plan = deploy_promote.promote_plan(green, {}, now="2026-10-29T00:00:00Z", version="2.0")
assert plan["firestore_updates"]["active_override_provider"] is None