Files
karaoke-gen/backend/tests/test_encoding_worker_preference.py
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00

226 lines
8.8 KiB
Python

"""Unit tests for the shared encoding-worker candidate preference logic.
This pure module drives BOTH runtime selection and deploy green selection, so its
ordering guarantees (fastest-first, cooldown demotion, machine_type inference) are
the contract both call-sites rely on.
"""
from datetime import datetime, timedelta, timezone
from backend.services.encoding_worker_preference import (
COOLDOWN_SECONDS,
SPEED_RANK,
UNKNOWN_RANK,
cooldown_key,
infer_machine_type,
ordered_candidates,
)
NOW = datetime(2026, 8, 15, 19, 0, 0, tzinfo=timezone.utc)
def _c(vm, zone, machine_type=None, **extra):
d = {"vm": vm, "zone": zone, "ip": f"10.0.0.{hash(vm) % 250 + 1}"}
if machine_type is not None:
d["machine_type"] = machine_type
d.update(extra)
return d
# ---------------------------------------------------------------------------
# machine_type inference
# ---------------------------------------------------------------------------
def test_explicit_machine_type_wins():
assert infer_machine_type(_c("whatever", "z", "c2d-highcpu-16")) == "c2d-highcpu-16"
def test_infer_from_vm_name_specific_before_generic():
# "n2d"/"c4d" must not be shadowed by "n2"/"c4".
assert infer_machine_type(_c("encoding-worker-fallback-n2da", "z")) == "n2d-highcpu-16"
assert infer_machine_type(_c("encoding-worker-fallback-n2f", "z")) == "n2-highcpu-16"
assert infer_machine_type(_c("encoding-worker-fallback-c4x", "z")) == "c4-highcpu-16"
def test_infer_unknown_returns_none():
# Bare a/b c4d fallbacks (no family token) can't be inferred from the name.
assert infer_machine_type(_c("encoding-worker-fallback-a", "z")) is None
def test_runtime_dict_shape_uses_vm_name_key():
assert infer_machine_type({"vm_name": "x-n2d-y", "zone": "z"}) == "n2d-highcpu-16"
# ---------------------------------------------------------------------------
# fastest-first ordering
# ---------------------------------------------------------------------------
def test_orders_fastest_first():
pool = [
_c("n2", "us-central1-c", "n2-highcpu-16"),
_c("c4d", "us-central1-c", "c4d-highcpu-16"),
_c("c2d", "us-central1-f", "c2d-highcpu-16"),
_c("c4", "us-central1-a", "c4-highcpu-16"),
]
got = [c["vm"] for c in ordered_candidates(pool, now=NOW)]
assert got == ["c4d", "c4", "c2d", "n2"]
def test_unknown_type_sorts_last_but_kept():
pool = [
_c("mystery", "z"), # no machine_type, unknown name
_c("c4d", "z", "c4d-highcpu-16"),
]
got = [c["vm"] for c in ordered_candidates(pool, now=NOW)]
assert got == ["c4d", "mystery"]
assert SPEED_RANK["c4d"] < UNKNOWN_RANK
def test_stable_within_equal_rank_preserves_zone_spread():
# Two n2 in different zones keep input order (caller's secondary preference).
pool = [
_c("n2f", "us-central1-f", "n2-highcpu-16"),
_c("n2c", "us-central1-c", "n2-highcpu-16"),
]
got = [c["vm"] for c in ordered_candidates(pool, now=NOW)]
assert got == ["n2f", "n2c"]
# ---------------------------------------------------------------------------
# cooldown / availability awareness
# ---------------------------------------------------------------------------
def test_recent_stockout_demotes_type_to_back():
pool = [
_c("c4d", "us-central1-c", "c4d-highcpu-16"),
_c("c4", "us-central1-a", "c4-highcpu-16"),
_c("n2", "us-central1-c", "n2-highcpu-16"),
]
# c4d stocked out 1 minute ago → demoted behind the still-healthy types.
capacity_state = {"c4d-highcpu-16@us-central1-c": (NOW - timedelta(minutes=1)).isoformat()}
got = [c["vm"] for c in ordered_candidates(pool, capacity_state, now=NOW)]
assert got == ["c4", "n2", "c4d"]
def test_cooldown_expires_and_snaps_back_to_fastest():
pool = [
_c("c4d", "us-central1-c", "c4d-highcpu-16"),
_c("c4", "us-central1-a", "c4-highcpu-16"),
]
# Stockout older than COOLDOWN → c4d is hot again and returns to the top.
old = (NOW - timedelta(seconds=COOLDOWN_SECONDS + 60)).isoformat()
capacity_state = {"c4d-highcpu-16@us-central1-c": old}
got = [c["vm"] for c in ordered_candidates(pool, capacity_state, now=NOW)]
assert got == ["c4d", "c4"]
def test_two_types_stocked_out_still_serves_from_deep_pool():
# Both newest-gen types (c4d, c4) exhausted; deep pools keep serving in order.
pool = [
_c("c4d", "us-central1-c", "c4d-highcpu-16"),
_c("c4", "us-central1-a", "c4-highcpu-16"),
_c("c2d", "us-central1-f", "c2d-highcpu-16"),
_c("n2d", "us-central1-a", "n2d-highcpu-16"),
]
recent = (NOW - timedelta(minutes=2)).isoformat()
capacity_state = {
"c4d-highcpu-16@us-central1-c": recent,
"c4-highcpu-16@us-central1-a": recent,
}
got = [c["vm"] for c in ordered_candidates(pool, capacity_state, now=NOW)]
# Hot deep pools first (fastest-first among them), then the two cold types.
assert got == ["c2d", "n2d", "c4d", "c4"]
def test_type_only_capacity_key_demotes_all_zones():
pool = [
_c("c4d-c", "us-central1-c", "c4d-highcpu-16"),
_c("n2", "us-central1-c", "n2-highcpu-16"),
]
capacity_state = {"c4d-highcpu-16": (NOW - timedelta(minutes=1)).isoformat()}
got = [c["vm"] for c in ordered_candidates(pool, capacity_state, now=NOW)]
assert got == ["n2", "c4d-c"]
def test_empty_capacity_state_is_pure_speed_order():
pool = [_c("n2", "z", "n2-highcpu-16"), _c("c4d", "z", "c4d-highcpu-16")]
assert [c["vm"] for c in ordered_candidates(pool, {}, now=NOW)] == ["c4d", "n2"]
assert [c["vm"] for c in ordered_candidates(pool, None, now=NOW)] == ["c4d", "n2"]
def test_naive_timestamp_tolerated():
pool = [_c("c4d", "us-central1-c", "c4d-highcpu-16"), _c("n2", "z", "n2-highcpu-16")]
naive = (NOW - timedelta(minutes=1)).replace(tzinfo=None).isoformat()
capacity_state = {"c4d-highcpu-16@us-central1-c": naive}
got = [c["vm"] for c in ordered_candidates(pool, capacity_state, now=NOW)]
assert got == ["n2", "c4d"]
def test_malformed_timestamp_ignored():
pool = [_c("c4d", "us-central1-c", "c4d-highcpu-16"), _c("n2", "z", "n2-highcpu-16")]
capacity_state = {"c4d-highcpu-16@us-central1-c": "not-a-date"}
got = [c["vm"] for c in ordered_candidates(pool, capacity_state, now=NOW)]
assert got == ["c4d", "n2"] # unparseable → not in cooldown
def test_empty_pool():
assert ordered_candidates([], now=NOW) == []
def test_cooldown_key_shape():
assert cooldown_key(_c("x", "us-central1-c", "c4d-highcpu-16")) == "c4d-highcpu-16@us-central1-c"
assert cooldown_key(_c("x", None, "n2-highcpu-16")) == "n2-highcpu-16"
assert cooldown_key(_c("no-token", "z")) is None
# --- AWS encoding worker (c7a) + runtime rank overrides -----------------------
AWS = dict(provider="aws", instance_id="i-0abc", region="us-east-2")
def test_aws_worker_is_last_resort_by_default():
pool = [
_c("aws-encoding-worker-1", "us-east-2b", "c7a-8xlarge", **AWS),
_c("encoding-worker-fallback-c2df", "us-central1-f", "c2d-highcpu-16"),
_c("encoding-worker-a", "us-central1-c", "c4d-highcpu-16"),
]
ranked = ordered_candidates(pool, now=NOW)
assert [c["vm"] for c in ranked] == [
"encoding-worker-a", "encoding-worker-fallback-c2df", "aws-encoding-worker-1",
]
def test_rank_override_prefers_aws_worker():
pool = [
_c("encoding-worker-a", "us-central1-c", "c4d-highcpu-16"),
_c("encoding-worker-fallback-c2df", "us-central1-f", "c2d-highcpu-16"),
_c("aws-encoding-worker-1", "us-east-2b", "c7a-8xlarge", **AWS),
]
ranked = ordered_candidates(pool, now=NOW, rank_overrides={"c7a": 5})
assert ranked[0]["vm"] == "aws-encoding-worker-1"
# The rest keep the built-in order behind it.
assert [c["vm"] for c in ranked[1:]] == ["encoding-worker-a", "encoding-worker-fallback-c2df"]
def test_stocked_out_aws_worker_still_demoted_despite_override():
aws = _c("aws-encoding-worker-1", "us-east-2b", "c7a-8xlarge", **AWS)
pool = [aws, _c("encoding-worker-a", "us-central1-c", "c4d-highcpu-16")]
state = {cooldown_key(aws): (NOW - timedelta(seconds=60)).isoformat()}
ranked = ordered_candidates(pool, capacity_state=state, now=NOW, rank_overrides={"c7a": 5})
assert ranked[-1]["vm"] == "aws-encoding-worker-1"
def test_malformed_rank_override_falls_back_to_builtin_rank():
pool = [
_c("aws-encoding-worker-1", "us-east-2b", "c7a-8xlarge", **AWS),
_c("encoding-worker-a", "us-central1-c", "c4d-highcpu-16"),
]
ranked = ordered_candidates(pool, now=NOW, rank_overrides={"c7a": "first please"})
assert ranked[0]["vm"] == "encoding-worker-a"
def test_aws_cooldown_key_is_firestore_safe():
key = cooldown_key(_c("aws-encoding-worker-1", "us-east-2b", "c7a-8xlarge", **AWS))
assert key == "c7a-8xlarge@us-east-2b"
assert "." not in key
assert SPEED_RANK["c7a"] > SPEED_RANK["c2d"]