Files
karaoke-gen/backend/tests/test_encoding_worker_manager_aws.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 de5dfcd50d feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:42:35 -04:00

147 lines
5.6 KiB
Python

"""EncodingWorkerManager with the AWS encoding worker in the pool.
Covers routing AWS VMs to the EC2 adapter (GCE untouched), the shared fallback
parser passing provider fields through, ranking with Firestore
speed_rank_overrides, the ranked warmup, and the override's provider tag.
"""
import json
from unittest.mock import MagicMock
import pytest
from backend.services.aws_encoding_compute import AwsStartOperation
from backend.services.encoding_errors import EncodingWorkerCapacityError
from backend.services.encoding_worker_manager import EncodingWorkerManager, parse_fallback_vms
AWS_ENTRY = {
"vm": "aws-encoding-worker-1", "zone": "us-east-2b", "ip": "3.3.3.3",
"machine_type": "c7a-8xlarge", "provider": "aws", "instance_id": "i-0abc", "region": "us-east-2",
}
C2D_ENTRY = {"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "2.2.2.2",
"machine_type": "c2d-highcpu-16"}
def _db(**config):
data = {
"primary_vm": "encoding-worker-a", "primary_ip": "1.1.1.1", "primary_version": "1.0",
"secondary_vm": "encoding-worker-b", "secondary_ip": "1.1.1.2",
"last_activity_at": None, "capacity_state": {}, **config,
}
db = MagicMock()
snap = MagicMock(exists=True)
snap.to_dict.return_value = data
doc = MagicMock()
doc.get.return_value = snap
db.collection.return_value.document.return_value = doc
return db, doc
def _manager(db, aws=None, entries=(C2D_ENTRY, AWS_ENTRY)):
gce = MagicMock(name="gce")
aws = aws or MagicMock(name="aws")
aws.handles.side_effect = lambda vm: vm == AWS_ENTRY["vm"]
m = EncodingWorkerManager(db=db, compute_client=gce, fallback_vms_raw=json.dumps(list(entries)), aws_compute=aws)
return m, gce, aws
def test_parse_fallback_vms_passes_aws_fields_through():
parsed = parse_fallback_vms(json.dumps([C2D_ENTRY, AWS_ENTRY]))
assert parsed[0] == {**C2D_ENTRY, "kind": "fallback"}
assert parsed[1]["provider"] == "aws"
assert parsed[1]["instance_id"] == "i-0abc" and parsed[1]["region"] == "us-east-2"
@pytest.mark.parametrize("raw", [None, "", "not json", "{}", "null"])
def test_parse_fallback_vms_never_raises(raw):
assert parse_fallback_vms(raw) == []
def test_aws_vm_ops_go_to_adapter_and_gce_untouched():
db, _ = _db()
m, gce, aws = _manager(db)
aws.get.return_value = MagicMock(status="RUNNING")
gce.get.return_value = MagicMock(status="TERMINATED")
assert m.get_vm_status("aws-encoding-worker-1", zone="us-east-2b") == "RUNNING"
assert m.get_vm_status("encoding-worker-a") == "TERMINATED"
aws.get.assert_called_once()
gce.get.assert_called_once()
def test_aws_capacity_error_raises_typed_capacity_error():
db, _ = _db()
m, _, aws = _manager(db)
aws.get.return_value = MagicMock(status="TERMINATED")
def _no_capacity():
from botocore.exceptions import ClientError
raise ClientError({"Error": {"Code": "InsufficientInstanceCapacity", "Message": "none"}}, "StartInstances")
aws.start.return_value = AwsStartOperation(_no_capacity)
with pytest.raises(EncodingWorkerCapacityError):
m.start_vm("aws-encoding-worker-1", zone="us-east-2b")
def test_ranked_candidates_default_keeps_gce_primary_first_and_aws_last():
db, _ = _db()
m, _, _ = _manager(db)
ranked = m.ranked_candidates()
assert [c.vm_name for c in ranked] == ["encoding-worker-a", "encoding-worker-fallback-c2df", "aws-encoding-worker-1"]
assert ranked[0].is_primary and ranked[-1].provider == "aws"
def test_ranked_candidates_honour_speed_rank_overrides():
db, _ = _db(speed_rank_overrides={"c7a": 5})
m, _, _ = _manager(db)
assert m.ranked_candidates()[0].vm_name == "aws-encoding-worker-1"
def test_ensure_preferred_running_is_primary_path_when_primary_ranks_first():
db, _ = _db()
m, _, _ = _manager(db)
m.ensure_primary_running = MagicMock(return_value={"started": False})
m.ensure_any_running = MagicMock()
m.ensure_preferred_running()
m.ensure_primary_running.assert_called_once()
m.ensure_any_running.assert_not_called()
def test_ensure_preferred_running_starts_aws_when_preferred():
db, _ = _db(speed_rank_overrides={"c7a": 5})
m, _, _ = _manager(db)
m.ensure_primary_running = MagicMock()
m.ensure_any_running = MagicMock(return_value={"started": True, "vm_name": "aws-encoding-worker-1"})
m.ensure_preferred_running()
m.ensure_primary_running.assert_not_called()
(candidates,), _ = m.ensure_any_running.call_args
assert candidates[0].vm_name == "aws-encoding-worker-1"
def test_active_override_records_provider():
db, doc = _db()
m, _, _ = _manager(db)
aws_candidate = next(c for c in m.ranked_candidates() if c.provider == "aws")
m._set_active_override(aws_candidate)
update = doc.update.call_args[0][0]
assert update["active_override_vm"] == "aws-encoding-worker-1"
assert update["active_override_provider"] == "aws"
db2, doc2 = _db(active_override_vm="aws-encoding-worker-1", active_override_provider="aws")
m2, _, _ = _manager(db2)
m2._clear_active_override()
assert doc2.update.call_args[0][0]["active_override_provider"] is None
def test_aws_control_plane_fault_fails_over_to_gce():
db, _ = _db(speed_rank_overrides={"c7a": 5})
m, gce, aws = _manager(db, entries=(AWS_ENTRY,))
aws.get.side_effect = RuntimeError("could not load aws-encoding-controller-credentials")
gce.get.return_value = MagicMock(status="RUNNING")
m.update_activity = MagicMock()
m._clear_active_override = MagicMock()
m._clear_stockout = MagicMock()
result = m.ensure_any_running(m.ranked_candidates())
assert result["vm_name"] == "encoding-worker-a"