mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
147 lines
5.6 KiB
Python
147 lines
5.6 KiB
Python
"""EncodingWorkerManager with the AWS encoding worker in the pool.
|
|
|
|
Covers routing AWS VMs to the EC2 adapter (GCE untouched), the shared fallback
|
|
parser passing provider fields through, ranking with Firestore
|
|
speed_rank_overrides, the ranked warmup, and the override's provider tag.
|
|
"""
|
|
import json
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
|
|
from backend.services.aws_encoding_compute import AwsStartOperation
|
|
from backend.services.encoding_errors import EncodingWorkerCapacityError
|
|
from backend.services.encoding_worker_manager import EncodingWorkerManager, parse_fallback_vms
|
|
|
|
AWS_ENTRY = {
|
|
"vm": "aws-encoding-worker-1", "zone": "us-east-2b", "ip": "3.3.3.3",
|
|
"machine_type": "c7a-8xlarge", "provider": "aws", "instance_id": "i-0abc", "region": "us-east-2",
|
|
}
|
|
C2D_ENTRY = {"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "2.2.2.2",
|
|
"machine_type": "c2d-highcpu-16"}
|
|
|
|
|
|
def _db(**config):
|
|
data = {
|
|
"primary_vm": "encoding-worker-a", "primary_ip": "1.1.1.1", "primary_version": "1.0",
|
|
"secondary_vm": "encoding-worker-b", "secondary_ip": "1.1.1.2",
|
|
"last_activity_at": None, "capacity_state": {}, **config,
|
|
}
|
|
db = MagicMock()
|
|
snap = MagicMock(exists=True)
|
|
snap.to_dict.return_value = data
|
|
doc = MagicMock()
|
|
doc.get.return_value = snap
|
|
db.collection.return_value.document.return_value = doc
|
|
return db, doc
|
|
|
|
|
|
def _manager(db, aws=None, entries=(C2D_ENTRY, AWS_ENTRY)):
|
|
gce = MagicMock(name="gce")
|
|
aws = aws or MagicMock(name="aws")
|
|
aws.handles.side_effect = lambda vm: vm == AWS_ENTRY["vm"]
|
|
m = EncodingWorkerManager(db=db, compute_client=gce, fallback_vms_raw=json.dumps(list(entries)), aws_compute=aws)
|
|
return m, gce, aws
|
|
|
|
|
|
def test_parse_fallback_vms_passes_aws_fields_through():
|
|
parsed = parse_fallback_vms(json.dumps([C2D_ENTRY, AWS_ENTRY]))
|
|
assert parsed[0] == {**C2D_ENTRY, "kind": "fallback"}
|
|
assert parsed[1]["provider"] == "aws"
|
|
assert parsed[1]["instance_id"] == "i-0abc" and parsed[1]["region"] == "us-east-2"
|
|
|
|
|
|
@pytest.mark.parametrize("raw", [None, "", "not json", "{}", "null"])
|
|
def test_parse_fallback_vms_never_raises(raw):
|
|
assert parse_fallback_vms(raw) == []
|
|
|
|
|
|
def test_aws_vm_ops_go_to_adapter_and_gce_untouched():
|
|
db, _ = _db()
|
|
m, gce, aws = _manager(db)
|
|
aws.get.return_value = MagicMock(status="RUNNING")
|
|
gce.get.return_value = MagicMock(status="TERMINATED")
|
|
|
|
assert m.get_vm_status("aws-encoding-worker-1", zone="us-east-2b") == "RUNNING"
|
|
assert m.get_vm_status("encoding-worker-a") == "TERMINATED"
|
|
aws.get.assert_called_once()
|
|
gce.get.assert_called_once()
|
|
|
|
|
|
def test_aws_capacity_error_raises_typed_capacity_error():
|
|
db, _ = _db()
|
|
m, _, aws = _manager(db)
|
|
aws.get.return_value = MagicMock(status="TERMINATED")
|
|
|
|
def _no_capacity():
|
|
from botocore.exceptions import ClientError
|
|
raise ClientError({"Error": {"Code": "InsufficientInstanceCapacity", "Message": "none"}}, "StartInstances")
|
|
|
|
aws.start.return_value = AwsStartOperation(_no_capacity)
|
|
with pytest.raises(EncodingWorkerCapacityError):
|
|
m.start_vm("aws-encoding-worker-1", zone="us-east-2b")
|
|
|
|
|
|
def test_ranked_candidates_default_keeps_gce_primary_first_and_aws_last():
|
|
db, _ = _db()
|
|
m, _, _ = _manager(db)
|
|
ranked = m.ranked_candidates()
|
|
assert [c.vm_name for c in ranked] == ["encoding-worker-a", "encoding-worker-fallback-c2df", "aws-encoding-worker-1"]
|
|
assert ranked[0].is_primary and ranked[-1].provider == "aws"
|
|
|
|
|
|
def test_ranked_candidates_honour_speed_rank_overrides():
|
|
db, _ = _db(speed_rank_overrides={"c7a": 5})
|
|
m, _, _ = _manager(db)
|
|
assert m.ranked_candidates()[0].vm_name == "aws-encoding-worker-1"
|
|
|
|
|
|
def test_ensure_preferred_running_is_primary_path_when_primary_ranks_first():
|
|
db, _ = _db()
|
|
m, _, _ = _manager(db)
|
|
m.ensure_primary_running = MagicMock(return_value={"started": False})
|
|
m.ensure_any_running = MagicMock()
|
|
m.ensure_preferred_running()
|
|
m.ensure_primary_running.assert_called_once()
|
|
m.ensure_any_running.assert_not_called()
|
|
|
|
|
|
def test_ensure_preferred_running_starts_aws_when_preferred():
|
|
db, _ = _db(speed_rank_overrides={"c7a": 5})
|
|
m, _, _ = _manager(db)
|
|
m.ensure_primary_running = MagicMock()
|
|
m.ensure_any_running = MagicMock(return_value={"started": True, "vm_name": "aws-encoding-worker-1"})
|
|
m.ensure_preferred_running()
|
|
m.ensure_primary_running.assert_not_called()
|
|
(candidates,), _ = m.ensure_any_running.call_args
|
|
assert candidates[0].vm_name == "aws-encoding-worker-1"
|
|
|
|
|
|
def test_active_override_records_provider():
|
|
db, doc = _db()
|
|
m, _, _ = _manager(db)
|
|
aws_candidate = next(c for c in m.ranked_candidates() if c.provider == "aws")
|
|
m._set_active_override(aws_candidate)
|
|
update = doc.update.call_args[0][0]
|
|
assert update["active_override_vm"] == "aws-encoding-worker-1"
|
|
assert update["active_override_provider"] == "aws"
|
|
|
|
db2, doc2 = _db(active_override_vm="aws-encoding-worker-1", active_override_provider="aws")
|
|
m2, _, _ = _manager(db2)
|
|
m2._clear_active_override()
|
|
assert doc2.update.call_args[0][0]["active_override_provider"] is None
|
|
|
|
|
|
def test_aws_control_plane_fault_fails_over_to_gce():
|
|
db, _ = _db(speed_rank_overrides={"c7a": 5})
|
|
m, gce, aws = _manager(db, entries=(AWS_ENTRY,))
|
|
aws.get.side_effect = RuntimeError("could not load aws-encoding-controller-credentials")
|
|
gce.get.return_value = MagicMock(status="RUNNING")
|
|
m.update_activity = MagicMock()
|
|
m._clear_active_override = MagicMock()
|
|
m._clear_stockout = MagicMock()
|
|
|
|
result = m.ensure_any_running(m.ranked_candidates())
|
|
|
|
assert result["vm_name"] == "encoding-worker-a"
|