mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
116 lines
3.9 KiB
Python
116 lines
3.9 KiB
Python
"""Unit tests for the EC2 adapter behind the AWS encoding worker.
|
|
|
|
The adapter must look exactly like the compute_v1.InstancesClient slice the
|
|
manager uses, so the manager's GCE-shaped status checks, capacity cooldowns
|
|
and NotFound skips apply to the AWS worker unchanged.
|
|
"""
|
|
import json
|
|
|
|
import boto3
|
|
import pytest
|
|
from botocore.stub import Stubber
|
|
from google.api_core import exceptions as gcp_exceptions
|
|
|
|
from backend.services.aws_encoding_compute import (
|
|
AwsComputeAdapter,
|
|
AwsVm,
|
|
_load_credentials,
|
|
aws_vms_from_pool,
|
|
)
|
|
from backend.services.encoding_errors import CAPACITY_ERROR_CODES
|
|
|
|
VM = "aws-encoding-worker-1"
|
|
IID = "i-0123456789abcdef0"
|
|
|
|
|
|
@pytest.fixture
|
|
def ec2():
|
|
client = boto3.client("ec2", region_name="us-east-2", aws_access_key_id="x", aws_secret_access_key="y")
|
|
with Stubber(client) as stub:
|
|
yield client, stub
|
|
|
|
|
|
def _adapter(client):
|
|
return AwsComputeAdapter({VM: AwsVm(IID, "us-east-2")}, client_factory=lambda region: client)
|
|
|
|
|
|
def _describe(state):
|
|
return {"Reservations": [{"Instances": [{"InstanceId": IID, "State": {"Name": state, "Code": 0}}]}]}
|
|
|
|
|
|
@pytest.mark.parametrize("ec2_state,gce_status", [
|
|
("running", "RUNNING"),
|
|
("pending", "STAGING"),
|
|
("stopping", "STOPPING"),
|
|
("stopped", "TERMINATED"),
|
|
("shutting-down", "STOPPING"),
|
|
])
|
|
def test_get_maps_ec2_state_to_gce_status(ec2, ec2_state, gce_status):
|
|
client, stub = ec2
|
|
stub.add_response("describe_instances", _describe(ec2_state), {"InstanceIds": [IID]})
|
|
assert _adapter(client).get(project="p", zone="us-east-2b", instance=VM).status == gce_status
|
|
|
|
|
|
def test_get_missing_instance_raises_not_found(ec2):
|
|
client, stub = ec2
|
|
stub.add_client_error("describe_instances", service_error_code="InvalidInstanceID.NotFound")
|
|
with pytest.raises(gcp_exceptions.NotFound):
|
|
_adapter(client).get(instance=VM)
|
|
|
|
|
|
def test_unregistered_vm_raises_not_found(ec2):
|
|
client, _ = ec2
|
|
with pytest.raises(gcp_exceptions.NotFound):
|
|
_adapter(client).get(instance="encoding-worker-a")
|
|
|
|
|
|
def test_start_success_leaves_no_error_code(ec2):
|
|
client, stub = ec2
|
|
stub.add_response("start_instances", {"StartingInstances": []}, {"InstanceIds": [IID]})
|
|
op = _adapter(client).start(instance=VM)
|
|
op.result(timeout=30)
|
|
assert op.error_code == ""
|
|
|
|
|
|
def test_start_capacity_error_is_a_capacity_code(ec2):
|
|
client, stub = ec2
|
|
stub.add_client_error(
|
|
"start_instances", service_error_code="InsufficientInstanceCapacity",
|
|
service_message="no c7a.8xlarge capacity in us-east-2b",
|
|
)
|
|
op = _adapter(client).start(instance=VM)
|
|
op.result()
|
|
assert op.error_code == "InsufficientInstanceCapacity"
|
|
assert op.error_code in CAPACITY_ERROR_CODES
|
|
assert "capacity" in op.error_message
|
|
|
|
|
|
def test_stop_calls_stop_instances(ec2):
|
|
client, stub = ec2
|
|
stub.add_response("stop_instances", {"StoppingInstances": []}, {"InstanceIds": [IID]})
|
|
_adapter(client).stop(instance=VM)
|
|
stub.assert_no_pending_responses()
|
|
|
|
|
|
def test_handles_only_registered_vms():
|
|
adapter = AwsComputeAdapter({VM: AwsVm(IID, "us-east-2")}, client_factory=lambda r: None)
|
|
assert adapter.handles(VM)
|
|
assert not adapter.handles("encoding-worker-a")
|
|
|
|
|
|
def test_aws_vms_from_pool_keeps_only_complete_aws_entries():
|
|
pool = [
|
|
{"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "1.2.3.4"},
|
|
{"vm": VM, "zone": "us-east-2b", "ip": "5.6.7.8", "provider": "aws", "instance_id": IID, "region": "us-east-2"},
|
|
{"vm": "aws-incomplete", "zone": "us-east-2b", "ip": "9.9.9.9", "provider": "aws"},
|
|
]
|
|
assert aws_vms_from_pool(pool) == {VM: AwsVm(IID, "us-east-2")}
|
|
|
|
|
|
def test_credentials_env_override(monkeypatch):
|
|
monkeypatch.setenv(
|
|
"AWS_ENCODING_CONTROLLER_CREDENTIALS",
|
|
json.dumps({"access_key_id": "AKIA", "secret_access_key": "s3cr3t", "extra": 1}),
|
|
)
|
|
assert _load_credentials("nomadkaraoke") == {"access_key_id": "AKIA", "secret_access_key": "s3cr3t"}
|