mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
138 lines
5.8 KiB
Python
138 lines
5.8 KiB
Python
"""Nomad Karaoke AWS account: the EC2 encoding worker.
|
|
|
|
Paid for by the AWS Activate credits ($5k, acct 790139457751, exp 2028-10-31).
|
|
The worker runs the same ``gce_encoding`` HTTP service as the GCE encoding VMs
|
|
and is one more candidate in the backend's encoding-worker pool (provider
|
|
"aws"). Plan: karaoke-gen docs/archive/2026-10-29-aws-encoding-worker-plan.md.
|
|
|
|
- Persistent Spot instance, ``stop`` on interruption and on OS shutdown (the idle
|
|
watchdog), so the EBS root (venv, caches) survives like a GCE boot disk.
|
|
- Elastic IP + port 8080 (API-key auth, same as the GCE firewall).
|
|
- GCP access via Workload Identity Federation as encoding-worker@ (the
|
|
``nomad-encoding-worker`` role is the federated identity; see
|
|
infrastructure/modules/aws_encoding_wif.py in the GCP stack).
|
|
- ``nomad-encoding-controller`` IAM user: the backend's start/stop/describe
|
|
rights, scoped by tag. Its access key is created by hand and stored in GCP
|
|
Secret Manager ``aws-encoding-controller-credentials`` (see README).
|
|
"""
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
import pulumi
|
|
import pulumi_aws as aws
|
|
|
|
PROJECT_TAG = {"Project": "nomad-encoding"}
|
|
VM_NAME = "aws-encoding-worker-1"
|
|
INSTANCE_TYPE = "c7a.8xlarge" # 32 vCPU / 64 GB — see the plan doc for sizing
|
|
AZ = "us-east-2b"
|
|
WIF_PROVIDER = (
|
|
"projects/718638054799/locations/global/workloadIdentityPools/"
|
|
"aws-encoding/providers/aws-790139457751"
|
|
)
|
|
|
|
# --- Worker identity (federated into GCP; also SSM for operator shell) ----------
|
|
role = aws.iam.Role(
|
|
"worker-role",
|
|
name="nomad-encoding-worker",
|
|
assume_role_policy=json.dumps({
|
|
"Version": "2012-10-17",
|
|
"Statement": [{"Effect": "Allow", "Principal": {"Service": "ec2.amazonaws.com"}, "Action": "sts:AssumeRole"}],
|
|
}),
|
|
tags=PROJECT_TAG,
|
|
)
|
|
aws.iam.RolePolicyAttachment(
|
|
"worker-ssm",
|
|
role=role.name,
|
|
policy_arn="arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
|
|
)
|
|
profile = aws.iam.InstanceProfile("worker-profile", name="nomad-encoding-worker", role=role.name)
|
|
|
|
# --- Backend controller (start/stop/describe the tagged worker) -----------------
|
|
controller = aws.iam.User("controller", name="nomad-encoding-controller", tags=PROJECT_TAG)
|
|
aws.iam.UserPolicy(
|
|
"controller-policy",
|
|
user=controller.name,
|
|
policy=json.dumps({
|
|
"Version": "2012-10-17",
|
|
"Statement": [
|
|
{
|
|
"Effect": "Allow",
|
|
"Action": ["ec2:StartInstances", "ec2:StopInstances"],
|
|
"Resource": "arn:aws:ec2:*:790139457751:instance/*",
|
|
"Condition": {"StringEquals": {"aws:ResourceTag/Project": "nomad-encoding"}},
|
|
},
|
|
{"Effect": "Allow", "Action": ["ec2:DescribeInstances"], "Resource": "*"},
|
|
],
|
|
}),
|
|
)
|
|
|
|
# --- Network ---------------------------------------------------------------------
|
|
vpc = aws.ec2.get_vpc(default=True)
|
|
subnet = aws.ec2.get_subnet(filters=[
|
|
aws.ec2.GetSubnetFilterArgs(name="vpc-id", values=[vpc.id]),
|
|
aws.ec2.GetSubnetFilterArgs(name="availability-zone", values=[AZ]),
|
|
aws.ec2.GetSubnetFilterArgs(name="default-for-az", values=["true"]),
|
|
])
|
|
sg = aws.ec2.SecurityGroup(
|
|
"worker-sg",
|
|
name="nomad-encoding-worker",
|
|
description="Encoding worker HTTP (API-key auth), same as the GCE encoding-worker firewall",
|
|
vpc_id=vpc.id,
|
|
ingress=[aws.ec2.SecurityGroupIngressArgs(protocol="tcp", from_port=8080, to_port=8080, cidr_blocks=["0.0.0.0/0"])],
|
|
egress=[aws.ec2.SecurityGroupEgressArgs(protocol="-1", from_port=0, to_port=0, cidr_blocks=["0.0.0.0/0"])],
|
|
tags=PROJECT_TAG,
|
|
)
|
|
|
|
# --- Instance --------------------------------------------------------------------
|
|
ami = aws.ssm.get_parameter(name="/aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id")
|
|
provision = Path(__file__).parent.parent.joinpath("encoding-worker", "aws", "provision.sh").read_text()
|
|
user_data = provision.replace(
|
|
"set -euxo pipefail\n",
|
|
f"set -euxo pipefail\nexport WIF_PROVIDER='{WIF_PROVIDER}' VM_NAME='{VM_NAME}'\n",
|
|
1,
|
|
)
|
|
|
|
instance = aws.ec2.Instance(
|
|
"worker",
|
|
ami=ami.value,
|
|
instance_type=INSTANCE_TYPE,
|
|
subnet_id=subnet.id,
|
|
vpc_security_group_ids=[sg.id],
|
|
iam_instance_profile=profile.name,
|
|
user_data=user_data,
|
|
instance_initiated_shutdown_behavior="stop",
|
|
instance_market_options=aws.ec2.InstanceInstanceMarketOptionsArgs(
|
|
market_type="spot",
|
|
spot_options=aws.ec2.InstanceInstanceMarketOptionsSpotOptionsArgs(
|
|
spot_instance_type="persistent",
|
|
instance_interruption_behavior="stop",
|
|
),
|
|
),
|
|
metadata_options=aws.ec2.InstanceMetadataOptionsArgs(http_tokens="required", http_put_response_hop_limit=2),
|
|
root_block_device=aws.ec2.InstanceRootBlockDeviceArgs(
|
|
volume_size=100, volume_type="gp3", iops=6000, throughput=400, delete_on_termination=True,
|
|
tags={**PROJECT_TAG, "Name": VM_NAME},
|
|
),
|
|
tags={**PROJECT_TAG, "Name": VM_NAME},
|
|
# The root volume is the worker's state (like a GCE boot disk): a newer
|
|
# Ubuntu AMI or an edited provision.sh must never silently replace it.
|
|
opts=pulumi.ResourceOptions(ignore_changes=["ami", "user_data"]),
|
|
)
|
|
eip = aws.ec2.Eip("worker-eip", domain="vpc", tags={**PROJECT_TAG, "Name": VM_NAME})
|
|
aws.ec2.EipAssociation("worker-eip-assoc", instance_id=instance.id, allocation_id=eip.id)
|
|
|
|
pulumi.export("vm_name", VM_NAME)
|
|
pulumi.export("instance_id", instance.id)
|
|
pulumi.export("public_ip", eip.public_ip)
|
|
pulumi.export("availability_zone", instance.availability_zone)
|
|
pulumi.export(
|
|
"fallback_vms_entry",
|
|
pulumi.Output.all(instance.id, eip.public_ip, instance.availability_zone).apply(
|
|
lambda a: json.dumps({
|
|
"vm": VM_NAME, "zone": a[2], "ip": a[1], "machine_type": INSTANCE_TYPE.replace(".", "-"),
|
|
"provider": "aws", "instance_id": a[0], "region": "us-east-2",
|
|
})
|
|
),
|
|
)
|