feat(encoding): AWS EC2 encoding worker as a pool candidate (v0.288.0) (#1236)

## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.

The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.

## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
  - EC2 states map to GCE statuses.
  - EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
  - A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
  - An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
  - The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
  - A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
  - `infrastructure/encoding-worker/aws/` adds:
    - `provision.sh`, as EC2 user-data;
    - `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
  - The secret `aws-encoding-controller-credentials`.
  - Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
  - an Elastic IP;
  - a security group on 8080 (API key, same as the GCE firewall);
  - the worker role with SSM;
  - a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".

## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.

Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.

## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
  - manager routing and ranking with overrides;
  - `ensure_preferred_running` branches;
  - failover on an AWS control-plane fault;
  - provider tagging;
  - deploy_promote exclusions;
  - the idle function skipping AWS;
  - watchdog decisions;
  - the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-10-29 17:42:35 -04:00
1 parent 65940c6f57
commit de5dfcd50d
36 files changed
+1610 -103

No files matched your search

+18 -1
View File
@@ -87,7 +87,7 @@ jobs:
# ============================================
runner-manager-tests:
name: "Infra - runner_manager function tests"
name: "Infra - Cloud Function tests"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.infra == 'true' || needs.changes.outputs.deps == 'true')
@@ -100,6 +100,9 @@ jobs:
# google.cloud namespace package is genuinely installed.
- run: pip install pytest -r infrastructure/functions/runner_manager/requirements.txt
- run: pytest infrastructure/functions/runner_manager -q -p no:cacheprovider -o addopts=""
- run: pip install -r infrastructure/functions/encoding_worker_idle/requirements.txt
# Separate invocation: both functions have a top-level `main` module.
- run: pytest infrastructure/functions/encoding_worker_idle -q -p no:cacheprovider -o addopts=""
frontend-lint:
name: "Frontend - Lint (Rules of Hooks)"
@@ -1399,6 +1402,11 @@ jobs:
echo "Uploading startup.sh..."
gsutil cp infrastructure/encoding-worker/startup.sh \
"${BUCKET}/encoding-worker/startup.sh"
# AWS encoding worker (EC2): its startup + idle self-stop, fetched on boot
gsutil cp infrastructure/encoding-worker/aws/startup-aws.sh \
"${BUCKET}/encoding-worker/startup-aws.sh"
gsutil cp infrastructure/encoding-worker/aws/idle-watchdog.py \
"${BUCKET}/encoding-worker/idle-watchdog.py"
# 4. Write version manifest for verification
echo "Writing version manifest..."
@@ -1425,6 +1433,15 @@ jobs:
print("::error::Encoding worker config not found in Firestore")
raise SystemExit(1)
data = doc.to_dict()
if (data.get("active_override_provider") or "").lower() == "aws":
# The AWS encoding worker (EC2) is serving by preference. This
# GCE blue-green flow can't gcloud-start/stop it and must not
# clear its override; it picks up the new wheel on its next job.
# Treat it as "no GCE override" so every step below ignores it.
print(f"AWS worker {data.get('active_override_vm')} is the serving override; "
"leaving it untouched by the GCE blue-green deploy")
for key in ("active_override_vm", "active_override_ip", "active_override_zone"):
data[key] = None
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
for key in ["primary_vm", "primary_ip", "primary_version",
"secondary_vm", "secondary_ip",
+6
View File
@@ -171,6 +171,12 @@ gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke
For job stuck at `encoding` status, see [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
**AWS encoding worker** (`aws-encoding-worker-1`, EC2 c7a.8xlarge Spot, us-east-2) is a pool
candidate too (`provider: aws` in `encoding-worker-fallback-vms`): last resort by default,
preferred when Firestore `config/encoding-worker.speed_rank_overrides = {"c7a": 5}`. Not
blue-green deployed, idle-stops itself, shell via SSM, logs are on the box (journald). See
`infrastructure/aws/README.md`.
### What Doesn't Work
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
+2 -2
View File
@@ -51,7 +51,7 @@ async def warmup_encoding_worker(
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Start the primary encoding worker VM if it's stopped.
"""Start the preferred encoding worker if it's stopped (normally the GCE primary).
Called on lyrics-review page load so the VM is warm by the time the
reviewer previews a render. ``job_id`` scopes auth to the job under review.
@@ -63,7 +63,7 @@ async def warmup_encoding_worker(
# WHOLE instance for ~11s on every review-page open with a cold VM,
# stalling that same page's audio/lyrics requests behind it
# (2026-09-19, follow-up to the concurrent-load investigation).
result = await asyncio.to_thread(manager.ensure_primary_running)
result = await asyncio.to_thread(manager.ensure_preferred_running)
if result["started"]:
logger.info(f"Started encoding worker VM: {result['vm_name']}")
return result
+167
View File
@@ -0,0 +1,167 @@
"""EC2 adapter for the AWS encoding worker.
The AWS worker (an EC2 Spot instance running the same ``gce_encoding`` HTTP
service) is just another candidate in the encoding-worker pool. Rather than
teach every caller about EC2, this module mimics the slice of
``google.cloud.compute_v1.InstancesClient`` that ``EncodingWorkerManager`` uses
(``get`` / ``start`` / ``stop``), so the manager routes AWS VMs here and the
GCE code paths stay untouched:
- EC2 states map to GCE statuses (``running`` → ``RUNNING``, ``stopped`` →
``TERMINATED``, …) so the existing RUNNING/STAGING/stopped checks just work.
- ``start`` returns an operation object shaped like a compute ExtendedOperation
(``result()``, ``error_code``, ``error_message``); EC2 capacity errors surface
as codes in ``CAPACITY_ERROR_CODES`` so stockout cooldowns apply unchanged.
- A missing instance raises ``google.api_core.exceptions.NotFound``, which the
manager already treats as "skip this candidate".
Credentials: an IAM user limited to Start/Stop/Describe on instances tagged
``Project=nomad-encoding``, stored as JSON ``{"access_key_id", "secret_access_key"}``
in Secret Manager ``aws-encoding-controller-credentials`` (env
``AWS_ENCODING_CONTROLLER_CREDENTIALS`` overrides, for tests/local runs).
"""
from __future__ import annotations
import json
import logging
import os
import threading
from dataclasses import dataclass
from typing import Any, Dict, Optional
from google.api_core import exceptions as gcp_exceptions
logger = logging.getLogger(__name__)
CREDENTIALS_SECRET = "aws-encoding-controller-credentials"
CREDENTIALS_ENV = "AWS_ENCODING_CONTROLLER_CREDENTIALS"
# EC2 instance state → GCE instance status.
_EC2_TO_GCE_STATUS = {
"pending": "STAGING",
"running": "RUNNING",
"stopping": "STOPPING",
"stopped": "TERMINATED",
"shutting-down": "STOPPING",
"terminated": "TERMINATED",
}
@dataclass(frozen=True)
class AwsVm:
"""Where an AWS worker lives. Built from a fallback-pool entry."""
instance_id: str
region: str
@dataclass
class _Instance:
status: str
class AwsStartOperation:
"""Duck-types a compute ExtendedOperation for ``_wait_for_compute_operation``."""
def __init__(self, run):
self._run = run
self.error_code = ""
self.error_message = ""
def result(self, timeout: Optional[float] = None) -> None:
from botocore.exceptions import ClientError
try:
self._run()
except ClientError as e:
err = e.response.get("Error", {})
self.error_code = err.get("Code", "") or "ClientError"
self.error_message = err.get("Message", "") or str(e)
def _load_credentials(project_id: str) -> Dict[str, str]:
raw = os.environ.get(CREDENTIALS_ENV)
if not raw:
from google.cloud import secretmanager
client = secretmanager.SecretManagerServiceClient()
name = f"projects/{project_id}/secrets/{CREDENTIALS_SECRET}/versions/latest"
raw = client.access_secret_version(name=name).payload.data.decode("utf-8")
creds = json.loads(raw)
return {"access_key_id": creds["access_key_id"], "secret_access_key": creds["secret_access_key"]}
class AwsComputeAdapter:
"""``get``/``start``/``stop`` for registered AWS worker VMs, by VM name."""
def __init__(self, vms: Dict[str, AwsVm], project_id: str = "nomadkaraoke", client_factory=None):
self._vms = dict(vms)
self._project_id = project_id
self._client_factory = client_factory
self._clients: Dict[str, Any] = {}
self._lock = threading.Lock()
def handles(self, vm_name: str) -> bool:
return vm_name in self._vms
def _client(self, region: str):
with self._lock:
if region not in self._clients:
if self._client_factory is not None:
self._clients[region] = self._client_factory(region)
else:
import boto3
creds = _load_credentials(self._project_id)
self._clients[region] = boto3.client(
"ec2",
region_name=region,
aws_access_key_id=creds["access_key_id"],
aws_secret_access_key=creds["secret_access_key"],
)
return self._clients[region]
def _vm(self, instance: str) -> AwsVm:
vm = self._vms.get(instance)
if vm is None:
raise gcp_exceptions.NotFound(f"AWS worker {instance} is not registered")
return vm
def get(self, project: str = "", zone: str = "", instance: str = "") -> _Instance:
from botocore.exceptions import ClientError
vm = self._vm(instance)
try:
resp = self._client(vm.region).describe_instances(InstanceIds=[vm.instance_id])
except ClientError as e:
if e.response.get("Error", {}).get("Code", "").startswith("InvalidInstanceID"):
raise gcp_exceptions.NotFound(f"AWS worker {instance} ({vm.instance_id}) not found") from e
raise
reservations = resp.get("Reservations") or []
if not reservations or not reservations[0].get("Instances"):
raise gcp_exceptions.NotFound(f"AWS worker {instance} ({vm.instance_id}) not found")
state = reservations[0]["Instances"][0]["State"]["Name"]
return _Instance(status=_EC2_TO_GCE_STATUS.get(state, state.upper()))
def start(self, project: str = "", zone: str = "", instance: str = "") -> AwsStartOperation:
vm = self._vm(instance)
logger.info("Starting AWS worker %s (%s in %s)", instance, vm.instance_id, vm.region)
return AwsStartOperation(lambda: self._client(vm.region).start_instances(InstanceIds=[vm.instance_id]))
def stop(self, project: str = "", zone: str = "", instance: str = "") -> None:
vm = self._vm(instance)
logger.info("Stopping AWS worker %s (%s in %s)", instance, vm.instance_id, vm.region)
self._client(vm.region).stop_instances(InstanceIds=[vm.instance_id])
def aws_vms_from_pool(entries) -> Dict[str, AwsVm]:
"""``{vm_name: AwsVm}`` for the ``provider == "aws"`` entries of a fallback pool."""
vms: Dict[str, AwsVm] = {}
for e in entries or []:
if (e.get("provider") or "").lower() != "aws":
continue
name = e.get("vm") or e.get("vm_name")
if name and e.get("instance_id") and e.get("region"):
vms[name] = AwsVm(instance_id=e["instance_id"], region=e["region"])
return vms
+6
View File
@@ -12,6 +12,12 @@ CAPACITY_ERROR_CODES: FrozenSet[str] = frozenset({
"ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS",
"STOCKOUT",
"QUOTA_EXCEEDED",
# AWS EC2 (the AWS encoding worker; see aws_encoding_compute.py)
"InsufficientInstanceCapacity",
"InsufficientHostCapacity",
"InsufficientCapacity",
"VcpuLimitExceeded",
"MaxSpotInstanceCountExceeded",
})
+13 -79
View File
@@ -294,43 +294,14 @@ class EncodingService:
yield
def _fallback_entries(self) -> list:
"""Parse ENCODING_WORKER_FALLBACK_VMS into candidate dicts.
"""Parse ENCODING_WORKER_FALLBACK_VMS into candidate dicts (never raises).
Schema: '[{"vm":"encoding-worker-fallback-c2df","zone":"us-central1-f",
"ip":"34.x.x.x","machine_type":"c2d-highcpu-16"}, ...]'. machine_type is
optional (inferred from the VM name for legacy entries). Malformed input
degrades to fewer/no fallbacks — this runs during connection recovery and
must never raise.
See ``encoding_worker_manager.parse_fallback_vms`` for the schema (GCE
entries, plus AWS ones carrying provider/instance_id/region).
"""
import json as _json
raw = self.settings.encoding_worker_fallback_vms
if not raw:
return []
try:
parsed = _json.loads(raw)
except (ValueError, TypeError) as e:
logger.warning(f"Invalid ENCODING_WORKER_FALLBACK_VMS JSON: {e}")
return []
# A non-list root (null, dict, scalar) parses fine but would blow up
# `for item in parsed` and escape during connection recovery — degrade
# to no fallbacks instead.
if not isinstance(parsed, list):
logger.warning("ENCODING_WORKER_FALLBACK_VMS must be a JSON list; ignoring")
return []
entries = []
for item in parsed:
try:
entries.append({
"vm": item["vm"],
"zone": item["zone"],
"ip": item["ip"],
"machine_type": item.get("machine_type"),
"kind": "fallback",
})
except (KeyError, TypeError) as e:
logger.warning(f"Skipping malformed fallback candidate {item}: {e}")
continue
return entries
from backend.services.encoding_worker_manager import parse_fallback_vms
return parse_fallback_vms(self.settings.encoding_worker_fallback_vms)
def _vm_for_worker_url(self, worker_url: Optional[str]) -> Optional[tuple]:
"""Map a worker base URL (http://<ip>:8080) back to its (vm_name, zone).
@@ -394,13 +365,12 @@ class EncodingService:
def _build_worker_candidates(self) -> list:
"""Build the ranked candidate list for ensure_any_running.
Reads optional fallback VMs from settings (env var
ENCODING_WORKER_FALLBACK_VMS, JSON list of {vm, zone, ip, machine_type?}).
The primary + all fallbacks are ranked by the shared preference logic
(fastest-first, demoting recently-stocked-out types) so runtime selection
stays in lock-step with the deploy green selection. Returns an empty list
when no worker manager / no configured fallbacks — caller falls back to
the single-VM ensure_primary_running path.
Delegates to ``EncodingWorkerManager.ranked_candidates`` (primary +
ENCODING_WORKER_FALLBACK_VMS, fastest-first, demoting recently
stocked-out types, honouring ``speed_rank_overrides``) so runtime
selection, the warmups and the deploy green selection rank identically.
Returns an empty list when there is no worker manager or its config
can't be read — caller falls back to ensure_primary_running.
"""
if not self._worker_manager:
return []
@@ -409,43 +379,7 @@ class EncodingService:
except Exception as e: # noqa: BLE001
logger.warning(f"Could not read worker config for candidates: {_format_exception(e)}")
return []
from backend.services.encoding_worker_manager import EncodingWorkerCandidate
from backend.services.encoding_worker_preference import (
PRIMARY_MACHINE_TYPE,
ordered_candidates,
)
# Build a dict pool (primary + fallbacks), each tagged with machine_type so
# the shared preference logic can rank them. The primary/secondary pair is
# always c4d; the config may override via primary_machine_type.
primary_mt = getattr(config, "primary_machine_type", None) or PRIMARY_MACHINE_TYPE
pool = [{
"vm": config.primary_vm,
"zone": self._worker_manager._zone,
"ip": config.primary_ip,
"machine_type": primary_mt,
"kind": "primary",
"is_primary": True,
}]
# Optional capacity-fallback VMs in alternate zones/families (see
# _fallback_entries). Empty when unconfigured.
pool.extend(self._fallback_entries())
# Rank fastest-first, demoting types that recently stocked out. Keeps the
# primary (c4d) at the top whenever it has capacity.
ranked = ordered_candidates(pool, capacity_state=config.capacity_state)
return [
EncodingWorkerCandidate(
vm_name=c["vm"],
zone=c["zone"],
ip=c["ip"],
machine_type=c.get("machine_type"),
is_primary=c.get("is_primary", False),
)
for c in ranked
]
return self._worker_manager.ranked_candidates(config)
def _load_credentials(self):
"""Load encoding worker URL and API key from config/secrets."""
+165 -4
View File
@@ -112,6 +112,9 @@ class EncodingWorkerConfig:
# shared preference logic to demote recently-exhausted types. See
# backend/services/encoding_worker_preference.py.
capacity_state: dict = field(default_factory=dict)
# {machine_family: rank} replacing SPEED_RANK at runtime, e.g. {"c7a": 5}
# makes the AWS worker preferred. Absent/empty ⇒ the built-in ranking.
speed_rank_overrides: dict = field(default_factory=dict)
@property
def primary_url(self) -> str:
@@ -155,6 +158,8 @@ class EncodingWorkerCandidate:
# the preference logic may rank a fallback ahead of a stocked-out primary, so
# "first candidate" is no longer synonymous with "primary".
is_primary: bool = False
# "aws" for the EC2 worker (routed to AwsComputeAdapter); None ⇒ GCE.
provider: Optional[str] = None
@property
def url(self) -> str:
@@ -165,14 +170,148 @@ class EncodingWorkerCandidate:
return {"vm": self.vm_name, "zone": self.zone, "machine_type": self.machine_type}
def parse_fallback_vms(raw: Optional[str]) -> list:
"""Parse ENCODING_WORKER_FALLBACK_VMS into candidate dicts. Never raises.
Schema: '[{"vm","zone","ip","machine_type"?}, ...]'. AWS workers add
``"provider": "aws", "instance_id", "region"`` (passed through so the
manager can route them to the EC2 adapter). machine_type is optional
(inferred from the VM name for legacy entries). Malformed input degrades to
fewer/no fallbacks — this runs during connection recovery.
"""
import json as _json
if not raw:
return []
try:
parsed = _json.loads(raw)
except (ValueError, TypeError) as e:
logger.warning("Invalid ENCODING_WORKER_FALLBACK_VMS JSON: %s", e)
return []
if not isinstance(parsed, list):
logger.warning("ENCODING_WORKER_FALLBACK_VMS must be a JSON list; ignoring")
return []
entries = []
for item in parsed:
try:
entry = {
"vm": item["vm"],
"zone": item["zone"],
"ip": item["ip"],
"machine_type": item.get("machine_type"),
"kind": "fallback",
}
for key in ("provider", "instance_id", "region"):
if item.get(key):
entry[key] = item[key]
entries.append(entry)
except (KeyError, TypeError) as e:
logger.warning("Skipping malformed fallback candidate %s: %s", item, e)
continue
return entries
class EncodingWorkerManager:
"""Manages blue-green encoding worker VMs via Firestore config and Compute API."""
def __init__(self, db, compute_client=None, project_id="nomadkaraoke", zone="us-central1-c"):
def __init__(
self,
db,
compute_client=None,
project_id="nomadkaraoke",
zone="us-central1-c",
fallback_vms_raw: Optional[str] = None,
aws_compute=None,
):
self._db = db
self._compute = compute_client
self._project_id = project_id
self._zone = zone
# Raw ENCODING_WORKER_FALLBACK_VMS JSON; None → read from settings lazily.
self._fallback_vms_raw = fallback_vms_raw
self._aws_compute = aws_compute # injectable for tests; built lazily otherwise
def fallback_entries(self) -> list:
"""The configured capacity-fallback candidates (GCE and AWS), never raising."""
raw = self._fallback_vms_raw
if raw is None:
try:
from backend.config import get_settings
raw = get_settings().encoding_worker_fallback_vms
except Exception as e: # noqa: BLE001
logger.warning("Could not read ENCODING_WORKER_FALLBACK_VMS: %s", e)
raw = None
return parse_fallback_vms(raw)
def _compute_for(self, vm_name: str):
"""The compute client for a VM: the EC2 adapter for AWS workers, else GCE."""
if self._aws_compute is None:
from backend.services.aws_encoding_compute import AwsComputeAdapter, aws_vms_from_pool
self._aws_compute = AwsComputeAdapter(
aws_vms_from_pool(self.fallback_entries()), project_id=self._project_id
)
if self._aws_compute.handles(vm_name):
return self._aws_compute
return self._compute
def ranked_candidates(self, config: Optional["EncodingWorkerConfig"] = None) -> list:
"""Primary + fallbacks ranked by the shared preference logic.
Fastest-first (SPEED_RANK, overridable per family via
``config/encoding-worker.speed_rank_overrides``), demoting types that
recently stocked out. Shared by the encode path and the warmups so they
always start the worker that requests will actually be routed to.
"""
from backend.services.encoding_worker_preference import (
PRIMARY_MACHINE_TYPE,
ordered_candidates,
)
config = config or self.get_config()
primary_mt = getattr(config, "primary_machine_type", None) or PRIMARY_MACHINE_TYPE
pool = [{
"vm": config.primary_vm,
"zone": self._zone,
"ip": config.primary_ip,
"machine_type": primary_mt,
"kind": "primary",
"is_primary": True,
}]
pool.extend(self.fallback_entries())
ranked = ordered_candidates(
pool,
capacity_state=config.capacity_state,
rank_overrides=getattr(config, "speed_rank_overrides", None),
)
return [
EncodingWorkerCandidate(
vm_name=c["vm"],
zone=c["zone"],
ip=c["ip"],
machine_type=c.get("machine_type"),
is_primary=c.get("is_primary", False),
provider=c.get("provider"),
)
for c in ranked
]
def ensure_preferred_running(self) -> dict:
"""Warm the worker requests will be routed to (ranked, with failover).
When the GCE primary ranks first (the normal case) this is exactly
``ensure_primary_running`` — unchanged behaviour. When something else
ranks first (a ``speed_rank_overrides`` preferring the AWS worker, or the
primary's type in stockout cooldown) it runs the same ranked
``ensure_any_running`` the encode path uses, so the worker that requests
will be routed to is the one warmed, and the active override points at it.
"""
config = self.get_config()
candidates = self.ranked_candidates(config)
if not candidates or candidates[0].is_primary:
return self.ensure_primary_running()
return self.ensure_any_running(candidates)
def _doc_ref(self):
"""Return a reference to the config/encoding-worker document."""
@@ -209,6 +348,7 @@ class EncodingWorkerManager:
active_override_zone=data.get("active_override_zone"),
active_override_set_at=data.get("active_override_set_at"),
capacity_state=data.get("capacity_state") or {},
speed_rank_overrides=data.get("speed_rank_overrides") or {},
)
def update_activity(self) -> None:
@@ -271,7 +411,7 @@ class EncodingWorkerManager:
Pass `zone` for fallback VMs that live in alternate zones.
"""
instance = self._compute.get(
instance = self._compute_for(vm_name).get(
project=self._project_id,
zone=zone or self._zone,
instance=vm_name,
@@ -303,7 +443,7 @@ class EncodingWorkerManager:
logger.info("VM %s is already %s, skipping start", vm_name, status)
return
logger.info("Starting VM %s in zone %s (current status: %s)", vm_name, target_zone, status)
operation = self._compute.start(
operation = self._compute_for(vm_name).start(
project=self._project_id,
zone=target_zone,
instance=vm_name,
@@ -318,7 +458,7 @@ class EncodingWorkerManager:
logger.info("VM %s is already %s, skipping stop", vm_name, status)
return
logger.info("Stopping VM %s in zone %s (current status: %s)", vm_name, target_zone, status)
self._compute.stop(
self._compute_for(vm_name).stop(
project=self._project_id,
zone=target_zone,
instance=vm_name,
@@ -459,6 +599,22 @@ class EncodingWorkerManager:
zone=candidate.zone,
code="NOT_FOUND",
) from nf
except EncodingWorkerStartError:
raise
except Exception as err: # noqa: BLE001
if getattr(candidate, "provider", None) != "aws":
raise # GCE: unchanged behaviour
# An AWS control-plane fault (bad/missing controller creds,
# throttling, botocore error) must fail over to the GCE
# candidates, not abort selection — when the AWS worker is
# preferred it ranks first, so escaping here would block all
# encoding.
raise EncodingWorkerStartError(
f"VM {candidate.vm_name} status check failed in {candidate.zone}: {err}",
vm_name=candidate.vm_name,
zone=candidate.zone,
code="STATUS_CHECK_FAILED",
) from err
started = False
if status not in ("RUNNING", "STAGING"):
logger.info(
@@ -575,6 +731,10 @@ class EncodingWorkerManager:
"active_override_ip": candidate.ip,
"active_override_zone": candidate.zone,
"active_override_set_at": now,
# "aws" for the EC2 worker; the CI blue-green deploy leaves an AWS
# override serving (it can't gcloud-stop it, and it hot-updates its
# wheel per job). None for GCE.
"active_override_provider": getattr(candidate, "provider", None),
})
logger.info(
"Set active_override to %s in %s (capacity fallback)",
@@ -597,6 +757,7 @@ class EncodingWorkerManager:
"active_override_ip": None,
"active_override_zone": None,
"active_override_set_at": None,
"active_override_provider": None,
})
logger.info("Cleared active_override (primary is healthy again)")
+18 -3
View File
@@ -51,6 +51,11 @@ SPEED_RANK: Dict[str, int] = {
"c2d": 50,
"n2d": 60,
"n2": 70,
# AWS EC2 c7a (AMD Genoa, us-east-2) — the AWS encoding worker
# (provider "aws"; machine_type "c7a-8xlarge", "." → "-" for Firestore-safe
# cooldown keys). Last resort by default; make it preferred at runtime with
# config/encoding-worker.speed_rank_overrides = {"c7a": 5}.
"c7a": 90,
}
# Rank for a candidate whose machine type is unknown and cannot be inferred:
@@ -106,8 +111,14 @@ def infer_machine_type(candidate: Dict[str, Any]) -> Optional[str]:
return None
def _speed_rank(candidate: Dict[str, Any]) -> int:
return SPEED_RANK.get(machine_family(infer_machine_type(candidate)), UNKNOWN_RANK)
def _speed_rank(candidate: Dict[str, Any], rank_overrides: Optional[Dict[str, Any]] = None) -> int:
family = machine_family(infer_machine_type(candidate))
if rank_overrides and family in rank_overrides:
try:
return int(rank_overrides[family])
except (TypeError, ValueError):
pass # a malformed override must never break worker selection
return SPEED_RANK.get(family, UNKNOWN_RANK)
def cooldown_key(candidate: Dict[str, Any]) -> Optional[str]:
@@ -164,6 +175,7 @@ def ordered_candidates(
capacity_state: Optional[Dict[str, Any]] = None,
now: Optional[datetime] = None,
cooldown_seconds: float = COOLDOWN_SECONDS,
rank_overrides: Optional[Dict[str, Any]] = None,
) -> List[Dict[str, Any]]:
"""Rank ``pool`` fastest-first, demoting recently-stocked-out types to the back.
@@ -175,6 +187,9 @@ def ordered_candidates(
Absent/empty ⇒ pure fastest-first order (today's behaviour).
now: current UTC time (injectable for tests); defaults to ``datetime.now(UTC)``.
cooldown_seconds: demotion window; defaults to ``COOLDOWN_SECONDS``.
rank_overrides: ``{family: rank}`` from Firestore
``config/encoding-worker.speed_rank_overrides``; replaces SPEED_RANK
for those families (e.g. ``{"c7a": 5}`` prefers the AWS worker).
The speed sort is STABLE, so equal-rank candidates keep their input order —
callers express a secondary preference (e.g. zone spread) purely via input
@@ -182,7 +197,7 @@ def ordered_candidates(
"""
if now is None:
now = datetime.now(timezone.utc)
ranked = sorted(pool, key=_speed_rank) # stable: fastest first
ranked = sorted(pool, key=lambda c: _speed_rank(c, rank_overrides)) # stable: fastest first
hot = [c for c in ranked if not _recently_stocked_out(c, capacity_state, now, cooldown_seconds)]
cold = [c for c in ranked if _recently_stocked_out(c, capacity_state, now, cooldown_seconds)]
return hot + cold
+14 -2
View File
@@ -707,6 +707,18 @@ def verify_wheel_imports():
return True
def _gcs_cli() -> list:
"""The GCS CLI for wheel listing/copying on this worker.
GCE workers use ``gsutil`` (unchanged). The AWS worker authenticates to GCP
via Workload Identity Federation, which ``gsutil`` doesn't support, so it
uses ``gcloud storage`` (same ``ls``/``cp`` syntax).
"""
if os.environ.get("ENCODING_WORKER_PROVIDER", "").lower() == "aws":
return ["gcloud", "storage"]
return ["gsutil"]
def ensure_latest_wheel():
'''Ensure the latest karaoke-gen wheel is installed & verified (thread-safe).
@@ -748,7 +760,7 @@ def _ensure_latest_wheel_locked():
# job and blow the timeout on fresh workers — the historical cause of
# "no wheel found" / partial installs. Listing + single copy is ~2s.
ls_result = subprocess.run(
["gsutil", "ls", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl"],
[*_gcs_cli(), "ls", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl"],
capture_output=True, text=True, timeout=60
)
# Filter out karaoke_gen-current.whl (not a valid PEP 427 wheel name)
@@ -787,7 +799,7 @@ def _ensure_latest_wheel_locked():
)
cp_result = subprocess.run(
["gsutil", "cp", latest_uri, wheel_path],
[*_gcs_cli(), "cp", latest_uri, wheel_path],
capture_output=True, text=True, timeout=120
)
if cp_result.returncode != 0:
+1 -1
View File
@@ -624,7 +624,7 @@ class WorkerService:
compute_client=compute_client,
project_id=self.settings.google_cloud_project,
)
result = manager.ensure_primary_running()
result = manager.ensure_preferred_running()
if result["started"]:
logger.info(f"[job:{job_id}] Encoding worker warmup: started VM {result['vm_name']}")
else:
+115
View File
@@ -0,0 +1,115 @@
"""Unit tests for the EC2 adapter behind the AWS encoding worker.
The adapter must look exactly like the compute_v1.InstancesClient slice the
manager uses, so the manager's GCE-shaped status checks, capacity cooldowns
and NotFound skips apply to the AWS worker unchanged.
"""
import json
import boto3
import pytest
from botocore.stub import Stubber
from google.api_core import exceptions as gcp_exceptions
from backend.services.aws_encoding_compute import (
AwsComputeAdapter,
AwsVm,
_load_credentials,
aws_vms_from_pool,
)
from backend.services.encoding_errors import CAPACITY_ERROR_CODES
VM = "aws-encoding-worker-1"
IID = "i-0123456789abcdef0"
@pytest.fixture
def ec2():
client = boto3.client("ec2", region_name="us-east-2", aws_access_key_id="x", aws_secret_access_key="y")
with Stubber(client) as stub:
yield client, stub
def _adapter(client):
return AwsComputeAdapter({VM: AwsVm(IID, "us-east-2")}, client_factory=lambda region: client)
def _describe(state):
return {"Reservations": [{"Instances": [{"InstanceId": IID, "State": {"Name": state, "Code": 0}}]}]}
@pytest.mark.parametrize("ec2_state,gce_status", [
("running", "RUNNING"),
("pending", "STAGING"),
("stopping", "STOPPING"),
("stopped", "TERMINATED"),
("shutting-down", "STOPPING"),
])
def test_get_maps_ec2_state_to_gce_status(ec2, ec2_state, gce_status):
client, stub = ec2
stub.add_response("describe_instances", _describe(ec2_state), {"InstanceIds": [IID]})
assert _adapter(client).get(project="p", zone="us-east-2b", instance=VM).status == gce_status
def test_get_missing_instance_raises_not_found(ec2):
client, stub = ec2
stub.add_client_error("describe_instances", service_error_code="InvalidInstanceID.NotFound")
with pytest.raises(gcp_exceptions.NotFound):
_adapter(client).get(instance=VM)
def test_unregistered_vm_raises_not_found(ec2):
client, _ = ec2
with pytest.raises(gcp_exceptions.NotFound):
_adapter(client).get(instance="encoding-worker-a")
def test_start_success_leaves_no_error_code(ec2):
client, stub = ec2
stub.add_response("start_instances", {"StartingInstances": []}, {"InstanceIds": [IID]})
op = _adapter(client).start(instance=VM)
op.result(timeout=30)
assert op.error_code == ""
def test_start_capacity_error_is_a_capacity_code(ec2):
client, stub = ec2
stub.add_client_error(
"start_instances", service_error_code="InsufficientInstanceCapacity",
service_message="no c7a.8xlarge capacity in us-east-2b",
)
op = _adapter(client).start(instance=VM)
op.result()
assert op.error_code == "InsufficientInstanceCapacity"
assert op.error_code in CAPACITY_ERROR_CODES
assert "capacity" in op.error_message
def test_stop_calls_stop_instances(ec2):
client, stub = ec2
stub.add_response("stop_instances", {"StoppingInstances": []}, {"InstanceIds": [IID]})
_adapter(client).stop(instance=VM)
stub.assert_no_pending_responses()
def test_handles_only_registered_vms():
adapter = AwsComputeAdapter({VM: AwsVm(IID, "us-east-2")}, client_factory=lambda r: None)
assert adapter.handles(VM)
assert not adapter.handles("encoding-worker-a")
def test_aws_vms_from_pool_keeps_only_complete_aws_entries():
pool = [
{"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "1.2.3.4"},
{"vm": VM, "zone": "us-east-2b", "ip": "5.6.7.8", "provider": "aws", "instance_id": IID, "region": "us-east-2"},
{"vm": "aws-incomplete", "zone": "us-east-2b", "ip": "9.9.9.9", "provider": "aws"},
]
assert aws_vms_from_pool(pool) == {VM: AwsVm(IID, "us-east-2")}
def test_credentials_env_override(monkeypatch):
monkeypatch.setenv(
"AWS_ENCODING_CONTROLLER_CREDENTIALS",
json.dumps({"access_key_id": "AKIA", "secret_access_key": "s3cr3t", "extra": 1}),
)
assert _load_credentials("nomadkaraoke") == {"access_key_id": "AKIA", "secret_access_key": "s3cr3t"}
+54
View File
@@ -0,0 +1,54 @@
"""Tests for the AWS encoding worker's idle self-stop decision."""
import importlib.util
from datetime import datetime, timedelta, timezone
from pathlib import Path
_spec = importlib.util.spec_from_file_location("idle_watchdog", Path(__file__).resolve().parents[2] / "infrastructure" / "encoding-worker" / "aws" / "idle-watchdog.py")
wd = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(wd)
NOW = datetime(2026, 10, 29, 12, 0, tzinfo=timezone.utc)
VM = "aws-encoding-worker-1"
def _decide(active_jobs=0, config=None, unknown_since=None):
return wd.decide(now=NOW, active_jobs=active_jobs, config=config, vm_name=VM, unknown_since=unknown_since)[0]
def _routed(minutes_ago):
return {"active_override_vm": VM, "last_activity_at": (NOW - timedelta(minutes=minutes_ago)).isoformat()}
def test_busy_worker_keeps_running_even_if_not_routed():
assert _decide(active_jobs=2, config={"active_override_vm": "encoding-worker-a"}) == "keep"
def test_idle_and_not_routed_stops():
assert _decide(config={"active_override_vm": None}) == "stop"
def test_routed_with_recent_activity_keeps_running():
assert _decide(config=_routed(2)) == "keep"
def test_routed_but_idle_past_timeout_stops():
assert _decide(config=_routed(6)) == "stop"
def test_routed_without_activity_timestamp_stops():
assert _decide(config={"active_override_vm": VM}) == "stop"
def test_naive_timestamp_treated_as_utc():
assert _decide(config={"active_override_vm": VM, "last_activity_at": "2026-10-29T11:58:00"}) == "keep"
def test_unreadable_health_starts_grace_then_stops():
assert _decide(active_jobs=None, config=_routed(1)) == "unknown"
assert _decide(active_jobs=None, config=_routed(1), unknown_since=NOW - timedelta(minutes=5)) == "unknown"
assert _decide(active_jobs=None, config=_routed(1), unknown_since=NOW - timedelta(minutes=16)) == "stop"
def test_unreadable_firestore_starts_grace_then_stops():
assert _decide(config=None) == "unknown"
assert _decide(config=None, unknown_since=NOW - timedelta(minutes=20)) == "stop"
+40
View File
@@ -219,3 +219,43 @@ class TestSelectGreenRankingSharedWithRuntime:
cands = select_green_candidates(_config(active_override_vm=None), self.POOL)
c4 = next(c for c in cands if c["vm"] == "encoding-worker-fallback-c4a")
assert c4["machine_type"] == "c4-highcpu-16"
# --- AWS encoding worker: never touched by the GCE blue-green deploy ----------
_AWS = {"vm": "aws-encoding-worker-1", "zone": "us-east-2b", "ip": "3.3.3.3",
"machine_type": "c7a-8xlarge", "provider": "aws", "instance_id": "i-0abc", "region": "us-east-2"}
def test_aws_worker_is_never_a_green_candidate():
config = {"secondary_vm": "encoding-worker-b", "secondary_ip": "1.1.1.2"}
fallbacks = [{"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "2.2.2.2"}, _AWS]
names = [c["vm"] for c in deploy_promote.select_green_candidates(config, fallbacks)]
assert "aws-encoding-worker-1" not in names
assert names == ["encoding-worker-b", "encoding-worker-fallback-c2df"]
def test_secondary_promotion_leaves_aws_override_serving():
config = {
"primary_vm": "encoding-worker-a", "primary_ip": "1.1.1.1", "primary_version": "1.0",
"active_override_vm": "aws-encoding-worker-1", "active_override_ip": "3.3.3.3",
"active_override_zone": "us-east-2b", "active_override_provider": "aws",
}
green = {"vm": "encoding-worker-b", "ip": "1.1.1.2", "zone": "us-central1-c", "kind": "secondary"}
plan = deploy_promote.promote_plan(green, config, now="2026-10-29T00:00:00Z", version="2.0")
assert "active_override_vm" not in plan["firestore_updates"]
assert [d["vm"] for d in plan["drain_and_stop"]] == ["encoding-worker-a"]
def test_fallback_promotion_does_not_drain_aws_override():
config = {"active_override_vm": "aws-encoding-worker-1", "active_override_zone": "us-east-2b",
"active_override_provider": "aws"}
green = {"vm": "encoding-worker-fallback-c2df", "ip": "2.2.2.2", "zone": "us-central1-f", "kind": "fallback"}
plan = deploy_promote.promote_plan(green, config, now="2026-10-29T00:00:00Z", version="2.0")
assert plan["drain_and_stop"] == []
def test_fallback_promotion_clears_provider_tag():
green = {"vm": "encoding-worker-fallback-c2df", "ip": "2.2.2.2", "zone": "us-central1-f", "kind": "fallback"}
plan = deploy_promote.promote_plan(green, {}, now="2026-10-29T00:00:00Z", version="2.0")
assert plan["firestore_updates"]["active_override_provider"] is None
+10 -3
View File
@@ -1351,12 +1351,19 @@ class TestBuildWorkerCandidates:
cfg.primary_ip = "10.0.0.1"
cfg.primary_machine_type = None # → defaults to c4d
cfg.capacity_state = capacity_state or {}
cfg.speed_rank_overrides = {}
return cfg
def _service_with_fallbacks(self, encoding_service, fallbacks_json, capacity_state=None):
mgr = MagicMock()
mgr._zone = "us-central1-c"
mgr.get_config.return_value = self._config(capacity_state)
# Ranking lives in EncodingWorkerManager.ranked_candidates (shared with the
# warmups), so use a real manager with a mocked config read.
from backend.services.encoding_worker_manager import EncodingWorkerManager
mgr = EncodingWorkerManager(
db=MagicMock(), compute_client=MagicMock(), zone="us-central1-c",
fallback_vms_raw=fallbacks_json,
)
mgr.get_config = MagicMock(return_value=self._config(capacity_state))
encoding_service._worker_manager = mgr
encoding_service.settings.encoding_worker_fallback_vms = fallbacks_json
return encoding_service
@@ -0,0 +1,146 @@
"""EncodingWorkerManager with the AWS encoding worker in the pool.
Covers routing AWS VMs to the EC2 adapter (GCE untouched), the shared fallback
parser passing provider fields through, ranking with Firestore
speed_rank_overrides, the ranked warmup, and the override's provider tag.
"""
import json
from unittest.mock import MagicMock
import pytest
from backend.services.aws_encoding_compute import AwsStartOperation
from backend.services.encoding_errors import EncodingWorkerCapacityError
from backend.services.encoding_worker_manager import EncodingWorkerManager, parse_fallback_vms
AWS_ENTRY = {
"vm": "aws-encoding-worker-1", "zone": "us-east-2b", "ip": "3.3.3.3",
"machine_type": "c7a-8xlarge", "provider": "aws", "instance_id": "i-0abc", "region": "us-east-2",
}
C2D_ENTRY = {"vm": "encoding-worker-fallback-c2df", "zone": "us-central1-f", "ip": "2.2.2.2",
"machine_type": "c2d-highcpu-16"}
def _db(**config):
data = {
"primary_vm": "encoding-worker-a", "primary_ip": "1.1.1.1", "primary_version": "1.0",
"secondary_vm": "encoding-worker-b", "secondary_ip": "1.1.1.2",
"last_activity_at": None, "capacity_state": {}, **config,
}
db = MagicMock()
snap = MagicMock(exists=True)
snap.to_dict.return_value = data
doc = MagicMock()
doc.get.return_value = snap
db.collection.return_value.document.return_value = doc
return db, doc
def _manager(db, aws=None, entries=(C2D_ENTRY, AWS_ENTRY)):
gce = MagicMock(name="gce")
aws = aws or MagicMock(name="aws")
aws.handles.side_effect = lambda vm: vm == AWS_ENTRY["vm"]
m = EncodingWorkerManager(db=db, compute_client=gce, fallback_vms_raw=json.dumps(list(entries)), aws_compute=aws)
return m, gce, aws
def test_parse_fallback_vms_passes_aws_fields_through():
parsed = parse_fallback_vms(json.dumps([C2D_ENTRY, AWS_ENTRY]))
assert parsed[0] == {**C2D_ENTRY, "kind": "fallback"}
assert parsed[1]["provider"] == "aws"
assert parsed[1]["instance_id"] == "i-0abc" and parsed[1]["region"] == "us-east-2"
@pytest.mark.parametrize("raw", [None, "", "not json", "{}", "null"])
def test_parse_fallback_vms_never_raises(raw):
assert parse_fallback_vms(raw) == []
def test_aws_vm_ops_go_to_adapter_and_gce_untouched():
db, _ = _db()
m, gce, aws = _manager(db)
aws.get.return_value = MagicMock(status="RUNNING")
gce.get.return_value = MagicMock(status="TERMINATED")
assert m.get_vm_status("aws-encoding-worker-1", zone="us-east-2b") == "RUNNING"
assert m.get_vm_status("encoding-worker-a") == "TERMINATED"
aws.get.assert_called_once()
gce.get.assert_called_once()
def test_aws_capacity_error_raises_typed_capacity_error():
db, _ = _db()
m, _, aws = _manager(db)
aws.get.return_value = MagicMock(status="TERMINATED")
def _no_capacity():
from botocore.exceptions import ClientError
raise ClientError({"Error": {"Code": "InsufficientInstanceCapacity", "Message": "none"}}, "StartInstances")
aws.start.return_value = AwsStartOperation(_no_capacity)
with pytest.raises(EncodingWorkerCapacityError):
m.start_vm("aws-encoding-worker-1", zone="us-east-2b")
def test_ranked_candidates_default_keeps_gce_primary_first_and_aws_last():
db, _ = _db()
m, _, _ = _manager(db)
ranked = m.ranked_candidates()
assert [c.vm_name for c in ranked] == ["encoding-worker-a", "encoding-worker-fallback-c2df", "aws-encoding-worker-1"]
assert ranked[0].is_primary and ranked[-1].provider == "aws"
def test_ranked_candidates_honour_speed_rank_overrides():
db, _ = _db(speed_rank_overrides={"c7a": 5})
m, _, _ = _manager(db)
assert m.ranked_candidates()[0].vm_name == "aws-encoding-worker-1"
def test_ensure_preferred_running_is_primary_path_when_primary_ranks_first():
db, _ = _db()
m, _, _ = _manager(db)
m.ensure_primary_running = MagicMock(return_value={"started": False})
m.ensure_any_running = MagicMock()
m.ensure_preferred_running()
m.ensure_primary_running.assert_called_once()
m.ensure_any_running.assert_not_called()
def test_ensure_preferred_running_starts_aws_when_preferred():
db, _ = _db(speed_rank_overrides={"c7a": 5})
m, _, _ = _manager(db)
m.ensure_primary_running = MagicMock()
m.ensure_any_running = MagicMock(return_value={"started": True, "vm_name": "aws-encoding-worker-1"})
m.ensure_preferred_running()
m.ensure_primary_running.assert_not_called()
(candidates,), _ = m.ensure_any_running.call_args
assert candidates[0].vm_name == "aws-encoding-worker-1"
def test_active_override_records_provider():
db, doc = _db()
m, _, _ = _manager(db)
aws_candidate = next(c for c in m.ranked_candidates() if c.provider == "aws")
m._set_active_override(aws_candidate)
update = doc.update.call_args[0][0]
assert update["active_override_vm"] == "aws-encoding-worker-1"
assert update["active_override_provider"] == "aws"
db2, doc2 = _db(active_override_vm="aws-encoding-worker-1", active_override_provider="aws")
m2, _, _ = _manager(db2)
m2._clear_active_override()
assert doc2.update.call_args[0][0]["active_override_provider"] is None
def test_aws_control_plane_fault_fails_over_to_gce():
db, _ = _db(speed_rank_overrides={"c7a": 5})
m, gce, aws = _manager(db, entries=(AWS_ENTRY,))
aws.get.side_effect = RuntimeError("could not load aws-encoding-controller-credentials")
gce.get.return_value = MagicMock(status="RUNNING")
m.update_activity = MagicMock()
m._clear_active_override = MagicMock()
m._clear_stockout = MagicMock()
result = m.ensure_any_running(m.ranked_candidates())
assert result["vm_name"] == "encoding-worker-a"
@@ -170,3 +170,56 @@ def test_cooldown_key_shape():
assert cooldown_key(_c("x", "us-central1-c", "c4d-highcpu-16")) == "c4d-highcpu-16@us-central1-c"
assert cooldown_key(_c("x", None, "n2-highcpu-16")) == "n2-highcpu-16"
assert cooldown_key(_c("no-token", "z")) is None
# --- AWS encoding worker (c7a) + runtime rank overrides -----------------------
AWS = dict(provider="aws", instance_id="i-0abc", region="us-east-2")
def test_aws_worker_is_last_resort_by_default():
pool = [
_c("aws-encoding-worker-1", "us-east-2b", "c7a-8xlarge", **AWS),
_c("encoding-worker-fallback-c2df", "us-central1-f", "c2d-highcpu-16"),
_c("encoding-worker-a", "us-central1-c", "c4d-highcpu-16"),
]
ranked = ordered_candidates(pool, now=NOW)
assert [c["vm"] for c in ranked] == [
"encoding-worker-a", "encoding-worker-fallback-c2df", "aws-encoding-worker-1",
]
def test_rank_override_prefers_aws_worker():
pool = [
_c("encoding-worker-a", "us-central1-c", "c4d-highcpu-16"),
_c("encoding-worker-fallback-c2df", "us-central1-f", "c2d-highcpu-16"),
_c("aws-encoding-worker-1", "us-east-2b", "c7a-8xlarge", **AWS),
]
ranked = ordered_candidates(pool, now=NOW, rank_overrides={"c7a": 5})
assert ranked[0]["vm"] == "aws-encoding-worker-1"
# The rest keep the built-in order behind it.
assert [c["vm"] for c in ranked[1:]] == ["encoding-worker-a", "encoding-worker-fallback-c2df"]
def test_stocked_out_aws_worker_still_demoted_despite_override():
aws = _c("aws-encoding-worker-1", "us-east-2b", "c7a-8xlarge", **AWS)
pool = [aws, _c("encoding-worker-a", "us-central1-c", "c4d-highcpu-16")]
state = {cooldown_key(aws): (NOW - timedelta(seconds=60)).isoformat()}
ranked = ordered_candidates(pool, capacity_state=state, now=NOW, rank_overrides={"c7a": 5})
assert ranked[-1]["vm"] == "aws-encoding-worker-1"
def test_malformed_rank_override_falls_back_to_builtin_rank():
pool = [
_c("aws-encoding-worker-1", "us-east-2b", "c7a-8xlarge", **AWS),
_c("encoding-worker-a", "us-central1-c", "c4d-highcpu-16"),
]
ranked = ordered_candidates(pool, now=NOW, rank_overrides={"c7a": "first please"})
assert ranked[0]["vm"] == "encoding-worker-a"
def test_aws_cooldown_key_is_firestore_safe():
key = cooldown_key(_c("aws-encoding-worker-1", "us-east-2b", "c7a-8xlarge", **AWS))
assert key == "c7a-8xlarge@us-east-2b"
assert "." not in key
assert SPEED_RANK["c7a"] > SPEED_RANK["c2d"]
+3 -3
View File
@@ -23,12 +23,12 @@ class TestWarmupHandler:
def test_success_passthrough(self):
manager = MagicMock()
manager.ensure_primary_running.return_value = {"started": True, "vm_name": "encoding-worker-a"}
manager.ensure_preferred_running.return_value = {"started": True, "vm_name": "encoding-worker-a"}
assert self._run(manager)["started"] is True
def test_start_error_is_soft(self):
manager = MagicMock()
manager.ensure_primary_running.side_effect = EncodingWorkerStartError("zone exhausted")
manager.ensure_preferred_running.side_effect = EncodingWorkerStartError("zone exhausted")
result = self._run(manager)
assert result["started"] is False
assert "zone exhausted" in result["error"]
@@ -41,7 +41,7 @@ class TestWarmupHandler:
time.sleep(0.5)
return {"started": False}
manager.ensure_primary_running.side_effect = slow_warmup
manager.ensure_preferred_running.side_effect = slow_warmup
async def scenario():
ticks = 0
+14
View File
@@ -733,3 +733,17 @@ class TestVerifyWheelImports:
monkeypatch.setattr(main.subprocess, "run", _timeout)
assert main.verify_wheel_imports() is False
class TestGcsCli:
"""The AWS worker (WIF auth) can't use gsutil; GCE workers keep it."""
def test_gce_worker_uses_gsutil(self, monkeypatch):
from backend.services.gce_encoding import main as worker
monkeypatch.delenv("ENCODING_WORKER_PROVIDER", raising=False)
assert worker._gcs_cli() == ["gsutil"]
def test_aws_worker_uses_gcloud_storage(self, monkeypatch):
from backend.services.gce_encoding import main as worker
monkeypatch.setenv("ENCODING_WORKER_PROVIDER", "aws")
assert worker._gcs_cli() == ["gcloud", "storage"]
+17
View File
@@ -476,6 +476,23 @@ and raised as `EncodingJobLostError`, which `run_with_lost_job_resubmit` resubmi
— the resubmit's connection failure then starts the Spot VM again, or the
on-demand fallback if Spot capacity is gone.
### AWS encoding worker (EC2, provider "aws")
**`aws-encoding-worker-1`** is a c7a.8xlarge persistent Spot instance in us-east-2. It runs the same `gce_encoding` service and is one more entry in the `encoding-worker-fallback-vms` pool (`"provider": "aws"`, plus `instance_id` and `region`). It is paid for by AWS Activate credits. IaC lives in `infrastructure/aws/` (its own Pulumi stack; see its README).
- **Control:**
- The manager routes AWS VMs to `backend/services/aws_encoding_compute.py`, an EC2 adapter that mimics the compute client. EC2 states map to GCE statuses, and EC2 capacity errors join `CAPACITY_ERROR_CODES`.
- So failover, overrides, lost-job detection and stockout cooldowns apply unchanged.
- The override records `active_override_provider: "aws"`. The CI blue-green deploy and the idle Cloud Function both ignore the AWS worker.
- **Ranking:**
- It is a last resort by default (`SPEED_RANK["c7a"] = 90`).
- Firestore `config/encoding-worker.speed_rank_overrides = {"c7a": 5}` makes it preferred with no deploy.
- Both warmups (`ensure_preferred_running`) follow the same ranking, so they warm the worker requests are routed to.
- **Auth and lifecycle:**
- GCP access is Workload Identity Federation as `encoding-worker@`: no keys.
- It stops itself when idle (`infrastructure/encoding-worker/aws/idle-watchdog.py`).
- It gets new code at boot and per job, with no blue-green deploy.
### Architecture
```text
@@ -0,0 +1,91 @@
# AWS encoding worker: implementation plan (2026-10-29)
**Goal:** run gen's encoding worker on EC2 Spot, paid for with the AWS Activate credits (acct 790139457751, expiring 2028-10-31). Then run distribution (Dropbox/YouTube/Drive uploads) from AWS too, so the large output egress moves off the GCP bill. **Constraint:** job→published must not get slower, and the move runs in parallel with GCP first.
Evidence: workspace `docs/archive/2026-10-29-aws-encoding-worker-migration-plan.md`. A c7a.4xlarge matched or beat the GCE c4d-highcpu-16 on 3 replayed prod jobs.
## Phase 1: AWS box as an encoding-worker candidate (this PR set)
The box runs the **unchanged worker HTTP service** (`backend.services.gce_encoding.main`). It joins the existing candidate pool, so failover, overrides, lost-job detection and cooldowns all apply unchanged.
### Design
**Instance**
- `aws-encoding-worker-1`: **c7a.8xlarge** (32 vCPU / 64 GB), us-east-2.
- A **persistent Spot** request with `stop` on interruption, matching GCE's STOP-on-preempt, plus an Elastic IP.
- Port 8080 open with API-key auth, the same as the GCE firewall today.
- The root EBS volume persists across stop/start, like the GCE boot disks. It is provisioned once by user-data (a port of packer `provision.sh`).
- **Why 8xlarge:** ~100 h/mo × ~$0.46/h ≈ $46/mo of *credits* (we accrue ~$205/mo of credits). It encodes faster than today's 16 vCPU and fits two heavy jobs in RAM.
**GCP auth**
- Workload Identity Federation: an `aws-encoding` pool plus an AWS provider for acct 790139457751.
- The `nomad-encoding-worker` role is bound to the existing **encoding-worker SA** (`roles/iam.workloadIdentityUser`). No SA keys.
- The box uses an `external_account` credential config (not secret) with IMDSv2, for both the client libraries and `gcloud`.
**Backend control of EC2**
- An IAM user `nomad-encoding-controller` limited to `ec2:Start/Stop/DescribeInstances` on instances tagged `Project=nomad-encoding`.
- Its keys go in Secret Manager `aws-encoding-controller-credentials`, the same pattern as backup-to-aws; the backend SA can read it.
**Registry**
- The box is an entry in the existing `encoding-worker-fallback-vms` secret:
`{"vm":"aws-encoding-worker-1","zone":"us-east-2b","ip":<EIP>,"machine_type":"c7a-8xlarge","provider":"aws","instance_id":"i-…","region":"us-east-2"}`
- The `.` in the AWS type becomes `-`, so Firestore cooldown keys stay valid.
**Manager**
- `AwsComputeAdapter` (`backend/services/aws_encoding_compute.py`) mimics the `compute_v1.InstancesClient` surface the manager uses (`get/start/stop`). It maps EC2 states to GCE ones (`running→RUNNING`, `pending→STAGING`, `stopping→STOPPING`, `stopped→TERMINATED`), EC2 capacity errors to `CAPACITY_ERROR_CODES`, and `InvalidInstanceID.NotFound` to `NotFound`.
- `EncodingWorkerManager._compute_for(vm)` routes AWS VMs to the adapter. GCE code paths are untouched.
**Ranking**
- `SPEED_RANK["c7a"] = 90`, so by default AWS is a **last-resort** fallback (stage A).
- A new Firestore field `config/encoding-worker.speed_rank_overrides` (e.g. `{"c7a": 5}`) makes it **preferred** (stage C) with no deploy. Rollback deletes the field.
**Warmups**
- `worker_service._warmup_encoding_worker` and `POST /internal/encoding-worker/warmup` currently start only the GCE primary.
- They will use the same ranked path as the encode itself (`manager.ensure_preferred_running()`). Otherwise, with AWS preferred, they would warm a VM that isn't routed to.
**Exclusions from GCE-only machinery**
- CI blue-green green selection: `deploy_promote.select_green_candidates` and the jq in ci.yml.
- The idle Cloud Function's `_parse_fallback_vms`.
- The box gets new code through the existing per-job `ensure_latest_wheel` hot update, so it needs no blue-green deploy.
**Idle stop**
- A systemd timer **on the box**, independent of the uvicorn process, runs every 2 min.
- It stops the instance (`shutdown -h now`, instance-initiated shutdown = stop) when either:
- `/health` reports `active_jobs == 0`, and `last_activity_at` is older than 5 min or the box isn't the routed worker; or
- `/health` has failed for 15 min.
- This mirrors the idle function's rules.
**Worker boot on AWS**
- `startup-aws.sh` has no GCE metadata server. It fetches the API key and wheel with `gcloud` over WIF and sets `ENCODING_WORKER_VM_NAME`.
- `ensure_latest_wheel` uses `gsutil`, which doesn't support WIF credential files. On AWS (`ENCODING_WORKER_PROVIDER=aws`) it uses `gcloud storage`; the GCE path is unchanged.
### Rollout
- **A:** deploy and register the box at rank 90. It's only exercised on c4d+c2d stockouts.
- **B:** set `speed_rank_overrides {"c7a": 5}` for a supervised window. Run the daily E2E plus admin test jobs, and compare render/encode times and outputs (including the portrait parity check) against GCE.
- **C:** leave the override on; AWS is preferred. After a week clean, delete the c4d pair (Pulumi), leaving c2df as the GCP fallback.
### Testing
- **Unit:**
- adapter (botocore `Stubber`): state map, capacity/NotFound mapping, start/stop idempotency
- manager routing (`_compute_for`), `ensure_preferred_running`
- preference `rank_overrides`, `c7a` family
- fallback parsing passthrough of `provider`/`instance_id`/`region`
- deploy_promote and idle-fn exclusion of aws entries
- the idle watchdog decision function
- `ensure_latest_wheel` command selection
- Extend the existing test files listed in the GCE-coupling map.
- **Integration (prod, supervised):** stage B runs real jobs through the AWS worker. The prod daily E2E (payment + happy path) is the end-to-end check.
## Phase 2: distribution from AWS (next PR set, after Phase 1 is verified)
The video worker (`backend.workers.video_worker`) runs **on the box** with local encoding (`LocalEncodingBackend`):
- Inputs come from GCS once (~300 MB).
- Finals are encoded locally, uploaded to Dropbox/YouTube/Drive from AWS, and pushed to GCS `finals/` (AWS egress, on credits).
- This removes the ~$18/mo of Cloud Run internet egress and the video-encoding-job CPU (~$11/mo).
Needs:
- a `/run-worker` endpoint and env/secrets on the box (the video-encoding-job env list);
- a trigger branch in `WorkerService.trigger_video_worker`, with Cloud Run as the fallback;
- the box's WIF identity mapped to the backend SA's permissions;
- **Spot-interruption recovery for in-flight runs.** Today nothing recovers a video worker that dies after `GENERATING_VIDEO`; build a supervised re-dispatch first;
- the deferred YouTube queue processor runs on the API, so it still egresses from GCP (low volume).
+6
View File
@@ -163,6 +163,12 @@ encoding_worker_iam_bindings = worker_sas.grant_encoding_worker_permissions(
encoding_worker_sa, bucket
)
# AWS encoding worker (EC2) → GCP as encoding-worker@ via Workload Identity
# Federation (no keys). See modules/aws_encoding_wif.py.
from modules import aws_encoding_wif # noqa: E402
aws_encoding_wif_resources = aws_encoding_wif.create_aws_encoding_wif(encoding_worker_sa)
gdrive_validator_sa = worker_sas.create_gdrive_validator_service_account()
gdrive_validator_iam_bindings = worker_sas.grant_gdrive_validator_permissions(
gdrive_validator_sa
+2
View File
@@ -0,0 +1,2 @@
config:
aws:region: us-east-2
+6
View File
@@ -0,0 +1,6 @@
name: nomad-aws
runtime:
name: python
options:
virtualenv: venv
description: Nomad Karaoke AWS account (790139457751) — the EC2 encoding worker. See README.md.
+43
View File
@@ -0,0 +1,43 @@
# nomad-aws: Nomad Karaoke's AWS account (790139457751)
This stack holds the **AWS encoding worker**, an EC2 Spot instance running the same `gce_encoding` HTTP service as the GCE encoding VMs. It is paid for by AWS Activate credits ($5k, expiring 2028-10-31). The plan and benchmark are in `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
## What's here
| Resource | Purpose |
|---|---|
| `aws-encoding-worker-1` | c7a.8xlarge **persistent Spot**, us-east-2b. Stops on interruption and on OS shutdown; the EBS root survives. The Elastic IP serves `:8080`. |
| `nomad-encoding-worker` role | The box's identity. GCP federates it to `encoding-worker@nomadkaraoke` (WIF pool `aws-encoding`, in the GCP stack's `modules/aws_encoding_wif.py`). It also has SSM for a shell. |
| `nomad-encoding-controller` user | Backend rights: start/stop instances tagged `Project=nomad-encoding`, plus describe. |
`provision.sh` (in `infrastructure/encoding-worker/aws/`) runs once as user-data. Every boot after that runs `bootstrap.sh` → `startup-aws.sh`, which CI uploads to GCS. The new wheel installs at boot and per job, so there is **no blue-green deploy** for this box. `idle-watchdog.py` stops the box when it's idle.
## Apply
`pulumi up` runs locally and never in CI, with Nomad's AWS keys from the workspace `.envrc`:
```bash
cd infrastructure/aws && python3 -m venv venv && ./venv/bin/pip install -r requirements.txt
AWS_ACCESS_KEY_ID=$NOMAD_AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$NOMAD_AWS_SECRET_ACCESS_KEY \
pulumi up --stack nomadkaraoke/prod
```
`ami` and `user_data` are `ignore_changes`, so a newer Ubuntu or an edited `provision.sh` never replaces the worker's disk. To re-provision on purpose, use `pulumi up --replace <instance urn>`.
## One-time wiring after the first `up`
1. **Controller key → GCP secret** (needs `/gcp-admin`):
```bash
aws iam create-access-key --user-name nomad-encoding-controller \
| jq -c '{access_key_id:.AccessKey.AccessKeyId, secret_access_key:.AccessKey.SecretAccessKey}' \
| scripts/gcp-admin gcloud secrets versions add aws-encoding-controller-credentials --data-file=- --project=nomadkaraoke
```
2. **Register the worker in the pool:** append `pulumi stack output fallback_vms_entry` to the JSON list in secret `encoding-worker-fallback-vms`. The backend picks up `:latest` on new instances, and the next deploy picks it up everywhere.
3. **Rank:** by default the AWS worker is a **last resort** (`SPEED_RANK["c7a"] = 90`). To prefer it, set Firestore `config/encoding-worker.speed_rank_overrides = {"c7a": 5}`. Delete the field to roll back; no deploy is needed either way.
## Operating it
- **Shell:** `aws ssm start-session --target <instance_id> --region us-east-2`.
- **Logs:** `journalctl -u encoding-worker`, `/var/log/encoding-worker-startup.log`, `journalctl -u encoding-worker-idle`. They are **not** in Cloud Logging.
- **Health:** `curl http://<eip>:8080/health`.
- **Spot interruption:** EC2 *stops* the box. An in-flight encode/render is detected as lost (the box is no longer RUNNING) and resubmitted. If c7a capacity is short, the start fails with `InsufficientInstanceCapacity`; that is recorded in `capacity_state`, demoted for 15 min, and GCE takes over.
+137
View File
@@ -0,0 +1,137 @@
"""Nomad Karaoke AWS account: the EC2 encoding worker.
Paid for by the AWS Activate credits ($5k, acct 790139457751, exp 2028-10-31).
The worker runs the same ``gce_encoding`` HTTP service as the GCE encoding VMs
and is one more candidate in the backend's encoding-worker pool (provider
"aws"). Plan: karaoke-gen docs/archive/2026-10-29-aws-encoding-worker-plan.md.
- Persistent Spot instance, ``stop`` on interruption and on OS shutdown (the idle
watchdog), so the EBS root (venv, caches) survives like a GCE boot disk.
- Elastic IP + port 8080 (API-key auth, same as the GCE firewall).
- GCP access via Workload Identity Federation as encoding-worker@ (the
``nomad-encoding-worker`` role is the federated identity; see
infrastructure/modules/aws_encoding_wif.py in the GCP stack).
- ``nomad-encoding-controller`` IAM user: the backend's start/stop/describe
rights, scoped by tag. Its access key is created by hand and stored in GCP
Secret Manager ``aws-encoding-controller-credentials`` (see README).
"""
import json
from pathlib import Path
import pulumi
import pulumi_aws as aws
PROJECT_TAG = {"Project": "nomad-encoding"}
VM_NAME = "aws-encoding-worker-1"
INSTANCE_TYPE = "c7a.8xlarge" # 32 vCPU / 64 GB — see the plan doc for sizing
AZ = "us-east-2b"
WIF_PROVIDER = (
"projects/718638054799/locations/global/workloadIdentityPools/"
"aws-encoding/providers/aws-790139457751"
)
# --- Worker identity (federated into GCP; also SSM for operator shell) ----------
role = aws.iam.Role(
"worker-role",
name="nomad-encoding-worker",
assume_role_policy=json.dumps({
"Version": "2012-10-17",
"Statement": [{"Effect": "Allow", "Principal": {"Service": "ec2.amazonaws.com"}, "Action": "sts:AssumeRole"}],
}),
tags=PROJECT_TAG,
)
aws.iam.RolePolicyAttachment(
"worker-ssm",
role=role.name,
policy_arn="arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
)
profile = aws.iam.InstanceProfile("worker-profile", name="nomad-encoding-worker", role=role.name)
# --- Backend controller (start/stop/describe the tagged worker) -----------------
controller = aws.iam.User("controller", name="nomad-encoding-controller", tags=PROJECT_TAG)
aws.iam.UserPolicy(
"controller-policy",
user=controller.name,
policy=json.dumps({
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["ec2:StartInstances", "ec2:StopInstances"],
"Resource": "arn:aws:ec2:*:790139457751:instance/*",
"Condition": {"StringEquals": {"aws:ResourceTag/Project": "nomad-encoding"}},
},
{"Effect": "Allow", "Action": ["ec2:DescribeInstances"], "Resource": "*"},
],
}),
)
# --- Network ---------------------------------------------------------------------
vpc = aws.ec2.get_vpc(default=True)
subnet = aws.ec2.get_subnet(filters=[
aws.ec2.GetSubnetFilterArgs(name="vpc-id", values=[vpc.id]),
aws.ec2.GetSubnetFilterArgs(name="availability-zone", values=[AZ]),
aws.ec2.GetSubnetFilterArgs(name="default-for-az", values=["true"]),
])
sg = aws.ec2.SecurityGroup(
"worker-sg",
name="nomad-encoding-worker",
description="Encoding worker HTTP (API-key auth), same as the GCE encoding-worker firewall",
vpc_id=vpc.id,
ingress=[aws.ec2.SecurityGroupIngressArgs(protocol="tcp", from_port=8080, to_port=8080, cidr_blocks=["0.0.0.0/0"])],
egress=[aws.ec2.SecurityGroupEgressArgs(protocol="-1", from_port=0, to_port=0, cidr_blocks=["0.0.0.0/0"])],
tags=PROJECT_TAG,
)
# --- Instance --------------------------------------------------------------------
ami = aws.ssm.get_parameter(name="/aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id")
provision = Path(__file__).parent.parent.joinpath("encoding-worker", "aws", "provision.sh").read_text()
user_data = provision.replace(
"set -euxo pipefail\n",
f"set -euxo pipefail\nexport WIF_PROVIDER='{WIF_PROVIDER}' VM_NAME='{VM_NAME}'\n",
1,
)
instance = aws.ec2.Instance(
"worker",
ami=ami.value,
instance_type=INSTANCE_TYPE,
subnet_id=subnet.id,
vpc_security_group_ids=[sg.id],
iam_instance_profile=profile.name,
user_data=user_data,
instance_initiated_shutdown_behavior="stop",
instance_market_options=aws.ec2.InstanceInstanceMarketOptionsArgs(
market_type="spot",
spot_options=aws.ec2.InstanceInstanceMarketOptionsSpotOptionsArgs(
spot_instance_type="persistent",
instance_interruption_behavior="stop",
),
),
metadata_options=aws.ec2.InstanceMetadataOptionsArgs(http_tokens="required", http_put_response_hop_limit=2),
root_block_device=aws.ec2.InstanceRootBlockDeviceArgs(
volume_size=100, volume_type="gp3", iops=6000, throughput=400, delete_on_termination=True,
tags={**PROJECT_TAG, "Name": VM_NAME},
),
tags={**PROJECT_TAG, "Name": VM_NAME},
# The root volume is the worker's state (like a GCE boot disk): a newer
# Ubuntu AMI or an edited provision.sh must never silently replace it.
opts=pulumi.ResourceOptions(ignore_changes=["ami", "user_data"]),
)
eip = aws.ec2.Eip("worker-eip", domain="vpc", tags={**PROJECT_TAG, "Name": VM_NAME})
aws.ec2.EipAssociation("worker-eip-assoc", instance_id=instance.id, allocation_id=eip.id)
pulumi.export("vm_name", VM_NAME)
pulumi.export("instance_id", instance.id)
pulumi.export("public_ip", eip.public_ip)
pulumi.export("availability_zone", instance.availability_zone)
pulumi.export(
"fallback_vms_entry",
pulumi.Output.all(instance.id, eip.public_ip, instance.availability_zone).apply(
lambda a: json.dumps({
"vm": VM_NAME, "zone": a[2], "ip": a[1], "machine_type": INSTANCE_TYPE.replace(".", "-"),
"provider": "aws", "instance_id": a[0], "region": "us-east-2",
})
),
)
+2
View File
@@ -0,0 +1,2 @@
pulumi>=3.0.0,<4.0.0
pulumi-aws>=7.0.0,<8.0.0
@@ -0,0 +1,109 @@
#!/usr/bin/env python3
"""Idle self-stop for the AWS encoding worker (systemd timer, every 2 min).
The GCE workers are stopped by the encoding-worker-idle Cloud Function; the AWS
worker isn't a GCE VM, so it stops itself with the same rules. This runs as its
own systemd unit (not inside uvicorn), so a wedged worker process still gets
stopped:
- busy (``/health`` active_jobs > 0) → keep running
- idle and not the routed worker → stop
- idle, routed, last_activity_at older than 5 min → stop
- health or Firestore unreadable for 15 min → stop (never run forever)
"Stop" = ``shutdown -h now``; the instance's shutdown behaviour is ``stop``, so
the EBS root (venv, caches) survives and the backend restarts it on demand.
CI uploads this to gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/idle-watchdog.py.
"""
from __future__ import annotations
import json
import subprocess
import urllib.request
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Optional
IDLE_TIMEOUT = timedelta(minutes=5)
UNKNOWN_GRACE = timedelta(minutes=15)
STATE_FILE = Path("/var/lib/encoding-worker-idle/unknown-since")
VM_NAME_FILE = Path("/opt/encoding-worker/vm-name")
def decide(
*,
now: datetime,
active_jobs: Optional[int],
config: Optional[dict],
vm_name: str,
unknown_since: Optional[datetime],
) -> tuple[str, str]:
"""Return ("stop" | "keep" | "unknown", reason). Pure, for tests.
``active_jobs``/``config`` are None when /health or Firestore couldn't be
read. "unknown" asks the caller to start (or keep) the grace clock.
"""
if active_jobs is not None and active_jobs > 0:
return "keep", f"{active_jobs} active job(s)"
if active_jobs is None or config is None:
what = "health" if active_jobs is None else "Firestore config"
if unknown_since and now - unknown_since >= UNKNOWN_GRACE:
return "stop", f"{what} unreadable since {unknown_since.isoformat()}"
return "unknown", f"{what} unreadable"
if config.get("active_override_vm") != vm_name:
return "stop", f"idle and not the routed worker (routed: {config.get('active_override_vm')})"
last = config.get("last_activity_at")
try:
last_dt = datetime.fromisoformat(last) if last else None
except (TypeError, ValueError):
last_dt = None
if last_dt is not None and last_dt.tzinfo is None:
last_dt = last_dt.replace(tzinfo=timezone.utc)
if last_dt is None or now - last_dt >= IDLE_TIMEOUT:
return "stop", f"idle since {last or 'never'}"
return "keep", f"routed, last activity {last}"
def _active_jobs() -> Optional[int]:
try:
with urllib.request.urlopen("http://127.0.0.1:8080/health", timeout=10) as r:
return int(json.load(r).get("active_jobs", 0))
except Exception: # noqa: BLE001
return None
def _config() -> Optional[dict]:
try:
from google.cloud import firestore
snap = firestore.Client(project="nomadkaraoke").collection("config").document("encoding-worker").get()
return snap.to_dict() if snap.exists else None
except Exception: # noqa: BLE001
return None
def main() -> None:
now = datetime.now(timezone.utc)
vm_name = VM_NAME_FILE.read_text().strip() if VM_NAME_FILE.exists() else ""
try:
unknown_since = datetime.fromisoformat(STATE_FILE.read_text().strip())
except (OSError, ValueError):
unknown_since = None
action, reason = decide(
now=now, active_jobs=_active_jobs(), config=_config(), vm_name=vm_name, unknown_since=unknown_since
)
print(f"idle-watchdog: {action}: {reason}")
if action == "unknown":
if unknown_since is None:
STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
STATE_FILE.write_text(now.isoformat())
return
STATE_FILE.unlink(missing_ok=True)
if action == "stop":
subprocess.run(["shutdown", "-h", "now"], check=False)
if __name__ == "__main__":
main()
+117
View File
@@ -0,0 +1,117 @@
#!/bin/bash
# One-time provisioning of the AWS encoding worker (EC2 user-data, first boot).
#
# AWS counterpart of infrastructure/packer/scripts/provision.sh: same static
# ffmpeg, fonts, Python 3.13 venv with the full karaoke-gen dependency tree
# (CPU-only torch), and the same systemd unit layout under /opt/encoding-worker
# (self_heal.running_on_worker_vm keys off /opt/encoding-worker/bootstrap.sh).
# The root EBS volume persists across stop/start, so this runs once.
#
# GCP auth: Workload Identity Federation (no keys). The credential config is a
# non-secret JSON pointing at the AWS IMDSv2 role creds; gcloud and the client
# libraries both use it to impersonate encoding-worker@nomadkaraoke.
set -euxo pipefail
exec > >(tee -a /var/log/encoding-worker-provision.log) 2>&1
WORKER_DIR=/opt/encoding-worker
BUCKET=gs://karaoke-gen-storage-nomadkaraoke
WIF_PROVIDER="${WIF_PROVIDER:?}" # projects/<num>/locations/global/workloadIdentityPools/<pool>/providers/<provider>
WORKER_SA="${WORKER_SA:-encoding-worker@nomadkaraoke.iam.gserviceaccount.com}"
VM_NAME="${VM_NAME:-aws-encoding-worker-1}"
export DEBIAN_FRONTEND=noninteractive
apt-get update -q
apt-get install -y -q curl xz-utils unzip git build-essential libffi-dev libssl-dev \
fonts-noto fonts-noto-cjk fontconfig apt-transport-https ca-certificates gnupg
# Google Cloud CLI (gcloud storage / secrets over WIF)
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" \
> /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update -q && apt-get install -y -q google-cloud-cli
# Static FFmpeg 7.x (same source as the GCE image)
curl -fsSL https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ff.tar.xz
tar -xf /tmp/ff.tar.xz -C /tmp && cp /tmp/ffmpeg-*-amd64-static/ff{mpeg,probe} /usr/local/bin/ && rm -rf /tmp/ff*
mkdir -p "${WORKER_DIR}"
gcloud iam workload-identity-pools create-cred-config "${WIF_PROVIDER}" \
--service-account="${WORKER_SA}" --aws --enable-imdsv2 \
--output-file="${WORKER_DIR}/gcp-wif.json"
gcloud auth login --cred-file="${WORKER_DIR}/gcp-wif.json" --quiet
gcloud config set project nomadkaraoke --quiet
export GOOGLE_APPLICATION_CREDENTIALS="${WORKER_DIR}/gcp-wif.json"
# Python 3.13 venv (uv-managed interpreter; --seed so pip exists for startup/ensure_latest_wheel)
curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
UV_PYTHON_INSTALL_DIR=/opt/python uv venv --seed --python 3.13 "${WORKER_DIR}/venv"
VERSION=$(gcloud storage cat "${BUCKET}/encoding-worker/version.txt" | tr -d '[:space:]')
WHEEL="/tmp/karaoke_gen-${VERSION}-py3-none-any.whl"
gcloud storage cp "${BUCKET}/wheels/karaoke_gen-current.whl" "${WHEEL}"
"${WORKER_DIR}/venv/bin/pip" install --quiet fastapi uvicorn google-cloud-storage aiofiles aiohttp packaging
"${WORKER_DIR}/venv/bin/pip" install --quiet \
--index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple "${WHEEL}"
rm -f "${WHEEL}"
# Bootstrap (ExecStartPre): fetch the latest startup-aws.sh from GCS, else the baked copy.
cat > "${WORKER_DIR}/bootstrap.sh" <<'BOOT'
#!/bin/bash
set -e
if gcloud storage cp gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/startup-aws.sh /opt/encoding-worker/startup-aws.sh.new; then
mv /opt/encoding-worker/startup-aws.sh.new /opt/encoding-worker/startup-aws.sh
fi
exec bash /opt/encoding-worker/startup-aws.sh
BOOT
chmod +x "${WORKER_DIR}/bootstrap.sh"
gcloud storage cp "${BUCKET}/encoding-worker/startup-aws.sh" "${WORKER_DIR}/startup-aws.sh"
gcloud storage cp "${BUCKET}/encoding-worker/idle-watchdog.py" "${WORKER_DIR}/idle-watchdog.py"
echo "${VM_NAME}" > "${WORKER_DIR}/vm-name"
cat > /etc/systemd/system/encoding-worker.service <<UNIT
[Unit]
Description=Karaoke encoding worker (AWS)
After=network-online.target
Wants=network-online.target
[Service]
User=root
WorkingDirectory=${WORKER_DIR}
ExecStartPre=${WORKER_DIR}/bootstrap.sh
ExecStart=${WORKER_DIR}/venv/bin/uvicorn backend.services.gce_encoding.main:app --host 0.0.0.0 --port 8080
Restart=always
RestartSec=5
TimeoutStartSec=600
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
EnvironmentFile=${WORKER_DIR}/env
[Install]
WantedBy=multi-user.target
UNIT
cat > /etc/systemd/system/encoding-worker-idle.service <<UNIT
[Unit]
Description=Stop this instance when the encoding worker is idle
[Service]
Type=oneshot
Environment="GOOGLE_APPLICATION_CREDENTIALS=${WORKER_DIR}/gcp-wif.json"
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
ExecStart=${WORKER_DIR}/venv/bin/python ${WORKER_DIR}/idle-watchdog.py
UNIT
cat > /etc/systemd/system/encoding-worker-idle.timer <<UNIT
[Unit]
Description=Encoding worker idle check (every 2 min)
[Timer]
OnBootSec=10min
OnUnitActiveSec=2min
[Install]
WantedBy=timers.target
UNIT
touch "${WORKER_DIR}/env"
systemctl daemon-reload
systemctl enable --now encoding-worker.service encoding-worker-idle.timer
echo "PROVISIONED $(date -u +%FT%TZ)"
+65
View File
@@ -0,0 +1,65 @@
#!/bin/bash
# AWS encoding worker startup (run by bootstrap.sh on every service start).
#
# AWS counterpart of ../startup.sh. Differences: no GCE metadata server (GCP
# auth is Workload Identity Federation via gcp-wif.json + the instance's IMDSv2
# role), `gcloud storage` instead of gsutil (gsutil can't use WIF credential
# files), and the VM name comes from /opt/encoding-worker/vm-name.
#
# CI uploads this to gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/startup-aws.sh
set -e
BUCKET="gs://karaoke-gen-storage-nomadkaraoke"
WORKER_DIR="/opt/encoding-worker"
VENV="${WORKER_DIR}/venv"
TMP_DIR="/tmp/encoding-worker-startup"
export GOOGLE_APPLICATION_CREDENTIALS="${WORKER_DIR}/gcp-wif.json"
exec > >(tee -a /var/log/encoding-worker-startup.log) 2>&1
echo "=== AWS encoding worker startup: $(date) ==="
rm -rf "${TMP_DIR}" && mkdir -p "${TMP_DIR}"
echo "[1/5] Fetching API key from Secret Manager (WIF)..."
ENCODING_API_KEY=""
for attempt in $(seq 1 10); do
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key --project=nomadkaraoke 2>/dev/null || echo "")
[ -n "$ENCODING_API_KEY" ] && break
echo "API key fetch failed (attempt ${attempt}/10), retrying in 5s..."
sleep 5
done
if [ -z "$ENCODING_API_KEY" ]; then
echo "FATAL: No API key from Secret Manager; exiting so systemd retries"
exit 1
fi
VM_NAME=$(cat "${WORKER_DIR}/vm-name" 2>/dev/null || hostname)
cat > "${WORKER_DIR}/env" <<ENV
ENCODING_API_KEY=${ENCODING_API_KEY}
ENCODING_WORKER_VM_NAME=${VM_NAME}
ENCODING_WORKER_PROVIDER=aws
GOOGLE_APPLICATION_CREDENTIALS=${WORKER_DIR}/gcp-wif.json
ENV
chmod 600 "${WORKER_DIR}/env"
# Keep the idle watchdog current with the deployed code (best effort).
gcloud storage cp "${BUCKET}/encoding-worker/idle-watchdog.py" "${WORKER_DIR}/idle-watchdog.py" || true
echo "[2/5] Downloading version manifest..."
EXPECTED_VERSION=$(gcloud storage cat "${BUCKET}/encoding-worker/version.txt" 2>/dev/null | tr -d '[:space:]' || echo "")
echo "Expected version: ${EXPECTED_VERSION:-unknown}"
echo "[3/5] Downloading current wheel..."
WHEEL_PATH="${TMP_DIR}/karaoke_gen-${EXPECTED_VERSION:-0.0.0}-py3-none-any.whl"
gcloud storage cp "${BUCKET}/wheels/karaoke_gen-current.whl" "${WHEEL_PATH}"
echo "[4/5] Installing wheel (code only; deps are baked by provision.sh)..."
"${VENV}/bin/pip" install --upgrade --quiet --force-reinstall --no-deps "${WHEEL_PATH}"
echo "[5/5] Verifying installed version..."
INSTALLED_VERSION=$("${VENV}/bin/pip" show karaoke-gen 2>/dev/null | grep "^Version:" | cut -d' ' -f2 | tr -d '[:space:]')
if [ -n "$EXPECTED_VERSION" ] && [ "${INSTALLED_VERSION}" != "${EXPECTED_VERSION}" ]; then
echo "FATAL: version mismatch (expected ${EXPECTED_VERSION}, installed ${INSTALLED_VERSION})"
exit 1
fi
rm -rf "${TMP_DIR}"
echo "=== Startup complete: karaoke-gen v${INSTALLED_VERSION} on ${VM_NAME} ==="
@@ -85,9 +85,13 @@ def select_green_candidates(
# Only complete entries are usable — a fallback missing vm/zone/ip would
# produce an unstartable candidate (e.g. `gcloud ... --zone=None`) and
# silently drop capacity from the selection path.
# AWS workers (provider "aws") are excluded: this workflow starts/stops
# greens with `gcloud compute`, and the AWS worker hot-updates its wheel per
# job instead of being blue-green deployed.
usable = [
f for f in (fallback_vms or [])
if f.get("vm") and f.get("zone") and f.get("ip") and f.get("vm") != override_vm
and (f.get("provider") or "").lower() != "aws"
]
for f in usable:
pool.append({
@@ -101,6 +105,10 @@ def select_green_candidates(
return ordered_candidates(pool, capacity_state=config.get("capacity_state"))
def _is_aws_override(config: Dict[str, Any]) -> bool:
return (config.get("active_override_provider") or "").lower() == "aws"
def promote_plan(
green: Dict[str, Any],
config: Dict[str, Any],
@@ -152,14 +160,17 @@ def promote_plan(
"ip": config.get("primary_ip"),
"zone": config.get("primary_zone") or DEFAULT_C4D_ZONE,
})
# A stale override must be cleared so active_url follows the fresh primary.
if config.get("active_override_vm"):
# A stale override must be cleared so active_url follows the fresh primary —
# except an AWS override, which is serving by preference (rank override),
# can't be gcloud-stopped, and picks up the new wheel on its next job.
if config.get("active_override_vm") and not _is_aws_override(config):
updates.update({
"active_override_vm": None,
"active_override_ip": None,
"active_override_zone": None,
"active_override_set_at": None,
"active_override_version": None,
"active_override_provider": None,
})
drain.append({
"vm": config["active_override_vm"],
@@ -175,11 +186,14 @@ def promote_plan(
"active_override_zone": green["zone"],
"active_override_set_at": now,
"active_override_version": version,
# The green is a GCE VM: never leave a stale "aws" tag on it (CI hides
# an AWS override from this flow, so it may be the override replaced).
"active_override_provider": None,
"deploy_in_progress": False,
"deploy_in_progress_since": None,
}
old_override = config.get("active_override_vm")
if old_override and old_override != green["vm"]:
if old_override and old_override != green["vm"] and not _is_aws_override(config):
drain.append({
"vm": old_override,
"ip": config.get("active_override_ip"),
@@ -144,6 +144,10 @@ def _parse_fallback_vms():
fallbacks = []
for item in parsed:
try:
if (item.get("provider") or "").lower() == "aws":
# The AWS worker is not a GCE VM; it idle-stops itself
# (infrastructure/encoding-worker/aws/idle-watchdog.py).
continue
fallbacks.append((item["vm"], item["zone"]))
except (KeyError, TypeError) as exc:
logger.warning(f"Skipping malformed fallback entry {item}: {exc}")
@@ -505,6 +505,18 @@ class TestParseFallbackVms:
monkeypatch.setenv("ENCODING_WORKER_FALLBACK_VMS", "not json")
assert main._parse_fallback_vms() == []
def test_skips_aws_worker(self, monkeypatch):
# The AWS encoding worker isn't a GCE VM; it idle-stops itself.
monkeypatch.setenv(
"ENCODING_WORKER_FALLBACK_VMS",
json.dumps([
{"vm": "fb-a", "zone": "us-central1-a", "ip": "1.1.1.1"},
{"vm": "aws-encoding-worker-1", "zone": "us-east-2b", "ip": "3.3.3.3",
"provider": "aws", "instance_id": "i-0abc", "region": "us-east-2"},
]),
)
assert main._parse_fallback_vms() == [("fb-a", "us-central1-a")]
def test_returns_tuples_of_vm_zone(self, monkeypatch):
monkeypatch.setenv(
"ENCODING_WORKER_FALLBACK_VMS",
@@ -0,0 +1,59 @@
"""Workload Identity Federation for the AWS encoding worker.
The AWS encoding worker (EC2 in Nomad's AWS account) authenticates to GCP as
the existing ``encoding-worker`` service account with no keys: the instance's
IAM role ``nomad-encoding-worker`` is exchanged for a short-lived GCP token via
this pool/provider. The box's credential config is generated from the provider
name in infrastructure/encoding-worker/aws/provision.sh.
"""
import pulumi
import pulumi_gcp as gcp
from config import PROJECT_ID
AWS_ACCOUNT_ID = "790139457751"
AWS_WORKER_ROLE = "nomad-encoding-worker"
POOL_ID = "aws-encoding"
PROVIDER_ID = f"aws-{AWS_ACCOUNT_ID}"
def create_aws_encoding_wif(encoding_worker_sa: gcp.serviceaccount.Account) -> dict:
pool = gcp.iam.WorkloadIdentityPool(
"aws-encoding-wif-pool",
workload_identity_pool_id=POOL_ID,
display_name="AWS encoding worker",
description="EC2 encoding worker in Nomad's AWS account (karaoke-gen)",
)
provider = gcp.iam.WorkloadIdentityPoolProvider(
"aws-encoding-wif-provider",
workload_identity_pool_id=pool.workload_identity_pool_id,
workload_identity_pool_provider_id=PROVIDER_ID,
display_name="Nomad AWS account",
aws=gcp.iam.WorkloadIdentityPoolProviderAwsArgs(account_id=AWS_ACCOUNT_ID),
attribute_mapping={
"google.subject": "assertion.arn",
"attribute.aws_role": (
"assertion.arn.contains('assumed-role') ? "
"assertion.arn.extract('{account_arn}assumed-role/') + 'assumed-role/' + "
"assertion.arn.extract('assumed-role/{role_name}/') : assertion.arn"
),
},
# Only the worker's role in Nomad's account may federate.
attribute_condition=(
f"attribute.aws_role == 'arn:aws:sts::{AWS_ACCOUNT_ID}:assumed-role/{AWS_WORKER_ROLE}'"
),
)
binding = gcp.serviceaccount.IAMMember(
"aws-encoding-worker-wif-user",
service_account_id=encoding_worker_sa.name,
role="roles/iam.workloadIdentityUser",
member=pool.name.apply(
lambda name: (
f"principalSet://iam.googleapis.com/{name}/attribute.aws_role/"
f"arn:aws:sts::{AWS_ACCOUNT_ID}:assumed-role/{AWS_WORKER_ROLE}"
)
),
)
pulumi.export("aws_encoding_wif_provider", provider.name)
return {"pool": pool, "provider": provider, "binding": binding}
+4
View File
@@ -50,6 +50,10 @@ def create_secrets() -> dict[str, secretmanager.Secret]:
# comma-delimited and the JSON value contains commas; the value itself is
# non-secret (just public IPs of fallback VMs).
"encoding-worker-fallback-vms",
# AWS encoding worker: IAM user keys ({"access_key_id","secret_access_key"})
# the backend uses to start/stop/describe the EC2 worker
# (backend/services/aws_encoding_compute.py). Version added by hand.
"aws-encoding-controller-credentials",
# Notifications
"pushbullet-api-key",
Generated
+71 -1
View File
@@ -660,6 +660,46 @@ files = [
[package.dependencies]
numpy = {version = ">=1.19.0,<3.0.0", markers = "python_version >= \"3.9\""}
[[package]]
name = "boto3"
version = "1.43.122"
description = "The AWS SDK for Python (Boto3)"
optional = false
python-versions = ">=3.10"
groups = ["main"]
files = [
{file = "boto3-1.43.122-py3-none-any.whl", hash = "sha256:cca51b3650080594a1df344b8a2e76007eab6d5fa50504a0b622dfdc14cdeaa1"},
{file = "boto3-1.43.122.tar.gz", hash = "sha256:c923650d09e4c445564ca14d5ce89466c3083884b6f6c189f0f445fc17f02eb5"},
]
[package.dependencies]
botocore = ">=1.43.122,<1.44.0"
jmespath = ">=0.7.1,<2.0.0"
s3transfer = ">=0.19.0,<0.20.0"
[package.extras]
crt = ["botocore[crt] (>=1.21.0,<2.0a0)"]
[[package]]
name = "botocore"
version = "1.43.122"
description = "Low-level, data-driven core of boto 3."
optional = false
python-versions = ">=3.10"
groups = ["main"]
files = [
{file = "botocore-1.43.122-py3-none-any.whl", hash = "sha256:668e608edf8847fc4c3c6b38688e183c3bcf19b8a1f7f58d33c03a6f3039bd49"},
{file = "botocore-1.43.122.tar.gz", hash = "sha256:2fc53b68b35314b1738cad46e6170d0556496863913451428fc3afccb8c7fa95"},
]
[package.dependencies]
jmespath = ">=0.7.1,<2.0.0"
python-dateutil = ">=2.1,<3.0.0"
urllib3 = ">=1.25.4,<2.2.0 || >2.2.0,<3"
[package.extras]
crt = ["awscrt (==0.38.0)"]
[[package]]
name = "build"
version = "1.3.0"
@@ -3579,6 +3619,18 @@ files = [
click = ">=8.1.8"
rapidfuzz = ">=3.9.7"
[[package]]
name = "jmespath"
version = "1.1.0"
description = "JSON Matching Expressions"
optional = false
python-versions = ">=3.9"
groups = ["main"]
files = [
{file = "jmespath-1.1.0-py3-none-any.whl", hash = "sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64"},
{file = "jmespath-1.1.0.tar.gz", hash = "sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d"},
]
[[package]]
name = "joblib"
version = "1.5.3"
@@ -7979,6 +8031,24 @@ oletools = ">=0.56"
dev = ["coverage (>=7.2.2,<7.3.0)", "lxml (>=6.0,<7.0)", "mypy (>=1.1,<2.0)", "pdoc3 (>=0.10.0,<0.11.0)"]
msg-parse = ["extract_msg (>=0.27,<1.0)"]
[[package]]
name = "s3transfer"
version = "0.19.3"
description = "An Amazon S3 Transfer Manager"
optional = false
python-versions = ">=3.10"
groups = ["main"]
files = [
{file = "s3transfer-0.19.3-py3-none-any.whl", hash = "sha256:24eb7535804306355e31c952a90022bf5b2f9d760f9a11062c7493a217c08553"},
{file = "s3transfer-0.19.3.tar.gz", hash = "sha256:8326acc3d056d7f7b8a37110c7d2aea41f67e83f6ae79cd3b5fbcca34c726406"},
]
[package.dependencies]
botocore = ">=1.37.4,<2.0a.0"
[package.extras]
crt = ["botocore[crt] (>=1.37.4,<2.0a.0)"]
[[package]]
name = "safetensors"
version = "0.7.0"
@@ -10738,4 +10808,4 @@ local-whisper = ["whisper-timestamped"]
[metadata]
lock-version = "2.1"
python-versions = ">=3.10,<3.14"
content-hash = "ea37b6f69b799819b84da84381a2e2971bbcac8d409c375d94a26fa8f230696b"
content-hash = "59291df081a5b57850eefee645e27751b7fd78491e517c5eb180a82c43e2c219"
+3 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.287.2"
version = "0.288.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
@@ -98,6 +98,8 @@ httpx = ">=0.25.0"
tenacity = "^8.2.0" # Retry logic for external service calls
google-cloud-firestore = "2.26.0"
google-cloud-compute = "^1.0.0"
# AWS encoding worker control (EC2 start/stop/describe) — backend/services/aws_encoding_compute.py
boto3 = "^1.34"
google-cloud-storage = "3.10.1"
google-cloud-secret-manager = "2.26.0"
google-cloud-monitoring = "2.29.1" # Cloud Monitoring for YouTube API quota metrics