mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
118 lines
5.0 KiB
Bash
Executable File
118 lines
5.0 KiB
Bash
Executable File
#!/bin/bash
|
|
# One-time provisioning of the AWS encoding worker (EC2 user-data, first boot).
|
|
#
|
|
# AWS counterpart of infrastructure/packer/scripts/provision.sh: same static
|
|
# ffmpeg, fonts, Python 3.13 venv with the full karaoke-gen dependency tree
|
|
# (CPU-only torch), and the same systemd unit layout under /opt/encoding-worker
|
|
# (self_heal.running_on_worker_vm keys off /opt/encoding-worker/bootstrap.sh).
|
|
# The root EBS volume persists across stop/start, so this runs once.
|
|
#
|
|
# GCP auth: Workload Identity Federation (no keys). The credential config is a
|
|
# non-secret JSON pointing at the AWS IMDSv2 role creds; gcloud and the client
|
|
# libraries both use it to impersonate encoding-worker@nomadkaraoke.
|
|
set -euxo pipefail
|
|
exec > >(tee -a /var/log/encoding-worker-provision.log) 2>&1
|
|
|
|
WORKER_DIR=/opt/encoding-worker
|
|
BUCKET=gs://karaoke-gen-storage-nomadkaraoke
|
|
WIF_PROVIDER="${WIF_PROVIDER:?}" # projects/<num>/locations/global/workloadIdentityPools/<pool>/providers/<provider>
|
|
WORKER_SA="${WORKER_SA:-encoding-worker@nomadkaraoke.iam.gserviceaccount.com}"
|
|
VM_NAME="${VM_NAME:-aws-encoding-worker-1}"
|
|
|
|
export DEBIAN_FRONTEND=noninteractive
|
|
apt-get update -q
|
|
apt-get install -y -q curl xz-utils unzip git build-essential libffi-dev libssl-dev \
|
|
fonts-noto fonts-noto-cjk fontconfig apt-transport-https ca-certificates gnupg
|
|
|
|
# Google Cloud CLI (gcloud storage / secrets over WIF)
|
|
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
|
|
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" \
|
|
> /etc/apt/sources.list.d/google-cloud-sdk.list
|
|
apt-get update -q && apt-get install -y -q google-cloud-cli
|
|
|
|
# Static FFmpeg 7.x (same source as the GCE image)
|
|
curl -fsSL https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ff.tar.xz
|
|
tar -xf /tmp/ff.tar.xz -C /tmp && cp /tmp/ffmpeg-*-amd64-static/ff{mpeg,probe} /usr/local/bin/ && rm -rf /tmp/ff*
|
|
|
|
mkdir -p "${WORKER_DIR}"
|
|
gcloud iam workload-identity-pools create-cred-config "${WIF_PROVIDER}" \
|
|
--service-account="${WORKER_SA}" --aws --enable-imdsv2 \
|
|
--output-file="${WORKER_DIR}/gcp-wif.json"
|
|
gcloud auth login --cred-file="${WORKER_DIR}/gcp-wif.json" --quiet
|
|
gcloud config set project nomadkaraoke --quiet
|
|
export GOOGLE_APPLICATION_CREDENTIALS="${WORKER_DIR}/gcp-wif.json"
|
|
|
|
# Python 3.13 venv (uv-managed interpreter; --seed so pip exists for startup/ensure_latest_wheel)
|
|
curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
|
|
UV_PYTHON_INSTALL_DIR=/opt/python uv venv --seed --python 3.13 "${WORKER_DIR}/venv"
|
|
VERSION=$(gcloud storage cat "${BUCKET}/encoding-worker/version.txt" | tr -d '[:space:]')
|
|
WHEEL="/tmp/karaoke_gen-${VERSION}-py3-none-any.whl"
|
|
gcloud storage cp "${BUCKET}/wheels/karaoke_gen-current.whl" "${WHEEL}"
|
|
"${WORKER_DIR}/venv/bin/pip" install --quiet fastapi uvicorn google-cloud-storage aiofiles aiohttp packaging
|
|
"${WORKER_DIR}/venv/bin/pip" install --quiet \
|
|
--index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple "${WHEEL}"
|
|
rm -f "${WHEEL}"
|
|
|
|
# Bootstrap (ExecStartPre): fetch the latest startup-aws.sh from GCS, else the baked copy.
|
|
cat > "${WORKER_DIR}/bootstrap.sh" <<'BOOT'
|
|
#!/bin/bash
|
|
set -e
|
|
if gcloud storage cp gs://karaoke-gen-storage-nomadkaraoke/encoding-worker/startup-aws.sh /opt/encoding-worker/startup-aws.sh.new; then
|
|
mv /opt/encoding-worker/startup-aws.sh.new /opt/encoding-worker/startup-aws.sh
|
|
fi
|
|
exec bash /opt/encoding-worker/startup-aws.sh
|
|
BOOT
|
|
chmod +x "${WORKER_DIR}/bootstrap.sh"
|
|
gcloud storage cp "${BUCKET}/encoding-worker/startup-aws.sh" "${WORKER_DIR}/startup-aws.sh"
|
|
gcloud storage cp "${BUCKET}/encoding-worker/idle-watchdog.py" "${WORKER_DIR}/idle-watchdog.py"
|
|
echo "${VM_NAME}" > "${WORKER_DIR}/vm-name"
|
|
|
|
cat > /etc/systemd/system/encoding-worker.service <<UNIT
|
|
[Unit]
|
|
Description=Karaoke encoding worker (AWS)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
User=root
|
|
WorkingDirectory=${WORKER_DIR}
|
|
ExecStartPre=${WORKER_DIR}/bootstrap.sh
|
|
ExecStart=${WORKER_DIR}/venv/bin/uvicorn backend.services.gce_encoding.main:app --host 0.0.0.0 --port 8080
|
|
Restart=always
|
|
RestartSec=5
|
|
TimeoutStartSec=600
|
|
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
|
|
EnvironmentFile=${WORKER_DIR}/env
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
UNIT
|
|
|
|
cat > /etc/systemd/system/encoding-worker-idle.service <<UNIT
|
|
[Unit]
|
|
Description=Stop this instance when the encoding worker is idle
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
Environment="GOOGLE_APPLICATION_CREDENTIALS=${WORKER_DIR}/gcp-wif.json"
|
|
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
|
|
ExecStart=${WORKER_DIR}/venv/bin/python ${WORKER_DIR}/idle-watchdog.py
|
|
UNIT
|
|
|
|
cat > /etc/systemd/system/encoding-worker-idle.timer <<UNIT
|
|
[Unit]
|
|
Description=Encoding worker idle check (every 2 min)
|
|
|
|
[Timer]
|
|
OnBootSec=10min
|
|
OnUnitActiveSec=2min
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|
|
UNIT
|
|
|
|
touch "${WORKER_DIR}/env"
|
|
systemctl daemon-reload
|
|
systemctl enable --now encoding-worker.service encoding-worker-idle.timer
|
|
echo "PROVISIONED $(date -u +%FT%TZ)"
|