Files
karaoke-gen/infrastructure/compute/encoding_worker_vm.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 a1cbe1c0d4 feat(encoding): retire the GCE c4d pair; run on AWS + c2df only (0.293.0) (#1255)
## Why
Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has
handled every encode and distribution job. 27 jobs went to it, with no
Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related
in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs
~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP
fallback.

## What
Deleting the pair needs code changes first, because several paths
assumed it exists:
- **`encoding_worker_manager`**
  - The primary is optional: an empty `primary_vm` means none.
- `ranked_candidates` skips a missing primary and always sets
`is_primary`, so with no primary a successful start records the AWS/c2df
override. Previously the legacy index-0 rule would have cleared it.
- `ensure_primary_running` and `ensure_preferred_running` start a ranked
candidate. If there are no candidates they raise `NO_CANDIDATES`.
- `demote_active_worker` looks up the override's `machine_type`, so the
AWS box can now be cooled down. It then clears the override so the retry
goes elsewhere.
- **`encoding_service`**
  - When `active_url` is empty, it targets the top-ranked candidate.
- It also invalidates its cached URL after a single-candidate fallback.
- **CI deploy (`ci.yml` + `deploy_promote.py`)**
- While AWS serves, a c2df green is validate-only: it writes
`validated_version`, keeps the AWS override, and stops the green. Before
this, it would have replaced the AWS override with c2df.
- The worker-change gate now diffs against `validated_version`, because
`primary_version` stops moving.
- An in-place restart of a serving override first waits for it to drain
(15-min cap). If it is still busy, the restart is skipped.
- **Idle function:** a missing VM is logged as a WARNING returning
`NOT_FOUND`, not an ERROR.
- **Error monitor:** watches c2df instead of a/b.
- **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows
exactly 4 deletes. A separate, unrelated
`youtube-comment-autoreply-scheduler` drift shows up too; it will not be
applied.
- **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING.

## Rollout (after merge + deploy)
1. Back up `config/encoding-worker`, then set `primary_vm`,
`primary_ip`, `secondary_vm` and `secondary_ip` to `""`.
2. Apply `pulumi destroy --target` on the 4 a/b URNs only.
3. Redeploy the idle function with `deploy.sh`.
4. Verify: one prod encode on AWS, warmup, and the logs.

**Rollback:** in Firestore, delete `speed_rank_overrides` and set
`aws_video_worker=false`. Restoring the pair would need a revert plus
`pulumi up`, which gives new static IPs.

## Testing
- New `backend/tests/test_encoding_worker_no_primary.py` (13 tests),
plus new deploy_promote, idle-function and error-monitor tests.
- Full backend suite: 7188 passed. The 8 `test_emulator_integration`
failures also fail on main, because of a local emulator from another
session.
- CodeRabbit CLI: no findings. An independent review found 2 issues,
both fixed: the demote retry path, and the in-place restart drain.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe

@coderabbitai ignore

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 18:50:54 -05:00

121 lines
4.9 KiB
Python

"""
Encoding Worker VM resources — the GCE capacity-fallback fleet.
The c4d Spot blue-green pair (encoding-worker-a/-b) was retired 2026-11; the AWS
EC2 worker (infrastructure/aws/) is now the preferred encoder. What remains here
is the on-demand fallback (c2df), provisioned stopped and started on demand.
History: docs/superpowers/specs/2026-03-24-blue-green-encoding-worker-design.md
"""
import pulumi
from pulumi_gcp import compute, serviceaccount
from config import REGION, PROJECT_ID, MachineTypes, DiskSizes, EncodingWorkerConfig
from .startup_scripts import read_script
def create_encoding_worker_fallback_ips() -> list[compute.Address]:
"""Create static IPs for the capacity-fallback VMs.
These IPs back the on-demand fallback that absorbs a Spot/c4d stockout of
the primary pair (alternate zone AND machine family). One IP per entry in
EncodingWorkerConfig.FALLBACKS (order-aligned).
"""
ips = []
for fb in EncodingWorkerConfig.FALLBACKS:
ip_name = EncodingWorkerConfig.fallback_ip_name(fb["suffix"])
ip = compute.Address(
ip_name,
name=ip_name,
region=REGION,
address_type="EXTERNAL",
# Keep this string byte-identical to the originally-deployed value so
# `pulumi up` does not diff (and needlessly churn) the existing IPs.
description=f"Static external IP for {ip_name} (zone-fallback)",
)
ips.append(ip)
return ips
def create_encoding_worker_fallback_vms(
ips: list[compute.Address],
service_account: serviceaccount.Account,
) -> list[compute.Instance]:
"""Create capacity-fallback encoding worker VMs.
Provisioned stopped — only started by the application when the Spot primary
pair can't start (ZONE_RESOURCE_POOL_EXHAUSTED / Spot stockout). On-demand
(default STANDARD scheduling — never preempted), in a different machine
family and zone from the c4d pair (incident 2026-08-12). Each entry's
machine_type and disk_type come from EncodingWorkerConfig.FALLBACKS — c2d/n2
use pd-balanced because they don't support hyperdisk-balanced. Cost when
stopped is just the 50 GB boot disk.
"""
startup_script = read_script("encoding_worker.sh")
custom_image = f"projects/{PROJECT_ID}/global/images/family/encoding-worker"
vms = []
for fb, ip in zip(EncodingWorkerConfig.FALLBACKS, ips):
vm_name = EncodingWorkerConfig.fallback_vm_name(fb["suffix"])
zone = f"{REGION}-{fb['zone_suffix']}"
# Hyperdisk provisioned IOPS/throughput are deliberately NOT set: setting
# them forces a full VM replacement (boot-disk init params are immutable),
# and recreating a c4d VM depends on the exact capacity the fallback fleet
# exists to absorb. hyperdisk-balanced disks are pinned to the free
# baseline (3000 IOPS / 140 MB/s) live via `gcloud compute disks update`
# after any image rebuild that recreates them; pd-balanced (n2) needs no
# such tuning. See docs/archive/2026-06-14-gcp-cost-reduction-plan.md.
vm = compute.Instance(
vm_name,
name=vm_name,
machine_type=fb["machine_type"],
zone=zone,
boot_disk=compute.InstanceBootDiskArgs(
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
image=custom_image,
size=DiskSizes.ENCODING_WORKER,
type=fb["disk_type"],
),
),
network_interfaces=[compute.InstanceNetworkInterfaceArgs(
network="default",
access_configs=[compute.InstanceNetworkInterfaceAccessConfigArgs(
nat_ip=ip.address,
)],
)],
service_account=compute.InstanceServiceAccountArgs(
email=service_account.email,
scopes=["cloud-platform"],
),
metadata_startup_script=startup_script,
tags=["encoding-worker"],
allow_stopping_for_update=True,
advanced_machine_features=compute.InstanceAdvancedMachineFeaturesArgs(
threads_per_core=2,
),
# Pulumi default is "running" — we want fallbacks stopped at create.
desired_status="TERMINATED",
)
vms.append(vm)
return vms
def create_encoding_worker_firewall() -> compute.Firewall:
"""Create firewall rules for encoding worker VMs.
Opens port 8080 for the HTTP API. Both VMs share the same
firewall rule via the 'encoding-worker' network tag.
"""
return compute.Firewall(
"encoding-worker-firewall",
name="encoding-worker-allow-http",
network="default",
allows=[
compute.FirewallAllowArgs(protocol="tcp", ports=["8080"]),
],
source_ranges=["0.0.0.0/0"],
target_tags=["encoding-worker"],
description="Allow HTTP access to encoding workers (auth required)",
)