Files
karaoke-gen/infrastructure/compute/encoding_worker_vm.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 87a36bd2a5 feat(encoding): GCP cost cuts — Spot 16-vCPU workers, 3-VM fleet, 5-min idle, gated worker deploys (v0.245.0) (#1061)
## Summary
Approved GCP cost cuts for the encoding pipeline (target: whole project
< $300/mo; encoding VMs were ~$300/mo for ~5-20 jobs/day). Details +
before/after table: `docs/archive/2026-09-26-encoding-cost-cuts.md`.

| | Before | After |
|---|---|---|
| Primary pair a/b | c4d-highcpu-32 on-demand | **c4d-highcpu-16 Spot**
(STOP on preemption) |
| Fallbacks | 8 VMs / 6 families | **1 on-demand**
`encoding-worker-fallback-c2df` (c2d-highcpu-16, us-central1-f) |
| Idle shutdown | 15 min / check every 5 | **5 min / check every 2** |
| CI worker blue-green | every push to main | only when
`infrastructure/encoding-worker/worker_code_paths.txt` paths changed
since the serving version (+ `workflow_dispatch`
`force_encoding_worker_deploy`) |
| Nomad Dropbox upload | whole folder | minus lossless 4K **MP4**
(tenant folders unchanged) |

**Infra already applied locally (targeted Pulumi):** 7 fallback VMs +
disks + 7 static IPs deleted; a/b + c2df resized in place; idle function
env + scheduler updated. The Spot switch was done in place with `gcloud
compute instances set-scheduling` on the stopped VMs + `pulumi refresh`
(pulumi-gcp treats a `scheduling` diff as *replace*, and recreating a
c4d VM risks a stockout). `encoding-worker-fallback-vms` secret → v6
(c2df only). CI's pulumi run should be a no-op for these resources.

### Scope change (per Andrew)
**Task 5 dropped:** review previews stay on the encoding VM
(`USE_GCE_PREVIEW_ENCODING=true` unchanged) and the review-page warmup
is kept, since it deliberately warms the VM for the preview and the
final render. To make sure the new 5-min idle timeout doesn't undermine
that, the review page now also heartbeats **every 2 min** while the tab
is visible and was used in the last 15 min, and re-warms on tab refocus
(`useEncodingWorkerKeepAlive`). Before this, the heartbeat only fired on
lyric edits. The idle checker keeps the routed VM alive while
`last_activity_at` < 5 min, and both warmup and heartbeat write it.
Tests cover both sides.

### Code changes
- **Spot preemption mid-encode**: when a poll fails,
`wait_for_completion` checks the pinned worker VM. If it is
STOPPING/TERMINATED it raises `EncodingJobLostError`, so
`run_with_lost_job_resubmit` resubmits the encode, and that resubmit
restarts the Spot VM or falls back to c2df. Before, the failure showed
up after ~7 min as a "lost contact" error that was not resubmitted.
- **Stale fallback lists**: a candidate VM that no longer exists
(`NotFound`) is skipped instead of aborting the whole failover. This
protects the window before running instances pick up the new secret.
- `SPEED_RANK` is now keyed by machine family, so the ranking is
independent of VM size.
- **Idle-stop vs. in-flight encode** (review follow-up): the idle
function's `/health active_jobs` only counts RUNNING jobs, and
`last_activity_at` was only written when a VM was started. With a 5-min
window, a long encode's VM could be stopped in the seconds between the
job finishing and the next poll. The poll loop now refreshes
`last_activity_at` at most once a minute. A VM seen in
STAGING/PROVISIONING after it accepted a job also counts as lost. CI
`Read Firestore config` now writes empty values instead of "None".
- **Worker deploy gate**:
`backend/tests/test_encoding_worker_code_paths.py` derives the worker's
transitive `backend.*` import set and fails if `worker_code_paths.txt`
doesn't cover it. Every deploy still uploads the wheel and `startup.sh`,
so a cold-booted VM always runs the latest code.
- **Dropbox**: nothing automated reads lossless files from Dropbox. The
lossless **MKV** is still uploaded because the completion email and the
live Fiverr auto-reply bot both tell customers it is in the Dropbox
folder, and the VocalStar tenant expects lossless files. To drop the MKV
as well, update those two texts first, then add the MKV suffix to
`DROPBOX_SKIP_OUTPUT_SUFFIXES`.

## Testing
- New/updated unit tests: preemption → lost job → resubmit, NotFound
candidate skip, family ranking, idle 5-min/heartbeat window, fleet-shape
invariants, worker-path coverage, Dropbox exclusion, keep-alive hook
(Jest, fake timers).
- `pytest backend/tests --ignore=backend/tests/emulator`: 4802 passed.
The only failures are 6 tests that need the spaCy `en_core_web_sm`
model, which isn't installed in my local venv; unrelated to this PR.
- Jest lyrics-review + lib/lyrics-review: 423 passed. Infra tests
(`test_encoding_worker_config.py`, idle function): all pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 23:20:54 -04:00

218 lines
9.0 KiB
Python

"""
Encoding Worker VM resources — blue-green deployment pair.
Manages two identical VMs (a/b) for zero-downtime deployments.
Only one is active (primary) at a time; the other is stopped (secondary).
Both auto-shutdown when idle to minimize cost, and both run on SPOT capacity
(preempted → STOPPED, restarted on demand; lost encodes are resubmitted). One
on-demand fallback VM in a different family/zone absorbs Spot stockouts.
See docs/superpowers/specs/2026-03-24-blue-green-encoding-worker-design.md
"""
import pulumi
from pulumi_gcp import compute, serviceaccount
from config import REGION, ENCODING_WORKER_ZONE, PROJECT_ID, MachineTypes, DiskSizes, EncodingWorkerConfig
from .startup_scripts import read_script
def create_encoding_worker_ips() -> list[compute.Address]:
"""Create static IPs for both encoding worker VMs."""
ips = []
for name in EncodingWorkerConfig.IP_NAMES:
ip = compute.Address(
name,
name=name,
region=REGION,
address_type="EXTERNAL",
description=f"Static external IP for {name}",
)
ips.append(ip)
return ips
def create_encoding_worker_vms(
ips: list[compute.Address],
service_account: serviceaccount.Account,
) -> list[compute.Instance]:
"""Create the blue-green encoding worker VM pair."""
startup_script = read_script("encoding_worker.sh")
custom_image = f"projects/{PROJECT_ID}/global/images/family/encoding-worker"
vms = []
for vm_name, ip in zip(EncodingWorkerConfig.VM_NAMES, ips):
vm = compute.Instance(
vm_name,
name=vm_name,
machine_type=MachineTypes.ENCODING_WORKER,
zone=ENCODING_WORKER_ZONE,
boot_disk=compute.InstanceBootDiskArgs(
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
image=custom_image,
size=DiskSizes.ENCODING_WORKER,
type="hyperdisk-balanced",
# NOTE: Hyperdisk provisioned IOPS/throughput are deliberately
# NOT set here. Specifying them forces a full VM *replacement*
# (the provider treats boot-disk init params as immutable), and
# recreating a primary depends on us-central1-c capacity — the
# exact shortage the fallback fleet exists to absorb. So the
# disks are pinned to the Hyperdisk Balanced free baseline
# (3000 IOPS / 140 MB/s) live via `gcloud compute disks update`
# instead — non-disruptive, and pulumi does not manage this
# attribute so there is no drift. They were over-provisioned at
# 3600 IOPS / 290 MB/s (~$8.70/disk/mo wasted). Re-apply the
# gcloud tune after any worker-image rebuild that recreates the
# disks. See docs/archive/2026-06-14-gcp-cost-reduction-plan.md.
),
),
network_interfaces=[compute.InstanceNetworkInterfaceArgs(
network="default",
access_configs=[compute.InstanceNetworkInterfaceAccessConfigArgs(
nat_ip=ip.address,
)],
)],
service_account=compute.InstanceServiceAccountArgs(
email=service_account.email,
scopes=["cloud-platform"],
),
metadata_startup_script=startup_script,
tags=["encoding-worker"],
allow_stopping_for_update=True,
advanced_machine_features=compute.InstanceAdvancedMachineFeaturesArgs(
threads_per_core=2,
),
scheduling=_primary_pair_scheduling(),
)
vms.append(vm)
return vms
def _primary_pair_scheduling() -> compute.InstanceSchedulingArgs:
"""Scheduling for the a/b pair: Spot with STOP-on-preemption.
STOP (not DELETE) keeps the VM, its disk and static IP, so a preempted
worker is just a TERMINATED instance that the backend's ensure-running path
restarts like any idle-stopped worker. automatic_restart must be False and
on_host_maintenance TERMINATE for Spot.
"""
if not EncodingWorkerConfig.PRIMARY_PAIR_SPOT:
return compute.InstanceSchedulingArgs(
provisioning_model="STANDARD",
preemptible=False,
automatic_restart=True,
on_host_maintenance="MIGRATE",
)
return compute.InstanceSchedulingArgs(
provisioning_model="SPOT",
preemptible=True,
automatic_restart=False,
on_host_maintenance="TERMINATE",
instance_termination_action="STOP",
)
def create_encoding_worker_fallback_ips() -> list[compute.Address]:
"""Create static IPs for the capacity-fallback VMs.
These IPs back the on-demand fallback that absorbs a Spot/c4d stockout of
the primary pair (alternate zone AND machine family). One IP per entry in
EncodingWorkerConfig.FALLBACKS (order-aligned).
"""
ips = []
for fb in EncodingWorkerConfig.FALLBACKS:
ip_name = EncodingWorkerConfig.fallback_ip_name(fb["suffix"])
ip = compute.Address(
ip_name,
name=ip_name,
region=REGION,
address_type="EXTERNAL",
# Keep this string byte-identical to the originally-deployed value so
# `pulumi up` does not diff (and needlessly churn) the existing a/b IPs.
description=f"Static external IP for {ip_name} (zone-fallback)",
)
ips.append(ip)
return ips
def create_encoding_worker_fallback_vms(
ips: list[compute.Address],
service_account: serviceaccount.Account,
) -> list[compute.Instance]:
"""Create capacity-fallback encoding worker VMs.
Provisioned stopped — only started by the application when the Spot primary
pair can't start (ZONE_RESOURCE_POOL_EXHAUSTED / Spot stockout). On-demand
(default STANDARD scheduling — never preempted), in a different machine
family and zone from the c4d pair (incident 2026-08-12). Each entry's
machine_type and disk_type come from EncodingWorkerConfig.FALLBACKS — c2d/n2
use pd-balanced because they don't support hyperdisk-balanced. Cost when
stopped is just the 50 GB boot disk.
"""
startup_script = read_script("encoding_worker.sh")
custom_image = f"projects/{PROJECT_ID}/global/images/family/encoding-worker"
vms = []
for fb, ip in zip(EncodingWorkerConfig.FALLBACKS, ips):
vm_name = EncodingWorkerConfig.fallback_vm_name(fb["suffix"])
zone = f"{REGION}-{fb['zone_suffix']}"
# Hyperdisk provisioned IOPS/throughput are deliberately NOT set: setting
# them forces a full VM replacement (boot-disk init params are immutable),
# and recreating a c4d VM depends on the exact capacity the fallback fleet
# exists to absorb. hyperdisk-balanced disks are pinned to the free
# baseline (3000 IOPS / 140 MB/s) live via `gcloud compute disks update`
# after any image rebuild that recreates them; pd-balanced (n2) needs no
# such tuning. See docs/archive/2026-06-14-gcp-cost-reduction-plan.md.
vm = compute.Instance(
vm_name,
name=vm_name,
machine_type=fb["machine_type"],
zone=zone,
boot_disk=compute.InstanceBootDiskArgs(
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
image=custom_image,
size=DiskSizes.ENCODING_WORKER,
type=fb["disk_type"],
),
),
network_interfaces=[compute.InstanceNetworkInterfaceArgs(
network="default",
access_configs=[compute.InstanceNetworkInterfaceAccessConfigArgs(
nat_ip=ip.address,
)],
)],
service_account=compute.InstanceServiceAccountArgs(
email=service_account.email,
scopes=["cloud-platform"],
),
metadata_startup_script=startup_script,
tags=["encoding-worker"],
allow_stopping_for_update=True,
advanced_machine_features=compute.InstanceAdvancedMachineFeaturesArgs(
threads_per_core=2,
),
# Pulumi default is "running" — we want fallbacks stopped at create.
desired_status="TERMINATED",
)
vms.append(vm)
return vms
def create_encoding_worker_firewall() -> compute.Firewall:
"""Create firewall rules for encoding worker VMs.
Opens port 8080 for the HTTP API. Both VMs share the same
firewall rule via the 'encoding-worker' network tag.
"""
return compute.Firewall(
"encoding-worker-firewall",
name="encoding-worker-allow-http",
network="default",
allows=[
compute.FirewallAllowArgs(protocol="tcp", ports=["8080"]),
],
source_ranges=["0.0.0.0/0"],
target_tags=["encoding-worker"],
description="Allow HTTP access to encoding workers (auth required)",
)