Compare commits

...
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 5.5 57a30979a7 fix(ci): prune keeps newest 2 ACTIVE runner images; docs: executable swap commands + final results
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 16:26:31 -04:00
Andrew BeveridgeandClaude Opus 5.5 70dd49da4a chore(infra): encoding-worker disk cost cut + fix runner image pruning
- Encoding-worker boot disks 100 -> 50 GB (DiskSizes.ENCODING_WORKER); Packer
  image disk_size 100 -> 30 so the image no longer forces a 100 GB floor.
  Test enforces image size <= worker disk size.
- build-runner-images.yml: prune step used an invalid gcloud filter and
  silently pruned nothing (31 images accumulated). Now deletes images beyond
  the newest 2 per family, matches the family exactly (gcloud family= also
  matched gpu-windows), and never deletes a disk's source image.
- Docs: rollout procedure (stockout-safe in-place boot-disk swap), lessons,
  cost doc, packer README.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 16:12:25 -04:00
Andrew BeveridgeandClaude Opus 5.5 f26fb3fd85 chore(infra): shrink encoding-worker boot disks 100->50 GB, Packer image 100->30 GB
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 16:12:25 -04:00
11 changed files with 235 additions and 23 deletions

No files matched your search

+41 -16
View File
@@ -14,7 +14,7 @@ name: Build GHA Runner Images
# 3. Stop the VM, snapshot its boot disk to a new image in the `gha-runner-<variant>`
# image family with name gha-runner-<variant>-<timestamp>.
# 4. Delete the build VM (boot disk auto-deletes).
# 5. Deprecate older images in the family, keeping the newest 3 ACTIVE.
# 5. Delete older images in the family, keeping the newest 2 (rollback = previous month).
on:
workflow_dispatch:
@@ -24,7 +24,7 @@ on:
required: false
default: 'all'
skip_prune:
description: 'Skip deprecation of old images (debugging)'
description: 'Skip pruning (deletion) of old images (debugging)'
type: boolean
required: false
default: false
@@ -41,7 +41,7 @@ env:
BUILD_REGION: us-central1
BUILD_ZONE: us-central1-a
BASE_IMAGE: projects/debian-cloud/global/images/family/debian-12
KEEP_IMAGES_PER_FAMILY: 3
KEEP_IMAGES_PER_FAMILY: 2
jobs:
resolve-matrix:
@@ -271,7 +271,7 @@ jobs:
run: |
gcloud compute instances delete "$VM" --zone="$ZONE" --project="$PROJECT" --quiet || true
- name: Deprecate old images (keep newest ${{ env.KEEP_IMAGES_PER_FAMILY }})
- name: Prune old images (keep newest ${{ env.KEEP_IMAGES_PER_FAMILY }})
if: ${{ github.event.inputs.skip_prune != 'true' }}
env:
PROJECT: ${{ env.GCP_PROJECT }}
@@ -279,20 +279,45 @@ jobs:
KEEP: ${{ env.KEEP_IMAGES_PER_FAMILY }}
run: |
set -euo pipefail
# List ACTIVE images in this family, newest first, skip the top $KEEP.
to_deprecate=$(gcloud compute images list \
# DELETE (not deprecate — deprecated images still bill for storage)
# every image in this family beyond the newest $KEEP. The dispatcher
# boots from the family URL, which always resolves to the newest image,
# so older ones are only rollback insurance (~$0.05/GB-mo; gpu images
# are ~21 GB each). Previously this step used an invalid
# `-deprecated.state:*` filter: gcloud errored to stderr, stdout was
# empty, and it silently reported "Nothing to deprecate" every month
# while images piled up (31 runner images, ~$18/mo, by 2026-09).
# Match the family EXACTLY client-side: a gcloud `family=gha-runner-gpu`
# filter is a word-boundary regex that ALSO matches gha-runner-gpu-windows.
# Rank by ACTIVE (non-deprecated) images: the family URL skips deprecated
# images, so the newest $KEEP active ones are the real rollback path.
# Deprecated images never count toward $KEEP.
rows=$(gcloud compute images list \
--project="$PROJECT" \
--filter="family=$FAMILY AND status=READY AND -deprecated.state:*" \
--no-standard-images \
--show-deprecated \
--sort-by='~creationTimestamp' \
--format='value(name)' | tail -n +$((KEEP+1)))
if [[ -z "$to_deprecate" ]]; then
echo "Nothing to deprecate."
--format='value(name,family,deprecated.state)' | awk -v f="$FAMILY" '$2==f')
active=$(echo "$rows" | awk 'NF==2 {print $1}')
if [[ -z "$active" ]]; then
echo "::error::No active images found in family $FAMILY (list failed?) — refusing to prune."
exit 1
fi
keep=$(echo "$active" | head -n "$KEEP")
to_delete=$(echo "$rows" | awk '{print $1}' | grep -vxF "$keep" || true)
if [[ -z "$to_delete" ]]; then
echo "Nothing to prune (<= $KEEP images in $FAMILY)."
exit 0
fi
for img in $to_deprecate; do
echo "Deprecating $img"
gcloud compute images deprecate "$img" \
--project="$PROJECT" \
--state=DEPRECATED \
--replacement="projects/$PROJECT/global/images/family/$FAMILY"
# Never delete an image some existing disk was created from (e.g. a
# long-lived VM still on an older image).
in_use=$(gcloud compute disks list --project="$PROJECT" \
--format='value(sourceImage.basename())' | sort -u)
for img in $to_delete; do
if grep -qx "$img" <<<"$in_use"; then
echo "Keeping $img (source image of an existing disk)"
continue
fi
echo "Deleting $img"
gcloud compute images delete "$img" --project="$PROJECT" --quiet
done
+7 -1
View File
@@ -242,7 +242,13 @@ GitHub schedules onto any runner whose labels are a superset of the job's.
Each image is tagged `gha-runner-<variant>-<YYYYMMDD-HHMMSS>` and joined to the
matching image family. The dispatcher always selects from the family, so the
newest non-deprecated image wins automatically. The build workflow keeps the
newest 3 images per family and deprecates the rest.
newest 2 images per family and **deletes** the rest (deprecated images still bill
for storage; images that an existing disk was created from are never deleted).
Until 2026-09-26 this step used an invalid gcloud filter and silently pruned
nothing — 31 runner images (~360 GB archived, ~$18/mo) had accumulated; they were pruned by hand to
2 per family when the step was fixed. Note a gcloud `family=gha-runner-gpu`
filter also matches `gha-runner-gpu-windows` (word-boundary regex), so the step
matches the family exactly client-side.
Build cadence: monthly cron (`0 2 1 * *` UTC) + `workflow_dispatch`. Manual:
+2 -1
View File
@@ -85,7 +85,8 @@ The `concurrency=1` setting may have been added deliberately for performance rea
(c4d/c4/n4d/c2d/n2d/n2) for stockout resilience.
- **Idle auto-shutdown is live** (JIT start on the lyrics-review page; a Cloud
Function stops idle VMs after 15 min). When idle, cost is just boot disks
(~$10/VM/mo); compute is billed only while encoding — so the historical
(50 GB since 2026-09-26, ~$4-5/VM/mo; was 100 GB / ~$10 — see
docs/archive/2026-09-26-encoding-worker-disk-cost.md); compute is billed only while encoding — so the historical
"~$700/mo running 24/7" no longer applies.
**Opportunities (status):**
+17
View File
@@ -2033,3 +2033,20 @@ Also: colour-code the bars with the **exact** `HIGHLIGHT_CLASSES` tints the Adva
pills use (not hand-picked solids) so the two views read identically, and persist the
view mode under a new `lyricsReviewViewMode` enum that migrates the old
`lyricsReviewAdvancedMode` boolean.
### Encoding-worker disks: shrink via in-place boot-disk swap, not Pulumi replace (2026-09-26)
- Workers used ~8-15 GB of their 100 GB boot disks. Shrunk to 50 GB (Packer image
`disk_size` 100→30 — a disk can't be smaller than its source image).
- Changing `boot_disk.initialize_params.size` in Pulumi means replace (or an
impossible in-place shrink); replacing a c4d/n4d/c2d VM needs capacity that may be
stocked out for hours. Instead: while TERMINATED, detach + delete the old boot disk,
create a new one from the new image, attach `--boot --device-name=persistent-disk-0`,
then `pulumi refresh --target` the instance → no diff. No compute allocation needed.
- New hyperdisks default to paid IOPS/throughput above the free 3000/140 baseline —
always pass `--provisioned-iops=3000 --provisioned-throughput=140` (Pulumi doesn't
set them because boot-disk init params force replacement).
- A gcloud `family=X` filter is a word-boundary regex (`gha-runner-gpu` also matches
`gha-runner-gpu-windows`), and an invalid filter errors to stderr with empty stdout
— pipelines that treat empty output as "nothing to do" silently no-op. The runner
image prune step did this for months (31 images, ~$18/mo).
@@ -0,0 +1,116 @@
# Encoding-worker disk cost reduction (2026-09-26)
## Problem
The 10 encoding-worker VMs (c4d primary pair + 8 stockout fallbacks) are stopped
between jobs, so their cost is mostly their boot disks. Last-30-day billing:
Hyperdisk Balanced capacity ~$48 (601 GiB-mo) + provisioned throughput ~$12 +
provisioned IOPS ~$6; Balanced PD capacity ~$45 (includes 2 runner disks). Plus
~$18/mo "Storage Image" for 39 custom images.
## Findings
- **Disks were ~85% empty.** `encoding-worker-b`, 8 months in service: 15 GB used
of 99 GB (7 GB venv, 3 GB pip cache, 0.7 GB logs). Fresh disks from the current
image use 8 GB.
- **Scratch need is small.** Each `/encode` downloads its whole `jobs/<id>/` folder
into `/tmp` on the boot disk. Across 1,904 job folders: median 0.43 GB, p99 1.3 GB,
max 4.0 GB. Downloads happen before the (serialized) heavy lane, so queued jobs
hold their inputs at the same time. The deepest queue seen in 30 days was 8.
Realistic peak: ~8 GB base + ~10-15 GB scratch.
- **Only two disks had paid IOPS/throughput.** `-c4a` and `-n4db`, created 2026-08-15
after the June IOPS tune, got the Hyperdisk defaults of 3600 IOPS / 290 MB/s. The
other hyperdisks were already on the free baseline of 3000 / 140.
- **Image size set the disk floor.** Packer built the image with `disk_size = 100`.
GCE cannot create a disk smaller than its source image, so every worker had to be
at least 100 GB.
- **Runner image pruning had never run.** The monthly `build-runner-images.yml`
step used an invalid filter (`-deprecated.state:*`). gcloud wrote the error to
stderr and the step printed "Nothing to deprecate". Deprecating wouldn't have
saved money anyway, because deprecated images are still billed.
## Changes
| Lever | Change | Est. saving |
|---|---|---|
| Boot disk size | 100 GB to **50 GB** on all 10 workers (`DiskSizes.ENCODING_WORKER`); Packer image `disk_size` 100 to **30** | ~$44/mo |
| Hyperdisk IOPS/throughput | `-c4a`/`-n4db` 3600/290 to 3000/140 (free baseline). New hyperdisks are created at 3000/140 explicitly | ~$18/mo |
| Custom images | Deleted 23 old runner images (kept newest 2 per family). Fixed the prune step so it deletes beyond the newest 2, matches the family exactly, and skips images that are a disk's source | ~$13/mo |
**Total: ~$75/mo.**
Levers considered and **not** taken:
- **Removing fallback VMs.** Not done. They are the stockout insurance, and machine-family diversity is deliberate.
- **Ephemeral boot disks (create from image at start).** Not done. This adds disk-creation and hydration latency to every cold start, plus another failure mode during stockouts. The saving (~$4/VM/mo after the shrink) isn't worth that.
- **pd-standard for the fallbacks.** Not done. A 50 GB pd-standard disk gets ~75 read IOPS, which would slow boot and pip installs badly. c4/c4d/n4d VMs require Hyperdisk anyway.
- **Smaller than 50 GB.** Not done. It saves only ~$1/disk/mo per 10 GB, and a disk-full error during a bulk-queue encode costs far more than that.
## Performance
pd-balanced performance scales with size: 100 GB gives 3600 IOPS / 168 MB/s, and
50 GB gives 3300 / 154. Hyperdisk Balanced at 3000 / 140 is the same at any size.
Encoding is CPU-bound: a full 4K finalization writes ~200 MB of outputs in ~2 min.
Real full `/encode` of the same job (`bench/disk-swap-input`, the Aug-2026 benchmark
song; formats 4k lossless + 4k lossy + 720p), wall time in seconds:
| VM | CPU platform | Before (100 GB, old image) | After (50 GB, new image) | start to healthy |
|---|---|---|---|---|
| fallback-c2df (pd-balanced) | AMD Milan | 142.6, 138.9 | 142.2, 136.9 | 34.9 s to 33.5 s |
| fallback-c4a (hyperdisk) | Intel Emerald Rapids | 132.1, 121.4 | 147.4, 144.5, then a re-run of 143.9, 138.7, 138.6 | 39.1 s to 50-53 s |
| fallback-n2da (pd-balanced) | AMD Milan | failed: transient metadata-server SSL error (pre-existing, see v0.217.0) | 155.7 | to 41.7 s |
| fallback-n2f (pd-balanced) | Intel Cascade Lake | 247.2 (Aug benchmark) | 246.8 | to 39.7 s |
| fallback-a (hyperdisk) | AMD Turin (c4d) | n/a | 99.0 | to 42.3 s |
| fallback-b (hyperdisk) | AMD Turin (c4d) | n/a | 105.4 | to 52.5 s |
| encoding-worker-b (hyperdisk) | AMD Turin (c4d) | n/a | 100.8 | to 35.9 s |
| encoding-worker-a (hyperdisk, primary) | AMD Turin (c4d) | n/a | 98.9 | to 36.9 s |
On c4a the after-median of 139 s is exactly the 138.6 s August median, which was
measured on the old 100 GB disk. The two "before" samples today were at the fast
end of the ~35% run-to-run noise the August benchmark documented. Hyperdisk
performance at 3000/140 doesn't depend on disk size, so the disk change can't
explain the difference.
## Rollout procedure: in-place boot-disk swap
Changing `boot_disk.initialize_params.size` makes Pulumi either replace the VM or
try an in-place resize. An in-place resize can't shrink a disk, and a replacement
has to re-allocate the VM, which for c4d/n4d/c2d can be blocked for hours by a
stockout. Each VM was left in place and only its boot disk was swapped, while the
VM was `TERMINATED`. No compute capacity is needed for this.
```bash
# Per VM, only if status == TERMINATED and config/encoding-worker.deploy_in_progress is false
gcloud compute instances detach-disk $VM --zone=$Z --disk=$VM
gcloud compute disks delete $VM --zone=$Z --quiet
# hyperdisk-balanced (c4d/c4/n4d):
gcloud compute disks create $VM --zone=$Z --image=$NEW_IMAGE --image-project=nomadkaraoke \
--size=50GB --type=hyperdisk-balanced --provisioned-iops=3000 --provisioned-throughput=140
# pd-balanced (n2/n2d/c2d):
gcloud compute disks create $VM --zone=$Z --image=$NEW_IMAGE --image-project=nomadkaraoke \
--size=50GB --type=pd-balanced
gcloud compute instances attach-disk $VM --zone=$Z --disk=$VM --boot --device-name=persistent-disk-0
gcloud compute instances set-disk-auto-delete $VM --zone=$Z --disk=$VM --auto-delete
# then: pulumi refresh --target <instance urn> → state picks up size 50 / new image;
# pulumi preview shows no diff for that VM (the family-image URL diff-suppresses).
```
The old disk is deleted without a snapshot on purpose. Worker boot disks hold no
state that can't be rebuilt: the image, the wheel downloaded at boot, and caches.
Jobs live in GCS and Firestore.
Rollback: repeat the swap using the previous image. `encoding-worker-1786843294`
is kept for this.
Verification per VM: start it, check `/health` (wheel version), run a real `/encode`,
check `df`, then stop it. VMs that hit a stockout at start were swapped but could
only be verified later.
## Status
All 10 workers were swapped to 50 GB on `encoding-worker-1790444464` and their
Pulumi state was refreshed, and `pulumi preview` shows no encoding-VM diff. The old
encoding-worker images were deleted, keeping `encoding-worker-1786843294` for
rollback. Custom images went from 39 to 10. `-n4db` (n4d, us-central1-b) and `-n2c`
(n2, us-central1-c) couldn't be boot-verified because both zones were stocked out
during the rollout. They got the same swap as the 8 verified VMs.
+1 -1
View File
@@ -173,7 +173,7 @@ gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
High-performance VM for video encoding:
- Machine: c4-standard-8 (8 vCPU, 32GB RAM)
- Disk: 100GB hyperdisk-balanced
- Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
- Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts
**API Endpoints:**
+1 -1
View File
@@ -122,7 +122,7 @@ def create_encoding_worker_fallback_vms(
every lane at once (incident 2026-08-12). Each entry's machine_type and
disk_type come from EncodingWorkerConfig.FALLBACKS — n2 uses pd-balanced
because it does not support hyperdisk-balanced. Cost when stopped is just
the boot disk (~$10/mo each).
the 50 GB boot disk (~$4-5/mo each).
"""
startup_script = read_script("encoding_worker.sh")
custom_image = f"projects/{PROJECT_ID}/global/images/family/encoding-worker"
+13 -2
View File
@@ -71,7 +71,18 @@ class DiskSizes:
"""Disk size configurations in GB."""
GITHUB_RUNNER = 200 # Large for Docker builds/caches
ENCODING_WORKER = 100 # For temp video files during encoding
# Encoding-worker boot disk (OS + baked venv ~12-15 GB, plus per-job scratch in
# /tmp: each queued /encode downloads its whole jobs/<id>/ folder — p99 ~1.3 GB,
# max seen ~4 GB — and up to ~9 jobs have been queued on one worker at once).
# Was 100 GB, but a worker 8 months into service used only 15 GB of it; 50 GB
# keeps >3x headroom over realistic peak while halving the 10-VM disk bill.
# Must be >= the Packer image's disk_size (infrastructure/packer/
# encoding-worker.pkr.hcl), since GCE cannot create a disk smaller than its
# source image. Changing this forces VM *replacement* in Pulumi — see
# docs/archive/2026-09-26-encoding-worker-disk-cost.md for the in-place
# boot-disk swap procedure that avoids re-allocating (possibly stocked-out) VMs.
ENCODING_WORKER = 50
ENCODING_WORKER_IMAGE = 30 # Packer image disk_size (keep in sync with .pkr.hcl)
FLACFETCH = 30 # For torrent storage
@@ -306,7 +317,7 @@ class EncodingWorkerConfig:
# Capacity-resilience fallback fleet. Each VM is provisioned STOPPED in an
# alternate zone / machine family and is started on demand only when the
# primary zone rejects a start with ZONE_RESOURCE_POOL_EXHAUSTED. Cost when
# stopped is just the boot disk (~$10/mo each).
# stopped is just the 50 GB boot disk (~$4-5/mo each).
#
# Machine-family diversity is DELIBERATE. The primary pair and the two c4d
# fallbacks are all c4d-highcpu-32, so a region-wide c4d stockout (observed
+10
View File
@@ -94,6 +94,16 @@ Rebuild the image when:
- Font packages change
- bootstrap.sh changes (extremely rare)
**Image disk size:** `disk_size` in the template (30 GB) is the *minimum* boot-disk
size of any VM created from the image — GCE cannot create a disk smaller than its
source image. Keep it ≤ `DiskSizes.ENCODING_WORKER` in `infrastructure/config.py`
(50 GB); `infrastructure/test_encoding_worker_config.py` enforces this.
**After a rebuild, existing VMs keep their old disks** (Pulumi only uses the new
image when a VM/disk is recreated). To roll a new image onto the stopped workers
without re-allocating VMs (stockout-safe), use the in-place boot-disk swap in
`docs/archive/2026-09-26-encoding-worker-disk-cost.md`.
**No rebuild needed** for:
- Application code changes (wheel downloaded at runtime)
- Startup logic changes (startup.sh downloaded from GCS)
@@ -44,7 +44,12 @@ source "googlecompute" "encoding-worker" {
# Use n2 machine type for building (c4 requires hyperdisk which isn't supported by Packer)
# The resulting image will work on c4-standard-8 in production
machine_type = "n2-standard-8"
disk_size = 100
# Image disk size = the MINIMUM boot-disk size any VM created from this image
# can have (GCE cannot create a disk smaller than its source image). Keep it
# small (the provisioned OS + venv is ~12 GB) so the worker boot disk size is
# decided by infrastructure/config.py::DiskSizes.ENCODING_WORKER (50 GB), not
# pinned here. Was 100, which forced every worker onto a 100 GB disk.
disk_size = 30
disk_type = "pd-ssd" # Use SSD for faster builds
# Image naming - uses timestamp for versioning, family for latest
@@ -112,3 +112,24 @@ def test_original_c4d_fallbacks_unchanged():
"machine_type": MachineTypes.ENCODING_WORKER, "disk_type": "hyperdisk-balanced"}
assert b == {"suffix": "b", "zone_suffix": "b",
"machine_type": MachineTypes.ENCODING_WORKER, "disk_type": "hyperdisk-balanced"}
def test_worker_boot_disk_fits_packer_image_size():
"""GCE cannot create a boot disk smaller than its source image's disk size.
The Packer image's `disk_size` must stay <= DiskSizes.ENCODING_WORKER, or
every worker (re)creation from the new image fails. Also keeps the
ENCODING_WORKER_IMAGE mirror constant honest."""
import re
from pathlib import Path
from config import DiskSizes
hcl = (Path(__file__).parent / "packer" / "encoding-worker.pkr.hcl").read_text()
match = re.search(r"^\s*disk_size\s*=\s*(\d+)", hcl, re.MULTILINE)
assert match, "disk_size not found in encoding-worker.pkr.hcl"
image_size = int(match.group(1))
assert image_size == DiskSizes.ENCODING_WORKER_IMAGE
assert image_size <= DiskSizes.ENCODING_WORKER
# Headroom floor: OS + baked venv is ~12-15 GB and per-job scratch lives in
# /tmp on the boot disk. Don't let a future "cost cut" starve encodes.
assert DiskSizes.ENCODING_WORKER >= 40