feat(encoding): GCP cost cuts — Spot 16-vCPU workers, 3-VM fleet, 5-min idle, gated worker deploys (v0.245.0) (#1061)

## Summary
Approved GCP cost cuts for the encoding pipeline (target: whole project
< $300/mo; encoding VMs were ~$300/mo for ~5-20 jobs/day). Details +
before/after table: `docs/archive/2026-09-26-encoding-cost-cuts.md`.

| | Before | After |
|---|---|---|
| Primary pair a/b | c4d-highcpu-32 on-demand | **c4d-highcpu-16 Spot**
(STOP on preemption) |
| Fallbacks | 8 VMs / 6 families | **1 on-demand**
`encoding-worker-fallback-c2df` (c2d-highcpu-16, us-central1-f) |
| Idle shutdown | 15 min / check every 5 | **5 min / check every 2** |
| CI worker blue-green | every push to main | only when
`infrastructure/encoding-worker/worker_code_paths.txt` paths changed
since the serving version (+ `workflow_dispatch`
`force_encoding_worker_deploy`) |
| Nomad Dropbox upload | whole folder | minus lossless 4K **MP4**
(tenant folders unchanged) |

**Infra already applied locally (targeted Pulumi):** 7 fallback VMs +
disks + 7 static IPs deleted; a/b + c2df resized in place; idle function
env + scheduler updated. The Spot switch was done in place with `gcloud
compute instances set-scheduling` on the stopped VMs + `pulumi refresh`
(pulumi-gcp treats a `scheduling` diff as *replace*, and recreating a
c4d VM risks a stockout). `encoding-worker-fallback-vms` secret → v6
(c2df only). CI's pulumi run should be a no-op for these resources.

### Scope change (per Andrew)
**Task 5 dropped:** review previews stay on the encoding VM
(`USE_GCE_PREVIEW_ENCODING=true` unchanged) and the review-page warmup
is kept, since it deliberately warms the VM for the preview and the
final render. To make sure the new 5-min idle timeout doesn't undermine
that, the review page now also heartbeats **every 2 min** while the tab
is visible and was used in the last 15 min, and re-warms on tab refocus
(`useEncodingWorkerKeepAlive`). Before this, the heartbeat only fired on
lyric edits. The idle checker keeps the routed VM alive while
`last_activity_at` < 5 min, and both warmup and heartbeat write it.
Tests cover both sides.

### Code changes
- **Spot preemption mid-encode**: when a poll fails,
`wait_for_completion` checks the pinned worker VM. If it is
STOPPING/TERMINATED it raises `EncodingJobLostError`, so
`run_with_lost_job_resubmit` resubmits the encode, and that resubmit
restarts the Spot VM or falls back to c2df. Before, the failure showed
up after ~7 min as a "lost contact" error that was not resubmitted.
- **Stale fallback lists**: a candidate VM that no longer exists
(`NotFound`) is skipped instead of aborting the whole failover. This
protects the window before running instances pick up the new secret.
- `SPEED_RANK` is now keyed by machine family, so the ranking is
independent of VM size.
- **Idle-stop vs. in-flight encode** (review follow-up): the idle
function's `/health active_jobs` only counts RUNNING jobs, and
`last_activity_at` was only written when a VM was started. With a 5-min
window, a long encode's VM could be stopped in the seconds between the
job finishing and the next poll. The poll loop now refreshes
`last_activity_at` at most once a minute. A VM seen in
STAGING/PROVISIONING after it accepted a job also counts as lost. CI
`Read Firestore config` now writes empty values instead of "None".
- **Worker deploy gate**:
`backend/tests/test_encoding_worker_code_paths.py` derives the worker's
transitive `backend.*` import set and fails if `worker_code_paths.txt`
doesn't cover it. Every deploy still uploads the wheel and `startup.sh`,
so a cold-booted VM always runs the latest code.
- **Dropbox**: nothing automated reads lossless files from Dropbox. The
lossless **MKV** is still uploaded because the completion email and the
live Fiverr auto-reply bot both tell customers it is in the Dropbox
folder, and the VocalStar tenant expects lossless files. To drop the MKV
as well, update those two texts first, then add the MKV suffix to
`DROPBOX_SKIP_OUTPUT_SUFFIXES`.

## Testing
- New/updated unit tests: preemption → lost job → resubmit, NotFound
candidate skip, family ranking, idle 5-min/heartbeat window, fleet-shape
invariants, worker-path coverage, Dropbox exclusion, keep-alive hook
(Jest, fake timers).
- `pytest backend/tests --ignore=backend/tests/emulator`: 4802 passed.
The only failures are 6 tests that need the spaCy `en_core_web_sm`
model, which isn't installed in my local venv; unrelated to this PR.
- Jest lyrics-review + lib/lyrics-review: 423 passed. Infra tests
(`test_encoding_worker_config.py`, idle function): all pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-09-26 23:20:54 -04:00
1 parent c138c9d1ac
commit 87a36bd2a5
35 files changed
+1358 -335

No files matched your search

+58 -6
View File
@@ -7,6 +7,15 @@ on:
branches: [main]
types: [opened, synchronize, reopened, labeled]
workflow_dispatch:
inputs:
force_encoding_worker_deploy:
# Manual override for the encoding-worker deploy gate (see "Decide
# encoding-worker deploy" in deploy-backend). Dispatch on main re-runs
# package-build + deploy-backend; with this ticked the GCE worker is
# blue-green redeployed even if no worker code changed.
description: "Force a blue-green redeploy of the GCE encoding worker"
type: boolean
default: false
permissions:
contents: write
@@ -824,7 +833,7 @@ jobs:
package-build:
name: "Package - Build"
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
if: github.ref == 'refs/heads/main' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
steps:
- name: Free disk space (only if low)
# GitHub-hosted runners (free for this public repo) currently have ~86 GB
@@ -1079,7 +1088,7 @@ jobs:
name: "Deploy - Backend (Cloud Run)"
needs: [package-build]
runs-on: [self-hosted, linux, gcp, docker-build]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
if: github.ref == 'refs/heads/main' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
permissions:
contents: read
id-token: write
@@ -1289,7 +1298,9 @@ jobs:
for key in ["primary_vm", "primary_ip", "primary_version",
"secondary_vm", "secondary_ip",
"active_override_vm", "active_override_ip", "active_override_zone"]:
f.write(f"{key}={data.get(key, '')}\n")
# `or ''`: a key stored as null must be empty, not the string
# "None" (which later steps would treat as a real VM name).
f.write(f"{key}={data.get(key) or ''}\n")
f.write(f"deploy_in_progress={str(data.get('deploy_in_progress', False)).lower()}\n")
# capacity_state is a map — emit as single-line JSON so the green
# selection step can demote recently-stocked-out machine types.
@@ -1298,8 +1309,47 @@ jobs:
f"active_override={data.get('active_override_vm')}")
PYEOF
# Gate the blue-green worker deploy on worker-affecting changes. It boots a
# fresh green VM + runs a real test encode, which used to happen on EVERY
# push to main (~5x/day) even for frontend/backend-only changes. The wheel,
# startup.sh and version.txt above are still uploaded every time, so any
# worker that cold-boots (they idle-stop after 5 min) runs the latest code.
# Diff base = the version the serving worker was last validated at (not
# github.event.before), so a cancelled/skipped deploy can never lose a
# worker change. Any doubt → deploy (fail open).
- name: Decide encoding-worker deploy
id: worker_gate
env:
FORCE: ${{ github.event_name == 'workflow_dispatch' && inputs.force_encoding_worker_deploy == true }}
SERVING_VERSION: ${{ steps.config.outputs.primary_version }}
OVERRIDE_VM: ${{ steps.config.outputs.active_override_vm }}
run: |
mapfile -t PATHS < <(grep -v '^\s*#' infrastructure/encoding-worker/worker_code_paths.txt | grep -v '^\s*$')
DEPLOY=true
if [ "$FORCE" = "true" ]; then
REASON="manual override (workflow_dispatch force_encoding_worker_deploy)"
elif [ -n "$OVERRIDE_VM" ] && [ "$OVERRIDE_VM" != "None" ]; then
REASON="capacity override $OVERRIDE_VM is serving — refresh it"
elif [ -z "$SERVING_VERSION" ] || [ "$SERVING_VERSION" = "None" ]; then
REASON="serving worker version unknown"
elif ! git fetch --no-tags --depth=1 origin "refs/tags/v${SERVING_VERSION}:refs/tags/v${SERVING_VERSION}"; then
REASON="tag v${SERVING_VERSION} not fetchable"
elif ! CHANGED=$(git diff --name-only "v${SERVING_VERSION}" HEAD -- "${PATHS[@]}"); then
REASON="git diff against v${SERVING_VERSION} failed"
elif [ -z "$CHANGED" ]; then
DEPLOY=false
REASON="no worker-affecting changes since serving v${SERVING_VERSION}"
else
REASON="worker-affecting changes since v${SERVING_VERSION}"
echo "Changed worker paths:"; echo "$CHANGED" | head -50
fi
echo "deploy=${DEPLOY}" >> "$GITHUB_OUTPUT"
echo "Encoding-worker blue-green deploy: ${DEPLOY} (${REASON})"
echo "### Encoding worker deploy: ${DEPLOY}" >> "$GITHUB_STEP_SUMMARY"
echo "${REASON}" >> "$GITHUB_STEP_SUMMARY"
- name: Wait for concurrent deploy
if: steps.config.outputs.deploy_in_progress == 'true'
if: steps.worker_gate.outputs.deploy == 'true' && steps.config.outputs.deploy_in_progress == 'true'
run: |
echo "Another deploy is in progress, waiting..."
python3 << 'PYEOF'
@@ -1318,6 +1368,7 @@ jobs:
PYEOF
- name: Set deploy in progress
if: steps.worker_gate.outputs.deploy == 'true'
run: |
python3 << 'PYEOF'
from datetime import datetime, timezone
@@ -1338,6 +1389,7 @@ jobs:
# infrastructure/encoding-worker/deploy_promote.py.
- name: Select & start green target
id: green
if: steps.worker_gate.outputs.deploy == 'true'
# Never hard-fail the whole backend deploy just because no encoding
# worker could be validated — emit green_ok=false and let the gated
# health/encode/promote steps skip while the Cloud Run deploy proceeds.
@@ -1578,7 +1630,7 @@ jobs:
# always() so the deploy_in_progress flag is ALWAYS cleared, even when the
# green step found no worker (its downstream steps skip, but the flag was
# already set). Skips only on a fully successful promote (passed == true).
if: always() && steps.encode-test.outputs.passed != 'true'
if: always() && steps.worker_gate.outputs.deploy == 'true' && steps.encode-test.outputs.passed != 'true'
run: |
GREEN_VM="${{ steps.green.outputs.green_vm }}"
GREEN_ZONE="${{ steps.green.outputs.green_zone }}"
@@ -1719,7 +1771,7 @@ jobs:
# step guarantees the lock is released so the next deploy doesn't stall 10 min
# in "Wait for concurrent deploy". Idempotent with the other clears.
- name: Clear encoding-worker deploy lock
if: always()
if: always() && steps.worker_gate.outputs.deploy == 'true'
run: |
python3 -c "
from google.cloud import firestore