## Summary Retires the self-hosted `gha-build-*` runner VMs (e2-standard-8, `docker-build` label, ~$21/mo). `deploy-backend` now runs on free `ubuntu-latest`; the Docker images are built by **Cloud Build in us-central1**. ## Why not build on the hosted runner? (egress math) The app images are built `FROM` bases in us-central1 Artifact Registry, and the app layer has a `RUN pip install`, so the full base must be pulled each deploy: - CPU base `karaoke-backend-base`: **4.22 GB** compressed - GPU base `karaoke-backend-gpu-base`: **13.1 GB** compressed - ~120 main deploys/month (119 in Sept, 11 so far today) On a GitHub-hosted runner that is GCP internet egress: 17.3 GB × 120 × $0.12/GB ≈ **$250/mo** (CPU-only would still be ≈ $61/mo). Both are worse than the $21 VM. Cloud Build in us-central1 pulls same-region (free). Build-minutes on the default-pool e2-standard-2 machine are in the **2,500 min/month free tier** ($0.006/min after that). A test build of the real images (CPU and GPU in parallel) took **6m55s**, about the same as today's ~7.3 min of sequential builds on the VM. That's ~840 min/mo, so the expected cost is ~$0 (worst case ~$5/mo). ## Changes - `infrastructure/cloudbuild/backend-images.yaml` + `build-backend-images.sh`: buildx (docker-container) builds with the **same tags and registry caches** as before. CPU and GPU build as parallel steps, and each build gets a fresh metadata-server token. - `ci.yml` `deploy-backend`: `runs-on: ubuntu-latest`. It keeps the hash checks (manifest + ~10 KB config blob only) and submits `gcloud builds submit --async`, then streams the logs and checks the status. - Base rebuilds (≈ monthly) use e2-highcpu-8 with 300 GB disk and a 2 h timeout. - A Cloud Build still running when the job is cancelled or fails is cancelled. CI concurrency cancels superseded deploys, and an orphaned build would otherwise still push `:latest`. - Removed the disk-space step, the local base pull, and the one-time us-east4→us-central1 GPU-base copy. The copy finished 2026-09-26, and on a hosted runner it would stream 13 GB through egress. - `build-runner-images.yml`: `variants=all` no longer bakes the unused `build` family. It can still be built with an explicit `variants=build`. - Docs: EPHEMERAL-GHA-RUNNERS, SELF-HOSTED-RUNNERS, GCP-COST-OPTIMIZATION. No Pulumi or IAM changes were needed. `github-actions-deployer` already has `cloudbuild.builds.editor`, `storage.admin` and `iam.serviceAccountUser`, and the default Cloud Build SA (compute default) already has `artifactregistry.writer` on both repos. ## Rollback (one commit) `git revert <squash-merge sha of this PR>` and push to main. That restores `runs-on: [self-hosted, linux, gcp, docker-build]` and the old in-job buildx steps. Nothing outside the repo was changed: the runner-manager `build` family is untouched, and the `gha-runner-build` GCE images (20260812, 20260901) are kept so the dispatcher can still launch `gha-build-*` VMs. If the build image has been pruned by then, re-bake it with Actions → Build GHA Runner Images → `variants=build` (only `variants=all` no longer includes it). ## Testing - I ran the real CPU and GPU builds from this branch's config in Cloud Build us-central1, with `cbtest-*` tags and no cache writes. Both succeeded in 6m55s, and the us-east4 push uploaded only the app layers. - `actionlint` and `shellcheck` report no new findings. - Full verification is the first main deploy after merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Backend image builds now run through a managed build service in the central US region, with CPU and GPU images published as part of the deployment workflow. * The default “all variants” selection now excludes the build variant; explicitly requested variants remain available. * **Documentation** * Updated runner and cost documentation to reflect the current build and deployment setup. * **Maintenance** * Updated the package version to 0.261.2. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
28 KiB
Ephemeral GitHub Actions Runners
Operational runbook for the create-on-demand GHA runner dispatcher. Replaces the previous fixed pool of 7 long-lived VMs with single-use ephemeral VMs to eliminate ~$230/mo of always-billed pd-ssd boot disks.
STATUS — 2026-05-17
Cutover live. RUNNER_MODE=ephemeral deployed on the github-runner-manager
Cloud Function at 2026-05-17T04:56Z. Scheduler tick at 04:57:06 confirmed
routing to "ephemeral mode orphan cleanup" with no errors. The 7 legacy
runner VMs are still provisioned and in TERMINATED state — they cost only
their pd-ssd boot disks (~$32/mo each = ~$224/mo) until Phase 4 deletes them
after the 1-week soak.
Current images (use --image-family=gha-runner-<variant> — GCE auto-selects newest):
| Family | Latest image | Built |
|---|---|---|
gha-runner-general |
gha-runner-general-20260517-040403 |
04:04 UTC |
gha-runner-build |
gha-runner-build-20260517-043133 |
04:31 UTC |
gha-runner-gpu |
gha-runner-gpu-20260517-033723 |
03:37 UTC |
Old images (pre-poetry-fix) still in the families but auto-superseded.
PRs landed this session: #765 (core), #766 (variant resolution), #767 (serial-console wait), #768 (gpu python — superseded), #769 (visibility), #770 (image-create + python 3.13.2 — superseded), #771 (gpu pyenv unification), #772 (apt-daily race).
Self-serve status check
Paste this anytime to see the full picture:
echo "=== $(date -u +%FT%TZ) ===" && \
echo && echo "--- Active workflow runs ---" && \
gh run list --workflow=build-runner-images.yml --limit=3 && \
echo && echo "--- Live VMs (build + smoke + ephemeral runners) ---" && \
gcloud compute instances list --project=nomadkaraoke \
--filter='labels.purpose=gha-runner-image-builder OR labels.purpose=gha-runner-smoke OR labels.purpose=gha-ephemeral-runner' \
--format='table(name,status,creationTimestamp.date(),labels.purpose,labels.family)' && \
echo && echo "--- READY images ---" && \
gcloud compute images list --project=nomadkaraoke \
--filter='family~"^gha-runner-(general|build|gpu)$" AND status=READY' \
--format='table(name,family,creationTimestamp.date())' && \
echo && echo "--- Dispatcher mode + recent log lines ---" && \
gcloud functions describe github-runner-manager --gen2 --region=us-central1 \
--project=nomadkaraoke --format='value(serviceConfig.environmentVariables.RUNNER_MODE)' && \
gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="github-runner-manager"' \
--project=nomadkaraoke --limit=10 --format='value(timestamp,textPayload)' --freshness=30m
To watch the live console of any VM (no SSH, works always):
gcloud compute instances get-serial-port-output <vm-name> \
--zone=us-central1-a --project=nomadkaraoke --port=1 \
| grep -E "runner-image|SMOKE-|###" | tail -50
⏰ Followups (1 week from cutover — 2026-05-24)
If the dispatcher has been creating ephemeral VMs successfully and the monitoring alerts haven't fired, proceed with Phase 4 decommission:
1. Confirm no issues during the soak. Run the status check above plus:
# Count successful ephemeral VM dispatches (should be ≥ number of CI runs in past week)
gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="github-runner-manager" AND textPayload:"Dispatched ephemeral runner"' \
--project=nomadkaraoke --freshness=7d --format='value(timestamp)' | wc -l
# Check the 2 alert policies weren't firing
gcloud alpha monitoring policies list --project=nomadkaraoke \
--filter='displayName:"GHA Runner Dispatcher"' --format='value(displayName,enabled)'
# Verify no long-lived ephemeral VMs
gcloud compute instances list --project=nomadkaraoke \
--filter='labels.purpose=gha-ephemeral-runner' --format='table(name,status,creationTimestamp.date())'
2. If everything looks clean, proceed with Phase 4 (delete the 7 legacy VMs and their pd-ssd boot disks — this is what unlocks the ~$220/mo saving):
cd karaoke-gen/infrastructure
# Edit infrastructure/config.py:
# NUM_GITHUB_RUNNERS = 0
# NUM_GPU_RUNNERS = 0
# Edit infrastructure/__main__.py: comment out / remove the three calls
# github_runners.create_github_runners(...)
# github_runners.create_build_runner(...)
# github_runners.create_gpu_runners(...)
# Edit infrastructure/modules/runner_manager.py: remove the idle-check scheduler
# (no longer needed once ephemeral is the only mode).
# Edit infrastructure/functions/runner_manager/main.py: remove the legacy
# start_runners / check_and_stop_idle_runners code paths and the
# RUNNER_MODE branching (keep webhook signature verification).
pulumi up # destructive — requires admin@ creds (claude-readonly blocks compute.instances.delete)
# Verify legacy VMs and disks are gone
gcloud compute instances list --filter="name~'github-(runner|gpu|build)'" --project=nomadkaraoke # expect empty
gcloud compute disks list --filter="users:'github-'" --project=nomadkaraoke # expect empty
3. If issues emerged during the soak, roll back instead:
cd karaoke-gen/infrastructure
unset GOOGLE_APPLICATION_CREDENTIALS # use admin@ creds, not claude-readonly
pulumi config set karaoke-gen-infrastructure:runnerMode legacy
pulumi up --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:cloudfunctionsv2/function:Function::runner-manager-function'
Legacy VMs are still in Pulumi state and TERMINATED; the next workflow_job webhook will start them.
Iteration-cost lessons learned from this session
Hard-won insights for the next person debugging GCE image builds:
-
Always stream the provisioner log to /dev/ttyS0 (GCE serial console). It's readable via
gcloud compute instances get-serial-port-output --port=1with zero SSH/IAP dependency. SSH-over-IAP turned out to be wildly unreliable from a residential network — multiple iterations failed silently because I couldn't see what was happening on the build VM. Solved byexec > >(tee /var/log/X.log /dev/ttyS0 2>/dev/null) 2>&1at the top of the script + an ERR trap that writes### FAILED rc=N line=L cmd=...to the same channel. Wait steps poll the serial console for the marker, NOT SSH. -
gcloud compute ssh --command='test -f /file'can lie under IAP load. Witnessed: false-positive marker detected at 4 min on a 25-min provisioner because the SSH session closed before the remote command completed but the exit code defaulted to 0. Trust file content (e.g., regex^[0-9]{4}-against an ISO timestamp), not exit codes alone. -
gh run watchaggressively polls the GitHub API and rate-limits quickly. Usegcloud-based polling (image presence, VM presence) for waits whenever possible — they don't hit GitHub's quota. -
Compiling Python from source on Debian 12 is fragile. Python 3.13.0 has an install-time ensurepip regression (
FileNotFoundErroron_WHEEL_PKG_DIR). 3.13.2 fixes that but produces a broken_socketC extension when--enable-optimizationsis off. Use pyenv for all variants instead — pyenv's build wrapper has hardening that the rawmake installpath lacks. -
Fresh Debian VMs have an apt-daily race.
apt-daily.serviceandapt-daily-upgrade.serviceauto-start ~30s into boot and hold/var/lib/apt/lists/lock+/var/lib/dpkg/lockfor several minutes. Any concurrentapt-get(especiallyinstalldependencies.sh) fails withE: Could not get lock. Stop+disable+mask both timers early in the script, then wait for the lock before doing anything apt-related. -
Don't auto-name boot disks then try to extract the name via a format selector. I used
--format='value(disks[0].source.scope(disks).segment(1))'which silently returned empty in one run and brokegcloud compute images create --source-disk=. The VM name == boot disk name by default; use it directly. -
Pulumi's
cloudfunctionsv2.Functionwon't redeploy on source-bundle change unless you pin the generation. Replacing theBucketObjectupdates GCS but doesn't trigger the function. Addgeneration=source_archive.generationtostorage_sourceso Pulumi sees the change as a Function diff and redeploys. -
Smoke-test as the actual production user. Earlier smoke tests SSH'd as the OS-login user, hit "permission denied" on the runner directory + docker socket, and falsely reported failures. Production runs
sudo -u runner ./run.sh ..., so the smoke test must verify tools-as-runner. The v2 smoke test uses startup-script +sudo -iu runnerfor this.
Which jobs use self-hosted runners (2026-09-26)
karaoke-gen's test/package jobs (package/backend unit + integration + emulator
tests, package builds, full frontend E2E) moved to GitHub-hosted
ubuntu-latest runners — free for this public repo, and they never needed GCP
credentials. That removed ~390 gha-general-* VMs/month. Only jobs that need
the GCE runners still request them: the GPU integration tests in
python-audio-separator (gpu). No workflow currently requests the plain general
label; the gha-runner-general family is kept as a fallback.
2026-10-01: deploy-backend also moved off the docker-build (gha-build-*,
e2-standard-8) runners to ubuntu-latest; its Docker images are now built by
Cloud Build in us-central1 (infrastructure/cloudbuild/). Building on the
hosted runner itself was rejected: the app images are built FROM ~4.2 GB (CPU)
and ~13 GB (GPU) bases in us-central1 AR, so every deploy would pull 17 GB over
GCP internet egress ($2/deploy, ~$250/mo). Cloud Build pulls them same-region
for free and its e2-standard-2 minutes are in the 2,500 min/month free tier
(~7 min/deploy). The build family stays in the dispatcher but is unused and
excluded from the monthly image bake (variants=all).
High-level architecture
GitHub webhook (workflow_job.queued)
│
▼
runner-dispatcher Cloud Function (github-runner-manager)
│ ─ pick image family from labels (general | build | gpu)
│ ─ mint a JIT runner config from GitHub
│ ─ compute.instances.insert with --auto-delete boot disk
│ ─ return 200
▼
Fresh GCE VM boots from custom image
│ ─ startup-script reads JIT config from metadata
│ ─ runs ./run.sh --jitconfig (registers + runs one job + de-registers)
│ ─ on exit, trap fires `shutdown -h +1`
▼
Boot disk auto-deletes; VM is gone
Every 5 min: orphan cleanup + stalled-job re-dispatch pass
│ ─ list VMs labelled purpose=gha-ephemeral-runner
│ ─ cross-reference with org runners list
│ ─ delete VMs older than 30 min with no registration
│ ─ delete VMs older than 2 h regardless (hung job protection)
│ ─ de-register zombie GHA runners with no live VM
│ ─ scan every org repo for self-hosted jobs still `queued`
│ ─ per family: if a job is queued 5–60 min and queued jobs outnumber
│ spare runners (booting VMs + idle online runners), launch the
│ shortfall (max 5 per pass)
Stalled-job re-dispatch (2026-09-28)
GitHub sends workflow_job.queued once and never redelivers a failed
webhook, and the webhook path launches exactly one VM. Three incidents left a
deploy job queued indefinitely (prod half-deployed each time):
- 2026-08-08 / 2026-09-25 — the dispatched GPU-family VM was preempted before it registered; nothing replaced it.
- 2026-09-27 — the
deploy-backendwebhook got a Cloud Run 429 ("no available instance"): a CI burst had all 5 function instances busy, each blocked up to 90s confirming its VM insert. No VM was ever created.
Fixes: max_instance_count 5 → 20 (RunnerManagerConfig.FUNCTION_MAX_INSTANCES),
and the scheduled pass (now every 5 min) re-dispatches stranded jobs regardless
of cause. JIT runners aren't bound to a job, so it compares per-family demand
with spare supply rather than tracking individual jobs. It stops retrying a job
after 60 min so fleet-wide runner breakage (see "stale baked runner version"
below) can't become a VM-launch loop. Env knobs on the function:
REDISPATCH_ENABLED (kill switch), REDISPATCH_AFTER_MINUTES (5),
REDISPATCH_GIVE_UP_MINUTES (60), MAX_REDISPATCH_PER_TICK (5). The scheduler
response JSON includes a redispatch object (stalled_jobs, given_up,
dispatched, dispatch_failed); log lines start Re-dispatch.
Modes
Controlled by Pulumi config karaoke-gen-infrastructure:runnerMode:
-
legacy(default) — keeps the original start/stop pool behaviour. The Cloud Function usesstart_runners/check_and_stop_idle_runnersagainst the existinggithub-runner-{1,2,3},github-build-runner, andgithub-gpu-runner-{1,2,3}VMs. No behavioural change vs. pre-this-PR. -
ephemeral— webhook creates a fresh VM per job via JIT registration; scheduler runs orphan cleanup. The legacy VMs are not touched (and should be left in their existing TERMINATED state until Phase 4).
Switching modes is a two-step operation handled entirely by Pulumi:
pulumi config set karaoke-gen-infrastructure:runnerMode ephemeral
pulumi up --target 'urn:...:Function::runner-manager-function'
(replace ephemeral ↔ legacy to roll back; takes ~2 min for the function to
re-deploy).
Image families
Four GCE image families are maintained by
.github/workflows/build-runner-images.yml:
| Family | Base | Bakes | Built on |
|---|---|---|---|
gha-runner-general |
Debian 12 | Docker, gcloud, runner v2.332, Python 3.13 (pyenv), Node 20, Java 21, FFmpeg, Poetry | n2-standard-8, 50GB pd-balanced |
gha-runner-build |
Debian 12 | Same as general + pre-pulled karaoke-backend-base/:latest + fake-gcs-server |
n2-standard-8, 50GB pd-balanced |
gha-runner-gpu |
Debian 12 | NVIDIA driver + CUDA 12.4, Python 3.13 (compiled), FFmpeg+libsamplerate, audio-separator models (~14GB at /opt/audio-separator-models) |
n1-standard-4 + T4, 200GB pd-balanced |
gha-runner-gpu-windows |
Windows Server 2022 | NVIDIA GRID driver (WDDM — required for DirectML; datacenter driver = TCC = no DirectX), Python 3.12, Git, FFmpeg, Poetry, runner (win-x64), DirectML test models at C:\audio-separator-models |
n1-standard-4 + T4, 100GB pd-balanced |
The Windows variant is provisioned by
runner-image-provision.ps1
delivered via the windows-startup-script-ps1 metadata key (Windows ignores
startup-script). It exists for python-audio-separator's windows-directml
integration tests (RoFormer-on-Windows, issue #292 there).
Label routing: every runner advertises self-hosted, <os>, x64, gcp.
Families additionally advertise large-disk (general/build), docker-build
(build), or gpu (GPU families). The dispatcher resolves families with
precedence windows → gpu → build → general, so jobs MUST include an OS label
in runs-on ([self-hosted, linux, gpu] / [self-hosted, windows, gpu]) —
GitHub schedules onto any runner whose labels are a superset of the job's.
Each image is tagged gha-runner-<variant>-<YYYYMMDD-HHMMSS> and joined to the
matching image family. The dispatcher always selects from the family, so the
newest non-deprecated image wins automatically. The build workflow keeps the
newest image per family (KEEP_IMAGES_PER_FAMILY: 1 since 2026-09-26 — GPU/windows
images are 15-21 GiB each) and deletes the rest (deprecated images still bill
for storage; images that an existing disk was created from are never deleted).
Until 2026-09-26 this step used an invalid gcloud filter and silently pruned
nothing — 31 runner images (~360 GB archived, ~$18/mo) had accumulated; they were pruned by hand to
2 per family when the step was fixed. Note a gcloud family=gha-runner-gpu
filter also matches gha-runner-gpu-windows (word-boundary regex), so the step
matches the family exactly client-side.
Build cadence: monthly cron (0 2 1 * * UTC) + workflow_dispatch. Manual:
gh workflow run build-runner-images.yml -f variants=general,build
gh workflow run build-runner-images.yml -f variants=gpu
gh workflow run build-runner-images.yml # all variants
Pulumi does not manage images — they're produced by the GHA workflow. The
runtime dispatcher resolves projects/nomadkaraoke/global/images/family/gha-runner-<variant>
directly, which means image lifecycle and code lifecycle are decoupled.
Operator handoff: rollout sequence
The plan document is at
docs/archive/2026-05-16-ephemeral-gha-runners-plan.md
(this repo) and at nomadkaraoke/docs/archive/2026-05-16-ephemeral-gha-runners-plan.md
(workspace root, original). Phases 1–2 (code) and 5 (monitoring) are landed in
this PR. Phases 3 and 4 (cutover + decommission) are operator-driven.
Phase 1 — Build the images (one-time)
After this PR merges:
-
Trigger the workflow for the cheapest variant first to validate plumbing:
gh workflow run build-runner-images.yml -f variants=generalFirst build takes ~25 min on
general/build, ~60 min ongpu(model downloads dominate). -
Verify the image landed:
gcloud compute images describe-from-family gha-runner-general \ --project=nomadkaraoke \ --format='value(name,creationTimestamp,labels)' -
Repeat for
buildandgpu. -
Smoke test by launching a one-off VM manually:
gcloud compute instances create gha-smoke-test \ --zone=us-central1-a \ --machine-type=e2-standard-4 \ --image-family=gha-runner-general \ --image-project=nomadkaraoke \ --metadata=enable-oslogin=TRUE gcloud compute ssh gha-smoke-test --zone=us-central1-a \ --command='docker --version && python3 --version && /home/runner/actions-runner/bin/Runner.Listener --version' gcloud compute instances delete gha-smoke-test --zone=us-central1-a --quiet
Phase 2 — Apply the dispatcher changes (Pulumi)
This PR adds:
RUNNER_MODE,GCP_FALLBACK_ZONE,GITHUB_ORG,RUNNER_SERVICE_ACCOUNTenv vars ongithub-runner-manager- Replaces the Cloud Function source archive with the new
runner_manager/+ephemeral.pycode - Adds 2 log-based metrics + 2 alert policies for runner observability
Apply locally (the workspace's ADC points at a read-only SA — Pulumi from
this terminal will fail with 403 on storage.objects.create, so use an
account with write access, e.g. gcloud config configurations activate admin):
cd karaoke-gen/infrastructure
pulumi preview --diff
# expected: ~2 changes (runner-manager-source replace, runner-manager-function update)
# + 5 creates (log metrics, alert policies)
pulumi up --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:storage/bucketObject:BucketObject::runner-manager-source' \
--target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:cloudfunctionsv2/function:Function::runner-manager-function' \
--target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:logging/metric:Metric::runner-dispatcher-failures' \
--target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:logging/metric:Metric::runner-dispatcher-orphan-kills' \
--target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:monitoring/alertPolicy:AlertPolicy::runner-dispatcher-failures-alert' \
--target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:monitoring/alertPolicy:AlertPolicy::runner-dispatcher-long-lived-vm-alert'
After apply, RUNNER_MODE is still legacy — no behavioural change yet.
Why
--target? There's an unrelated pendingdivebar-sync-vmrecreate from the May 16 cost sprint. Until that PR's deploy runs, scoping to these targets keeps this rollout isolated.
Phase 3 — Cutover to ephemeral
Only after Phase 1 produced all 3 images AND Phase 2 dispatcher deployed.
-
Manual one-off test: invoke the dispatcher directly with a synthetic webhook payload (signature verification will reject this, but you can alternatively flip the mode just for this test by setting the env var on the function via
gcloud functions deploy-style override, or use a real PR — see step 3). -
Switch the dispatcher to ephemeral mode globally:
pulumi config set karaoke-gen-infrastructure:runnerMode ephemeral pulumi up --target 'urn:pulumi:prod::karaoke-gen-infrastructure::gcp:cloudfunctionsv2/function:Function::runner-manager-function'The 7 legacy VMs are currently in
TERMINATEDstate (the idle-check stopped them). WithRUNNER_MODE=ephemeralthe dispatcher no longer starts them — they sit idle until Phase 4 deletion. -
Trigger a real test run by opening (or pushing to) any branch on
karaoke-genorpython-audio-separator. Verify in Cloud Console that:runner-manager-functioninvocations succeed (no 503s)gcloud compute instances list --filter=labels.purpose=gha-ephemeral-runnershows VMs appearing and disappearing- The CI job completes within
1.5×the prior wall clock - The orphan-cleanup pass (every 15 min via
runner-manager-idle-checkscheduler) printskept_vms: []once jobs are done
-
Monitor the new alert policies for a week before Phase 4:
GHA Runner Dispatcher - Create Failures(any failure = page)GHA Runner Dispatcher - VM Alive > 2h30m(orphan cleanup didn't work)
Rollback: pulumi config set karaoke-gen-infrastructure:runnerMode legacy && pulumi up --target ...:runner-manager-function.
Legacy VMs are still in Pulumi state; the next workflow_job webhook will
start them.
Phase 4 — Decommission the legacy VMs (destructive)
Only after Phase 3 has run cleanly for ≥1 week with real CI traffic.
- In
karaoke-gen/infrastructure/config.py, setNUM_GITHUB_RUNNERS = 0andNUM_GPU_RUNNERS = 0. - In
karaoke-gen/infrastructure/__main__.py, remove thegithub_runners.create_github_runners(...),create_build_runner(...), andcreate_gpu_runners(...)calls along with their references. - In
karaoke-gen/infrastructure/modules/runner_manager.py, remove the idle-check scheduler (no longer needed once ephemeral is the only mode). - In
karaoke-gen/infrastructure/functions/runner_manager/main.py, delete the legacystart_runners/check_and_stop_idle_runnerspaths and theRUNNER_MODEbranching. Keep webhook signature verification. pulumi up(locally, since legacy VM delete needscompute.instances.deletewhichclaude-readonlyblocks).- Verify:
gcloud compute instances list --filter="name~'github-(runner|gpu|build)'" --project=nomadkaraoke # should return zero rows gcloud compute disks list --filter="users:'github-'" --project=nomadkaraoke # should return zero rows - Remove the unused
RUNNER_NAMESandGPU_RUNNER_NAMESenv vars from the Pulumi runner_manager module (cosmetic).
Savings realised: ~$220/mo (7×200GB pd-ssd at $32/mo each).
Follow-ups (not in initial PR)
- Dedicated image-builder service account. The build workflow currently
uses the
github-actions-deployer@SA (broad CI/CD perms). Best practice is a separategha-runner-image-builder@SA with onlyartifactregistry.readerlogging.logWriter. The build VM doesn't need write perms anywhere —gcloud compute images createis invoked from the GHA runner, not the VM. Adds 1 SA + 2 IAM bindings + a new repo secret.
- OIDC token verification for the scheduler entry point. The function
currently checks
Authorization: Bearer <jwt>is present and non-trivially long, which keeps random callers out but doesn't cryptographically verify the token. Adding full verification needsgoogle-authin the function's requirements.txt and a one-time JWKS fetch; defer until we observe abuse.
Known gotchas
- A stale baked runner version silently kills every ephemeral runner (incident
2026-05-24). GitHub deprecates older
actions/runnerversions and returns HTTP 403 on the broker poll ("Runner version vX.Y.Z is deprecated and cannot receive messages"). Ephemeral/JIT runners run withdisableUpdate, so they CANNOT self-update — the runner registers, gets rejected, the listener exits "no retry needed", and the EXIT-trap halts the VM. Net effect: VMs dispatch and boot fine, but CI jobs sitqueuedforever and each leaves a zombieofflineJIT registration in the org runner list. Diagnosis: read a failed VM's/home/runner/actions-runner/_diag/Runner_*.logfor the 403. (Note: serial console is unavailable once a VM self-halts, so capture logs by setting a dump startup-script and starting the stopped VM.) Fix: bumpRUNNER_VERSIONininfrastructure/scripts/runner-image-provision.sh(and legacycompute/startup_scripts/github_runner.sh) to the current release from https://github.com/actions/runner/releases/latest, then re-runbuild-runner-images.yml(workflow_dispatch — runs on GitHub-hosted runners, so it's not blocked by the broken self-hosted ones). The monthly rebuild cron (0 2 1 * *) is too slow vs. GitHub's deprecation pace — consider resolving "latest" at build time and/or alerting on deprecation 403s. - Source-bundle replacement doesn't auto-redeploy the Cloud Function (fixed
forward, but worth knowing). The original code uploaded a new source bundle
via
BucketObjectreplace, but thecloudfunctionsv2.Functionresource pointed at the bucket+name without pinning a generation, so Pulumi saw no diff on the Function and the live function kept serving the previously staged copy. Fixed by addinggeneration=source_archive.generationto the Function'sstorage_source. If you ever see a Pulumi apply complete cleanly but the live function still serving old code, rungcloud functions deploy github-runner-manager --gen2 --region=us-central1 --project=nomadkaraoke --source=gs://nomadkaraoke-runner-manager-source/runner-manager-source.zip --runtime=python312 --entry-point=handle_request --quiet. - claude-readonly ADC blocks Pulumi storage writes. The workspace's
GOOGLE_APPLICATION_CREDENTIALSpoints at a read-only SA. Pulumi 403s onstorage.objects.createwhen re-uploading the function source bundle. Apply Pulumi changes from a shell that usesadmin@nomadkaraoke.com(or temporarily unset the env var). Do NOT switch togcloudto bypass — seefeedback_claude_readonly_adc.mdin memory. - Pulumi state has a pending
divebar-sync-vmrecreate. Unrelated to this work; will be applied by the cost-sprint PR's CI deploy. - us-east4 GPU fallback uses external IPs, not Cloud NAT. There's no NAT in us-east4; provisioning one for a rarely-used fallback zone wasn't worth the complexity. Ephemeral external IPs are free while the VM is running.
- Pre-existing test failure in
infrastructure/functions/runner_manager/test_runner_manager.py::TestCheckGithubForPendingJobs::test_checks_both_queued_and_in_progress. Assumes 2 GitHub API calls; code now iterates 2 repos × 2 statuses = 4. Unrelated to this PR. - Existing legacy test
test_runner_manager.pystill passes in 25/26 scenarios; the newtest_ephemeral.pyadds 20 tests, all passing. - Image build VM uses
n2-standard-8to speed up apt + downloads, even though runtime general/build VMs aree2-standard-4/e2-standard-8. This is intentional — image build is ~25 min on n2; ~45 min on e2. - The
--ephemeralflag is implicit when./run.sh --jitconfigis used. Don't pass--ephemeralexplicitly with--jitconfig— GitHub's runner rejects the combination. - The provision script's GPU kernel-upgrade reboot is the slowest part of the GPU build. The first build will reboot once mid-provisioning; the workflow tolerates this via SSH retry-with-grace.
Files added by this work
karaoke-gen/infrastructure/scripts/runner-image-provision.sh— provision logickaraoke-gen/.github/workflows/build-runner-images.yml— image build pipelinekaraoke-gen/infrastructure/functions/runner_manager/ephemeral.py— new dispatcherkaraoke-gen/infrastructure/functions/runner_manager/test_ephemeral.py— 20 testskaraoke-gen/docs/EPHEMERAL-GHA-RUNNERS.md— this file
Files modified by this work
karaoke-gen/infrastructure/functions/runner_manager/main.py— route on RUNNER_MODEkaraoke-gen/infrastructure/modules/runner_manager.py— new env vars, runnerMode configkaraoke-gen/infrastructure/modules/monitoring.py— runner-dispatcher alertskaraoke-gen/infrastructure/__main__.py— wire monitoring resources