## Summary Retires the self-hosted `gha-build-*` runner VMs (e2-standard-8, `docker-build` label, ~$21/mo). `deploy-backend` now runs on free `ubuntu-latest`; the Docker images are built by **Cloud Build in us-central1**. ## Why not build on the hosted runner? (egress math) The app images are built `FROM` bases in us-central1 Artifact Registry, and the app layer has a `RUN pip install`, so the full base must be pulled each deploy: - CPU base `karaoke-backend-base`: **4.22 GB** compressed - GPU base `karaoke-backend-gpu-base`: **13.1 GB** compressed - ~120 main deploys/month (119 in Sept, 11 so far today) On a GitHub-hosted runner that is GCP internet egress: 17.3 GB × 120 × $0.12/GB ≈ **$250/mo** (CPU-only would still be ≈ $61/mo). Both are worse than the $21 VM. Cloud Build in us-central1 pulls same-region (free). Build-minutes on the default-pool e2-standard-2 machine are in the **2,500 min/month free tier** ($0.006/min after that). A test build of the real images (CPU and GPU in parallel) took **6m55s**, about the same as today's ~7.3 min of sequential builds on the VM. That's ~840 min/mo, so the expected cost is ~$0 (worst case ~$5/mo). ## Changes - `infrastructure/cloudbuild/backend-images.yaml` + `build-backend-images.sh`: buildx (docker-container) builds with the **same tags and registry caches** as before. CPU and GPU build as parallel steps, and each build gets a fresh metadata-server token. - `ci.yml` `deploy-backend`: `runs-on: ubuntu-latest`. It keeps the hash checks (manifest + ~10 KB config blob only) and submits `gcloud builds submit --async`, then streams the logs and checks the status. - Base rebuilds (≈ monthly) use e2-highcpu-8 with 300 GB disk and a 2 h timeout. - A Cloud Build still running when the job is cancelled or fails is cancelled. CI concurrency cancels superseded deploys, and an orphaned build would otherwise still push `:latest`. - Removed the disk-space step, the local base pull, and the one-time us-east4→us-central1 GPU-base copy. The copy finished 2026-09-26, and on a hosted runner it would stream 13 GB through egress. - `build-runner-images.yml`: `variants=all` no longer bakes the unused `build` family. It can still be built with an explicit `variants=build`. - Docs: EPHEMERAL-GHA-RUNNERS, SELF-HOSTED-RUNNERS, GCP-COST-OPTIMIZATION. No Pulumi or IAM changes were needed. `github-actions-deployer` already has `cloudbuild.builds.editor`, `storage.admin` and `iam.serviceAccountUser`, and the default Cloud Build SA (compute default) already has `artifactregistry.writer` on both repos. ## Rollback (one commit) `git revert <squash-merge sha of this PR>` and push to main. That restores `runs-on: [self-hosted, linux, gcp, docker-build]` and the old in-job buildx steps. Nothing outside the repo was changed: the runner-manager `build` family is untouched, and the `gha-runner-build` GCE images (20260812, 20260901) are kept so the dispatcher can still launch `gha-build-*` VMs. If the build image has been pruned by then, re-bake it with Actions → Build GHA Runner Images → `variants=build` (only `variants=all` no longer includes it). ## Testing - I ran the real CPU and GPU builds from this branch's config in Cloud Build us-central1, with `cbtest-*` tags and no cache writes. Both succeeded in 6m55s, and the us-east4 push uploaded only the app layers. - `actionlint` and `shellcheck` report no new findings. - Full verification is the first main deploy after merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Backend image builds now run through a managed build service in the central US region, with CPU and GPU images published as part of the deployment workflow. * The default “all variants” selection now excludes the build variant; explicitly requested variants remain available. * **Documentation** * Updated runner and cost documentation to reflect the current build and deployment setup. * **Maintenance** * Updated the package version to 0.261.2. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
13 KiB
Self-Hosted GitHub Actions Runners
Single source of truth for how Nomad Karaoke CI/CD runners work — architecture, cost model, auto-scaling, and troubleshooting.
Why Self-Hosted Runners?
GitHub-hosted runners work but are slower (2 vCPU, 7GB RAM) and costs scale linearly with CI minutes. Self-hosted runners on GCP give us:
- Faster builds — persistent Docker caches, more CPU/RAM
- GPU access — T4 GPUs for audio separation model testing
- Scale-to-zero — runners shut down automatically when idle, so we only pay for active CI time
- Dedicated build runner — 8 vCPU for Docker image builds
Architecture
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ GitHub Webhook │────>│ Cloud Function │────>│ Start Runner │
│ (workflow_job) │ │ (runner-manager) │ │ VMs │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│
v
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Cloud Scheduler│────>│ Cloud Function │────>│ Stop Idle │
│ (every 15 min) │ │ (runner-manager) │ │ Runner VMs │
└─────────────────┘ └──────────────────┘ └─────────────────┘
Components
| Component | Spec | Purpose |
|---|---|---|
| General runners | 3x github-runner-{1..3}, e2-standard-4, 200GB SSD |
Execute CI jobs (lint, test, type check, build) |
| Build runner | 1x github-build-runner, e2-standard-8, 200GB SSD |
Unused since 2026-10-01 — deploy-backend runs on ubuntu-latest and builds images in Cloud Build (see EPHEMERAL-GHA-RUNNERS.md) |
| GPU runners | 3x github-gpu-runner-{1..3}, n1-standard-4 + T4 GPU, 200GB SSD |
Audio separation model testing |
| Cloud Function | github-runner-manager, Gen2, Python 3.12, 5min timeout |
Start/stop VMs based on webhooks and idle checks |
| Cloud Scheduler | runner-manager-idle-check, every 5 min |
Orphan cleanup + stalled-job re-dispatch (see EPHEMERAL-GHA-RUNNERS.md) |
| Cloud NAT | github-runners-nat + github-runners-router |
Outbound internet for VMs (no external IPs) |
All runner VMs are in us-central1-a.
Service Accounts
| Account | Purpose | Key Permissions |
|---|---|---|
github-runner |
Runner VMs | artifactregistry.reader, secretmanager.secretAccessor |
runner-manager |
Cloud Function | compute.instanceAdmin.v1, iam.serviceAccountUser on github-runner, secret access |
runner-manager-scheduler |
Cloud Scheduler | cloudfunctions.invoker on the function |
Runner Labels
| Label Set | VMs | Used By |
|---|---|---|
[self-hosted, linux, x64, gcp, large-disk] |
general runners | Most CI jobs |
[self-hosted, linux, x64, gcp, large-disk, docker-build] |
build runner | Unused since 2026-10-01 (images build in Cloud Build) |
[self-hosted, linux, x64, gcp, gpu] |
GPU runners | Audio separation tests |
How Auto-Scaling Works
Starting Runners (Webhook-Driven)
- Developer pushes code or opens a PR
- GitHub Actions queues CI jobs with labels
[self-hosted, linux, gcp] - GitHub sends a
workflow_job.queuedwebhook to the Cloud Function - The function iterates over all runner VM names (from
RUNNER_NAMESenv var) - Any TERMINATED VMs are started in parallel via
ThreadPoolExecutor - If VMs are in STOPPING state (from an overlapping idle shutdown), the function polls until they become TERMINATED (up to 90s), then starts them
- VMs boot in ~2-3 minutes, run the startup script, register with GitHub, and pick up queued jobs
Tracking Activity (Metadata-Based)
When a job completes, GitHub sends a workflow_job.completed webhook. The function updates a last-activity metadata key on the specific runner that ran the job (identified by runner_name from the webhook payload). Only updating the specific runner avoids fingerprint conflicts from concurrent webhook calls.
Stopping Idle Runners (Scheduler-Driven)
Every 15 minutes, Cloud Scheduler triggers the function with ?action=check_idle:
- Check for pending work: Query GitHub API for queued/in-progress runs. If anything is pending, keep all runners alive (but do NOT refresh timestamps — that would reset idle tracking).
- Check each runner's idle time: For each RUNNING VM, read
last-activitymetadata. If missing, fall back tocreationTimestamp(the VM's creation date). - Stop idle runners: Any runner idle for 1+ hour is queued for shutdown. All stops execute in parallel to minimize the race window.
The Race Condition (and How We Handle It)
During active development, new work can arrive while the idle check is shutting down runners:
- Idle check fires all stop operations in parallel (~60s window)
- Webhook arrives with new
workflow_job.queuedevent - start_runners finds some VMs in
STOPPINGstate
Solution: start_runners() detects STOPPING VMs and polls them every 5 seconds (up to 90s) until they become TERMINATED, then starts them immediately.
Cost Model
Monthly Estimates
| Scenario | Runner Uptime | Monthly Cost |
|---|---|---|
| Active development (8h/day) | ~10-12h/day | ~$100-150 |
| Light usage (solo dev) | ~3-5h/day | ~$50-80 |
| No activity (all stopped) | 0h | ~$35 (NAT + scheduler only) |
Fixed Costs
| Component | Monthly Cost |
|---|---|
| Cloud NAT | ~$30 |
| Cloud Scheduler | <$1 |
| Cloud Function | <$5 |
| Total fixed | ~$35 |
Variable Costs
| VM Type | Hourly Cost | 24/7 Monthly |
|---|---|---|
| e2-standard-4 (general) × 3 | $0.402 | $289 |
| e2-standard-8 (build) × 1 | $0.268 | $193 |
| n1-standard-4 + T4 (GPU) × 3 | $1.62 | $1,166 |
| All 7 VMs | $2.29/hr | $1,648 |
This is why idle shutdown is critical — the difference between working and broken auto-scaling is ~$1,600/month.
Historical Bugs and Fixes (March 2026)
The initial runner manager implementation had several bugs that prevented idle shutdown. Runners were running 24/7 for weeks, wasting ~$1,600/month of GCP credits.
Bug 1: Metadata Writes Not Awaited
Problem: client.set_metadata() returns an Operation object. Without calling .result(), the write completes asynchronously and fingerprint conflicts from concurrent calls are swallowed silently. last-activity metadata never persisted.
Fix: Added operation.result(timeout=60) to all metadata writes.
Bug 2: "Set Now and Keep" Infinite Loop
Problem: When last-activity was missing, the function set it to "now" and kept the runner. Since the metadata write failed silently (Bug 1), every 15-minute idle check repeated: "No last-activity metadata, setting now" → keeps running → forever.
Fix: Use creationTimestamp as fallback instead of "now". VMs without metadata are treated as having been idle since creation.
Bug 3: Pending Jobs Reset All Timestamps
Problem: When any queued run existed, the function refreshed last-activity on ALL runners. This reset idle tracking so runners could never accumulate enough idle time to be stopped.
Fix: When jobs are pending, keep runners alive but do NOT refresh timestamps.
Bug 4: Missing IAM Permission
Problem: The runner-manager SA needed iam.serviceAccountUser on the github-runner SA to set metadata on VMs running as that account. Without it, metadata writes failed with SERVICE_ACCOUNT_ACCESS_DENIED.
Fix: Added roles/iam.serviceAccountUser binding in Pulumi.
Bug 5: No STOPPING State Handling
Problem: VMs in STOPPING state (from an overlapping idle check) were skipped by start_runners(), potentially leaving all runners unreachable.
Fix: Poll STOPPING VMs until they become TERMINATED (up to 90s), then start them.
Bug 6: Sequential Stops
Problem: Stopping VMs one at a time created a wide race window where new webhooks could overlap.
Fix: Stop all idle VMs in parallel using ThreadPoolExecutor.
Bug 7: All Runners Updated on Job Completion
Problem: in_progress and completed webhooks updated metadata on ALL running instances, causing fingerprint conflicts when multiple jobs completed simultaneously.
Fix: Only update the specific runner identified by runner_name in the webhook payload. Removed the in_progress handler entirely.
Bug 8: Flaky "docker: command not found" (Exit Code 127)
Problem: Backend Emulator Integration Tests intermittently failed with docker: command not found (exit code 127). Re-running the job always succeeded. Root cause: When a stopped VM restarts, the GHA runner systemd service (installed via svc.sh install on a previous boot) auto-starts immediately, accepting CI jobs before the GCE startup script has finished setting up Docker and other dependencies.
Fix (two-part):
- Startup script: Stop the runner service at the very top of the script, before any package installations, preventing premature job acceptance.
- CI workflow: Added a Docker readiness check step with 60-second retry loop in
backend-emulator-tests, providing defense-in-depth.
Troubleshooting
Runners Not Starting
# 1. Check VM status
gcloud compute instances list --filter="name~github-runner" --format="table(name,status)" --project=nomadkaraoke
# 2. Check GitHub registration
gh api /orgs/nomadkaraoke/actions/runners --jq '.runners[] | {name, status, busy}'
# 3. Check Cloud Function logs
gcloud functions logs read github-runner-manager --limit=20 --gen2 --region=us-central1 --project=nomadkaraoke
# 4. Manual start if needed
gcloud compute instances start github-runner-{1..3} github-build-runner --zone=us-central1-a --project=nomadkaraoke
Runners Not Stopping
# 1. Check last-activity metadata
for i in 1 2 3; do
echo -n "github-runner-$i: "
gcloud compute instances describe "github-runner-$i" --zone=us-central1-a --project=nomadkaraoke \
--format="json(metadata.items)" | python3 -c "
import sys,json; items=json.load(sys.stdin).get('metadata',{}).get('items',[])
[print(i['value']) for i in items if i['key']=='last-activity']" 2>/dev/null || echo "no metadata"
done
# 2. Check for pending jobs
gh run list --limit 5 --json status,name --repo nomadkaraoke/karaoke-gen
# 3. Check function logs for idle check results
gcloud functions logs read github-runner-manager --limit=30 --gen2 --region=us-central1 --project=nomadkaraoke | grep -E "(Scheduler|idle|stop|pending)"
Common causes:
last-activitymetadata not being set (check for IAM errors in function logs)- Scheduled workflows sending
workflow_job.queuedevents - Function timeout exceeded
Webhook Not Triggering
- Check https://github.com/organizations/nomadkaraoke/settings/hooks
- Look at "Recent Deliveries" for the Cloud Function webhook
- Common issues: secret mismatch, function URL changed after deploy
Infrastructure Files
| File | Purpose |
|---|---|
infrastructure/config.py |
NUM_GITHUB_RUNNERS, NUM_GPU_RUNNERS, RunnerManagerConfig |
infrastructure/compute/github_runners.py |
VM instances, Cloud NAT |
infrastructure/modules/runner_manager.py |
Cloud Function, Cloud Scheduler, IAM bindings |
infrastructure/functions/runner_manager/main.py |
Cloud Function code (webhook handler, idle check) |
infrastructure/compute/startup_scripts/github_runner.sh |
General runner VM bootstrap |
infrastructure/compute/startup_scripts/github_runner_gpu.sh |
GPU runner VM bootstrap |
Secrets
| Secret | Purpose | Rotation |
|---|---|---|
github-runner-pat |
Runner registration (PAT with admin:org > manage_runners:org) |
Create new PAT, update secret, reset VMs |
github-webhook-secret |
Webhook signature verification | Generate new hex, update secret + GitHub webhook config |
Manual Operations
Starting / Stopping All Runners
# Start all
gcloud compute instances start github-runner-{1..3} github-build-runner github-gpu-runner-{1..3} --zone=us-central1-a --project=nomadkaraoke
# Stop all
gcloud compute instances stop github-runner-{1..3} github-build-runner github-gpu-runner-{1..3} --zone=us-central1-a --project=nomadkaraoke
Verifying Auto-Stop Works
# After last CI job completes, wait 1+ hour, then check:
gcloud compute instances list --filter="name~github-runner OR name~github-build OR name~github-gpu" \
--format="table(name,status)" --project=nomadkaraoke
# Expected: all TERMINATED