Files
Andrew Beveridge ☄️andClaude Opus 5.5 dd795c66a0 ci(cost): build backend images in Cloud Build, run deploy-backend on ubuntu-latest (#1106)
## Summary
Retires the self-hosted `gha-build-*` runner VMs (e2-standard-8,
`docker-build` label, ~$21/mo). `deploy-backend` now runs on free
`ubuntu-latest`; the Docker images are built by **Cloud Build in
us-central1**.

## Why not build on the hosted runner? (egress math)
The app images are built `FROM` bases in us-central1 Artifact Registry,
and the app layer has a `RUN pip install`, so the full base must be
pulled each deploy:
- CPU base `karaoke-backend-base`: **4.22 GB** compressed
- GPU base `karaoke-backend-gpu-base`: **13.1 GB** compressed
- ~120 main deploys/month (119 in Sept, 11 so far today)

On a GitHub-hosted runner that is GCP internet egress: 17.3 GB × 120 ×
$0.12/GB ≈ **$250/mo** (CPU-only would still be ≈ $61/mo). Both are
worse than the $21 VM.

Cloud Build in us-central1 pulls same-region (free). Build-minutes on
the default-pool e2-standard-2 machine are in the **2,500 min/month free
tier** ($0.006/min after that). A test build of the real images (CPU and
GPU in parallel) took **6m55s**, about the same as today's ~7.3 min of
sequential builds on the VM. That's ~840 min/mo, so the expected cost is
~$0 (worst case ~$5/mo).

## Changes
- `infrastructure/cloudbuild/backend-images.yaml` +
`build-backend-images.sh`: buildx (docker-container) builds with the
**same tags and registry caches** as before. CPU and GPU build as
parallel steps, and each build gets a fresh metadata-server token.
- `ci.yml` `deploy-backend`: `runs-on: ubuntu-latest`. It keeps the hash
checks (manifest + ~10 KB config blob only) and submits `gcloud builds
submit --async`, then streams the logs and checks the status.
- Base rebuilds (≈ monthly) use e2-highcpu-8 with 300 GB disk and a 2 h
timeout.
- A Cloud Build still running when the job is cancelled or fails is
cancelled. CI concurrency cancels superseded deploys, and an orphaned
build would otherwise still push `:latest`.
- Removed the disk-space step, the local base pull, and the one-time
us-east4→us-central1 GPU-base copy. The copy finished 2026-09-26, and on
a hosted runner it would stream 13 GB through egress.
- `build-runner-images.yml`: `variants=all` no longer bakes the unused
`build` family. It can still be built with an explicit `variants=build`.
- Docs: EPHEMERAL-GHA-RUNNERS, SELF-HOSTED-RUNNERS,
GCP-COST-OPTIMIZATION.

No Pulumi or IAM changes were needed. `github-actions-deployer` already
has `cloudbuild.builds.editor`, `storage.admin` and
`iam.serviceAccountUser`, and the default Cloud Build SA (compute
default) already has `artifactregistry.writer` on both repos.

## Rollback (one commit)
`git revert <squash-merge sha of this PR>` and push to main. That
restores `runs-on: [self-hosted, linux, gcp, docker-build]` and the old
in-job buildx steps. Nothing outside the repo was changed: the
runner-manager `build` family is untouched, and the `gha-runner-build`
GCE images (20260812, 20260901) are kept so the dispatcher can still
launch `gha-build-*` VMs. If the build image has been pruned by then,
re-bake it with Actions → Build GHA Runner Images → `variants=build`
(only `variants=all` no longer includes it).

## Testing
- I ran the real CPU and GPU builds from this branch's config in Cloud
Build us-central1, with `cbtest-*` tags and no cache writes. Both
succeeded in 6m55s, and the us-east4 push uploaded only the app layers.
- `actionlint` and `shellcheck` report no new findings.
- Full verification is the first main deploy after merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Improvements**
* Backend image builds now run through a managed build service in the
central US region, with CPU and GPU images published as part of the
deployment workflow.
* The default “all variants” selection now excludes the build variant;
explicitly requested variants remain available.
* **Documentation**
* Updated runner and cost documentation to reflect the current build and
deployment setup.
* **Maintenance**
  * Updated the package version to 0.261.2.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 20:22:59 -04:00
..