mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary On 2026-10-27, **238 GCP `gha-general` VMs (~$67)** ran in one day. This pushed October's GCP bill past the $300 budget. **Cause:** - nomad-mobile's E2E jobs run on the utility MacBook (`[self-hosted, macOS, nomad-mbp]`). - The runner manager acted on *any* job labelled `self-hosted`, so each macOS job got a Linux VM that could never take it. - While the job waited for an MBP lane, `redispatch_stalled_jobs` launched another VM every 5 min for up to 60 min. - Each of those VMs then idled until the 2 h reaper deleted it. ## Changes - `ephemeral.can_serve(labels)`: GitHub assigns a job only to a runner that advertises every label the job requests, so a job is ours only if its labels ⊆ the resolved family's `runner_labels_for()`. - The new check gates the webhook (`main.py`), `list_queued_self_hosted_jobs` and `redispatch_stalled_jobs`. - I checked every `runs-on: [self-hosted, …]` label set in the org's workflows. All of them except the macOS lanes are subsets of our families' labels, so no existing job loses its VM. - CI: these function tests never ran in CI before. A new `runner-manager-tests` job runs them on infra or ci.yml changes and is enforced by CI Gate. - Docs: `docs/EPHEMERAL-GHA-RUNNERS.md` § "Only jobs our runners can serve". ## Testing - 6 new tests: macOS jobs, unknown labels, our families, the redispatch skip, queued-scan exclusion, and the webhook no-op. All 71 runner_manager tests pass locally. - Local CodeRabbit review: 2 cycles, both CI-wiring findings fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>