Files
karaoke-gen/.github
Andrew Beveridge ☄️andClaude Opus 5.5 1abdd8c860 fix(cost): runner-manager only dispatches VMs for jobs it can serve; enforce AR cleanup (v0.286.1) (#1228)
## Summary
On 2026-10-27, **238 GCP `gha-general` VMs (~$67)** ran in one day. This
pushed October's GCP bill past the $300 budget.

**Cause:**
- nomad-mobile's E2E jobs run on the utility MacBook (`[self-hosted,
macOS, nomad-mbp]`).
- The runner manager acted on *any* job labelled `self-hosted`, so each
macOS job got a Linux VM that could never take it.
- While the job waited for an MBP lane, `redispatch_stalled_jobs`
launched another VM every 5 min for up to 60 min.
- Each of those VMs then idled until the 2 h reaper deleted it.

## Changes
- `ephemeral.can_serve(labels)`: GitHub assigns a job only to a runner
that advertises every label the job requests, so a job is ours only if
its labels ⊆ the resolved family's `runner_labels_for()`.
- The new check gates the webhook (`main.py`),
`list_queued_self_hosted_jobs` and `redispatch_stalled_jobs`.
- I checked every `runs-on: [self-hosted, …]` label set in the org's
workflows. All of them except the macOS lanes are subsets of our
families' labels, so no existing job loses its VM.
- CI: these function tests never ran in CI before. A new
`runner-manager-tests` job runs them on infra or ci.yml changes and is
enforced by CI Gate.
- Docs: `docs/EPHEMERAL-GHA-RUNNERS.md` § "Only jobs our runners can
serve".

## Testing
- 6 new tests: macOS jobs, unknown labels, our families, the redispatch
skip, queued-scan exclusion, and the webhook no-op. All 71
runner_manager tests pass locally.
- Local CodeRabbit review: 2 cycles, both CI-wiring findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:35:44 -04:00
..
2024-09-17 12:18:12 -04:00
2025-12-11 03:49:32 -05:00