Files
karaoke-gen/infrastructure/scripts
Andrew Beveridge ☄️andClaude Opus 4.7 424b908d71 fix(runner-dispatcher): omit TERMINATE for e2 families (unblocks ephemeral cutover) (#776)
## Summary

Fixes three cutover bugs that left the ephemeral runner dispatcher
silently broken since 2026-05-17T04:56Z:

1. **`on_host_maintenance="TERMINATE"` set unconditionally** — e2
machine types reject TERMINATE unless preemptible, so 100% of
general/build VM creates failed. Fix: only set TERMINATE on GPU (n1)
families where it's required because attached GPUs can't live-migrate. 3
regression tests added in `test_ephemeral.py`.
2. **GPU `disk_size_gb=150` but image is 200GB** — GCE rejects boot
disks smaller than the source image. Fix: raise GPU disk to 200GB.
3. **Runner user has no passwordless sudo** — workflow steps like `sudo
apt-get install -y google-cloud-cli-firestore-emulator`
(backend-emulator-tests) and `sudo apt-get install -y ffmpeg`
(package-*, GPU integration tests) failed with "a terminal is required
to read the password". The legacy GHA runner VMs had NOPASSWD sudo. Fix:
add `/etc/sudoers.d/runner` in the image provision script.

Fixes 1 + 2 are dispatcher code (already applied via `pulumi up --target
...:runner-manager-source ...:runner-manager-function` from this
branch). Fix 3 requires a new image build (triggered:
build-runner-images.yml on this branch).

## Why the trivial file touches

`backend/main.py` and `karaoke_gen/__init__.py` carry one-line comments
to flip the `backend` and `package` `dorny/paths-filter` outputs so this
PR exercises **all five PR-triggered self-hosted jobs** end-to-end on
ephemeral runners. Verification PR (paired):
nomadkaraoke/python-audio-separator#286.

## Test plan

- [x] `pytest test_ephemeral.py` — 26 passed locally (3 new scheduling
tests)
- [x] Pulumi applied locally (both iterations, function update verified)
- [x] First ephemeral general VM create succeeded (4 RUNNING; previously
100% failure)
- [ ] Image rebuild completes for general+build+gpu with new sudo
provision
- [ ] PR CI passes with all 5 self-hosted jobs running on the new image
- [ ] python-audio-separator#286 GPU integration tests pass on the new
GPU image
- [ ] After merge: `deploy-backend` succeeds on ephemeral build runner;
prod /api/health/detailed reports new version
- [ ] Phase 4 decommission of 7 legacy VMs+disks (separate PR, ~$220/mo
saving)

## Context

- `karaoke-gen/docs/EPHEMERAL-GHA-RUNNERS.md`
- `docs/archive/2026-05-16-ephemeral-gha-runners-plan.md` (workspace)
- Memory: `project_cost_sprint_may2026.md`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 22:42:01 -04:00
..