Files
karaoke-gen/infrastructure
Andrew Beveridge ☄️andClaude Opus 5.5 1abdd8c860 fix(cost): runner-manager only dispatches VMs for jobs it can serve; enforce AR cleanup (v0.286.1) (#1228)
## Summary
On 2026-10-27, **238 GCP `gha-general` VMs (~$67)** ran in one day. This
pushed October's GCP bill past the $300 budget.

**Cause:**
- nomad-mobile's E2E jobs run on the utility MacBook (`[self-hosted,
macOS, nomad-mbp]`).
- The runner manager acted on *any* job labelled `self-hosted`, so each
macOS job got a Linux VM that could never take it.
- While the job waited for an MBP lane, `redispatch_stalled_jobs`
launched another VM every 5 min for up to 60 min.
- Each of those VMs then idled until the 2 h reaper deleted it.

## Changes
- `ephemeral.can_serve(labels)`: GitHub assigns a job only to a runner
that advertises every label the job requests, so a job is ours only if
its labels ⊆ the resolved family's `runner_labels_for()`.
- The new check gates the webhook (`main.py`),
`list_queued_self_hosted_jobs` and `redispatch_stalled_jobs`.
- I checked every `runs-on: [self-hosted, …]` label set in the org's
workflows. All of them except the macOS lanes are subsets of our
families' labels, so no existing job loses its VM.
- CI: these function tests never ran in CI before. A new
`runner-manager-tests` job runs them on infra or ci.yml changes and is
enforced by CI Gate.
- Docs: `docs/EPHEMERAL-GHA-RUNNERS.md` § "Only jobs our runners can
serve".

## Testing
- 6 new tests: macOS jobs, unknown labels, our families, the redispatch
skip, queued-scan exclusion, and the webhook no-op. All 71
runner_manager tests pass locally.
- Local CodeRabbit review: 2 cycles, both CI-wiring findings fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:35:44 -04:00
..

Infrastructure as Code with Pulumi

This directory is the home of Nomad Karaoke's shared platform IaC. It holds two Pulumi projects:

Project Path Stack Manages
karaoke-gen-infrastructure infrastructure/ prod The nomadkaraoke GCP project: gen's backend plus shared services (runner manager, divebar, KN sync, error monitor, monitoring, backups, IAM, secrets), and gen's Cloudflare edge security on api.nomadkaraoke.com
nomad-cloudflare infrastructure/cloudflare/ nomadkaraoke/prod The Nomad Karaoke Cloudflare account: zones, DNS, rulesets, email routing, tunnels, Zero Trust, Pages/Worker domains. See cloudflare/README.md

Boundary rule: infra that isn't specific to one app lives here. App repos keep only app-specific stacks (e.g. karaoke-decide's Cloud Run and its API-proxy Worker + route). Every Cloudflare zone ruleset phase has exactly one owner; see the exclusions table in cloudflare/README.md. CI never runs pulumi up for either project: apply locally, then merge.

Directory Structure

infrastructure/
├── __main__.py              # Entry point - wires modules together (~340 lines)
├── config.py                # Centralized configuration constants
├── Pulumi.yaml              # Pulumi project config
├── Pulumi.prod.yaml         # Production stack config
├── requirements.txt         # Python dependencies
│
├── modules/                 # Core GCP resources
│   ├── database.py         # Firestore database + indexes
│   ├── storage.py          # GCS bucket + lifecycle rules
│   ├── artifact_registry.py # Docker repository
│   ├── secrets.py          # Secret Manager secrets
│   ├── cloud_tasks.py      # Task queues
│   ├── cloud_run.py        # Domain mapping, Cloud Run Jobs
│   ├── monitoring.py       # Alert policies
│   └── iam/                # IAM by purpose
│       ├── backend_sa.py   # karaoke-backend service account
│       ├── github_actions_sa.py # WIF + deployer permissions
│       ├── claude_automation_sa.py # Claude Code automation
│       └── worker_sas.py   # VM service accounts
│
├── compute/                 # VM-based resources
│   ├── flacfetch_vm.py     # Audio download service
│   ├── encoding_worker_vm.py # Video encoding VM
│   ├── github_runners.py   # Self-hosted CI runners
│   └── startup_scripts/    # VM startup scripts (.sh)
│       ├── flacfetch.sh
│       ├── encoding_worker.sh
│       └── github_runner.sh
│
├── functions/               # Cloud Functions
│   └── gdrive_validator/
│       ├── main.py
│       ├── requirements.txt
│       └── deploy.sh       # Source upload script
│
└── monitoring/              # Monitoring configs
    ├── dashboard.json
    └── job-inspector.html

Resources Managed

Category Resources
Database Firestore (Native mode), 10 indexes, TTL policy
Storage GCS bucket with lifecycle rules
Registry Artifact Registry for Docker images
Secrets 18 Secret Manager secrets
IAM 7 service accounts, WIF, 50+ IAM bindings
Compute Encoding worker, Flacfetch, 10 GitHub runners
Tasks 5 Cloud Tasks queues
Monitoring 4 alert policies
Functions GDrive validator + scheduler

Quick Start

cd infrastructure

# Preview changes
pulumi preview --diff

# Apply changes
pulumi up

# View outputs
pulumi stack output

Prerequisites

  1. Pulumi CLI: brew install pulumi
  2. Pulumi Account: Sign up at https://app.pulumi.com
  3. Google Cloud SDK: With authenticated account
  4. Python 3.10+: For running Pulumi

Module Guide

config.py

Centralized configuration for the entire infrastructure:

  • Project ID, region, zone
  • Machine types and disk sizes
  • Google Drive folder ID
  • Number of GitHub runners

modules/database.py

Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.

modules/storage.py

GCS bucket with lifecycle rules:

  • 7-day cleanup of staging data
  • 24-hour cleanup of raw audio files

modules/iam/

Service accounts organized by purpose:

  • backend_sa.py: Cloud Run backend permissions
  • github_actions_sa.py: CI/CD via Workload Identity Federation
  • worker_sas.py: VM service accounts (flacfetch, encoding, runners)

compute/

VM definitions with startup scripts as separate files. Each module creates:

  • Static IP (where needed)
  • VM instance
  • Firewall rules

Adding Secret Values

After pulumi up, add actual secret values:

# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-

# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-

# GitHub runner PAT (admin:org scope for org-level runners)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-

GDrive Validator

The Cloud Function validates your public Google Drive folder daily:

  • Checks for duplicate sequence numbers
  • Validates filename formats
  • Detects sequence gaps
  • Sends Pushbullet notifications

Deploy:

cd functions/gdrive_validator
./deploy.sh

Note: deploy.sh uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.

Self-Hosted GitHub Runners

20 GCP VMs serve as organization-level self-hosted runners, available to all repos in the nomadkaraoke org:

  • 200GB SSD (vs 14GB on GitHub-hosted)
  • Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
  • Labels: self-hosted, linux, x64, gcp, large-disk

PAT Requirements: The github-runner-pat secret requires admin:org scope (specifically manage_runners:org) for organization-level runner registration.

Usage in workflows (any nomadkaraoke repo):

runs-on: [self-hosted, linux, gcp]

Troubleshooting:

# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
  "sudo cat /var/log/github-runner-startup.log"

# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
  "sudo systemctl status actions.runner.*"

Encoding Worker

High-performance VM for video encoding:

  • Machine: c4-standard-8 (8 vCPU, 32GB RAM)
  • Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
  • Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts

API Endpoints:

  • POST /encode - Full encoding job
  • POST /encode-preview - Quick 480p preview
  • GET /status/{job_id} - Job status
  • GET /health - Health check

Troubleshooting:

gcloud compute ssh encoding-worker --zone=us-central1-c -- \
  "sudo systemctl status encoding-worker"

State Management

Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:

pulumi refresh

Multi-Environment

pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up

pulumi stack select prod  # Switch stacks

Troubleshooting

"Resource already exists"

Import manually-created resources:

pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke

State conflicts

pulumi refresh  # Sync state with cloud

Permissions errors

Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.

Future Improvements

Phase 5: Custom Machine Images (Planned)

The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.

See: docs/PHASE5-PACKER-IMAGE-PLAN.md for the implementation plan.

Benefits:

  • Startup time: 10 min → 30 sec
  • Pre-baked: Python 3.13, FFmpeg, fonts
  • Hot updates: karaoke-gen wheel still downloaded at runtime