## ⚠️ Merge prerequisite (do BEFORE merging) The Secret Manager secret `decide-gen-partner-secret` must have a version before this merges, otherwise the CI deploy's `--set-secrets ...DECIDE_GEN_PARTNER_SECRET=decide-gen-partner-secret:latest` fails. ``` gcloud auth login admin@nomadkaraoke.com cd ~/Projects/nomadkaraoke/karaoke-gen-decide-partner-resolve/infrastructure GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token) pulumi up # review: should only add the decide-gen-partner-secret Secret openssl rand -hex 32 | tr -d '\n' | gcloud secrets versions add decide-gen-partner-secret --project=nomadkaraoke --data-file=- ``` No extra IAM grant needed: gen's API runs as `karaoke-backend@`, which already has project-level `secretmanager.secretAccessor`. ## Summary - `POST /api/kjbox/catalog/resolve` now also accepts karaoke-decide via header `X-Partner-Secret` (new setting `decide_partner_secret` / env `DECIDE_GEN_PARTNER_SECRET`). Other kjbox endpoints stay `X-Kjbox-Secret` only. decide gets its own AI-call budget (60/min) so kjbox nights are never starved; the verdict cache is shared. - Fake match judge / resolve for the local E2E stack (`FAKE_WORKERS`): `judge_kind` / `judge_canonical_*` knobs; no Gemini or catalog calls. - Error monitor now watches decide's new `karaoke-decide-search` service and `song-index-build` job. - Pulumi: declares the `decide-gen-partner-secret` Secret (value added manually); CI deploy mounts it. - Version 0.278.10. ## Testing - [x] Changed-module + kjbox/catalog/error-monitor/fake-worker backend tests pass (133 passed). Full `backend/tests` suite exceeds 10 min locally, so left to CI. ## Review - CodeRabbit CLI failed twice (WebSocket closed / connection error), so no local CodeRabbit review completed; I did a manual diff review instead (no issues found). Deliberately NO `@coderabbitai ignore`, so the GitHub bot reviews this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VsLw9cTTMBab1tiRxCsYrm --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Infrastructure as Code with Pulumi
This directory is the home of Nomad Karaoke's shared platform IaC. It holds two Pulumi projects:
| Project | Path | Stack | Manages |
|---|---|---|---|
karaoke-gen-infrastructure |
infrastructure/ |
prod |
The nomadkaraoke GCP project: gen's backend plus shared services (runner manager, divebar, KN sync, error monitor, monitoring, backups, IAM, secrets), and gen's Cloudflare edge security on api.nomadkaraoke.com |
nomad-cloudflare |
infrastructure/cloudflare/ |
nomadkaraoke/prod |
The Nomad Karaoke Cloudflare account: zones, DNS, rulesets, email routing, tunnels, Zero Trust, Pages/Worker domains. See cloudflare/README.md |
Boundary rule: infra that isn't specific to one app lives here. App repos keep only
app-specific stacks (e.g. karaoke-decide's Cloud Run and its API-proxy Worker + route). Every
Cloudflare zone ruleset phase has exactly one owner; see the exclusions table in cloudflare/README.md.
CI never runs pulumi up for either project: apply locally, then merge.
Directory Structure
infrastructure/
├── __main__.py # Entry point - wires modules together (~340 lines)
├── config.py # Centralized configuration constants
├── Pulumi.yaml # Pulumi project config
├── Pulumi.prod.yaml # Production stack config
├── requirements.txt # Python dependencies
│
├── modules/ # Core GCP resources
│ ├── database.py # Firestore database + indexes
│ ├── storage.py # GCS bucket + lifecycle rules
│ ├── artifact_registry.py # Docker repository
│ ├── secrets.py # Secret Manager secrets
│ ├── cloud_tasks.py # Task queues
│ ├── cloud_run.py # Domain mapping, Cloud Run Jobs
│ ├── monitoring.py # Alert policies
│ └── iam/ # IAM by purpose
│ ├── backend_sa.py # karaoke-backend service account
│ ├── github_actions_sa.py # WIF + deployer permissions
│ ├── claude_automation_sa.py # Claude Code automation
│ └── worker_sas.py # VM service accounts
│
├── compute/ # VM-based resources
│ ├── flacfetch_vm.py # Audio download service
│ ├── encoding_worker_vm.py # Video encoding VM
│ ├── github_runners.py # Self-hosted CI runners
│ └── startup_scripts/ # VM startup scripts (.sh)
│ ├── flacfetch.sh
│ ├── encoding_worker.sh
│ └── github_runner.sh
│
├── functions/ # Cloud Functions
│ └── gdrive_validator/
│ ├── main.py
│ ├── requirements.txt
│ └── deploy.sh # Source upload script
│
└── monitoring/ # Monitoring configs
├── dashboard.json
└── job-inspector.html
Resources Managed
| Category | Resources |
|---|---|
| Database | Firestore (Native mode), 10 indexes, TTL policy |
| Storage | GCS bucket with lifecycle rules |
| Registry | Artifact Registry for Docker images |
| Secrets | 18 Secret Manager secrets |
| IAM | 7 service accounts, WIF, 50+ IAM bindings |
| Compute | Encoding worker, Flacfetch, 10 GitHub runners |
| Tasks | 5 Cloud Tasks queues |
| Monitoring | 4 alert policies |
| Functions | GDrive validator + scheduler |
Quick Start
cd infrastructure
# Preview changes
pulumi preview --diff
# Apply changes
pulumi up
# View outputs
pulumi stack output
Prerequisites
- Pulumi CLI:
brew install pulumi - Pulumi Account: Sign up at https://app.pulumi.com
- Google Cloud SDK: With authenticated account
- Python 3.10+: For running Pulumi
Module Guide
config.py
Centralized configuration for the entire infrastructure:
- Project ID, region, zone
- Machine types and disk sizes
- Google Drive folder ID
- Number of GitHub runners
modules/database.py
Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.
modules/storage.py
GCS bucket with lifecycle rules:
- 7-day cleanup of staging data
- 24-hour cleanup of raw audio files
modules/iam/
Service accounts organized by purpose:
backend_sa.py: Cloud Run backend permissionsgithub_actions_sa.py: CI/CD via Workload Identity Federationworker_sas.py: VM service accounts (flacfetch, encoding, runners)
compute/
VM definitions with startup scripts as separate files. Each module creates:
- Static IP (where needed)
- VM instance
- Firewall rules
Adding Secret Values
After pulumi up, add actual secret values:
# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-
# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-
# GitHub runner PAT (admin:org scope for org-level runners)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-
GDrive Validator
The Cloud Function validates your public Google Drive folder daily:
- Checks for duplicate sequence numbers
- Validates filename formats
- Detects sequence gaps
- Sends Pushbullet notifications
Deploy:
cd functions/gdrive_validator
./deploy.sh
Note: deploy.sh uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.
Self-Hosted GitHub Runners
20 GCP VMs serve as organization-level self-hosted runners, available to all repos in the nomadkaraoke org:
- 200GB SSD (vs 14GB on GitHub-hosted)
- Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
- Labels:
self-hosted,linux,x64,gcp,large-disk
PAT Requirements: The github-runner-pat secret requires admin:org scope (specifically manage_runners:org) for organization-level runner registration.
Usage in workflows (any nomadkaraoke repo):
runs-on: [self-hosted, linux, gcp]
Troubleshooting:
# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo cat /var/log/github-runner-startup.log"
# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo systemctl status actions.runner.*"
Encoding Worker
High-performance VM for video encoding:
- Machine: c4-standard-8 (8 vCPU, 32GB RAM)
- Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
- Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts
API Endpoints:
POST /encode- Full encoding jobPOST /encode-preview- Quick 480p previewGET /status/{job_id}- Job statusGET /health- Health check
Troubleshooting:
gcloud compute ssh encoding-worker --zone=us-central1-c -- \
"sudo systemctl status encoding-worker"
State Management
Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:
pulumi refresh
Multi-Environment
pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up
pulumi stack select prod # Switch stacks
Troubleshooting
"Resource already exists"
Import manually-created resources:
pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke
State conflicts
pulumi refresh # Sync state with cloud
Permissions errors
Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.
Future Improvements
Phase 5: Custom Machine Images (Planned)
The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.
See: docs/PHASE5-PACKER-IMAGE-PLAN.md for the implementation plan.
Benefits:
- Startup time: 10 min → 30 sec
- Pre-baked: Python 3.13, FFmpeg, fonts
- Hot updates: karaoke-gen wheel still downloaded at runtime