## Summary Adopts the Nomad Karaoke Cloudflare account (`4585249d…`) into IaC as a second Pulumi project, `infrastructure/cloudflare/` (project `nomad-cloudflare`, stack `nomadkaraoke/prod`). It sits beside the GCP platform project, and `infrastructure/` is now documented as the **shared platform IaC home**. App repos keep only app-specific stacks. **130 resources, all adopted with `pulumi import`, with no recreation:** - 5 zones (protected) + `ssl` mode - 84 DNS records - 8 zone rulesets - Email Routing: address, enablement (protected), catch-alls, rules - `nomad-kjbox` / `nomad-hf2` tunnels (protected) + ingress - Zero Trust: org (protected), OTP IdP, service token, **3 reusable policies**, 2 apps - Pages projects + `gen` domain, Worker custom domains + interceptor route **Single ownership, explicitly excluded (see `infrastructure/cloudflare/README.md`):** - gen `edge_security` phases + `api`/`api-edge-test` DNS - runtime tenant domains - decide's route/Worker - read-only records - karaokehunt's `http_config_settings` (pulumi-cloudflare 6.22.1 drops `set_config bic:false`, so refresh fails) **Applied already (repo rule: apply locally before merge):** - Final `pulumi preview` shows **131 unchanged, 0 changes**. - The only live mutation was moving the two Access apps from app-scoped to reusable code-managed policies, with the same rules and 730h sessions. Verified afterwards: anonymous → 302 to `nomadkaraoke.cloudflareaccess.com`, and the kjbox service token → 200. - The post-import "updates" with no property diff were one-time state settlements. Each was confirmed as a no-op against a live before/after snapshot. ## Testing - `infrastructure/cloudflare/tests/test_build_import_list.py`: 4 passed (ownership exclusions, pagination, unique names). - `pulumi preview`: 131 unchanged. - Live spot-check of site, gen, api, decide, sing, singa, kjbox.cc, kjtip.me, karaokehunt and nomadkaroke. - Local CodeRabbit: cycle 1 found 1 issue (pagination), fixed; cycle 2 found 0. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014XpBykm3FJmoFTjuNQ18Bi --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
8.2 KiB
Infrastructure as Code with Pulumi
This directory is the home of Nomad Karaoke's shared platform IaC. It holds two Pulumi projects:
| Project | Path | Stack | Manages |
|---|---|---|---|
karaoke-gen-infrastructure |
infrastructure/ |
prod |
The nomadkaraoke GCP project: gen's backend plus shared services (runner manager, divebar, KN sync, error monitor, monitoring, backups, IAM, secrets), and gen's Cloudflare edge security on api.nomadkaraoke.com |
nomad-cloudflare |
infrastructure/cloudflare/ |
nomadkaraoke/prod |
The Nomad Karaoke Cloudflare account: zones, DNS, rulesets, email routing, tunnels, Zero Trust, Pages/Worker domains. See cloudflare/README.md |
Boundary rule: infra that isn't specific to one app lives here. App repos keep only
app-specific stacks (e.g. karaoke-decide's Cloud Run and its API-proxy Worker + route). Every
Cloudflare zone ruleset phase has exactly one owner; see the exclusions table in cloudflare/README.md.
CI never runs pulumi up for either project: apply locally, then merge.
Directory Structure
infrastructure/
├── __main__.py # Entry point - wires modules together (~340 lines)
├── config.py # Centralized configuration constants
├── Pulumi.yaml # Pulumi project config
├── Pulumi.prod.yaml # Production stack config
├── requirements.txt # Python dependencies
│
├── modules/ # Core GCP resources
│ ├── database.py # Firestore database + indexes
│ ├── storage.py # GCS bucket + lifecycle rules
│ ├── artifact_registry.py # Docker repository
│ ├── secrets.py # Secret Manager secrets
│ ├── cloud_tasks.py # Task queues
│ ├── cloud_run.py # Domain mapping, Cloud Run Jobs
│ ├── monitoring.py # Alert policies
│ └── iam/ # IAM by purpose
│ ├── backend_sa.py # karaoke-backend service account
│ ├── github_actions_sa.py # WIF + deployer permissions
│ ├── claude_automation_sa.py # Claude Code automation
│ └── worker_sas.py # VM service accounts
│
├── compute/ # VM-based resources
│ ├── flacfetch_vm.py # Audio download service
│ ├── encoding_worker_vm.py # Video encoding VM
│ ├── github_runners.py # Self-hosted CI runners
│ └── startup_scripts/ # VM startup scripts (.sh)
│ ├── flacfetch.sh
│ ├── encoding_worker.sh
│ └── github_runner.sh
│
├── functions/ # Cloud Functions
│ └── gdrive_validator/
│ ├── main.py
│ ├── requirements.txt
│ └── deploy.sh # Source upload script
│
└── monitoring/ # Monitoring configs
├── dashboard.json
└── job-inspector.html
Resources Managed
| Category | Resources |
|---|---|
| Database | Firestore (Native mode), 10 indexes, TTL policy |
| Storage | GCS bucket with lifecycle rules |
| Registry | Artifact Registry for Docker images |
| Secrets | 18 Secret Manager secrets |
| IAM | 7 service accounts, WIF, 50+ IAM bindings |
| Compute | Encoding worker, Flacfetch, 10 GitHub runners |
| Tasks | 5 Cloud Tasks queues |
| Monitoring | 4 alert policies |
| Functions | GDrive validator + scheduler |
Quick Start
cd infrastructure
# Preview changes
pulumi preview --diff
# Apply changes
pulumi up
# View outputs
pulumi stack output
Prerequisites
- Pulumi CLI:
brew install pulumi - Pulumi Account: Sign up at https://app.pulumi.com
- Google Cloud SDK: With authenticated account
- Python 3.10+: For running Pulumi
Module Guide
config.py
Centralized configuration for the entire infrastructure:
- Project ID, region, zone
- Machine types and disk sizes
- Google Drive folder ID
- Number of GitHub runners
modules/database.py
Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.
modules/storage.py
GCS bucket with lifecycle rules:
- 7-day cleanup of staging data
- 24-hour cleanup of raw audio files
modules/iam/
Service accounts organized by purpose:
backend_sa.py: Cloud Run backend permissionsgithub_actions_sa.py: CI/CD via Workload Identity Federationworker_sas.py: VM service accounts (flacfetch, encoding, runners)
compute/
VM definitions with startup scripts as separate files. Each module creates:
- Static IP (where needed)
- VM instance
- Firewall rules
Adding Secret Values
After pulumi up, add actual secret values:
# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-
# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-
# GitHub runner PAT (admin:org scope for org-level runners)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-
GDrive Validator
The Cloud Function validates your public Google Drive folder daily:
- Checks for duplicate sequence numbers
- Validates filename formats
- Detects sequence gaps
- Sends Pushbullet notifications
Deploy:
cd functions/gdrive_validator
./deploy.sh
Note: deploy.sh uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.
Self-Hosted GitHub Runners
20 GCP VMs serve as organization-level self-hosted runners, available to all repos in the nomadkaraoke org:
- 200GB SSD (vs 14GB on GitHub-hosted)
- Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
- Labels:
self-hosted,linux,x64,gcp,large-disk
PAT Requirements: The github-runner-pat secret requires admin:org scope (specifically manage_runners:org) for organization-level runner registration.
Usage in workflows (any nomadkaraoke repo):
runs-on: [self-hosted, linux, gcp]
Troubleshooting:
# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo cat /var/log/github-runner-startup.log"
# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo systemctl status actions.runner.*"
Encoding Worker
High-performance VM for video encoding:
- Machine: c4-standard-8 (8 vCPU, 32GB RAM)
- Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
- Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts
API Endpoints:
POST /encode- Full encoding jobPOST /encode-preview- Quick 480p previewGET /status/{job_id}- Job statusGET /health- Health check
Troubleshooting:
gcloud compute ssh encoding-worker --zone=us-central1-c -- \
"sudo systemctl status encoding-worker"
State Management
Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:
pulumi refresh
Multi-Environment
pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up
pulumi stack select prod # Switch stacks
Troubleshooting
"Resource already exists"
Import manually-created resources:
pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke
State conflicts
pulumi refresh # Sync state with cloud
Permissions errors
Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.
Future Improvements
Phase 5: Custom Machine Images (Planned)
The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.
See: docs/PHASE5-PACKER-IMAGE-PLAN.md for the implementation plan.
Benefits:
- Startup time: 10 min → 30 sec
- Pre-baked: Python 3.13, FFmpeg, fonts
- Hot updates: karaoke-gen wheel still downloaded at runtime