## Summary Andrew asked for #alerts to carry only real problems. Routine and informational posts now go to the new **#noise** channel. The GitHub org PR webhook was already moved to #noise separately. | Message | Before | After | |---|---|---| | Error monitor: New Error Pattern / Spike / Incident | #alerts | #alerts | | Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** | | Error monitor: 📊 Daily Error Digest | #alerts | **#noise** | | Nightly Backup Report (clean run) | #alerts | **#noise** | | Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED") | #alerts | #alerts | | ✅ E2E Daily Test PASSED | #alerts | **#noise** | | 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise** | | 🩹 self-heal PR needs CI approval click / could not open | #alerts | #alerts | | E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish completeness SHADOW, Gemini unavailable, OAuth credential alerts, quarterly DR drill reminder | #alerts | #alerts (unchanged) | ## How the noise webhook is resolved - **Backend / error monitor and the backup function** read it at runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager secret `discord-noise-webhook` (the name can be overridden with `DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed, because both service accounts already have project-wide `secretAccessor`. - **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL`. - **Nothing is silently dropped.** If the noise webhook can't be resolved, or a post to it fails, the message goes to #alerts instead. In the workflow this uses `curl -fsS` with a retry on the alert webhook. - The error monitor now rate-limits each channel separately (10 per run each), so noise can't use up the alert budget during an incident. ## Deploy notes - **Backup Cloud Function:** CI doesn't deploy its source. Run `infrastructure/functions/backup_to_aws/deploy.sh` after merge. - **Error monitor job:** runs `karaoke-backend:latest`, so it picks up the change once CI pushes the image. - **No `pulumi up` needed.** ## Testing - `tests/unit/services/error_monitor`: 376 passed. New tests cover noise vs alert routing, fallback when the noise webhook is missing or fails, separate rate limits, and secret resolution. - `backend/tests` (excluding emulator): 6418 passed, 36 skipped. - `infrastructure/functions/backup_to_aws`: 60 passed, including the new `test_discord_alert.py` and routing tests in `test_main.py`. These were run in a scratch venv because `functions_framework` isn't in the conda env, which was already the case before this change. - `actionlint` on `e2e-daily.yml`: no new findings. The only findings are info-level shellcheck notes that were already there. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Infrastructure as Code with Pulumi
This directory is the home of Nomad Karaoke's shared platform IaC. It holds two Pulumi projects:
| Project | Path | Stack | Manages |
|---|---|---|---|
karaoke-gen-infrastructure |
infrastructure/ |
prod |
The nomadkaraoke GCP project: gen's backend plus shared services (runner manager, divebar, KN sync, error monitor, monitoring, backups, IAM, secrets), and gen's Cloudflare edge security on api.nomadkaraoke.com |
nomad-cloudflare |
infrastructure/cloudflare/ |
nomadkaraoke/prod |
The Nomad Karaoke Cloudflare account: zones, DNS, rulesets, email routing, tunnels, Zero Trust, Pages/Worker domains. See cloudflare/README.md |
Boundary rule: infra that isn't specific to one app lives here. App repos keep only
app-specific stacks (e.g. karaoke-decide's Cloud Run and its API-proxy Worker + route). Every
Cloudflare zone ruleset phase has exactly one owner; see the exclusions table in cloudflare/README.md.
CI never runs pulumi up for either project: apply locally, then merge.
Directory Structure
infrastructure/
├── __main__.py # Entry point - wires modules together (~340 lines)
├── config.py # Centralized configuration constants
├── Pulumi.yaml # Pulumi project config
├── Pulumi.prod.yaml # Production stack config
├── requirements.txt # Python dependencies
│
├── modules/ # Core GCP resources
│ ├── database.py # Firestore database + indexes
│ ├── storage.py # GCS bucket + lifecycle rules
│ ├── artifact_registry.py # Docker repository
│ ├── secrets.py # Secret Manager secrets
│ ├── cloud_tasks.py # Task queues
│ ├── cloud_run.py # Domain mapping, Cloud Run Jobs
│ ├── monitoring.py # Alert policies
│ └── iam/ # IAM by purpose
│ ├── backend_sa.py # karaoke-backend service account
│ ├── github_actions_sa.py # WIF + deployer permissions
│ ├── claude_automation_sa.py # Claude Code automation
│ └── worker_sas.py # VM service accounts
│
├── compute/ # VM-based resources
│ ├── flacfetch_vm.py # Audio download service
│ ├── encoding_worker_vm.py # Video encoding VM
│ ├── github_runners.py # Self-hosted CI runners
│ └── startup_scripts/ # VM startup scripts (.sh)
│ ├── flacfetch.sh
│ ├── encoding_worker.sh
│ └── github_runner.sh
│
├── functions/ # Cloud Functions
│ └── gdrive_validator/
│ ├── main.py
│ ├── requirements.txt
│ └── deploy.sh # Source upload script
│
└── monitoring/ # Monitoring configs
├── dashboard.json
└── job-inspector.html
Resources Managed
| Category | Resources |
|---|---|
| Database | Firestore (Native mode), 10 indexes, TTL policy |
| Storage | GCS bucket with lifecycle rules |
| Registry | Artifact Registry for Docker images |
| Secrets | 18 Secret Manager secrets |
| IAM | 7 service accounts, WIF, 50+ IAM bindings |
| Compute | Encoding worker, Flacfetch, 10 GitHub runners |
| Tasks | 5 Cloud Tasks queues |
| Monitoring | 4 alert policies |
| Functions | GDrive validator + scheduler |
Quick Start
cd infrastructure
# Preview changes
pulumi preview --diff
# Apply changes
pulumi up
# View outputs
pulumi stack output
Prerequisites
- Pulumi CLI:
brew install pulumi - Pulumi Account: Sign up at https://app.pulumi.com
- Google Cloud SDK: With authenticated account
- Python 3.10+: For running Pulumi
Module Guide
config.py
Centralized configuration for the entire infrastructure:
- Project ID, region, zone
- Machine types and disk sizes
- Google Drive folder ID
- Number of GitHub runners
modules/database.py
Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.
modules/storage.py
GCS bucket with lifecycle rules:
- 7-day cleanup of staging data
- 24-hour cleanup of raw audio files
modules/iam/
Service accounts organized by purpose:
backend_sa.py: Cloud Run backend permissionsgithub_actions_sa.py: CI/CD via Workload Identity Federationworker_sas.py: VM service accounts (flacfetch, encoding, runners)
compute/
VM definitions with startup scripts as separate files. Each module creates:
- Static IP (where needed)
- VM instance
- Firewall rules
Adding Secret Values
After pulumi up, add actual secret values:
# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-
# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-
# GitHub runner PAT (admin:org scope for org-level runners)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-
GDrive Validator
The Cloud Function validates your public Google Drive folder daily:
- Checks for duplicate sequence numbers
- Validates filename formats
- Detects sequence gaps
- Sends Pushbullet notifications
Deploy:
cd functions/gdrive_validator
./deploy.sh
Note: deploy.sh uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.
Self-Hosted GitHub Runners
20 GCP VMs serve as organization-level self-hosted runners, available to all repos in the nomadkaraoke org:
- 200GB SSD (vs 14GB on GitHub-hosted)
- Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
- Labels:
self-hosted,linux,x64,gcp,large-disk
PAT Requirements: The github-runner-pat secret requires admin:org scope (specifically manage_runners:org) for organization-level runner registration.
Usage in workflows (any nomadkaraoke repo):
runs-on: [self-hosted, linux, gcp]
Troubleshooting:
# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo cat /var/log/github-runner-startup.log"
# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo systemctl status actions.runner.*"
Encoding Worker
High-performance VM for video encoding:
- Machine: c4-standard-8 (8 vCPU, 32GB RAM)
- Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
- Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts
API Endpoints:
POST /encode- Full encoding jobPOST /encode-preview- Quick 480p previewGET /status/{job_id}- Job statusGET /health- Health check
Troubleshooting:
gcloud compute ssh encoding-worker --zone=us-central1-c -- \
"sudo systemctl status encoding-worker"
State Management
Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:
pulumi refresh
Multi-Environment
pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up
pulumi stack select prod # Switch stacks
Troubleshooting
"Resource already exists"
Import manually-created resources:
pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke
State conflicts
pulumi refresh # Sync state with cloud
Permissions errors
Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.
Future Improvements
Phase 5: Custom Machine Images (Planned)
The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.
See: docs/PHASE5-PACKER-IMAGE-PLAN.md for the implementation plan.
Benefits:
- Startup time: 10 min → 30 sec
- Pre-baked: Python 3.13, FFmpeg, fonts
- Hot updates: karaoke-gen wheel still downloaded at runtime