Files
karaoke-gen/infrastructure
Andrew Beveridge ☄️andClaude Opus 5.5 ff9b716435 feat: first-touch UTM/referrer signup attribution + UTM on YouTube description (v0.269.0) (#1154)
## Summary
This is BACKLOG Frog #2 ("plug the gen revenue leaks"), attribution
track. Until now we stored no acquisition source, so we couldn't tell
which channel brings the people who pay.

- **Capture (frontend):** `lib/attribution.ts` stores the first-touch
`utm_*` params, the external referrer host and the landing path for 30
days. It runs in the `[locale]` layout and in `LocaleRedirect`, before
`location.replace` wipes `document.referrer`. The data is sent with
`POST /auth/magic-link`.
- **Store (backend):** `services/signup_attribution.py` whitelists,
length-caps and sanitises the data. On account creation it sets
`signup_attribution` and `signup_source`: the `utm_source`, or a
referrer channel label such as `youtube`, `google`, `website`, `decide`
or `kjbox`. Malformed input never blocks sign-in, and the request field
is bounded (16 keys, values up to 500 chars).
- **Report:** `GET /api/users/admin/attribution-summary?days=90` returns
signups, verified, made a job, paid and revenue, per source. The admin
user detail shows the signup source and attribution.
- **YouTube:** description template links now carry
`utm_source=youtube&utm_medium=description&utm_campaign=requests|make_cta`.
They're static, so they're still derivable from the title, and the daily
backfill rolls them out. The `requests.nomadkaraoke.com` Cloudflare
redirect now has `preserve_query_string` (Pulumi `nomad-cloudflare`,
**already applied** locally).
- **Docs:** STRIPE-SETUP.md gets a "Legacy Payment Links" section. The
old order-form link `dR67sx…` was **deactivated 2026-10-14** and points
people to gen; it had 0 completed checkouts in about 30 sessions since
Dec. Also updated API.md.

Companion: nomadkaraoke/public-website#34 makes
`nomadkaraoke.com/r/CODE` pass the UTM params through.

## Testing
- `make test`: backend 5986 passed, emulator 150, frontend Jest 1683,
package 2044. All green.
- New tests: `backend/tests/test_signup_attribution.py` (cleaning,
channel labels, magic-link wiring, bounded payload, summary endpoint)
and `frontend/lib/__tests__/attribution.test.ts`.
- Local CodeRabbit: 3 findings fixed (malformed referrer, NaN expiry,
unbounded payload); re-review found 0.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019KuzHAU4rxkv8JfD8NmHTN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:31:16 -04:00
..

Infrastructure as Code with Pulumi

This directory is the home of Nomad Karaoke's shared platform IaC. It holds two Pulumi projects:

Project Path Stack Manages
karaoke-gen-infrastructure infrastructure/ prod The nomadkaraoke GCP project: gen's backend plus shared services (runner manager, divebar, KN sync, error monitor, monitoring, backups, IAM, secrets), and gen's Cloudflare edge security on api.nomadkaraoke.com
nomad-cloudflare infrastructure/cloudflare/ nomadkaraoke/prod The Nomad Karaoke Cloudflare account: zones, DNS, rulesets, email routing, tunnels, Zero Trust, Pages/Worker domains. See cloudflare/README.md

Boundary rule: infra that isn't specific to one app lives here. App repos keep only app-specific stacks (e.g. karaoke-decide's Cloud Run and its API-proxy Worker + route). Every Cloudflare zone ruleset phase has exactly one owner; see the exclusions table in cloudflare/README.md. CI never runs pulumi up for either project: apply locally, then merge.

Directory Structure

infrastructure/
├── __main__.py              # Entry point - wires modules together (~340 lines)
├── config.py                # Centralized configuration constants
├── Pulumi.yaml              # Pulumi project config
├── Pulumi.prod.yaml         # Production stack config
├── requirements.txt         # Python dependencies
│
├── modules/                 # Core GCP resources
│   ├── database.py         # Firestore database + indexes
│   ├── storage.py          # GCS bucket + lifecycle rules
│   ├── artifact_registry.py # Docker repository
│   ├── secrets.py          # Secret Manager secrets
│   ├── cloud_tasks.py      # Task queues
│   ├── cloud_run.py        # Domain mapping, Cloud Run Jobs
│   ├── monitoring.py       # Alert policies
│   └── iam/                # IAM by purpose
│       ├── backend_sa.py   # karaoke-backend service account
│       ├── github_actions_sa.py # WIF + deployer permissions
│       ├── claude_automation_sa.py # Claude Code automation
│       └── worker_sas.py   # VM service accounts
│
├── compute/                 # VM-based resources
│   ├── flacfetch_vm.py     # Audio download service
│   ├── encoding_worker_vm.py # Video encoding VM
│   ├── github_runners.py   # Self-hosted CI runners
│   └── startup_scripts/    # VM startup scripts (.sh)
│       ├── flacfetch.sh
│       ├── encoding_worker.sh
│       └── github_runner.sh
│
├── functions/               # Cloud Functions
│   └── gdrive_validator/
│       ├── main.py
│       ├── requirements.txt
│       └── deploy.sh       # Source upload script
│
└── monitoring/              # Monitoring configs
    ├── dashboard.json
    └── job-inspector.html

Resources Managed

Category Resources
Database Firestore (Native mode), 10 indexes, TTL policy
Storage GCS bucket with lifecycle rules
Registry Artifact Registry for Docker images
Secrets 18 Secret Manager secrets
IAM 7 service accounts, WIF, 50+ IAM bindings
Compute Encoding worker, Flacfetch, 10 GitHub runners
Tasks 5 Cloud Tasks queues
Monitoring 4 alert policies
Functions GDrive validator + scheduler

Quick Start

cd infrastructure

# Preview changes
pulumi preview --diff

# Apply changes
pulumi up

# View outputs
pulumi stack output

Prerequisites

  1. Pulumi CLI: brew install pulumi
  2. Pulumi Account: Sign up at https://app.pulumi.com
  3. Google Cloud SDK: With authenticated account
  4. Python 3.10+: For running Pulumi

Module Guide

config.py

Centralized configuration for the entire infrastructure:

  • Project ID, region, zone
  • Machine types and disk sizes
  • Google Drive folder ID
  • Number of GitHub runners

modules/database.py

Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.

modules/storage.py

GCS bucket with lifecycle rules:

  • 7-day cleanup of staging data
  • 24-hour cleanup of raw audio files

modules/iam/

Service accounts organized by purpose:

  • backend_sa.py: Cloud Run backend permissions
  • github_actions_sa.py: CI/CD via Workload Identity Federation
  • worker_sas.py: VM service accounts (flacfetch, encoding, runners)

compute/

VM definitions with startup scripts as separate files. Each module creates:

  • Static IP (where needed)
  • VM instance
  • Firewall rules

Adding Secret Values

After pulumi up, add actual secret values:

# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-

# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-

# GitHub runner PAT (admin:org scope for org-level runners)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-

GDrive Validator

The Cloud Function validates your public Google Drive folder daily:

  • Checks for duplicate sequence numbers
  • Validates filename formats
  • Detects sequence gaps
  • Sends Pushbullet notifications

Deploy:

cd functions/gdrive_validator
./deploy.sh

Note: deploy.sh uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.

Self-Hosted GitHub Runners

20 GCP VMs serve as organization-level self-hosted runners, available to all repos in the nomadkaraoke org:

  • 200GB SSD (vs 14GB on GitHub-hosted)
  • Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
  • Labels: self-hosted, linux, x64, gcp, large-disk

PAT Requirements: The github-runner-pat secret requires admin:org scope (specifically manage_runners:org) for organization-level runner registration.

Usage in workflows (any nomadkaraoke repo):

runs-on: [self-hosted, linux, gcp]

Troubleshooting:

# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
  "sudo cat /var/log/github-runner-startup.log"

# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
  "sudo systemctl status actions.runner.*"

Encoding Worker

High-performance VM for video encoding:

  • Machine: c4-standard-8 (8 vCPU, 32GB RAM)
  • Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
  • Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts

API Endpoints:

  • POST /encode - Full encoding job
  • POST /encode-preview - Quick 480p preview
  • GET /status/{job_id} - Job status
  • GET /health - Health check

Troubleshooting:

gcloud compute ssh encoding-worker --zone=us-central1-c -- \
  "sudo systemctl status encoding-worker"

State Management

Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:

pulumi refresh

Multi-Environment

pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up

pulumi stack select prod  # Switch stacks

Troubleshooting

"Resource already exists"

Import manually-created resources:

pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke

State conflicts

pulumi refresh  # Sync state with cloud

Permissions errors

Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.

Future Improvements

Phase 5: Custom Machine Images (Planned)

The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.

See: docs/PHASE5-PACKER-IMAGE-PLAN.md for the implementation plan.

Benefits:

  • Startup time: 10 min → 30 sec
  • Pre-baked: Python 3.13, FFmpeg, fonts
  • Hot updates: karaoke-gen wheel still downloaded at runtime