Files
karaoke-gen/infrastructure
Andrew Beveridge ☄️andClaude Opus 5.5 fe89795aec fix: deliver worker-sent pushes/emails before Cloud Run Job exit; Web Push from workers (v0.284.1) (#1221)
Follow-up to #1220 (mobile Phase 6), from its whole-branch review.

## Problem
Cloud Run Job workers run inside `asyncio.run(...)`.
`JobManager._send_push_notification` (and the completion email / idle
reminder) used `loop.create_task(...)` and the worker returned
immediately, so `asyncio.run` cancelled the send. Prod logs: all 62
"Push notifications sent" lines in 14 days came from the API service,
none from worker jobs. Separately, worker jobs never had
`ENABLE_PUSH_NOTIFICATIONS` / VAPID keys, so even a delivered Web Push
would have been skipped.


## Customer impact check (emails)
Checked 2026-09-25 → 2026-10-25 (full log retention): **no customer
emails were lost.** 289 COMPLETE transitions → 11 deliberately
suppressed re-renders, 278 completion emails scheduled and 278 sent
(email_log has 297 completion records, all with Postmark MessageIDs;
Postmark outbound ≥ email_log). The email coroutines happen to do all
their I/O synchronously, so they finished in the task's first step
before `asyncio.run` cancelled; pushes hand off to a thread and were
cancelled. The drain removes that fragility for emails too. **Only
pushes were lost** (0 worker pushes in 14 days).

## Fix
- Notification tasks are tracked; every worker `main()` wraps its job in
`with_notification_drain` (bounded 10 s wait on our coroutines; provider
calls keep their own timeouts). The API server path stays
fire-and-forget.
- `failure_push_sent` is written only after the push was attempted;
in-flight id cleared via done-callback.
- Worker jobs (audio-download, audio-separation, bulk-search,
lyrics-transcription, video-encoding) get
`ENABLE_PUSH_NOTIFICATIONS=true` +
`VAPID_PUBLIC_KEY`/`VAPID_PRIVATE_KEY` (same secrets as the API).
CI-pinned jobs now `ignore_changes` their image (CI owns it), so a
targeted `pulumi up` changes env only.
- Expo messages carry `channelId: "jobs"` (the app's Android channel).

## Deploy
After merge, a targeted `pulumi up` from main on the 5 jobs. Preview
(read-only, from this branch): 5 updates, each only
+`ENABLE_PUSH_NOTIFICATIONS`/+`VAPID_*`, no removals, no image change,
283 unchanged.

**Behaviour change:** Web Push subscribers will start receiving the
worker-triggered pushes (video ready, needs review, failed) they were
always meant to get.

## Tests
6901 backend tests pass (emulator-integration file excluded: local env).
Tests reproduce the cancellation through each worker's real `main()`.
Per-task review + local CodeRabbit (no findings).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01CZf1cM9vefnNw5djARRhyA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:12:37 -04:00
..

Infrastructure as Code with Pulumi

This directory is the home of Nomad Karaoke's shared platform IaC. It holds two Pulumi projects:

Project Path Stack Manages
karaoke-gen-infrastructure infrastructure/ prod The nomadkaraoke GCP project: gen's backend plus shared services (runner manager, divebar, KN sync, error monitor, monitoring, backups, IAM, secrets), and gen's Cloudflare edge security on api.nomadkaraoke.com
nomad-cloudflare infrastructure/cloudflare/ nomadkaraoke/prod The Nomad Karaoke Cloudflare account: zones, DNS, rulesets, email routing, tunnels, Zero Trust, Pages/Worker domains. See cloudflare/README.md

Boundary rule: infra that isn't specific to one app lives here. App repos keep only app-specific stacks (e.g. karaoke-decide's Cloud Run and its API-proxy Worker + route). Every Cloudflare zone ruleset phase has exactly one owner; see the exclusions table in cloudflare/README.md. CI never runs pulumi up for either project: apply locally, then merge.

Directory Structure

infrastructure/
├── __main__.py              # Entry point - wires modules together (~340 lines)
├── config.py                # Centralized configuration constants
├── Pulumi.yaml              # Pulumi project config
├── Pulumi.prod.yaml         # Production stack config
├── requirements.txt         # Python dependencies
│
├── modules/                 # Core GCP resources
│   ├── database.py         # Firestore database + indexes
│   ├── storage.py          # GCS bucket + lifecycle rules
│   ├── artifact_registry.py # Docker repository
│   ├── secrets.py          # Secret Manager secrets
│   ├── cloud_tasks.py      # Task queues
│   ├── cloud_run.py        # Domain mapping, Cloud Run Jobs
│   ├── monitoring.py       # Alert policies
│   └── iam/                # IAM by purpose
│       ├── backend_sa.py   # karaoke-backend service account
│       ├── github_actions_sa.py # WIF + deployer permissions
│       ├── claude_automation_sa.py # Claude Code automation
│       └── worker_sas.py   # VM service accounts
│
├── compute/                 # VM-based resources
│   ├── flacfetch_vm.py     # Audio download service
│   ├── encoding_worker_vm.py # Video encoding VM
│   ├── github_runners.py   # Self-hosted CI runners
│   └── startup_scripts/    # VM startup scripts (.sh)
│       ├── flacfetch.sh
│       ├── encoding_worker.sh
│       └── github_runner.sh
│
├── functions/               # Cloud Functions
│   └── gdrive_validator/
│       ├── main.py
│       ├── requirements.txt
│       └── deploy.sh       # Source upload script
│
└── monitoring/              # Monitoring configs
    ├── dashboard.json
    └── job-inspector.html

Resources Managed

Category Resources
Database Firestore (Native mode), 10 indexes, TTL policy
Storage GCS bucket with lifecycle rules
Registry Artifact Registry for Docker images
Secrets 18 Secret Manager secrets
IAM 7 service accounts, WIF, 50+ IAM bindings
Compute Encoding worker, Flacfetch, 10 GitHub runners
Tasks 5 Cloud Tasks queues
Monitoring 4 alert policies
Functions GDrive validator + scheduler

Quick Start

cd infrastructure

# Preview changes
pulumi preview --diff

# Apply changes
pulumi up

# View outputs
pulumi stack output

Prerequisites

  1. Pulumi CLI: brew install pulumi
  2. Pulumi Account: Sign up at https://app.pulumi.com
  3. Google Cloud SDK: With authenticated account
  4. Python 3.10+: For running Pulumi

Module Guide

config.py

Centralized configuration for the entire infrastructure:

  • Project ID, region, zone
  • Machine types and disk sizes
  • Google Drive folder ID
  • Number of GitHub runners

modules/database.py

Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.

modules/storage.py

GCS bucket with lifecycle rules:

  • 7-day cleanup of staging data
  • 24-hour cleanup of raw audio files

modules/iam/

Service accounts organized by purpose:

  • backend_sa.py: Cloud Run backend permissions
  • github_actions_sa.py: CI/CD via Workload Identity Federation
  • worker_sas.py: VM service accounts (flacfetch, encoding, runners)

compute/

VM definitions with startup scripts as separate files. Each module creates:

  • Static IP (where needed)
  • VM instance
  • Firewall rules

Adding Secret Values

After pulumi up, add actual secret values:

# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-

# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-

# GitHub runner PAT (admin:org scope for org-level runners)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-

GDrive Validator

The Cloud Function validates your public Google Drive folder daily:

  • Checks for duplicate sequence numbers
  • Validates filename formats
  • Detects sequence gaps
  • Sends Pushbullet notifications

Deploy:

cd functions/gdrive_validator
./deploy.sh

Note: deploy.sh uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.

Self-Hosted GitHub Runners

20 GCP VMs serve as organization-level self-hosted runners, available to all repos in the nomadkaraoke org:

  • 200GB SSD (vs 14GB on GitHub-hosted)
  • Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
  • Labels: self-hosted, linux, x64, gcp, large-disk

PAT Requirements: The github-runner-pat secret requires admin:org scope (specifically manage_runners:org) for organization-level runner registration.

Usage in workflows (any nomadkaraoke repo):

runs-on: [self-hosted, linux, gcp]

Troubleshooting:

# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
  "sudo cat /var/log/github-runner-startup.log"

# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
  "sudo systemctl status actions.runner.*"

Encoding Worker

High-performance VM for video encoding:

  • Machine: c4-standard-8 (8 vCPU, 32GB RAM)
  • Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
  • Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts

API Endpoints:

  • POST /encode - Full encoding job
  • POST /encode-preview - Quick 480p preview
  • GET /status/{job_id} - Job status
  • GET /health - Health check

Troubleshooting:

gcloud compute ssh encoding-worker --zone=us-central1-c -- \
  "sudo systemctl status encoding-worker"

State Management

Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:

pulumi refresh

Multi-Environment

pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up

pulumi stack select prod  # Switch stacks

Troubleshooting

"Resource already exists"

Import manually-created resources:

pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke

State conflicts

pulumi refresh  # Sync state with cloud

Permissions errors

Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.

Future Improvements

Phase 5: Custom Machine Images (Planned)

The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.

See: docs/PHASE5-PACKER-IMAGE-PLAN.md for the implementation plan.

Benefits:

  • Startup time: 10 min → 30 sec
  • Pre-baked: Python 3.13, FFmpeg, fonts
  • Hot updates: karaoke-gen wheel still downloaded at runtime