## Summary
Phase 1 of moving encoding to AWS. EC2 is paid for by the AWS Activate
credits (acct 790139457751, expiring 2028-10-31).
- **Plan:** `docs/archive/2026-10-29-aws-encoding-worker-plan.md`.
- **Benchmark** (3 real prod jobs replayed): an EC2 c7a.4xlarge matched
or beat GCE c4d-highcpu-16.
The EC2 box runs the **unchanged** `gce_encoding` worker service and
joins the existing candidate pool as `provider: aws`. The existing
failover, active override, lost-job detection and stockout cooldowns all
apply to it as they are.
## Changes
- **`backend/services/aws_encoding_compute.py` (new):**
`AwsComputeAdapter` looks to the manager like the
`compute_v1.InstancesClient` methods it uses.
- EC2 states map to GCE statuses.
- EC2 capacity errors are added to `CAPACITY_ERROR_CODES`.
- A missing instance raises `NotFound`.
- Controller credentials come from Secret Manager
`aws-encoding-controller-credentials`.
- **`EncodingWorkerManager`:**
- `_compute_for(vm)` sends AWS VMs to the adapter; the GCE paths are
untouched.
- A shared `parse_fallback_vms` passes `provider`/`instance_id`/`region`
through.
- `ranked_candidates()` is used by both the encode path and the warmups.
- `ensure_preferred_running()` behaves **exactly like
`ensure_primary_running` whenever the GCE primary ranks first**, which
is today's case.
- An AWS control-plane fault fails over to the GCE candidates.
- **Ranking:** `SPEED_RANK["c7a"] = 90`, so AWS is the last resort by
default. Firestore `config/encoding-worker.speed_rank_overrides =
{"c7a": 5}` makes it preferred, with no deploy.
- **Isolation from GCE-only tooling:**
- The override now records `active_override_provider`.
- The CI blue-green deploy, `deploy_promote` and the idle Cloud Function
all ignore the AWS worker; it hot-updates its wheel per job.
- A GCE promotion clears any stale provider tag.
- **Worker:**
- Uses `gcloud storage` on AWS, because gsutil can't use WIF credential
files. GCE keeps gsutil.
- `infrastructure/encoding-worker/aws/` adds:
- `provision.sh`, as EC2 user-data;
- `startup-aws.sh`, uploaded by CI;
- `idle-watchdog.py`, an idle self-stop that runs as its own systemd
timer.
- **Infra (GCP), already applied with `/gcp-admin`:**
- The WIF pool `aws-encoding`, plus an AWS provider that only admits the
`nomad-encoding-worker` role, federated to `encoding-worker@`.
- The secret `aws-encoding-controller-credentials`.
- Preview and apply: 4 creates, 288 unchanged.
- **Infra (AWS):** a new Pulumi project `infrastructure/aws` (stack
`nomadkaraoke/prod`):
- the c7a.8xlarge **persistent Spot** worker, which stops on
interruption or shutdown;
- an Elastic IP;
- a security group on 8080 (API key, same as the GCE firewall);
- the worker role with SSM;
- a tag-scoped `nomad-encoding-controller` user.
- Preview: 10 creates. **I apply it after merge**, because the box
provisions from the deployed wheel and boot scripts.
- `boto3` is now an explicit dependency.
- CI also runs the `encoding_worker_idle` function tests, in the job
renamed "Infra - Cloud Function tests".
## Rollout (after merge)
1. **Stage A:** `pulumi up` the AWS stack. Store the controller key in
the secret, then add the worker's entry to
`encoding-worker-fallback-vms`. It's inert unless both c4d and c2d stock
out.
2. **Stage B:** a supervised `speed_rank_overrides {"c7a": 5}` window.
Run real jobs, then compare timings and outputs, including a visual
check of the portrait video.
3. **Stage C:** keep it preferred. After a clean week, delete the c4d
pair.
Phase 2, running distribution (Dropbox/YouTube/Drive uploads) from AWS,
follows in a separate PR.
## Testing
- **New tests:**
- adapter: botocore `Stubber`, covering the state map, capacity and
NotFound handling;
- manager routing and ranking with overrides;
- `ensure_preferred_running` branches;
- failover on an AWS control-plane fault;
- provider tagging;
- deploy_promote exclusions;
- the idle function skipping AWS;
- watchdog decisions;
- the gcs CLI choice.
- **Updated** the candidate and warmup-route tests to the new seams.
- **Full backend + unit suites** have no new failures. The local
emulator and cross-suite failures are identical on main.
- **Local CodeRabbit:** 2 findings, both fixed (failover on AWS faults;
stale provider tag).
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVSR5qgwsxTQtvGimKF
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Infrastructure as Code with Pulumi
This directory is the home of Nomad Karaoke's shared platform IaC. It holds two Pulumi projects:
| Project | Path | Stack | Manages |
|---|---|---|---|
karaoke-gen-infrastructure |
infrastructure/ |
prod |
The nomadkaraoke GCP project: gen's backend plus shared services (runner manager, divebar, KN sync, error monitor, monitoring, backups, IAM, secrets), and gen's Cloudflare edge security on api.nomadkaraoke.com |
nomad-cloudflare |
infrastructure/cloudflare/ |
nomadkaraoke/prod |
The Nomad Karaoke Cloudflare account: zones, DNS, rulesets, email routing, tunnels, Zero Trust, Pages/Worker domains. See cloudflare/README.md |
Boundary rule: infra that isn't specific to one app lives here. App repos keep only
app-specific stacks (e.g. karaoke-decide's Cloud Run and its API-proxy Worker + route). Every
Cloudflare zone ruleset phase has exactly one owner; see the exclusions table in cloudflare/README.md.
CI never runs pulumi up for either project: apply locally, then merge.
Directory Structure
infrastructure/
├── __main__.py # Entry point - wires modules together (~340 lines)
├── config.py # Centralized configuration constants
├── Pulumi.yaml # Pulumi project config
├── Pulumi.prod.yaml # Production stack config
├── requirements.txt # Python dependencies
│
├── modules/ # Core GCP resources
│ ├── database.py # Firestore database + indexes
│ ├── storage.py # GCS bucket + lifecycle rules
│ ├── artifact_registry.py # Docker repository
│ ├── secrets.py # Secret Manager secrets
│ ├── cloud_tasks.py # Task queues
│ ├── cloud_run.py # Domain mapping, Cloud Run Jobs
│ ├── monitoring.py # Alert policies
│ └── iam/ # IAM by purpose
│ ├── backend_sa.py # karaoke-backend service account
│ ├── github_actions_sa.py # WIF + deployer permissions
│ ├── claude_automation_sa.py # Claude Code automation
│ └── worker_sas.py # VM service accounts
│
├── compute/ # VM-based resources
│ ├── flacfetch_vm.py # Audio download service
│ ├── encoding_worker_vm.py # Video encoding VM
│ ├── github_runners.py # Self-hosted CI runners
│ └── startup_scripts/ # VM startup scripts (.sh)
│ ├── flacfetch.sh
│ ├── encoding_worker.sh
│ └── github_runner.sh
│
├── functions/ # Cloud Functions
│ └── gdrive_validator/
│ ├── main.py
│ ├── requirements.txt
│ └── deploy.sh # Source upload script
│
└── monitoring/ # Monitoring configs
├── dashboard.json
└── job-inspector.html
Resources Managed
| Category | Resources |
|---|---|
| Database | Firestore (Native mode), 10 indexes, TTL policy |
| Storage | GCS bucket with lifecycle rules |
| Registry | Artifact Registry for Docker images |
| Secrets | 18 Secret Manager secrets |
| IAM | 7 service accounts, WIF, 50+ IAM bindings |
| Compute | Encoding worker, Flacfetch, 10 GitHub runners |
| Tasks | 5 Cloud Tasks queues |
| Monitoring | 4 alert policies |
| Functions | GDrive validator + scheduler |
Quick Start
cd infrastructure
# Preview changes
pulumi preview --diff
# Apply changes
pulumi up
# View outputs
pulumi stack output
Prerequisites
- Pulumi CLI:
brew install pulumi - Pulumi Account: Sign up at https://app.pulumi.com
- Google Cloud SDK: With authenticated account
- Python 3.10+: For running Pulumi
Module Guide
config.py
Centralized configuration for the entire infrastructure:
- Project ID, region, zone
- Machine types and disk sizes
- Google Drive folder ID
- Number of GitHub runners
modules/database.py
Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.
modules/storage.py
GCS bucket with lifecycle rules:
- 7-day cleanup of staging data
- 24-hour cleanup of raw audio files
modules/iam/
Service accounts organized by purpose:
backend_sa.py: Cloud Run backend permissionsgithub_actions_sa.py: CI/CD via Workload Identity Federationworker_sas.py: VM service accounts (flacfetch, encoding, runners)
compute/
VM definitions with startup scripts as separate files. Each module creates:
- Static IP (where needed)
- VM instance
- Firewall rules
Adding Secret Values
After pulumi up, add actual secret values:
# API keys
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-
# Private tracker credentials
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-
# GitHub runner PAT (admin:org scope for org-level runners)
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-
GDrive Validator
The Cloud Function validates your public Google Drive folder daily:
- Checks for duplicate sequence numbers
- Validates filename formats
- Detects sequence gaps
- Sends Pushbullet notifications
Deploy:
cd functions/gdrive_validator
./deploy.sh
Note: deploy.sh uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.
Self-Hosted GitHub Runners
20 GCP VMs serve as organization-level self-hosted runners, available to all repos in the nomadkaraoke org:
- 200GB SSD (vs 14GB on GitHub-hosted)
- Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
- Labels:
self-hosted,linux,x64,gcp,large-disk
PAT Requirements: The github-runner-pat secret requires admin:org scope (specifically manage_runners:org) for organization-level runner registration.
Usage in workflows (any nomadkaraoke repo):
runs-on: [self-hosted, linux, gcp]
Troubleshooting:
# View startup logs
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo cat /var/log/github-runner-startup.log"
# Check runner service
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
"sudo systemctl status actions.runner.*"
Encoding Worker
High-performance VM for video encoding:
- Machine: c4-standard-8 (8 vCPU, 32GB RAM)
- Disk: 50GB boot disk (hyperdisk-balanced at the free 3000 IOPS / 140 MB/s baseline; pd-balanced on c2d/n2/n2d fallbacks). See docs/archive/2026-09-26-encoding-worker-disk-cost.md
- Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts
API Endpoints:
POST /encode- Full encoding jobPOST /encode-preview- Quick 480p previewGET /status/{job_id}- Job statusGET /health- Health check
Troubleshooting:
gcloud compute ssh encoding-worker --zone=us-central1-c -- \
"sudo systemctl status encoding-worker"
State Management
Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:
pulumi refresh
Multi-Environment
pulumi stack init dev
pulumi config set gcp:project nomadkaraoke-dev
pulumi up
pulumi stack select prod # Switch stacks
Troubleshooting
"Resource already exists"
Import manually-created resources:
pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke
State conflicts
pulumi refresh # Sync state with cloud
Permissions errors
Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.
Future Improvements
Phase 5: Custom Machine Images (Planned)
The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.
See: docs/PHASE5-PACKER-IMAGE-PLAN.md for the implementation plan.
Benefits:
- Startup time: 10 min → 30 sec
- Pre-baked: Python 3.13, FFmpeg, fonts
- Hot updates: karaoke-gen wheel still downloaded at runtime