mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Why Since 2026-10-31 the AWS EC2 worker (`aws-encoding-worker-1`) has handled every encode and distribution job. 27 jobs went to it, with no Cloud Run fallbacks, no `encoding_stuck`, and nothing encoding-related in #alerts. The c4d Spot pair (`encoding-worker-a`/`-b`) now only costs ~$8/mo of Hyperdisk. `encoding-worker-fallback-c2df` stays as the GCP fallback. ## What Deleting the pair needs code changes first, because several paths assumed it exists: - **`encoding_worker_manager`** - The primary is optional: an empty `primary_vm` means none. - `ranked_candidates` skips a missing primary and always sets `is_primary`, so with no primary a successful start records the AWS/c2df override. Previously the legacy index-0 rule would have cleared it. - `ensure_primary_running` and `ensure_preferred_running` start a ranked candidate. If there are no candidates they raise `NO_CANDIDATES`. - `demote_active_worker` looks up the override's `machine_type`, so the AWS box can now be cooled down. It then clears the override so the retry goes elsewhere. - **`encoding_service`** - When `active_url` is empty, it targets the top-ranked candidate. - It also invalidates its cached URL after a single-candidate fallback. - **CI deploy (`ci.yml` + `deploy_promote.py`)** - While AWS serves, a c2df green is validate-only: it writes `validated_version`, keeps the AWS override, and stops the green. Before this, it would have replaced the AWS override with c2df. - The worker-change gate now diffs against `validated_version`, because `primary_version` stops moving. - An in-place restart of a serving override first waits for it to drain (15-min cap). If it is still busy, the restart is skipped. - **Idle function:** a missing VM is logged as a WARNING returning `NOT_FOUND`, not an ERROR. - **Error monitor:** watches c2df instead of a/b. - **Pulumi:** drops the a/b VMs and static IPs. The prod preview shows exactly 4 deletes. A separate, unrelated `youtube-comment-autoreply-scheduler` drift shows up too; it will not be applied. - **Docs:** CLAUDE.md, ARCHITECTURE, DEVELOPMENT and TROUBLESHOOTING. ## Rollout (after merge + deploy) 1. Back up `config/encoding-worker`, then set `primary_vm`, `primary_ip`, `secondary_vm` and `secondary_ip` to `""`. 2. Apply `pulumi destroy --target` on the 4 a/b URNs only. 3. Redeploy the idle function with `deploy.sh`. 4. Verify: one prod encode on AWS, warmup, and the logs. **Rollback:** in Firestore, delete `speed_rank_overrides` and set `aws_video_worker=false`. Restoring the pair would need a revert plus `pulumi up`, which gives new static IPs. ## Testing - New `backend/tests/test_encoding_worker_no_primary.py` (13 tests), plus new deploy_promote, idle-function and error-monitor tests. - Full backend suite: 7188 passed. The 8 `test_emulator_integration` failures also fail on main, because of a local emulator from another session. - CodeRabbit CLI: no findings. An independent review found 2 issues, both fixed: the demote retry path, and the in-place restart drain. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012RYLVTkatGnrYFhCzq4PTe @coderabbitai ignore Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
365 lines
16 KiB
Python
365 lines
16 KiB
Python
"""
|
|
Centralized configuration for Pulumi infrastructure.
|
|
|
|
This module provides shared constants and configuration used across all infrastructure modules.
|
|
"""
|
|
|
|
import pulumi
|
|
import pulumi_gcp as gcp
|
|
|
|
# GCP project info - lazily loaded to work with Pulumi runtime
|
|
_project = None
|
|
|
|
|
|
def _get_project():
|
|
"""Get GCP project info (lazy-loaded for Pulumi runtime compatibility)."""
|
|
global _project
|
|
if _project is None:
|
|
_project = gcp.organizations.get_project()
|
|
return _project
|
|
|
|
|
|
# Project ID and number are accessed as properties
|
|
# These will resolve correctly during Pulumi execution
|
|
PROJECT_ID = "nomadkaraoke" # Hardcoded for now - matches existing behavior
|
|
PROJECT_NUMBER = None # Set lazily when needed
|
|
|
|
|
|
def get_project_number() -> str:
|
|
"""Get the GCP project number."""
|
|
return _get_project().number
|
|
|
|
# Region configuration
|
|
REGION = "us-central1"
|
|
ZONE = f"{REGION}-a"
|
|
|
|
# Default zone the encoding idle-shutdown function targets (GCP_ZONE). The c4d
|
|
# pair that lived here was retired 2026-11; fallbacks carry their own zones.
|
|
ENCODING_WORKER_ZONE = f"{REGION}-c"
|
|
|
|
# Pulumi config accessor
|
|
config = pulumi.Config()
|
|
|
|
|
|
class MachineTypes:
|
|
"""Machine type configurations for GCE instances."""
|
|
|
|
GITHUB_RUNNER = "e2-standard-4" # 4 vCPU, 16GB RAM
|
|
GITHUB_BUILD_RUNNER = "e2-standard-8" # 8 vCPU, 32GB RAM - dedicated Docker build runner
|
|
GITHUB_GPU_RUNNER = "n1-standard-4" # 4 vCPU, 15GB RAM - GPU runners need N1 series
|
|
# Encoding workers (2026-09-26 cost cut, docs/archive/2026-09-26-encoding-cost-cuts.md):
|
|
# 16 vCPU instead of 32 — an approved speed-for-cost trade (finalization x264
|
|
# only kept ~66-78% of 32 vCPUs busy; ffmpeg auto-threads adapt to 16). The
|
|
# worker serializes heavy jobs (ENCODING_HEAVY_CONCURRENCY=1), and highcpu-16
|
|
# still has ~30-32 GB RAM — enough for one 4K encode (~18 GB peak).
|
|
# (The c4d-highcpu-16 Spot blue-green pair was retired 2026-11 in favour of
|
|
# the AWS EC2 worker, infrastructure/aws/.)
|
|
# The ONE on-demand capacity fallback. Deliberately a DIFFERENT machine family
|
|
# and zone from the c4d Spot pair: c4d suffered a region-wide
|
|
# ZONE_RESOURCE_POOL_EXHAUSTED stockout (2026-08-12), and Spot capacity can
|
|
# vanish at any time. c2d (AMD Milan) draws from a deep, mature pool and is
|
|
# on-demand, so it is not preemptible. c2d does NOT support hyperdisk →
|
|
# pd-balanced boot disk.
|
|
ENCODING_WORKER_FALLBACK = "c2d-highcpu-16" # 16 vCPU / 32 GB, AMD Milan (Zen3), on-demand
|
|
FLACFETCH = "e2-small" # 0.5 vCPU, 2GB RAM
|
|
|
|
|
|
class DiskSizes:
|
|
"""Disk size configurations in GB."""
|
|
|
|
GITHUB_RUNNER = 200 # Large for Docker builds/caches
|
|
# Encoding-worker boot disk (OS + baked venv ~12-15 GB, plus per-job scratch in
|
|
# /tmp: each queued /encode downloads its whole jobs/<id>/ folder — p99 ~1.3 GB,
|
|
# max seen ~4 GB — and up to ~9 jobs have been queued on one worker at once).
|
|
# Was 100 GB, but a worker 8 months into service used only 15 GB of it; 50 GB
|
|
# keeps >3x headroom over realistic peak while halving the 10-VM disk bill.
|
|
# Must be >= the Packer image's disk_size (infrastructure/packer/
|
|
# encoding-worker.pkr.hcl), since GCE cannot create a disk smaller than its
|
|
# source image. Changing this forces VM *replacement* in Pulumi — see
|
|
# docs/archive/2026-09-26-encoding-worker-disk-cost.md for the in-place
|
|
# boot-disk swap procedure that avoids re-allocating (possibly stocked-out) VMs.
|
|
ENCODING_WORKER = 50
|
|
ENCODING_WORKER_IMAGE = 30 # Packer image disk_size (keep in sync with .pkr.hcl)
|
|
FLACFETCH = 30 # For torrent storage
|
|
|
|
|
|
class QueueConfigs:
|
|
"""Cloud Tasks queue rate limiting configurations."""
|
|
|
|
class Audio:
|
|
MAX_DISPATCHES_PER_SECOND = 10
|
|
MAX_CONCURRENT_DISPATCHES = 50
|
|
MAX_RETRY_DURATION = "1800s" # 30 min
|
|
|
|
class Lyrics:
|
|
MAX_DISPATCHES_PER_SECOND = 10
|
|
MAX_CONCURRENT_DISPATCHES = 50
|
|
MAX_RETRY_DURATION = "1200s" # 20 min
|
|
|
|
class Screens:
|
|
MAX_DISPATCHES_PER_SECOND = 50
|
|
MAX_CONCURRENT_DISPATCHES = 100
|
|
MAX_RETRY_DURATION = "300s" # 5 min
|
|
|
|
class Render:
|
|
MAX_DISPATCHES_PER_SECOND = 5
|
|
MAX_CONCURRENT_DISPATCHES = 20
|
|
MAX_RETRY_DURATION = "3600s" # 60 min
|
|
|
|
class Video:
|
|
MAX_DISPATCHES_PER_SECOND = 3
|
|
MAX_CONCURRENT_DISPATCHES = 10
|
|
MAX_RETRY_DURATION = "7200s" # 2 hours
|
|
|
|
|
|
# Number of self-hosted GitHub Action runners
|
|
# Reduced from 20 to 3 - runners auto-scale with start/stop management
|
|
NUM_GITHUB_RUNNERS = 3
|
|
NUM_GPU_RUNNERS = 3
|
|
|
|
# Runner labels
|
|
GENERAL_RUNNER_LABELS = "self-hosted,linux,x64,gcp,large-disk"
|
|
BUILD_RUNNER_LABELS = "self-hosted,linux,x64,gcp,large-disk,docker-build"
|
|
GPU_RUNNER_LABELS = "self-hosted,linux,x64,gcp,gpu"
|
|
|
|
# Google Drive folder ID for validator (public share folder)
|
|
GDRIVE_FOLDER_ID = "1laRKAyxo0v817SstfM5XkpbWiNKNAMSX"
|
|
|
|
|
|
class RunnerManagerConfig:
|
|
"""Configuration for the ephemeral GHA runner dispatcher Cloud Function."""
|
|
|
|
FUNCTION_NAME = "github-runner-manager"
|
|
FUNCTION_MEMORY = "512M" # Increased from 256M due to memory usage
|
|
FUNCTION_TIMEOUT = 300 # 5 minutes
|
|
# Scheduler cadence for the orphan-cleanup pass (reconciles ephemeral VMs
|
|
# against org-runner registrations, deletes stragglers).
|
|
# Also re-dispatches self-hosted jobs stuck `queued` (dropped webhook,
|
|
# VM that never registered), so the cadence bounds how long a job strands.
|
|
IDLE_CHECK_SCHEDULE = "*/5 * * * *" # Every 5 minutes
|
|
# Each webhook holds an instance for up to ~90s confirming the VM insert,
|
|
# so a CI burst needs headroom — at 5, bursts got Cloud Run 429s and GitHub
|
|
# never redelivers a dropped webhook (2026-09-27 stalled deploy).
|
|
FUNCTION_MAX_INSTANCES = 20
|
|
|
|
|
|
class SecretNames:
|
|
"""Secret Manager secret names."""
|
|
|
|
GITHUB_RUNNER_PAT = "github-runner-pat"
|
|
GITHUB_WEBHOOK_SECRET = "github-webhook-secret"
|
|
# Shared secret Cloudflare injects as a header on proxied requests; the
|
|
# backend edge-auth middleware rejects requests to public routes that lack
|
|
# it (blocks direct-to-origin bypass of the Cloudflare edge). See
|
|
# modules/edge_security.py and backend/middleware/edge_auth.py.
|
|
EDGE_ORIGIN_SECRET = "edge-origin-secret"
|
|
|
|
|
|
class CloudflareConfig:
|
|
"""
|
|
Configuration for the Cloudflare edge (WAF / rate limiting / origin lock).
|
|
|
|
Non-secret values (zone/account ids, hostnames) are read from Pulumi config
|
|
so they are version-controlled per stack without hardcoding account details.
|
|
The API token is a Pulumi *secret* config under the provider key
|
|
``cloudflare:apiToken`` (set with ``pulumi config set --secret``).
|
|
|
|
Set the ids once known:
|
|
pulumi config set edge:cloudflareZoneId <zone-id-for-nomadkaraoke.com>
|
|
pulumi config set edge:cloudflareAccountId <account-id>
|
|
pulumi config set --secret cloudflare:apiToken <token> # Zone WAF:Edit + DNS:Edit
|
|
"""
|
|
|
|
# Production backend host (already a Cloud Run domain mapping today).
|
|
PROD_API_HOST = "api.nomadkaraoke.com"
|
|
# Throwaway staging host used to validate the full edge stack against the
|
|
# SAME Cloud Run service before touching the prod `api` record. Torn down
|
|
# after cutover. See docs/archive/2026-07-20-edge-security-hardening-plan.md.
|
|
STAGING_API_HOST = "api-edge-test.nomadkaraoke.com"
|
|
# Cloud Run domain mappings both target this service origin.
|
|
DOMAIN_MAPPING_ROUTE = "karaoke-backend"
|
|
ORIGIN_CNAME_TARGET = "ghs.googlehosted.com"
|
|
|
|
# SSL/TLS NOTE (validated on staging 2026-07-20): the zone SSL mode is
|
|
# "full" (not strict). In "full" mode Cloudflare encrypts to the origin but
|
|
# does NOT validate the origin cert, so the Cloud Run domain-mapping managed
|
|
# cert's provisioning/renewal status is irrelevant through the proxy — the
|
|
# edge works even while that cert shows "pending". This sidesteps the 525
|
|
# "dragon" (which only bites in "full (strict)") WITHOUT needing a run.app
|
|
# Origin Rule. Keep the zone on "full" (never flip to strict without first
|
|
# provisioning/renewing the managed cert, which cannot issue behind CF).
|
|
|
|
# Header name Cloudflare injects and the backend checks (value = secret).
|
|
ORIGIN_AUTH_HEADER = "X-Edge-Auth"
|
|
|
|
# Rate limit: block a client IP that exceeds this many requests/period.
|
|
# Free plan constraints (enforced by the Cloudflare API): period must be 10s,
|
|
# and mitigation_timeout must equal the period (10s). A single flood control
|
|
# rule — the WAF path block (below) is the primary defense against the
|
|
# scanner class; this catches volumetric abuse.
|
|
#
|
|
# Sized for the legitimate worst case from ONE IP (2026-09-22 incident): an
|
|
# admin reloading 16 review tabs at once fires ~10 API calls per tab within
|
|
# seconds. At the old 50/10s the edge blocked a chunk of them — and because
|
|
# CF block pages carry no CORS headers, the browser saw opaque network
|
|
# failures ("Access denied" / "temporarily unavailable" screens) while the
|
|
# origin was fully healthy. 200/10s (20 req/s/IP) still stops volumetric
|
|
# abuse; OPTIONS preflights and /api/health are excluded from counting (see
|
|
# create_rate_limit_ruleset).
|
|
RATE_LIMIT_REQUESTS = 200
|
|
RATE_LIMIT_PERIOD_SECONDS = 10
|
|
RATE_LIMIT_MITIGATION_SECONDS = 10
|
|
|
|
# Paths that must never be rate-limited / header-gated at the edge:
|
|
# scheduler cron hits (OIDC-authed) to /api/internal/*.
|
|
INTERNAL_PATH_PREFIX = "/api/internal/"
|
|
|
|
@staticmethod
|
|
def managed_waf_enabled():
|
|
"""
|
|
Whether to deploy the Cloudflare Managed (OWASP-style) Ruleset.
|
|
|
|
The nomadkaraoke.com zone is on the **Free** plan, where managed
|
|
rulesets are NOT available (Pro+ only) — deploying one errors. So this
|
|
defaults to False. The custom exploit-path rules + rate limiting +
|
|
origin lock + bot mitigation (all Free-tier) already cover the
|
|
path-scanning threat this was built for. If the zone is upgraded to
|
|
Pro, set ``edge:managedWafEnabled true`` to turn on the OWASP ruleset.
|
|
"""
|
|
return pulumi.Config("edge").get_bool("managedWafEnabled") or False
|
|
|
|
# Cloudflare zone id for nomadkaraoke.com (not sensitive). Overridable via
|
|
# Pulumi config `edge:cloudflareZoneId`; defaults to the known zone so the
|
|
# edge module activates as soon as a WAF/DNS-scoped `cloudflare:apiToken`
|
|
# is set. NOTE: the zone is on the **Free** plan (managed WAF ruleset is
|
|
# Pro+; see managed_waf_enabled).
|
|
DEFAULT_ZONE_ID = "2324a342622e93343cf4e08227d82c02" # Nomad Karaoke account since 2026-10
|
|
|
|
@staticmethod
|
|
def enabled():
|
|
"""
|
|
Master activation switch for the Cloudflare edge module
|
|
(``edge:enabled`` bool, default False).
|
|
|
|
MUST stay False until a WAF/DNS-scoped ``cloudflare:apiToken`` is set —
|
|
otherwise `pulumi up` would try to create Cloudflare resources with an
|
|
unauthorized token and fail, blocking all infra deploys. Flip to True
|
|
only after the token + config are in place (see the cutover runbook).
|
|
"""
|
|
return pulumi.Config("edge").get_bool("enabled") or False
|
|
|
|
@staticmethod
|
|
def zone_id():
|
|
"""Cloudflare zone id for nomadkaraoke.com."""
|
|
return pulumi.Config("edge").get("cloudflareZoneId") or CloudflareConfig.DEFAULT_ZONE_ID
|
|
|
|
@staticmethod
|
|
def account_id():
|
|
"""Cloudflare account id (from Pulumi config)."""
|
|
return pulumi.Config("edge").get("cloudflareAccountId")
|
|
|
|
@staticmethod
|
|
def rollout_stage():
|
|
"""
|
|
Which hosts the edge WAF/rate-limit/header rules apply to (from Pulumi
|
|
config ``edge:rolloutStage``):
|
|
- "staging" : rules scoped to the staging host only.
|
|
- "prod" : rules apply to the prod host too (staging kept for soak).
|
|
Defaults to "staging" so a first apply is non-disruptive to prod.
|
|
|
|
NOTE: this does NOT flip the prod DNS proxy — that's a SEPARATE switch
|
|
(``proxy_prod_api``), so production edge rules can be provisioned and
|
|
verified while the API is still DNS-only, then the proxy flipped in an
|
|
isolated second apply (and rolled back without tearing down the rules).
|
|
"""
|
|
stage = pulumi.Config("edge").get("rolloutStage") or "staging"
|
|
return "prod" if stage in ("prod", "cutover") else "staging"
|
|
|
|
@staticmethod
|
|
def proxy_staging_api():
|
|
"""
|
|
Whether the staging ``api-edge-test`` record is proxied (``edge:proxyStaging``
|
|
bool, default False).
|
|
|
|
THE SSL DRAGON: a Cloud Run domain mapping provisions its Google-managed
|
|
cert via ACME, which needs the hostname to resolve DIRECTLY to
|
|
``ghs.googlehosted.com``. While the record is proxied (orange-cloud) DNS
|
|
resolves to Cloudflare, so ACME can't validate and the cert never issues.
|
|
Sequence: create grey (proxied=False) → wait for cert → set this True.
|
|
"""
|
|
return pulumi.Config("edge").get_bool("proxyStaging") or False
|
|
|
|
@staticmethod
|
|
def proxy_prod_api():
|
|
"""
|
|
Whether the prod ``api`` DNS record is proxied through Cloudflare
|
|
(``edge:proxyProdApi`` bool, default False).
|
|
|
|
This is the ONE prod-affecting, instantly-reversible flip. Sequence:
|
|
1. rolloutStage=prod, apply → prod edge rules live; api still DNS-only.
|
|
2. proxyProdApi=true, apply → api proxied (only this record changes).
|
|
Rollback = proxyProdApi=false, apply (edge rules stay provisioned).
|
|
"""
|
|
return pulumi.Config("edge").get_bool("proxyProdApi") or False
|
|
|
|
# GitHub repository for runner registration
|
|
GITHUB_REPO_OWNER = "nomadkaraoke"
|
|
GITHUB_REPO_NAME = "karaoke-gen"
|
|
|
|
|
|
class EncodingWorkerConfig:
|
|
"""Configuration for the GCE encoding worker fallback fleet + idle shutdown."""
|
|
# Stop idle workers fast: VMs bill per second while RUNNING, and with only
|
|
# ~5-20 jobs/day most of the old 15-min tail was paid idle time. The function
|
|
# still NEVER stops a VM with active jobs (fail-safe on unconfirmed /health).
|
|
IDLE_CHECK_SCHEDULE = "*/2 * * * *" # Every 2 minutes
|
|
IDLE_TIMEOUT_MINUTES = 5
|
|
FUNCTION_NAME = "encoding-worker-idle-shutdown"
|
|
FUNCTION_MEMORY = "512M" # Increased from 256M — OOM with gRPC/Firestore/Compute client libs
|
|
FUNCTION_TIMEOUT = 120 # 2 minutes
|
|
|
|
# Capacity-resilience fallback. Provisioned STOPPED and started on demand only
|
|
# when the Spot pair can't start (ZONE_RESOURCE_POOL_EXHAUSTED / Spot
|
|
# stockout). ON-DEMAND (never preempted) and a different family + zone from the
|
|
# c4d pair so one shortage can't take out both lanes. Cost when stopped is just
|
|
# the 50 GB boot disk.
|
|
#
|
|
# Was 8 fallbacks across 6 families (2026-08-15) — cut to 1 on 2026-09-26
|
|
# (fleet 10 → 3 VMs) to save the idle disk/IP spend; c2df was kept because it
|
|
# is a deep-pool, pd-balanced (cheaper disk) family in a separate zone.
|
|
#
|
|
# Each entry: name/IP suffix, zone suffix ({REGION}-{zone_suffix}),
|
|
# machine_type, boot disk_type. IPs and VMs are zipped by position. The
|
|
# runtime/deploy candidate PRIORITY is decided by the shared speed-rank +
|
|
# cooldown in backend/services/encoding_worker_preference.py.
|
|
FALLBACKS = [
|
|
{"suffix": "c2df", "zone_suffix": "f",
|
|
"machine_type": MachineTypes.ENCODING_WORKER_FALLBACK, "disk_type": "pd-balanced"},
|
|
]
|
|
|
|
# Derived name lists (kept for readability / any external reference).
|
|
FALLBACK_VM_NAMES = [f"encoding-worker-fallback-{fb['suffix']}" for fb in FALLBACKS]
|
|
FALLBACK_IP_NAMES = [f"encoding-worker-fallback-ip-{fb['suffix']}" for fb in FALLBACKS]
|
|
|
|
@staticmethod
|
|
def fallback_vm_name(suffix: str) -> str:
|
|
return f"encoding-worker-fallback-{suffix}"
|
|
|
|
@staticmethod
|
|
def fallback_ip_name(suffix: str) -> str:
|
|
return f"encoding-worker-fallback-ip-{suffix}"
|
|
|
|
|
|
class ErrorMonitorConfig:
|
|
"""Configuration for the error monitor Cloud Run Job."""
|
|
JOB_NAME = "nomad-error-monitor"
|
|
MEMORY = "512Mi"
|
|
CPU = "1"
|
|
TIMEOUT = "300s"
|
|
MAX_RETRIES = 0
|
|
# Hourly (was every 15 min; 2026-09-26 cost cut). Must match
|
|
# backend/services/error_monitor/config.py LOOKBACK_MINUTES (60).
|
|
MONITOR_SCHEDULE = "0 * * * *"
|
|
DIGEST_SCHEDULE = "0 8 * * *" # 08:00 UTC daily
|
|
SCHEDULER_SA_NAME = "error-monitor-scheduler"
|