mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Root-cause fix for Andrew's failed 16-tab reload test (2026-09-22 21:37Z), which the new degradation telemetry captured end-to-end (`client_events`: banner_reconnecting ×6, banner_unavailable ×2 with failed probes, waveform_slow with peaks-fallback). **Diagnosis:** the origin was completely healthy during the test — zero backend errors, 3 instances serving — but only ~207 API requests reached it (16 tabs should have sent 2–3× that; only 9 `correction-data` arrived). The missing traffic was blocked by **our own Cloudflare rate limit (50 req/10s/IP)**: 16 tabs × ~10 API calls + CORS preflights ≈ hundreds of edge requests in seconds. CF block pages have no CORS headers, so cross-origin fetches fail opaquely — producing "Access denied / Failed to load job details" (blocked job fetch), "temporarily unavailable" (blocked correction-data, 3 fast retries all inside the 10s block), and outage banners (the `/api/health` probes were themselves rate-limited, falsely "confirming" an outage). ## Changes **Edge (Pulumi `infrastructure/`)** - `RATE_LIMIT_REQUESTS` 50 → **200 per 10s/IP** — still flood control (20 req/s sustained blocks), but a legitimate many-tab power user passes. - Rate-limit expression now **excludes OPTIONS preflights** (double-counting; a blocked preflight kills the real request) and **`/api/health`** (the frontend's connectivity probe must reflect origin health — never edge-block state — or every tab shows the outage banner for a healthy origin). **Frontend** - `429` treated as transient; retry honors `Retry-After` or waits out the full 10s mitigation window (fast retries fail *and* prolong the block). - Network-failure retry backoff 0.6s/1.5s → **2s/9s**, so the final attempt can land after a 10s block expires (blocked cross-origin requests surface as network errors). - Review-page access check: `BackendUnavailableError` now shows **"Trouble reaching our servers" + Try again** instead of the misleading **"Access denied"** — unreachability is not an authorization verdict. ## Testing - Frontend: 2 new 429 tests (waits out window; honors Retry-After), network-backoff tests updated to the new schedule with probe-interference handled — **1309 passed**. Backend suite (untouched code): **4518 passed**. - Post-merge: Pulumi applies via CI (local `pulumi up` unavailable to the agent — read-only ADC can't write Pulumi state; flagged below). Will verify the ruleset values via the Cloudflare API afterwards and re-run the review load test. ## ⚠️ Deploy note `infrastructure/` changes normally get a local `pulumi up` before merge; the agent's read-only ADC can't write Pulumi state, so this relies on the CI apply on merge (idempotent single-ruleset update — low partial-state risk). Andrew: if you prefer, run `pulumi up` locally before merging. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
423 lines
18 KiB
Python
423 lines
18 KiB
Python
"""
|
||
Cloudflare edge security for the karaoke backend.
|
||
|
||
Puts Cloudflare's WAF, rate limiting and bot mitigation in front of
|
||
``api.nomadkaraoke.com`` (Cloud Run) and locks the origin so it only accepts
|
||
traffic that came through Cloudflare (via a secret header the edge injects).
|
||
|
||
WHY THIS EXISTS
|
||
The backend was reachable directly at ``ghs.googlehosted.com`` (Cloud Run
|
||
domain mapping, DNS-only / grey-cloud) with no WAF — a vuln scanner hammered
|
||
it with ``/.env`` / ``/.ssh/id_rsa`` style probes on 2026-07-20. This module
|
||
moves the API behind the Cloudflare edge that already fronts the rest of the
|
||
domain.
|
||
|
||
ZERO-DOWNTIME ROLLOUT (see docs/archive/2026-07-20-edge-security-hardening-plan.md)
|
||
Driven by TWO independent switches so provisioning and the DNS flip are
|
||
separate (``CloudflareConfig.rollout_stage()`` + ``.proxy_prod_api()``):
|
||
|
||
rolloutStage="staging" (default, NON-disruptive):
|
||
* adds a *second* Cloud Run domain mapping + DNS record for
|
||
``api-edge-test.nomadkaraoke.com`` (proxied) → same karaoke-backend
|
||
* edge rules (WAF / rate-limit / header inject) are scoped to the
|
||
staging host ONLY
|
||
* the prod ``api`` record is left exactly as today (DNS-only)
|
||
→ validate the whole Cloud-Run-×-Cloudflare interaction (incl. the
|
||
managed-cert / SSL "dragon") without any prod impact.
|
||
|
||
rolloutStage="prod" (still non-disruptive):
|
||
* edge rules additionally apply to the prod host, but the prod ``api``
|
||
record is STILL DNS-only (proxied stays False).
|
||
→ provision + verify prod WAF/header rules before touching DNS.
|
||
|
||
proxyProdApi=true (the single prod-affecting change, instantly reversible):
|
||
* flips ONLY the prod ``api`` record to proxied=True.
|
||
* requires rolloutStage=prod first (guarded) so the header transform +
|
||
rules cover the prod host before traffic is routed through the edge.
|
||
→ rollback = proxyProdApi=false + `pulumi up` (edge rules stay in place).
|
||
|
||
PROVIDER
|
||
pulumi-cloudflare 6.x (see infrastructure/requirements.txt). Resource arg
|
||
shapes were introspected against 6.18.0. Run ``pulumi preview`` after any
|
||
provider bump — Cloudflare restructures ruleset schemas between majors.
|
||
|
||
CREDENTIALS / CONFIG (per stack, not committed)
|
||
pulumi config set --secret cloudflare:apiToken <token> # Zone WAF:Edit + DNS:Edit + Zone Settings:Edit
|
||
pulumi config set edge:cloudflareZoneId <zone id for nomadkaraoke.com>
|
||
pulumi config set edge:cloudflareAccountId <account id>
|
||
pulumi config set edge:rolloutStage staging # then later: cutover
|
||
"""
|
||
|
||
from typing import Optional
|
||
|
||
import pulumi
|
||
import pulumi_cloudflare as cloudflare
|
||
import pulumi_gcp as gcp
|
||
from pulumi_gcp import cloudrun
|
||
|
||
from config import CloudflareConfig, REGION, PROJECT_ID
|
||
|
||
|
||
# Cloudflare's account-level "Cloudflare Managed Ruleset" (OWASP-style core
|
||
# protections). This id is stable across all Cloudflare accounts.
|
||
CLOUDFLARE_MANAGED_RULESET_ID = "efb7b8c949ac4650a09736fc376e9aee"
|
||
|
||
# Exploit / secret-exposure path substrings to hard-block at the edge regardless
|
||
# of source (2026-07-20 scanner signature + common web probes). We use the
|
||
# `contains` operator (not regex `matches`) because the regex operator requires a
|
||
# Business/WAF-Advanced plan; `contains` works on Free. Scanners use lowercase
|
||
# paths, matching these lowercase tokens.
|
||
EXPLOIT_PATH_TOKENS = [
|
||
"/.env",
|
||
"/.git/",
|
||
"/.aws/",
|
||
"/.ssh/",
|
||
"/etc/passwd",
|
||
"/etc/shadow",
|
||
"/etc/hosts",
|
||
"/proc/self",
|
||
"/proc/net",
|
||
"/var/log/",
|
||
"wp-config",
|
||
"wp-login",
|
||
"wp-admin",
|
||
"configuration.php",
|
||
".mysql_history",
|
||
".bash_history",
|
||
]
|
||
|
||
|
||
def _exploit_path_expression(host_expr: str) -> str:
|
||
"""Free-plan-safe firewall expression: host match AND any exploit substring."""
|
||
ors = " or ".join(
|
||
f'http.request.uri.path contains "{t}"' for t in EXPLOIT_PATH_TOKENS
|
||
)
|
||
return f"{host_expr} and ({ors})"
|
||
|
||
|
||
def _hosts_for_stage() -> list[str]:
|
||
"""Which hostnames the edge rules apply to, per rollout stage."""
|
||
if CloudflareConfig.rollout_stage() == "prod":
|
||
return [CloudflareConfig.PROD_API_HOST, CloudflareConfig.STAGING_API_HOST]
|
||
return [CloudflareConfig.STAGING_API_HOST]
|
||
|
||
|
||
def _host_match_expression(hosts: list[str]) -> str:
|
||
"""Cloudflare filter expression matching any of the given hosts."""
|
||
quoted = " ".join(f'"{h}"' for h in hosts)
|
||
return f"(http.host in {{{quoted}}})"
|
||
|
||
|
||
# --------------------------------------------------------------------------- #
|
||
# Staging Cloud Run domain mapping (same service as prod api)
|
||
# --------------------------------------------------------------------------- #
|
||
def create_staging_domain_mapping() -> cloudrun.DomainMapping:
|
||
"""
|
||
Second Cloud Run domain mapping so ``api-edge-test.nomadkaraoke.com`` serves
|
||
the SAME karaoke-backend service as prod. Lets us validate the full edge
|
||
stack end-to-end without touching the live ``api`` mapping. Torn down after
|
||
cutover (delete this resource + its DNS record, then `pulumi up`).
|
||
"""
|
||
return cloudrun.DomainMapping(
|
||
"karaoke-backend-edge-test-domain",
|
||
location=REGION,
|
||
name=CloudflareConfig.STAGING_API_HOST,
|
||
metadata=cloudrun.DomainMappingMetadataArgs(namespace=PROJECT_ID),
|
||
spec=cloudrun.DomainMappingSpecArgs(
|
||
route_name=CloudflareConfig.DOMAIN_MAPPING_ROUTE,
|
||
),
|
||
)
|
||
|
||
|
||
# --------------------------------------------------------------------------- #
|
||
# DNS records (Cloudflare)
|
||
# --------------------------------------------------------------------------- #
|
||
def create_dns_records(zone_id: str) -> dict[str, cloudflare.DnsRecord]:
|
||
"""
|
||
Manage the ``api`` (prod) and ``api-edge-test`` (staging) CNAMEs.
|
||
|
||
- staging: always proxied=True (that's the whole point — test the proxy).
|
||
- prod ``api``: proxied only once ``proxy_prod_api()`` is true; until then it
|
||
stays DNS-only, byte-for-byte the behaviour we have today.
|
||
|
||
NOTE: ``ttl`` must be 1 (automatic) whenever proxied=True — Cloudflare
|
||
rejects an explicit TTL on a proxied record.
|
||
"""
|
||
records: dict[str, cloudflare.DnsRecord] = {}
|
||
proxy_prod = CloudflareConfig.proxy_prod_api()
|
||
|
||
# Created grey-cloud (proxied=False) so the Cloud Run managed cert can
|
||
# provision via ACME; flip to proxied=True (edge:proxyStaging) once the cert
|
||
# is issued. See CloudflareConfig.proxy_staging_api (the SSL "dragon").
|
||
records["staging"] = cloudflare.DnsRecord(
|
||
"api-edge-test-dns",
|
||
zone_id=zone_id,
|
||
name="api-edge-test",
|
||
type="CNAME",
|
||
content=CloudflareConfig.ORIGIN_CNAME_TARGET,
|
||
ttl=1,
|
||
proxied=CloudflareConfig.proxy_staging_api(),
|
||
comment="Edge-security staging host → karaoke-backend (throwaway; see edge_security.py)",
|
||
)
|
||
|
||
# The prod `api` record is only brought under Pulumi management once we're
|
||
# provisioning for prod (rollout_stage="prod") — during staging validation
|
||
# we leave the live record completely untouched (no import needed yet).
|
||
if CloudflareConfig.rollout_stage() != "prod":
|
||
return records
|
||
|
||
# The prod `api` record ALREADY EXISTS in Cloudflare (id
|
||
# 6c31cba0080ff334a85cfff6c2927219, ttl=1/auto, proxied=False). Pulumi must
|
||
# IMPORT it into this resource before the first apply, otherwise it tries to
|
||
# create a duplicate / errors. Runbook has the import command. ttl is kept at
|
||
# 1 (auto) to match the live record and because a proxied record requires it.
|
||
# `proxied` is gated on its OWN switch (proxy_prod_api), independent of
|
||
# rollout_stage, so prod edge rules can be provisioned first and the proxy
|
||
# flip done (and rolled back) as an isolated one-record change.
|
||
records["prod"] = cloudflare.DnsRecord(
|
||
"api-dns",
|
||
zone_id=zone_id,
|
||
name="api",
|
||
type="CNAME",
|
||
content=CloudflareConfig.ORIGIN_CNAME_TARGET,
|
||
ttl=1,
|
||
proxied=proxy_prod,
|
||
comment="karaoke-backend (Cloud Run). Proxied via Cloudflare when proxyProdApi=true.",
|
||
)
|
||
|
||
return records
|
||
|
||
|
||
# --------------------------------------------------------------------------- #
|
||
# WAF: custom exploit-path block + Cloudflare Managed Ruleset
|
||
# --------------------------------------------------------------------------- #
|
||
def create_waf_rulesets(zone_id: str, hosts: list[str]) -> dict[str, cloudflare.Ruleset]:
|
||
host_expr = _host_match_expression(hosts)
|
||
rulesets: dict[str, cloudflare.Ruleset] = {}
|
||
|
||
# Custom firewall rules (evaluated before managed rules).
|
||
rulesets["custom"] = cloudflare.Ruleset(
|
||
"edge-waf-custom",
|
||
zone_id=zone_id,
|
||
name="karaoke-backend edge WAF (custom)",
|
||
description="Block secret-exposure / exploit path probes at the edge.",
|
||
kind="zone",
|
||
phase="http_request_firewall_custom",
|
||
rules=[
|
||
cloudflare.RulesetRuleArgs(
|
||
action="block",
|
||
description="Block exploit / secret-exposure paths",
|
||
expression=_exploit_path_expression(host_expr),
|
||
enabled=True,
|
||
),
|
||
],
|
||
)
|
||
|
||
# Cloudflare Managed Ruleset (OWASP-style). Only available on Pro+ plans —
|
||
# the zone is currently Free, so this is gated off by default (see
|
||
# CloudflareConfig.managed_waf_enabled). Deploy in LOG mode first so we can
|
||
# review false positives before switching the override action to "block".
|
||
if CloudflareConfig.managed_waf_enabled():
|
||
_managed_action = "log"
|
||
rulesets["managed"] = cloudflare.Ruleset(
|
||
"edge-waf-managed",
|
||
zone_id=zone_id,
|
||
name="karaoke-backend edge WAF (managed)",
|
||
description="Cloudflare Managed Ruleset (log first, then block).",
|
||
kind="zone",
|
||
phase="http_request_firewall_managed",
|
||
rules=[
|
||
cloudflare.RulesetRuleArgs(
|
||
action="execute",
|
||
description="Execute Cloudflare Managed Ruleset",
|
||
expression=host_expr,
|
||
enabled=True,
|
||
action_parameters=cloudflare.RulesetRuleActionParametersArgs(
|
||
id=CLOUDFLARE_MANAGED_RULESET_ID,
|
||
overrides=cloudflare.RulesetRuleActionParametersOverridesArgs(
|
||
action=_managed_action,
|
||
),
|
||
),
|
||
),
|
||
],
|
||
)
|
||
|
||
return rulesets
|
||
|
||
|
||
# --------------------------------------------------------------------------- #
|
||
# Rate limiting (per-IP), excluding scheduler cron paths
|
||
# --------------------------------------------------------------------------- #
|
||
def create_rate_limit_ruleset(zone_id: str, hosts: list[str]) -> cloudflare.Ruleset:
|
||
host_expr = _host_match_expression(hosts)
|
||
# Exclusions from rate-limit counting:
|
||
# - /api/internal/*: OIDC-authed scheduler hits.
|
||
# - OPTIONS: CORS preflights double every browser API call's edge count, and
|
||
# a blocked preflight kills the real request even when it would have
|
||
# passed. Preflights are cheap and carry no abuse value.
|
||
# - /api/health: the frontend's connectivity probe must reflect ORIGIN
|
||
# health. If probes get rate-limited alongside a burst, every open tab
|
||
# shows the outage banner for an origin that is perfectly healthy
|
||
# (2026-09-22 16-tab incident).
|
||
expression = (
|
||
f'{host_expr} and '
|
||
f'(not starts_with(http.request.uri.path, "{CloudflareConfig.INTERNAL_PATH_PREFIX}")) and '
|
||
f'(http.request.method ne "OPTIONS") and '
|
||
f'(http.request.uri.path ne "/api/health")'
|
||
)
|
||
return cloudflare.Ruleset(
|
||
"edge-ratelimit",
|
||
zone_id=zone_id,
|
||
name="karaoke-backend edge rate limit",
|
||
description="Throttle abusive clients by IP; exclude internal cron paths.",
|
||
kind="zone",
|
||
phase="http_ratelimit",
|
||
rules=[
|
||
cloudflare.RulesetRuleArgs(
|
||
action="block",
|
||
description=(
|
||
f"Block >{CloudflareConfig.RATE_LIMIT_REQUESTS} req / "
|
||
f"{CloudflareConfig.RATE_LIMIT_PERIOD_SECONDS}s per IP"
|
||
),
|
||
expression=expression,
|
||
enabled=True,
|
||
ratelimit=cloudflare.RulesetRuleRatelimitArgs(
|
||
# cf.colo.id is REQUIRED by Cloudflare (counting is processed
|
||
# per-colocation); ip.src makes it per source IP.
|
||
characteristics=["ip.src", "cf.colo.id"],
|
||
period=CloudflareConfig.RATE_LIMIT_PERIOD_SECONDS,
|
||
requests_per_period=CloudflareConfig.RATE_LIMIT_REQUESTS,
|
||
mitigation_timeout=CloudflareConfig.RATE_LIMIT_MITIGATION_SECONDS,
|
||
),
|
||
),
|
||
],
|
||
)
|
||
|
||
|
||
# --------------------------------------------------------------------------- #
|
||
# Origin lockdown: inject a secret header on every proxied request
|
||
# --------------------------------------------------------------------------- #
|
||
def create_origin_header_transform(
|
||
zone_id: str, hosts: list[str], secret: pulumi.Input[str]
|
||
) -> cloudflare.Ruleset:
|
||
"""
|
||
Add ``X-Edge-Auth: <secret>`` to requests as they leave the Cloudflare edge
|
||
for the origin. The backend edge-auth middleware rejects public requests
|
||
that lack it — so anyone hitting the raw ``*.run.app`` / ghs origin directly
|
||
(bypassing Cloudflare) is refused. A Cloud Run domain mapping can't use
|
||
ingress=internal, so this app-layer check is how we lock the origin.
|
||
"""
|
||
host_expr = _host_match_expression(hosts)
|
||
return cloudflare.Ruleset(
|
||
"edge-origin-header",
|
||
zone_id=zone_id,
|
||
name="karaoke-backend origin auth header",
|
||
description="Inject shared-secret header so origin can reject non-edge traffic.",
|
||
kind="zone",
|
||
phase="http_request_late_transform",
|
||
rules=[
|
||
cloudflare.RulesetRuleArgs(
|
||
action="rewrite",
|
||
description="Set origin auth header",
|
||
expression=host_expr,
|
||
enabled=True,
|
||
action_parameters=cloudflare.RulesetRuleActionParametersArgs(
|
||
headers={
|
||
CloudflareConfig.ORIGIN_AUTH_HEADER: cloudflare.RulesetRuleActionParametersHeadersArgs(
|
||
operation="set",
|
||
value=secret,
|
||
),
|
||
},
|
||
),
|
||
),
|
||
],
|
||
)
|
||
|
||
|
||
# --------------------------------------------------------------------------- #
|
||
# Zone-level bot mitigation / security level
|
||
# --------------------------------------------------------------------------- #
|
||
def create_zone_security_settings(zone_id: str) -> dict[str, cloudflare.ZoneSetting]:
|
||
"""
|
||
Raise baseline zone protections. These are zone-wide (Cloudflare doesn't
|
||
scope Security Level / Bot Fight Mode per-host), so they also benefit the
|
||
marketing site and other subdomains.
|
||
"""
|
||
settings: dict[str, cloudflare.ZoneSetting] = {}
|
||
settings["security_level"] = cloudflare.ZoneSetting(
|
||
"zone-security-level",
|
||
zone_id=zone_id,
|
||
setting_id="security_level",
|
||
value="medium",
|
||
)
|
||
return settings
|
||
|
||
|
||
# --------------------------------------------------------------------------- #
|
||
# Orchestrator
|
||
# --------------------------------------------------------------------------- #
|
||
def configure_edge_security() -> dict:
|
||
"""
|
||
Wire up the whole Cloudflare edge for the backend, honouring the rollout
|
||
stage. Returns a dict of created resources for export/inspection.
|
||
|
||
Safe no-op until explicitly enabled: returns ``{}`` and creates nothing
|
||
unless ``edge:enabled`` is True. This MUST gate on the opt-in flag (not the
|
||
zone id, which has a default) so that merging the module before a
|
||
WAF/DNS-scoped ``cloudflare:apiToken`` exists can't make `pulumi up` try to
|
||
create Cloudflare resources with an unauthorized token.
|
||
|
||
The origin-lock header value comes from the Pulumi secret config
|
||
``edge:originSecret`` (same string stored in Secret Manager as
|
||
``edge-origin-secret`` for the backend to read). If it isn't set, the WAF /
|
||
rate-limit / DNS resources are still created but the origin-header transform
|
||
is skipped (so the backend can stay in EDGE_AUTH_MODE=off/warn safely).
|
||
"""
|
||
if not CloudflareConfig.enabled():
|
||
pulumi.log.info(
|
||
"edge_security: edge:enabled is false — Cloudflare edge resources "
|
||
"not created. Set a WAF/DNS-scoped cloudflare:apiToken, then "
|
||
"`pulumi config set edge:enabled true` to activate."
|
||
)
|
||
return {}
|
||
|
||
zone_id = CloudflareConfig.zone_id()
|
||
# Guard the cutover ordering: proxying prod before the prod host is covered
|
||
# by the edge rules (esp. the X-Edge-Auth header transform) would send
|
||
# header-less requests to the origin — enforce mode would then 403 all prod
|
||
# traffic. Require rollout_stage=prod (rules provisioned + verified) first.
|
||
if CloudflareConfig.proxy_prod_api() and CloudflareConfig.rollout_stage() != "prod":
|
||
raise Exception(
|
||
"edge_security: edge:proxyProdApi=true requires edge:rolloutStage=prod "
|
||
"first (so prod-host WAF + X-Edge-Auth header rules exist before the "
|
||
"DNS proxy flip). Set rolloutStage=prod, apply, verify, THEN proxyProdApi=true."
|
||
)
|
||
|
||
hosts = _hosts_for_stage()
|
||
resources: dict = {
|
||
"stage": CloudflareConfig.rollout_stage(),
|
||
"proxy_prod": CloudflareConfig.proxy_prod_api(),
|
||
"hosts": hosts,
|
||
}
|
||
|
||
# Staging domain mapping only needs to exist while we're validating; keep it
|
||
# through cutover for soak, remove later.
|
||
resources["staging_domain_mapping"] = create_staging_domain_mapping()
|
||
resources["dns"] = create_dns_records(zone_id)
|
||
resources["waf"] = create_waf_rulesets(zone_id, hosts)
|
||
resources["rate_limit"] = create_rate_limit_ruleset(zone_id, hosts)
|
||
resources["zone_settings"] = create_zone_security_settings(zone_id)
|
||
|
||
origin_secret = pulumi.Config("edge").get_secret("originSecret")
|
||
if origin_secret is not None:
|
||
resources["origin_header"] = create_origin_header_transform(
|
||
zone_id, hosts, origin_secret
|
||
)
|
||
else:
|
||
pulumi.log.warn(
|
||
"edge_security: edge:originSecret not set — origin-header transform "
|
||
"skipped. Keep backend EDGE_AUTH_MODE=off/warn until it's set."
|
||
)
|
||
|
||
return resources
|