Files
Andrew Beveridge ☄️andClaude Fable 5 925ec79c56 fix: 16-tab burst tripped the Cloudflare rate limit — raise + scope it, survive edge blocks gracefully (#1033)
## Summary

Root-cause fix for Andrew's failed 16-tab reload test (2026-09-22
21:37Z), which the new degradation telemetry captured end-to-end
(`client_events`: banner_reconnecting ×6, banner_unavailable ×2 with
failed probes, waveform_slow with peaks-fallback).

**Diagnosis:** the origin was completely healthy during the test — zero
backend errors, 3 instances serving — but only ~207 API requests reached
it (16 tabs should have sent 2–3× that; only 9 `correction-data`
arrived). The missing traffic was blocked by **our own Cloudflare rate
limit (50 req/10s/IP)**: 16 tabs × ~10 API calls + CORS preflights ≈
hundreds of edge requests in seconds. CF block pages have no CORS
headers, so cross-origin fetches fail opaquely — producing "Access
denied / Failed to load job details" (blocked job fetch), "temporarily
unavailable" (blocked correction-data, 3 fast retries all inside the 10s
block), and outage banners (the `/api/health` probes were themselves
rate-limited, falsely "confirming" an outage).

## Changes

**Edge (Pulumi `infrastructure/`)**
- `RATE_LIMIT_REQUESTS` 50 → **200 per 10s/IP** — still flood control
(20 req/s sustained blocks), but a legitimate many-tab power user
passes.
- Rate-limit expression now **excludes OPTIONS preflights**
(double-counting; a blocked preflight kills the real request) and
**`/api/health`** (the frontend's connectivity probe must reflect origin
health — never edge-block state — or every tab shows the outage banner
for a healthy origin).

**Frontend**
- `429` treated as transient; retry honors `Retry-After` or waits out
the full 10s mitigation window (fast retries fail *and* prolong the
block).
- Network-failure retry backoff 0.6s/1.5s → **2s/9s**, so the final
attempt can land after a 10s block expires (blocked cross-origin
requests surface as network errors).
- Review-page access check: `BackendUnavailableError` now shows
**"Trouble reaching our servers" + Try again** instead of the misleading
**"Access denied"** — unreachability is not an authorization verdict.

## Testing

- Frontend: 2 new 429 tests (waits out window; honors Retry-After),
network-backoff tests updated to the new schedule with
probe-interference handled — **1309 passed**. Backend suite (untouched
code): **4518 passed**.
- Post-merge: Pulumi applies via CI (local `pulumi up` unavailable to
the agent — read-only ADC can't write Pulumi state; flagged below). Will
verify the ruleset values via the Cloudflare API afterwards and re-run
the review load test.

## ⚠️ Deploy note

`infrastructure/` changes normally get a local `pulumi up` before merge;
the agent's read-only ADC can't write Pulumi state, so this relies on
the CI apply on merge (idempotent single-ruleset update — low
partial-state risk). Andrew: if you prefer, run `pulumi up` locally
before merging.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-22 17:58:39 -04:00

423 lines
18 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""
Cloudflare edge security for the karaoke backend.
Puts Cloudflare's WAF, rate limiting and bot mitigation in front of
``api.nomadkaraoke.com`` (Cloud Run) and locks the origin so it only accepts
traffic that came through Cloudflare (via a secret header the edge injects).
WHY THIS EXISTS
The backend was reachable directly at ``ghs.googlehosted.com`` (Cloud Run
domain mapping, DNS-only / grey-cloud) with no WAF — a vuln scanner hammered
it with ``/.env`` / ``/.ssh/id_rsa`` style probes on 2026-07-20. This module
moves the API behind the Cloudflare edge that already fronts the rest of the
domain.
ZERO-DOWNTIME ROLLOUT (see docs/archive/2026-07-20-edge-security-hardening-plan.md)
Driven by TWO independent switches so provisioning and the DNS flip are
separate (``CloudflareConfig.rollout_stage()`` + ``.proxy_prod_api()``):
rolloutStage="staging" (default, NON-disruptive):
* adds a *second* Cloud Run domain mapping + DNS record for
``api-edge-test.nomadkaraoke.com`` (proxied) → same karaoke-backend
* edge rules (WAF / rate-limit / header inject) are scoped to the
staging host ONLY
* the prod ``api`` record is left exactly as today (DNS-only)
→ validate the whole Cloud-Run-×-Cloudflare interaction (incl. the
managed-cert / SSL "dragon") without any prod impact.
rolloutStage="prod" (still non-disruptive):
* edge rules additionally apply to the prod host, but the prod ``api``
record is STILL DNS-only (proxied stays False).
→ provision + verify prod WAF/header rules before touching DNS.
proxyProdApi=true (the single prod-affecting change, instantly reversible):
* flips ONLY the prod ``api`` record to proxied=True.
* requires rolloutStage=prod first (guarded) so the header transform +
rules cover the prod host before traffic is routed through the edge.
→ rollback = proxyProdApi=false + `pulumi up` (edge rules stay in place).
PROVIDER
pulumi-cloudflare 6.x (see infrastructure/requirements.txt). Resource arg
shapes were introspected against 6.18.0. Run ``pulumi preview`` after any
provider bump — Cloudflare restructures ruleset schemas between majors.
CREDENTIALS / CONFIG (per stack, not committed)
pulumi config set --secret cloudflare:apiToken <token> # Zone WAF:Edit + DNS:Edit + Zone Settings:Edit
pulumi config set edge:cloudflareZoneId <zone id for nomadkaraoke.com>
pulumi config set edge:cloudflareAccountId <account id>
pulumi config set edge:rolloutStage staging # then later: cutover
"""
from typing import Optional
import pulumi
import pulumi_cloudflare as cloudflare
import pulumi_gcp as gcp
from pulumi_gcp import cloudrun
from config import CloudflareConfig, REGION, PROJECT_ID
# Cloudflare's account-level "Cloudflare Managed Ruleset" (OWASP-style core
# protections). This id is stable across all Cloudflare accounts.
CLOUDFLARE_MANAGED_RULESET_ID = "efb7b8c949ac4650a09736fc376e9aee"
# Exploit / secret-exposure path substrings to hard-block at the edge regardless
# of source (2026-07-20 scanner signature + common web probes). We use the
# `contains` operator (not regex `matches`) because the regex operator requires a
# Business/WAF-Advanced plan; `contains` works on Free. Scanners use lowercase
# paths, matching these lowercase tokens.
EXPLOIT_PATH_TOKENS = [
"/.env",
"/.git/",
"/.aws/",
"/.ssh/",
"/etc/passwd",
"/etc/shadow",
"/etc/hosts",
"/proc/self",
"/proc/net",
"/var/log/",
"wp-config",
"wp-login",
"wp-admin",
"configuration.php",
".mysql_history",
".bash_history",
]
def _exploit_path_expression(host_expr: str) -> str:
"""Free-plan-safe firewall expression: host match AND any exploit substring."""
ors = " or ".join(
f'http.request.uri.path contains "{t}"' for t in EXPLOIT_PATH_TOKENS
)
return f"{host_expr} and ({ors})"
def _hosts_for_stage() -> list[str]:
"""Which hostnames the edge rules apply to, per rollout stage."""
if CloudflareConfig.rollout_stage() == "prod":
return [CloudflareConfig.PROD_API_HOST, CloudflareConfig.STAGING_API_HOST]
return [CloudflareConfig.STAGING_API_HOST]
def _host_match_expression(hosts: list[str]) -> str:
"""Cloudflare filter expression matching any of the given hosts."""
quoted = " ".join(f'"{h}"' for h in hosts)
return f"(http.host in {{{quoted}}})"
# --------------------------------------------------------------------------- #
# Staging Cloud Run domain mapping (same service as prod api)
# --------------------------------------------------------------------------- #
def create_staging_domain_mapping() -> cloudrun.DomainMapping:
"""
Second Cloud Run domain mapping so ``api-edge-test.nomadkaraoke.com`` serves
the SAME karaoke-backend service as prod. Lets us validate the full edge
stack end-to-end without touching the live ``api`` mapping. Torn down after
cutover (delete this resource + its DNS record, then `pulumi up`).
"""
return cloudrun.DomainMapping(
"karaoke-backend-edge-test-domain",
location=REGION,
name=CloudflareConfig.STAGING_API_HOST,
metadata=cloudrun.DomainMappingMetadataArgs(namespace=PROJECT_ID),
spec=cloudrun.DomainMappingSpecArgs(
route_name=CloudflareConfig.DOMAIN_MAPPING_ROUTE,
),
)
# --------------------------------------------------------------------------- #
# DNS records (Cloudflare)
# --------------------------------------------------------------------------- #
def create_dns_records(zone_id: str) -> dict[str, cloudflare.DnsRecord]:
"""
Manage the ``api`` (prod) and ``api-edge-test`` (staging) CNAMEs.
- staging: always proxied=True (that's the whole point — test the proxy).
- prod ``api``: proxied only once ``proxy_prod_api()`` is true; until then it
stays DNS-only, byte-for-byte the behaviour we have today.
NOTE: ``ttl`` must be 1 (automatic) whenever proxied=True — Cloudflare
rejects an explicit TTL on a proxied record.
"""
records: dict[str, cloudflare.DnsRecord] = {}
proxy_prod = CloudflareConfig.proxy_prod_api()
# Created grey-cloud (proxied=False) so the Cloud Run managed cert can
# provision via ACME; flip to proxied=True (edge:proxyStaging) once the cert
# is issued. See CloudflareConfig.proxy_staging_api (the SSL "dragon").
records["staging"] = cloudflare.DnsRecord(
"api-edge-test-dns",
zone_id=zone_id,
name="api-edge-test",
type="CNAME",
content=CloudflareConfig.ORIGIN_CNAME_TARGET,
ttl=1,
proxied=CloudflareConfig.proxy_staging_api(),
comment="Edge-security staging host → karaoke-backend (throwaway; see edge_security.py)",
)
# The prod `api` record is only brought under Pulumi management once we're
# provisioning for prod (rollout_stage="prod") — during staging validation
# we leave the live record completely untouched (no import needed yet).
if CloudflareConfig.rollout_stage() != "prod":
return records
# The prod `api` record ALREADY EXISTS in Cloudflare (id
# 6c31cba0080ff334a85cfff6c2927219, ttl=1/auto, proxied=False). Pulumi must
# IMPORT it into this resource before the first apply, otherwise it tries to
# create a duplicate / errors. Runbook has the import command. ttl is kept at
# 1 (auto) to match the live record and because a proxied record requires it.
# `proxied` is gated on its OWN switch (proxy_prod_api), independent of
# rollout_stage, so prod edge rules can be provisioned first and the proxy
# flip done (and rolled back) as an isolated one-record change.
records["prod"] = cloudflare.DnsRecord(
"api-dns",
zone_id=zone_id,
name="api",
type="CNAME",
content=CloudflareConfig.ORIGIN_CNAME_TARGET,
ttl=1,
proxied=proxy_prod,
comment="karaoke-backend (Cloud Run). Proxied via Cloudflare when proxyProdApi=true.",
)
return records
# --------------------------------------------------------------------------- #
# WAF: custom exploit-path block + Cloudflare Managed Ruleset
# --------------------------------------------------------------------------- #
def create_waf_rulesets(zone_id: str, hosts: list[str]) -> dict[str, cloudflare.Ruleset]:
host_expr = _host_match_expression(hosts)
rulesets: dict[str, cloudflare.Ruleset] = {}
# Custom firewall rules (evaluated before managed rules).
rulesets["custom"] = cloudflare.Ruleset(
"edge-waf-custom",
zone_id=zone_id,
name="karaoke-backend edge WAF (custom)",
description="Block secret-exposure / exploit path probes at the edge.",
kind="zone",
phase="http_request_firewall_custom",
rules=[
cloudflare.RulesetRuleArgs(
action="block",
description="Block exploit / secret-exposure paths",
expression=_exploit_path_expression(host_expr),
enabled=True,
),
],
)
# Cloudflare Managed Ruleset (OWASP-style). Only available on Pro+ plans —
# the zone is currently Free, so this is gated off by default (see
# CloudflareConfig.managed_waf_enabled). Deploy in LOG mode first so we can
# review false positives before switching the override action to "block".
if CloudflareConfig.managed_waf_enabled():
_managed_action = "log"
rulesets["managed"] = cloudflare.Ruleset(
"edge-waf-managed",
zone_id=zone_id,
name="karaoke-backend edge WAF (managed)",
description="Cloudflare Managed Ruleset (log first, then block).",
kind="zone",
phase="http_request_firewall_managed",
rules=[
cloudflare.RulesetRuleArgs(
action="execute",
description="Execute Cloudflare Managed Ruleset",
expression=host_expr,
enabled=True,
action_parameters=cloudflare.RulesetRuleActionParametersArgs(
id=CLOUDFLARE_MANAGED_RULESET_ID,
overrides=cloudflare.RulesetRuleActionParametersOverridesArgs(
action=_managed_action,
),
),
),
],
)
return rulesets
# --------------------------------------------------------------------------- #
# Rate limiting (per-IP), excluding scheduler cron paths
# --------------------------------------------------------------------------- #
def create_rate_limit_ruleset(zone_id: str, hosts: list[str]) -> cloudflare.Ruleset:
host_expr = _host_match_expression(hosts)
# Exclusions from rate-limit counting:
# - /api/internal/*: OIDC-authed scheduler hits.
# - OPTIONS: CORS preflights double every browser API call's edge count, and
# a blocked preflight kills the real request even when it would have
# passed. Preflights are cheap and carry no abuse value.
# - /api/health: the frontend's connectivity probe must reflect ORIGIN
# health. If probes get rate-limited alongside a burst, every open tab
# shows the outage banner for an origin that is perfectly healthy
# (2026-09-22 16-tab incident).
expression = (
f'{host_expr} and '
f'(not starts_with(http.request.uri.path, "{CloudflareConfig.INTERNAL_PATH_PREFIX}")) and '
f'(http.request.method ne "OPTIONS") and '
f'(http.request.uri.path ne "/api/health")'
)
return cloudflare.Ruleset(
"edge-ratelimit",
zone_id=zone_id,
name="karaoke-backend edge rate limit",
description="Throttle abusive clients by IP; exclude internal cron paths.",
kind="zone",
phase="http_ratelimit",
rules=[
cloudflare.RulesetRuleArgs(
action="block",
description=(
f"Block >{CloudflareConfig.RATE_LIMIT_REQUESTS} req / "
f"{CloudflareConfig.RATE_LIMIT_PERIOD_SECONDS}s per IP"
),
expression=expression,
enabled=True,
ratelimit=cloudflare.RulesetRuleRatelimitArgs(
# cf.colo.id is REQUIRED by Cloudflare (counting is processed
# per-colocation); ip.src makes it per source IP.
characteristics=["ip.src", "cf.colo.id"],
period=CloudflareConfig.RATE_LIMIT_PERIOD_SECONDS,
requests_per_period=CloudflareConfig.RATE_LIMIT_REQUESTS,
mitigation_timeout=CloudflareConfig.RATE_LIMIT_MITIGATION_SECONDS,
),
),
],
)
# --------------------------------------------------------------------------- #
# Origin lockdown: inject a secret header on every proxied request
# --------------------------------------------------------------------------- #
def create_origin_header_transform(
zone_id: str, hosts: list[str], secret: pulumi.Input[str]
) -> cloudflare.Ruleset:
"""
Add ``X-Edge-Auth: <secret>`` to requests as they leave the Cloudflare edge
for the origin. The backend edge-auth middleware rejects public requests
that lack it — so anyone hitting the raw ``*.run.app`` / ghs origin directly
(bypassing Cloudflare) is refused. A Cloud Run domain mapping can't use
ingress=internal, so this app-layer check is how we lock the origin.
"""
host_expr = _host_match_expression(hosts)
return cloudflare.Ruleset(
"edge-origin-header",
zone_id=zone_id,
name="karaoke-backend origin auth header",
description="Inject shared-secret header so origin can reject non-edge traffic.",
kind="zone",
phase="http_request_late_transform",
rules=[
cloudflare.RulesetRuleArgs(
action="rewrite",
description="Set origin auth header",
expression=host_expr,
enabled=True,
action_parameters=cloudflare.RulesetRuleActionParametersArgs(
headers={
CloudflareConfig.ORIGIN_AUTH_HEADER: cloudflare.RulesetRuleActionParametersHeadersArgs(
operation="set",
value=secret,
),
},
),
),
],
)
# --------------------------------------------------------------------------- #
# Zone-level bot mitigation / security level
# --------------------------------------------------------------------------- #
def create_zone_security_settings(zone_id: str) -> dict[str, cloudflare.ZoneSetting]:
"""
Raise baseline zone protections. These are zone-wide (Cloudflare doesn't
scope Security Level / Bot Fight Mode per-host), so they also benefit the
marketing site and other subdomains.
"""
settings: dict[str, cloudflare.ZoneSetting] = {}
settings["security_level"] = cloudflare.ZoneSetting(
"zone-security-level",
zone_id=zone_id,
setting_id="security_level",
value="medium",
)
return settings
# --------------------------------------------------------------------------- #
# Orchestrator
# --------------------------------------------------------------------------- #
def configure_edge_security() -> dict:
"""
Wire up the whole Cloudflare edge for the backend, honouring the rollout
stage. Returns a dict of created resources for export/inspection.
Safe no-op until explicitly enabled: returns ``{}`` and creates nothing
unless ``edge:enabled`` is True. This MUST gate on the opt-in flag (not the
zone id, which has a default) so that merging the module before a
WAF/DNS-scoped ``cloudflare:apiToken`` exists can't make `pulumi up` try to
create Cloudflare resources with an unauthorized token.
The origin-lock header value comes from the Pulumi secret config
``edge:originSecret`` (same string stored in Secret Manager as
``edge-origin-secret`` for the backend to read). If it isn't set, the WAF /
rate-limit / DNS resources are still created but the origin-header transform
is skipped (so the backend can stay in EDGE_AUTH_MODE=off/warn safely).
"""
if not CloudflareConfig.enabled():
pulumi.log.info(
"edge_security: edge:enabled is false — Cloudflare edge resources "
"not created. Set a WAF/DNS-scoped cloudflare:apiToken, then "
"`pulumi config set edge:enabled true` to activate."
)
return {}
zone_id = CloudflareConfig.zone_id()
# Guard the cutover ordering: proxying prod before the prod host is covered
# by the edge rules (esp. the X-Edge-Auth header transform) would send
# header-less requests to the origin — enforce mode would then 403 all prod
# traffic. Require rollout_stage=prod (rules provisioned + verified) first.
if CloudflareConfig.proxy_prod_api() and CloudflareConfig.rollout_stage() != "prod":
raise Exception(
"edge_security: edge:proxyProdApi=true requires edge:rolloutStage=prod "
"first (so prod-host WAF + X-Edge-Auth header rules exist before the "
"DNS proxy flip). Set rolloutStage=prod, apply, verify, THEN proxyProdApi=true."
)
hosts = _hosts_for_stage()
resources: dict = {
"stage": CloudflareConfig.rollout_stage(),
"proxy_prod": CloudflareConfig.proxy_prod_api(),
"hosts": hosts,
}
# Staging domain mapping only needs to exist while we're validating; keep it
# through cutover for soak, remove later.
resources["staging_domain_mapping"] = create_staging_domain_mapping()
resources["dns"] = create_dns_records(zone_id)
resources["waf"] = create_waf_rulesets(zone_id, hosts)
resources["rate_limit"] = create_rate_limit_ruleset(zone_id, hosts)
resources["zone_settings"] = create_zone_security_settings(zone_id)
origin_secret = pulumi.Config("edge").get_secret("originSecret")
if origin_secret is not None:
resources["origin_header"] = create_origin_header_transform(
zone_id, hosts, origin_secret
)
else:
pulumi.log.warn(
"edge_security: edge:originSecret not set — origin-header transform "
"skipped. Keep backend EDGE_AUTH_MODE=off/warn until it's set."
)
return resources