Files
karaoke-gen/infrastructure/modules/edge_security.py
T
Andrew Beveridge ☄️andClaude Opus 4.8 fb2bd6b108 Cloudflare edge security (WAF + rate limit + origin lock) for karaoke-backend (#886)
## Summary
Puts `api.nomadkaraoke.com` behind the Cloudflare edge that already
fronts the rest of the domain, after a vuln-scanner hammered the
directly-exposed Cloud Run origin on 2026-07-20 (`/.env`,
`/.ssh/id_rsa`, `wp-config.php`… — all 404'd, but the burst of 404s
tripped the "High Error Rate" alert). The API was the one
internet-facing surface with **no WAF, no rate limiting** — DNS-only
CNAME straight to `ghs.googlehosted.com`, Cloud Run ingress `all`.

**Zero-downtime by construction:** everything is gated behind
`edge:enabled` (default off) and staged via two independent switches
(`edge:rolloutStage` for rule scope, `edge:proxyProdApi` for the DNS
flip). The whole Cloud-Run×Cloudflare interaction is validated on a
throwaway staging host first; the only prod-affecting action is a
single, instantly-reversible DNS proxy flip. Full sequence in
`docs/archive/2026-07-20-edge-security-cutover-runbook.md`.

## Changes
- **`infrastructure/modules/edge_security.py`** (new) —
`pulumi_cloudflare` WAF (exploit-path block), per-IP rate limiting
(excludes `/api/internal/*`), origin-lock header transform, zone
security; staging + prod DNS records. Managed OWASP ruleset gated off
(zone is Free plan).
- **`backend/middleware/edge_auth.py`** (new) — off/warn/enforce origin
lock via a secret header Cloudflare injects; **fails closed (503) in
enforce** when the secret is missing; health/root exempt. +13 unit
tests.
- **`monitoring.py`** — retune the high-error-rate alert to **5xx-only**
(+ fix the rate-vs-ratio naming) so client/bot 404 bursts no longer
page.
- Wiring: `secrets.py` (edge-origin-secret container), `config.py`,
`__main__.py`, pinned `pulumi-cloudflare==6.18.0`, `ci.yml`
(`EDGE_AUTH_MODE=off` + secret ref), `Pulumi.prod.yaml` (encrypted
`edge:originSecret`).
- Docs: hardening plan + cutover runbook in `docs/archive/`.

## Already applied to prod (targeted `pulumi up`)
- ✅ `edge-origin-secret` Secret Manager container + v1 value
- ✅ 5xx error-rate alert retune (fixes the original paging issue
immediately)
- Everything else stays dormant: `edge:enabled=false`, backend
`EDGE_AUTH_MODE=off`.

## Testing
- [x] `make test` — full backend + frontend suite green (1085 frontend
tests; backend incl. 13 new edge-auth tests)
- [x] Cloudflare token verified against live API (DNS create+delete,
rulesets, zone settings)
- [x] `pulumi preview` clean (targeted apply of only the 2 additive
resources)
- [ ] Staging validation + prod cutover — follow the runbook (separate,
gated)

## Review
- [x] CodeRabbit CLI review completed locally; findings addressed
(fail-closed enforce, decoupled cutover, fixed origin-lock test,
threshold wording). One pre-existing PR_TITLE shell-interp finding in
the deploy step left out of scope.

## Notes for reviewer
- Nothing here touches prod behavior until someone sets `edge:enabled
true` and walks the runbook. The prod `api` DNS record must be `pulumi
import`ed (already exists) — documented in runbook step 4a.
- Pre-existing cross-worktree Cloud Function source drift is visible in
a full `pulumi preview`; not introduced here (applied via `--target` to
avoid it).

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 19:12:28 -04:00

413 lines
18 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""
Cloudflare edge security for the karaoke backend.
Puts Cloudflare's WAF, rate limiting and bot mitigation in front of
``api.nomadkaraoke.com`` (Cloud Run) and locks the origin so it only accepts
traffic that came through Cloudflare (via a secret header the edge injects).
WHY THIS EXISTS
The backend was reachable directly at ``ghs.googlehosted.com`` (Cloud Run
domain mapping, DNS-only / grey-cloud) with no WAF — a vuln scanner hammered
it with ``/.env`` / ``/.ssh/id_rsa`` style probes on 2026-07-20. This module
moves the API behind the Cloudflare edge that already fronts the rest of the
domain.
ZERO-DOWNTIME ROLLOUT (see docs/archive/2026-07-20-edge-security-hardening-plan.md)
Driven by TWO independent switches so provisioning and the DNS flip are
separate (``CloudflareConfig.rollout_stage()`` + ``.proxy_prod_api()``):
rolloutStage="staging" (default, NON-disruptive):
* adds a *second* Cloud Run domain mapping + DNS record for
``api-edge-test.nomadkaraoke.com`` (proxied) → same karaoke-backend
* edge rules (WAF / rate-limit / header inject) are scoped to the
staging host ONLY
* the prod ``api`` record is left exactly as today (DNS-only)
→ validate the whole Cloud-Run-×-Cloudflare interaction (incl. the
managed-cert / SSL "dragon") without any prod impact.
rolloutStage="prod" (still non-disruptive):
* edge rules additionally apply to the prod host, but the prod ``api``
record is STILL DNS-only (proxied stays False).
→ provision + verify prod WAF/header rules before touching DNS.
proxyProdApi=true (the single prod-affecting change, instantly reversible):
* flips ONLY the prod ``api`` record to proxied=True.
* requires rolloutStage=prod first (guarded) so the header transform +
rules cover the prod host before traffic is routed through the edge.
→ rollback = proxyProdApi=false + `pulumi up` (edge rules stay in place).
PROVIDER
pulumi-cloudflare 6.x (see infrastructure/requirements.txt). Resource arg
shapes were introspected against 6.18.0. Run ``pulumi preview`` after any
provider bump — Cloudflare restructures ruleset schemas between majors.
CREDENTIALS / CONFIG (per stack, not committed)
pulumi config set --secret cloudflare:apiToken <token> # Zone WAF:Edit + DNS:Edit + Zone Settings:Edit
pulumi config set edge:cloudflareZoneId <zone id for nomadkaraoke.com>
pulumi config set edge:cloudflareAccountId <account id>
pulumi config set edge:rolloutStage staging # then later: cutover
"""
from typing import Optional
import pulumi
import pulumi_cloudflare as cloudflare
import pulumi_gcp as gcp
from pulumi_gcp import cloudrun
from config import CloudflareConfig, REGION, PROJECT_ID
# Cloudflare's account-level "Cloudflare Managed Ruleset" (OWASP-style core
# protections). This id is stable across all Cloudflare accounts.
CLOUDFLARE_MANAGED_RULESET_ID = "efb7b8c949ac4650a09736fc376e9aee"
# Exploit / secret-exposure path substrings to hard-block at the edge regardless
# of source (2026-07-20 scanner signature + common web probes). We use the
# `contains` operator (not regex `matches`) because the regex operator requires a
# Business/WAF-Advanced plan; `contains` works on Free. Scanners use lowercase
# paths, matching these lowercase tokens.
EXPLOIT_PATH_TOKENS = [
"/.env",
"/.git/",
"/.aws/",
"/.ssh/",
"/etc/passwd",
"/etc/shadow",
"/etc/hosts",
"/proc/self",
"/proc/net",
"/var/log/",
"wp-config",
"wp-login",
"wp-admin",
"configuration.php",
".mysql_history",
".bash_history",
]
def _exploit_path_expression(host_expr: str) -> str:
"""Free-plan-safe firewall expression: host match AND any exploit substring."""
ors = " or ".join(
f'http.request.uri.path contains "{t}"' for t in EXPLOIT_PATH_TOKENS
)
return f"{host_expr} and ({ors})"
def _hosts_for_stage() -> list[str]:
"""Which hostnames the edge rules apply to, per rollout stage."""
if CloudflareConfig.rollout_stage() == "prod":
return [CloudflareConfig.PROD_API_HOST, CloudflareConfig.STAGING_API_HOST]
return [CloudflareConfig.STAGING_API_HOST]
def _host_match_expression(hosts: list[str]) -> str:
"""Cloudflare filter expression matching any of the given hosts."""
quoted = " ".join(f'"{h}"' for h in hosts)
return f"(http.host in {{{quoted}}})"
# --------------------------------------------------------------------------- #
# Staging Cloud Run domain mapping (same service as prod api)
# --------------------------------------------------------------------------- #
def create_staging_domain_mapping() -> cloudrun.DomainMapping:
"""
Second Cloud Run domain mapping so ``api-edge-test.nomadkaraoke.com`` serves
the SAME karaoke-backend service as prod. Lets us validate the full edge
stack end-to-end without touching the live ``api`` mapping. Torn down after
cutover (delete this resource + its DNS record, then `pulumi up`).
"""
return cloudrun.DomainMapping(
"karaoke-backend-edge-test-domain",
location=REGION,
name=CloudflareConfig.STAGING_API_HOST,
metadata=cloudrun.DomainMappingMetadataArgs(namespace=PROJECT_ID),
spec=cloudrun.DomainMappingSpecArgs(
route_name=CloudflareConfig.DOMAIN_MAPPING_ROUTE,
),
)
# --------------------------------------------------------------------------- #
# DNS records (Cloudflare)
# --------------------------------------------------------------------------- #
def create_dns_records(zone_id: str) -> dict[str, cloudflare.DnsRecord]:
"""
Manage the ``api`` (prod) and ``api-edge-test`` (staging) CNAMEs.
- staging: always proxied=True (that's the whole point — test the proxy).
- prod ``api``: proxied only once ``proxy_prod_api()`` is true; until then it
stays DNS-only, byte-for-byte the behaviour we have today.
NOTE: ``ttl`` must be 1 (automatic) whenever proxied=True — Cloudflare
rejects an explicit TTL on a proxied record.
"""
records: dict[str, cloudflare.DnsRecord] = {}
proxy_prod = CloudflareConfig.proxy_prod_api()
# Created grey-cloud (proxied=False) so the Cloud Run managed cert can
# provision via ACME; flip to proxied=True (edge:proxyStaging) once the cert
# is issued. See CloudflareConfig.proxy_staging_api (the SSL "dragon").
records["staging"] = cloudflare.DnsRecord(
"api-edge-test-dns",
zone_id=zone_id,
name="api-edge-test",
type="CNAME",
content=CloudflareConfig.ORIGIN_CNAME_TARGET,
ttl=1,
proxied=CloudflareConfig.proxy_staging_api(),
comment="Edge-security staging host → karaoke-backend (throwaway; see edge_security.py)",
)
# The prod `api` record is only brought under Pulumi management once we're
# provisioning for prod (rollout_stage="prod") — during staging validation
# we leave the live record completely untouched (no import needed yet).
if CloudflareConfig.rollout_stage() != "prod":
return records
# The prod `api` record ALREADY EXISTS in Cloudflare (id
# 6c31cba0080ff334a85cfff6c2927219, ttl=1/auto, proxied=False). Pulumi must
# IMPORT it into this resource before the first apply, otherwise it tries to
# create a duplicate / errors. Runbook has the import command. ttl is kept at
# 1 (auto) to match the live record and because a proxied record requires it.
# `proxied` is gated on its OWN switch (proxy_prod_api), independent of
# rollout_stage, so prod edge rules can be provisioned first and the proxy
# flip done (and rolled back) as an isolated one-record change.
records["prod"] = cloudflare.DnsRecord(
"api-dns",
zone_id=zone_id,
name="api",
type="CNAME",
content=CloudflareConfig.ORIGIN_CNAME_TARGET,
ttl=1,
proxied=proxy_prod,
comment="karaoke-backend (Cloud Run). Proxied via Cloudflare when proxyProdApi=true.",
)
return records
# --------------------------------------------------------------------------- #
# WAF: custom exploit-path block + Cloudflare Managed Ruleset
# --------------------------------------------------------------------------- #
def create_waf_rulesets(zone_id: str, hosts: list[str]) -> dict[str, cloudflare.Ruleset]:
host_expr = _host_match_expression(hosts)
rulesets: dict[str, cloudflare.Ruleset] = {}
# Custom firewall rules (evaluated before managed rules).
rulesets["custom"] = cloudflare.Ruleset(
"edge-waf-custom",
zone_id=zone_id,
name="karaoke-backend edge WAF (custom)",
description="Block secret-exposure / exploit path probes at the edge.",
kind="zone",
phase="http_request_firewall_custom",
rules=[
cloudflare.RulesetRuleArgs(
action="block",
description="Block exploit / secret-exposure paths",
expression=_exploit_path_expression(host_expr),
enabled=True,
),
],
)
# Cloudflare Managed Ruleset (OWASP-style). Only available on Pro+ plans —
# the zone is currently Free, so this is gated off by default (see
# CloudflareConfig.managed_waf_enabled). Deploy in LOG mode first so we can
# review false positives before switching the override action to "block".
if CloudflareConfig.managed_waf_enabled():
_managed_action = "log"
rulesets["managed"] = cloudflare.Ruleset(
"edge-waf-managed",
zone_id=zone_id,
name="karaoke-backend edge WAF (managed)",
description="Cloudflare Managed Ruleset (log first, then block).",
kind="zone",
phase="http_request_firewall_managed",
rules=[
cloudflare.RulesetRuleArgs(
action="execute",
description="Execute Cloudflare Managed Ruleset",
expression=host_expr,
enabled=True,
action_parameters=cloudflare.RulesetRuleActionParametersArgs(
id=CLOUDFLARE_MANAGED_RULESET_ID,
overrides=cloudflare.RulesetRuleActionParametersOverridesArgs(
action=_managed_action,
),
),
),
],
)
return rulesets
# --------------------------------------------------------------------------- #
# Rate limiting (per-IP), excluding scheduler cron paths
# --------------------------------------------------------------------------- #
def create_rate_limit_ruleset(zone_id: str, hosts: list[str]) -> cloudflare.Ruleset:
host_expr = _host_match_expression(hosts)
# Don't rate-limit OIDC-authed scheduler hits to /api/internal/*.
expression = (
f'{host_expr} and '
f'(not starts_with(http.request.uri.path, "{CloudflareConfig.INTERNAL_PATH_PREFIX}"))'
)
return cloudflare.Ruleset(
"edge-ratelimit",
zone_id=zone_id,
name="karaoke-backend edge rate limit",
description="Throttle abusive clients by IP; exclude internal cron paths.",
kind="zone",
phase="http_ratelimit",
rules=[
cloudflare.RulesetRuleArgs(
action="block",
description=(
f"Block >{CloudflareConfig.RATE_LIMIT_REQUESTS} req / "
f"{CloudflareConfig.RATE_LIMIT_PERIOD_SECONDS}s per IP"
),
expression=expression,
enabled=True,
ratelimit=cloudflare.RulesetRuleRatelimitArgs(
# cf.colo.id is REQUIRED by Cloudflare (counting is processed
# per-colocation); ip.src makes it per source IP.
characteristics=["ip.src", "cf.colo.id"],
period=CloudflareConfig.RATE_LIMIT_PERIOD_SECONDS,
requests_per_period=CloudflareConfig.RATE_LIMIT_REQUESTS,
mitigation_timeout=CloudflareConfig.RATE_LIMIT_MITIGATION_SECONDS,
),
),
],
)
# --------------------------------------------------------------------------- #
# Origin lockdown: inject a secret header on every proxied request
# --------------------------------------------------------------------------- #
def create_origin_header_transform(
zone_id: str, hosts: list[str], secret: pulumi.Input[str]
) -> cloudflare.Ruleset:
"""
Add ``X-Edge-Auth: <secret>`` to requests as they leave the Cloudflare edge
for the origin. The backend edge-auth middleware rejects public requests
that lack it — so anyone hitting the raw ``*.run.app`` / ghs origin directly
(bypassing Cloudflare) is refused. A Cloud Run domain mapping can't use
ingress=internal, so this app-layer check is how we lock the origin.
"""
host_expr = _host_match_expression(hosts)
return cloudflare.Ruleset(
"edge-origin-header",
zone_id=zone_id,
name="karaoke-backend origin auth header",
description="Inject shared-secret header so origin can reject non-edge traffic.",
kind="zone",
phase="http_request_late_transform",
rules=[
cloudflare.RulesetRuleArgs(
action="rewrite",
description="Set origin auth header",
expression=host_expr,
enabled=True,
action_parameters=cloudflare.RulesetRuleActionParametersArgs(
headers={
CloudflareConfig.ORIGIN_AUTH_HEADER: cloudflare.RulesetRuleActionParametersHeadersArgs(
operation="set",
value=secret,
),
},
),
),
],
)
# --------------------------------------------------------------------------- #
# Zone-level bot mitigation / security level
# --------------------------------------------------------------------------- #
def create_zone_security_settings(zone_id: str) -> dict[str, cloudflare.ZoneSetting]:
"""
Raise baseline zone protections. These are zone-wide (Cloudflare doesn't
scope Security Level / Bot Fight Mode per-host), so they also benefit the
marketing site and other subdomains.
"""
settings: dict[str, cloudflare.ZoneSetting] = {}
settings["security_level"] = cloudflare.ZoneSetting(
"zone-security-level",
zone_id=zone_id,
setting_id="security_level",
value="medium",
)
return settings
# --------------------------------------------------------------------------- #
# Orchestrator
# --------------------------------------------------------------------------- #
def configure_edge_security() -> dict:
"""
Wire up the whole Cloudflare edge for the backend, honouring the rollout
stage. Returns a dict of created resources for export/inspection.
Safe no-op until explicitly enabled: returns ``{}`` and creates nothing
unless ``edge:enabled`` is True. This MUST gate on the opt-in flag (not the
zone id, which has a default) so that merging the module before a
WAF/DNS-scoped ``cloudflare:apiToken`` exists can't make `pulumi up` try to
create Cloudflare resources with an unauthorized token.
The origin-lock header value comes from the Pulumi secret config
``edge:originSecret`` (same string stored in Secret Manager as
``edge-origin-secret`` for the backend to read). If it isn't set, the WAF /
rate-limit / DNS resources are still created but the origin-header transform
is skipped (so the backend can stay in EDGE_AUTH_MODE=off/warn safely).
"""
if not CloudflareConfig.enabled():
pulumi.log.info(
"edge_security: edge:enabled is false — Cloudflare edge resources "
"not created. Set a WAF/DNS-scoped cloudflare:apiToken, then "
"`pulumi config set edge:enabled true` to activate."
)
return {}
zone_id = CloudflareConfig.zone_id()
# Guard the cutover ordering: proxying prod before the prod host is covered
# by the edge rules (esp. the X-Edge-Auth header transform) would send
# header-less requests to the origin — enforce mode would then 403 all prod
# traffic. Require rollout_stage=prod (rules provisioned + verified) first.
if CloudflareConfig.proxy_prod_api() and CloudflareConfig.rollout_stage() != "prod":
raise Exception(
"edge_security: edge:proxyProdApi=true requires edge:rolloutStage=prod "
"first (so prod-host WAF + X-Edge-Auth header rules exist before the "
"DNS proxy flip). Set rolloutStage=prod, apply, verify, THEN proxyProdApi=true."
)
hosts = _hosts_for_stage()
resources: dict = {
"stage": CloudflareConfig.rollout_stage(),
"proxy_prod": CloudflareConfig.proxy_prod_api(),
"hosts": hosts,
}
# Staging domain mapping only needs to exist while we're validating; keep it
# through cutover for soak, remove later.
resources["staging_domain_mapping"] = create_staging_domain_mapping()
resources["dns"] = create_dns_records(zone_id)
resources["waf"] = create_waf_rulesets(zone_id, hosts)
resources["rate_limit"] = create_rate_limit_ruleset(zone_id, hosts)
resources["zone_settings"] = create_zone_security_settings(zone_id)
origin_secret = pulumi.Config("edge").get_secret("originSecret")
if origin_secret is not None:
resources["origin_header"] = create_origin_header_transform(
zone_id, hosts, origin_secret
)
else:
pulumi.log.warn(
"edge_security: edge:originSecret not set — origin-header transform "
"skipped. Keep backend EDGE_AUTH_MODE=off/warn until it's set."
)
return resources