Files
karaoke-gen/infrastructure/modules/monitoring.py
T
Andrew Beveridge ☄️andClaude Opus 4.8 fb2bd6b108 Cloudflare edge security (WAF + rate limit + origin lock) for karaoke-backend (#886)
## Summary
Puts `api.nomadkaraoke.com` behind the Cloudflare edge that already
fronts the rest of the domain, after a vuln-scanner hammered the
directly-exposed Cloud Run origin on 2026-07-20 (`/.env`,
`/.ssh/id_rsa`, `wp-config.php`… — all 404'd, but the burst of 404s
tripped the "High Error Rate" alert). The API was the one
internet-facing surface with **no WAF, no rate limiting** — DNS-only
CNAME straight to `ghs.googlehosted.com`, Cloud Run ingress `all`.

**Zero-downtime by construction:** everything is gated behind
`edge:enabled` (default off) and staged via two independent switches
(`edge:rolloutStage` for rule scope, `edge:proxyProdApi` for the DNS
flip). The whole Cloud-Run×Cloudflare interaction is validated on a
throwaway staging host first; the only prod-affecting action is a
single, instantly-reversible DNS proxy flip. Full sequence in
`docs/archive/2026-07-20-edge-security-cutover-runbook.md`.

## Changes
- **`infrastructure/modules/edge_security.py`** (new) —
`pulumi_cloudflare` WAF (exploit-path block), per-IP rate limiting
(excludes `/api/internal/*`), origin-lock header transform, zone
security; staging + prod DNS records. Managed OWASP ruleset gated off
(zone is Free plan).
- **`backend/middleware/edge_auth.py`** (new) — off/warn/enforce origin
lock via a secret header Cloudflare injects; **fails closed (503) in
enforce** when the secret is missing; health/root exempt. +13 unit
tests.
- **`monitoring.py`** — retune the high-error-rate alert to **5xx-only**
(+ fix the rate-vs-ratio naming) so client/bot 404 bursts no longer
page.
- Wiring: `secrets.py` (edge-origin-secret container), `config.py`,
`__main__.py`, pinned `pulumi-cloudflare==6.18.0`, `ci.yml`
(`EDGE_AUTH_MODE=off` + secret ref), `Pulumi.prod.yaml` (encrypted
`edge:originSecret`).
- Docs: hardening plan + cutover runbook in `docs/archive/`.

## Already applied to prod (targeted `pulumi up`)
- ✅ `edge-origin-secret` Secret Manager container + v1 value
- ✅ 5xx error-rate alert retune (fixes the original paging issue
immediately)
- Everything else stays dormant: `edge:enabled=false`, backend
`EDGE_AUTH_MODE=off`.

## Testing
- [x] `make test` — full backend + frontend suite green (1085 frontend
tests; backend incl. 13 new edge-auth tests)
- [x] Cloudflare token verified against live API (DNS create+delete,
rulesets, zone settings)
- [x] `pulumi preview` clean (targeted apply of only the 2 additive
resources)
- [ ] Staging validation + prod cutover — follow the runbook (separate,
gated)

## Review
- [x] CodeRabbit CLI review completed locally; findings addressed
(fail-closed enforce, decoupled cutover, fixed origin-lock test,
threshold wording). One pre-existing PR_TITLE shell-interp finding in
the deploy step left out of scope.

## Notes for reviewer
- Nothing here touches prod behavior until someone sets `edge:enabled
true` and walks the runbook. The prod `api` DNS record must be `pulumi
import`ed (already exists) — documented in runbook step 4a.
- Pre-existing cross-worktree Cloud Function source drift is visible in
a full `pulumi preview`; not introduced here (applied via `--target` to
avoid it).

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 19:12:28 -04:00

402 lines
19 KiB
Python

"""
Cloud Monitoring resources.
Manages alert policies for proactive monitoring and incident response. Every
alert policy attaches the notification channel(s) created by
``create_notification_channels`` — without that, a firing alert resolves into
the void (verified on 2026-05-17: three ephemeral-dispatcher create failures
fired the create-failures alert correctly, but the project had zero
notification channels attached, so nobody got paged for 14 hours).
"""
import pulumi
import pulumi_gcp as gcp
from config import PROJECT_ID
# Email address that receives all monitoring alerts. Cloud Monitoring delivers
# a one-time confirmation link to this address when the channel is created;
# the channel stays in PENDING state until verified.
_ALERT_EMAIL = "andrew@beveridge.uk"
def create_notification_channels() -> dict[str, gcp.monitoring.NotificationChannel]:
"""Create the notification channels attached to every alert policy.
Returns a dict so callers can pass selected channels to specific policies
in the future (e.g. critical-only routing). Today every policy uses
``channels["email"]``.
Swapping email for Discord later: Discord supports Slack-compatible
webhooks at ``<discord_webhook_url>/slack``. Add a second channel with
``type_="slack"`` and ``sensitive_labels={"auth_token": <url>}``; reference
it in the policies that should page Discord. Keep email as the durable
fallback.
"""
channels: dict[str, gcp.monitoring.NotificationChannel] = {}
channels["email"] = gcp.monitoring.NotificationChannel(
"alerts-email",
display_name="Andrew (email)",
type="email",
labels={"email_address": _ALERT_EMAIL},
description="Primary alert channel for Cloud Monitoring policies.",
enabled=True,
)
return channels
def _channel_names(channels: dict[str, gcp.monitoring.NotificationChannel]) -> list[pulumi.Output[str]]:
"""Return the .name Outputs of every channel (the form AlertPolicy wants)."""
return [ch.name for ch in channels.values()]
def create_alert_policies(
channels: dict[str, gcp.monitoring.NotificationChannel],
) -> dict[str, gcp.monitoring.AlertPolicy]:
"""
Create all Cloud Monitoring alert policies.
Args:
channels: Notification channels to attach (from
:func:`create_notification_channels`). All policies get every
channel.
Returns:
dict: Dictionary mapping alert names to AlertPolicy resources.
"""
notification_channels = _channel_names(channels)
alerts = {}
# Alert: Sustained server errors (5xx).
#
# NOTE on semantics: with ALIGN_RATE + REDUCE_SUM the threshold is a request
# *rate* (requests/second), NOT a true percentage — the old "Error rate >
# 10%" name was a misnomer. It also matched ALL non-2xx codes, so a burst of
# 404s from an internet vuln-scanner (2026-07-20: 0.167 req/s of 404s from a
# single IP hammering /.env, /.ssh/... ) tripped it despite the service being
# perfectly healthy.
#
# Fix: scope to `response_code_class="5xx"` so only genuine *server* errors
# count — client/bot 404s never page us. WAF/rate-limiting at the Cloudflare
# edge (see modules/edge_security.py) is the primary defense that keeps that
# scan traffic off the metric entirely. Threshold = >0.1 5xx req/s sustained
# over 5 min. With COMPARISON_GT the boundary is strict: 6 errors/min
# (=0.1/s) does NOT fire; >=7/min sustained for 5 min does → real regression.
alerts["error_rate"] = gcp.monitoring.AlertPolicy(
"high-error-rate-alert",
display_name="Karaoke Backend - High Server-Error Rate (5xx)",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Sustained 5xx server errors > 0.1/s (5 min)",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/request_count" AND metric.labels.response_code_class="5xx"',
comparison="COMPARISON_GT",
threshold_value=0.1,
duration="300s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_RATE",
cross_series_reducer="REDUCE_SUM",
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="3600s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="Sustained 5xx server errors on karaoke-backend (not client/bot 404s). Check Cloud Run logs for crashes, timeouts, or dependency failures.\n\nDashboard: https://console.cloud.google.com/run/detail/us-central1/karaoke-backend/logs?project=nomadkaraoke",
mime_type="text/markdown",
),
enabled=True,
notification_channels=notification_channels,
)
# Alert: Cloud Tasks Queue Backlog (tasks piling up)
alerts["queue_backlog"] = gcp.monitoring.AlertPolicy(
"queue-backlog-alert",
display_name="Karaoke Backend - Queue Backlog",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Queue depth > 50 tasks",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='resource.type="cloud_tasks_queue" AND metric.type="cloudtasks.googleapis.com/queue/depth"',
comparison="COMPARISON_GT",
threshold_value=50,
duration="600s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_MEAN",
cross_series_reducer="REDUCE_MAX",
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="1800s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="Cloud Tasks queue backlog detected. Tasks are piling up faster than workers can process.\n\nPossible causes:\n- Worker errors (check logs)\n- External API issues (Modal, AudioShake)\n- Cloud Run scaling limits\n\nDashboard: https://console.cloud.google.com/cloudtasks?project=nomadkaraoke",
mime_type="text/markdown",
),
enabled=True,
notification_channels=notification_channels,
)
# Alert: High Memory Utilization (workers may be crashing)
alerts["memory"] = gcp.monitoring.AlertPolicy(
"high-memory-alert",
display_name="Karaoke Backend - High Memory Usage",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Memory > 85%",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/container/memory/utilizations"',
comparison="COMPARISON_GT",
threshold_value=0.85,
duration="300s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_PERCENTILE_95",
cross_series_reducer="REDUCE_MAX",
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="1800s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="High memory utilization on karaoke-backend. Workers may be running out of memory.\n\nActions:\n- Check for memory leaks in worker code\n- Consider increasing Cloud Run memory limits\n- Review temp file cleanup",
mime_type="text/markdown",
),
enabled=True,
notification_channels=notification_channels,
)
# Alert: Cloud Run Service Unavailable
alerts["service_unavailable"] = gcp.monitoring.AlertPolicy(
"service-unavailable-alert",
display_name="Karaoke Backend - Service Unavailable",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="No healthy instances",
condition_absent=gcp.monitoring.AlertPolicyConditionConditionAbsentArgs(
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/container/instance_count"',
duration="300s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionAbsentAggregationArgs(
alignment_period="60s",
per_series_aligner="ALIGN_MEAN",
cross_series_reducer="REDUCE_SUM",
),
],
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="1800s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content="Karaoke backend service appears to be down - no healthy Cloud Run instances detected.\n\nImmediate actions:\n1. Check Cloud Run logs for crash reasons\n2. Verify recent deployments\n3. Check external dependencies (Modal, AudioShake)\n\nService URL: https://api.nomadkaraoke.com/api/health",
mime_type="text/markdown",
),
enabled=True,
notification_channels=notification_channels,
)
return alerts
def create_ephemeral_runner_observability(
channels: dict[str, gcp.monitoring.NotificationChannel],
) -> dict:
"""Log-based metrics + alerts for the ephemeral GHA runner dispatcher.
Adds two alerts that are useful during the Phase 3 soak and afterward:
1. Dispatcher VM-create failures (the dispatcher returns 503 + logs
``create_ephemeral_runner failed`` — any occurrence in 10 min fires).
2. Long-running ephemeral runner VMs (uptime > 2h30m on instances labelled
``purpose=gha-ephemeral-runner`` — catches hung jobs or cleanup bugs).
"""
notification_channels = _channel_names(channels)
resources: dict = {}
# ---- Log-based metric: dispatch failures -------------------------------
resources["dispatch_failure_metric"] = gcp.logging.Metric(
"runner-dispatcher-failures",
name="runner_dispatcher/create_failures",
description="Counts create_ephemeral_runner failures (any zone)",
# Function logs include the function name; runner-manager covers both
# legacy + ephemeral entry points.
filter=(
'resource.type="cloud_run_revision" '
'AND resource.labels.service_name="github-runner-manager" '
'AND textPayload:"create_ephemeral_runner failed"'
),
metric_descriptor=gcp.logging.MetricMetricDescriptorArgs(
metric_kind="DELTA",
value_type="INT64",
unit="1",
display_name="Runner dispatcher create failures",
),
)
# ---- Log-based metric: orphan kills (visibility, not alerting) ---------
resources["orphan_kill_metric"] = gcp.logging.Metric(
"runner-dispatcher-orphan-kills",
name="runner_dispatcher/orphan_kills",
description="Counts orphan VMs deleted by the dispatcher cleanup pass",
filter=(
'resource.type="cloud_run_revision" '
'AND resource.labels.service_name="github-runner-manager" '
'AND textPayload:"age=" AND textPayload:"deleting"'
),
metric_descriptor=gcp.logging.MetricMetricDescriptorArgs(
metric_kind="DELTA",
value_type="INT64",
unit="1",
display_name="Runner dispatcher orphan-VM deletions",
),
)
# ---- Alert: any dispatcher create failure in last 10 minutes -----------
resources["dispatch_failure_alert"] = gcp.monitoring.AlertPolicy(
"runner-dispatcher-failures-alert",
display_name="GHA Runner Dispatcher - Create Failures",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Any create_ephemeral_runner failure",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter=(
'metric.type="logging.googleapis.com/user/runner_dispatcher/create_failures" '
'AND resource.type="cloud_run_revision"'
),
comparison="COMPARISON_GT",
threshold_value=0,
duration="0s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="600s",
per_series_aligner="ALIGN_SUM",
cross_series_reducer="REDUCE_SUM",
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
auto_close="3600s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content=(
"Ephemeral runner dispatcher failed to create a VM.\n\n"
"Likely causes:\n"
"- GCE quota exhausted (CPU or NVIDIA_T4_GPUS in us-central1)\n"
"- GitHub JIT mint API rate-limited or PAT expired\n"
"- Image family deprecated/missing (gha-runner-{general,build,gpu})\n"
"- Both primary and fallback zones exhausted\n\n"
"Logs: filter `resource.labels.service_name=\"github-runner-manager\"` "
"in Cloud Logging for `create_ephemeral_runner failed`.\n\n"
"Rollback: not possible from this alert — Phase 4 deleted the "
"legacy long-lived VMs (PR #780). Recovery requires either "
"fixing the failure mode or recreating the legacy pool from "
"git history."
),
mime_type="text/markdown",
),
enabled=True,
notification_channels=notification_channels,
opts=pulumi.ResourceOptions(depends_on=[resources["dispatch_failure_metric"]]),
)
# ---- Alert: ephemeral VM uptime > 2h30m --------------------------------
# `compute.googleapis.com/instance/uptime_total` is a cumulative count of
# seconds the VM has been up. The orphan-cleanup pass kills VMs at
# MAX_VM_LIFETIME_MINUTES=120 on a 15-min scheduler tick, so worst-case
# lifetime is ~135min. ALIGN_MAX over a 5-min window plus the 300s
# duration means a threshold of exactly 7200s fires on every normal
# cleanup (the window containing the pre-delete sample still reports
# max>threshold for 5min after the VM is gone). Threshold set to 9000s
# (150min = worst-case + 15min buffer) so this alert only fires when
# cleanup is genuinely broken, not on routine cleanups that land within
# the expected window.
resources["long_lived_vm_alert"] = gcp.monitoring.AlertPolicy(
"runner-dispatcher-long-lived-vm-alert",
display_name="GHA Runner Dispatcher - VM Alive > 2h30m",
combiner="OR",
conditions=[
gcp.monitoring.AlertPolicyConditionArgs(
display_name="Ephemeral runner VM uptime > 2h30m",
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
filter=(
'metric.type="compute.googleapis.com/instance/uptime_total" '
'AND resource.type="gce_instance" '
'AND metadata.user_labels.purpose="gha-ephemeral-runner"'
),
comparison="COMPARISON_GT",
threshold_value=9000,
duration="300s",
aggregations=[
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
alignment_period="300s",
per_series_aligner="ALIGN_MAX",
cross_series_reducer="REDUCE_MAX",
group_by_fields=["resource.labels.instance_id"],
),
],
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
count=1,
),
),
),
],
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
# GCP minimum is 30 minutes — anything lower returns 400.
auto_close="1800s",
),
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
content=(
"An ephemeral GHA runner VM has been alive for more than 2h30m.\n"
"Orphan-cleanup should kill VMs at MAX_VM_LIFETIME_MINUTES (120)\n"
"on a 15-min scheduler tick — worst-case lifetime is ~135min.\n"
"If this alert fires, the cleanup pass is broken or the VM is\n"
"escaping it.\n\n"
"Investigate:\n"
"1. `gcloud compute instances list --filter=labels.purpose=gha-ephemeral-runner`\n"
"2. Check Cloud Function logs for the cleanup pass\n"
"3. Manually delete: `gcloud compute instances delete <name> --zone=<zone>`"
),
mime_type="text/markdown",
),
enabled=True,
notification_channels=notification_channels,
)
return resources