Files
Andrew Beveridge ☄️andClaude Opus 5.5 d6972f10aa infra(monitoring): alert when Postmark SMTP-fallback sends fail + document static-IP options (#1114)
## Summary
This is a follow-up to #1112/#1113 and decide#147. Postmark confirmed
(ticket #11562402) that the nginx 403s are an IP-reputation block on
shared Google us-central1 egress IPs, and they won't lift it. The SMTP
fallback is working in prod: on 2026-10-02 between 19:10 and 20:17 UTC,
12 of 12 blocked sends were delivered with 0 failures. To save money,
we're **deferring** a static egress IP and adding a free tripwire
instead.

## Changes
- `infrastructure/modules/monitoring.py`:
`create_email_delivery_observability()` adds:
- A log-based metric `email/postmark_smtp_fallback_failures`, plus an
alert policy ("Email - Postmark SMTP fallback failing (emails being
lost)") that fires on any failed fallback send in karaoke-backend **or**
karaoke-decide.
- The filter matches **both** `jsonPayload.message` (gen logs structured
JSON) and `textPayload` (decide logs plain text).
- `docs/archive/2026-10-02-postmark-ip-block-options.md`: full write-up
covering the evidence, Postmark's answer, how the fallback works, the
alert, and the static-IP options. It includes **measured costs**:
reusing the existing `github-runners-nat` with a reserved IP is
~$4–7/mo. It also has Pulumi steps, NAT sizing, and how to verify the
reserved IP is used.
- `docs/TROUBLESHOOTING.md` and `infrastructure/monitoring/README.md`
link to it.

## Testing / rollout
- Filter validated against real logs:
  - 0 failures, which is correct.
- Positive control on the gen JSON branch: 12/12 real fallback sends
matched.
  - Positive control on the decide text branch: 5/5 real lines matched.
- `pulumi preview` showed: +2 to create, 274 unchanged.
- **Already applied locally with `pulumi up`** (repo rule). The first
apply hit the log-metric propagation race, and re-running succeeded;
that gotcha is now documented in the code. The alert is live, enabled,
and has the email channel attached.
- Cost: $0 now. Alert policies cost $0.35/mo from 2027-09-01.

Local CodeRabbit review: 2 findings (Cloud NAT sizing, reserved-IP
verification in the options doc), both addressed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 18:42:29 -04:00
..

Monitoring Infrastructure

This directory contains Cloud Monitoring dashboard and alerting configurations for the karaoke-gen backend.


🎯 Quick Start: Debug a Failed Job

Bookmark this URL:

https://console.cloud.google.com/logs/query;query=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22karaoke-backend%22;duration=PT6H?project=nomadkaraoke

How to investigate a job:

  1. Open the bookmarked URL (Logs Explorer)

  2. Add your job ID to the query. In the query box, add a third line with your job ID:

    resource.type="cloud_run_revision"
    resource.labels.service_name="karaoke-backend"
    "[job:YOUR_JOB_ID"
    

    (Replace YOUR_JOB_ID with the actual ID, e.g. "[job:8f03f0ac")

  3. Click "Run query" or press Shift+Enter

  4. Enable "Stream logs" (toggle at top right) for real-time updates

What you'll see:

  • WORKER_START - When each worker (audio/lyrics/screens/video) begins
  • WORKER_END status=success/error - When it finishes with duration
  • All log messages from that job's processing
  • Any error tracebacks

Example queries:

All logs for job 8f03f0ac:

resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
"[job:8f03f0ac"

Only errors for a job:

resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
"[job:8f03f0ac"
severity>=ERROR

Only worker start/end events:

resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
"[job:8f03f0ac"
("WORKER_START" OR "WORKER_END")

Dashboard

The dashboard.json file defines a Cloud Monitoring dashboard with the following panels:

Overview Metrics

  • Jobs Created (Last Hour): Count of jobs created
  • Cloud Run Request Count: Request rate to the backend
  • Cloud Run Error Rate: Percentage of non-2xx responses

Performance Metrics

  • Request Latency (p95): 95th percentile response time
  • Cloud Run Instance Count: Active container instances
  • CPU Utilization: CPU usage across instances
  • Memory Utilization: Memory usage across instances

Cloud Tasks

  • Queue Depth: Number of tasks waiting in each queue
  • Execution Rate: Task processing rate per queue

Logs

  • Worker Logs: Searchable by job_id
  • Error Logs: All ERROR-level logs from the backend

Importing the Dashboard

To import the dashboard into Cloud Monitoring:

# Using gcloud CLI
gcloud monitoring dashboards create --config-from-file=infrastructure/monitoring/dashboard.json --project=nomadkaraoke

Or manually:

  1. Go to Cloud Console → Monitoring → Dashboards
  2. Click "Create Dashboard"
  3. Click "JSON Editor" and paste contents of dashboard.json

Alert Policies

Alert policies are defined in infrastructure/__main__.py as Pulumi resources:

High Error Rate

  • Condition: >10% of requests return errors for 5+ minutes
  • Severity: Warning
  • Auto-close: 1 hour after resolution

Queue Backlog

  • Condition: >50 tasks in any queue for 10+ minutes
  • Severity: Warning
  • Auto-close: 30 minutes after resolution

High Memory Usage

  • Condition: >85% memory utilization for 5+ minutes
  • Severity: Warning
  • Auto-close: 30 minutes after resolution

Service Unavailable

  • Condition: No healthy Cloud Run instances for 5+ minutes
  • Severity: Critical
  • Auto-close: 10 minutes after resolution

Email - Postmark SMTP fallback failing (emails being lost)

  • Defined in: modules/monitoring.py → create_email_delivery_observability()
  • Condition: any failed Postmark SMTP-fallback send in karaoke-backend or karaoke-decide in a 10-minute window. Counted by the log-based metric email/postmark_smtp_fallback_failures.
  • Gotcha: the filter matches both jsonPayload.message (karaoke-backend logs structured JSON) and textPayload (karaoke-decide logs plain text). A filter on only one field silently misses the other service.
  • Meaning: the Postmark API was edge-blocked (HTML 403) and the SMTP fallback failed, so users aren't getting magic links or order emails.
  • Action / options: docs/archive/2026-10-02-postmark-ip-block-options.md (static egress IP via the existing Cloud NAT, ~$4–7/mo)
  • Auto-close: 60 minutes after resolution

Notification Channels

Notification channels are managed as Pulumi IaC in infrastructure/modules/monitoring.py — see create_notification_channels() and the notification_channels=... arg attached to every alert policy.

Current channel: one email channel (alerts-email) delivering to the project owner.

Gotcha: Pulumi-created email channels don't auto-send the verification email

When an email channel is created through the Cloud Console, Cloud Monitoring automatically sends a verification link to the address. When it's created through the API or Pulumi, that auto-send does not happen — the channel lands in verificationStatus=UNVERIFIED and stays there silently. Alerts will fire and resolve but no email gets delivered.

After pulumi up creates a new email channel, trigger the verification email explicitly:

unset GOOGLE_APPLICATION_CREDENTIALS   # need admin creds, not claude-readonly
CHANNEL_ID=$(gcloud alpha monitoring channels list --project=nomadkaraoke \
  --filter='type=email AND displayName="Andrew (email)"' \
  --format='value(name.basename())')

curl -sS -X POST -d '{}' \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  "https://monitoring.googleapis.com/v3/projects/nomadkaraoke/notificationChannels/${CHANNEL_ID}:sendVerificationCode"

The email arrives within a minute or two. Subject is something like "Verify your email address for Google Cloud Alerting" and the body contains a code in the form G-NNNNNN plus a clickable link. Either path completes the verification — clicking the link is simplest, or POST the code to :verify:

curl -sS -X POST -d '{"code":"G-NNNNNN"}' \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  "https://monitoring.googleapis.com/v3/projects/nomadkaraoke/notificationChannels/${CHANNEL_ID}:verify"

Confirm afterwards:

curl -sS -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://monitoring.googleapis.com/v3/projects/nomadkaraoke/notificationChannels/${CHANNEL_ID}" \
  | python3 -c "import json,sys; print(json.load(sys.stdin).get('verificationStatus','(missing)'))"

Expected output: VERIFIED. Until that's true, gcloud alpha monitoring channels list --format='table(...,verificationStatus,...)' returns an empty verificationStatus column (which is itself a tell — verified channels populate the field).

Adding a second channel (e.g. Discord)

Discord supports Slack-compatible webhooks: append /slack to a Discord webhook URL and Discord parses Slack-formatted payloads. To add a Discord channel alongside email:

  1. Create a webhook in Discord (Server settings → Integrations → Webhooks).

  2. Store the URL in Secret Manager (the project already has discord-alert-webhook for the error-monitor service).

  3. In monitoring.py, add a second entry to create_notification_channels():

    channels["discord"] = gcp.monitoring.NotificationChannel(
        "alerts-discord",
        display_name="Alerts (Discord)",
        type="slack",
        sensitive_labels=gcp.monitoring.NotificationChannelSensitiveLabelsArgs(
            auth_token=discord_webhook_url + "/slack",
        ),
        enabled=True,
    )
    
  4. pulumi up. All alert policies pick up the new channel automatically because they iterate channels.values().

Log-Based Metrics

The structured logging in backend/services/structured_logging.py outputs metrics as JSON log entries. You can create log-based metrics to track:

  1. Go to Cloud Console → Logging → Log-based metrics
  2. Create new metric with filter:
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
jsonPayload.metric_name="jobs_total"
  1. Set extraction field to jsonPayload.metric_value

Repeat for other metrics:

  • worker_invocations_total
  • job_stage_duration_seconds
  • external_api_calls_total
  • external_api_duration_seconds
  • gcs_operations_total

Useful Log Queries

Find all logs for a specific job:

resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
jsonPayload.job_id="YOUR_JOB_ID"

Find worker errors:

resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
textPayload:"WORKER_END" AND textPayload:"status=error"

Find slow workers (>5 min):

resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
textPayload:"WORKER_END" AND textPayload:"duration=" 

Then add duration filter in the UI.

Cloud Trace Integration

Structured logs include trace correlation fields:

  • logging.googleapis.com/trace: Links to Cloud Trace
  • logging.googleapis.com/spanId: Current span ID

To view correlated traces:

  1. Find a log entry in Cloud Logging
  2. Click the "View in Trace" link in the log entry
  3. See the full distributed trace across services