## Summary This is a follow-up to #1112/#1113 and decide#147. Postmark confirmed (ticket #11562402) that the nginx 403s are an IP-reputation block on shared Google us-central1 egress IPs, and they won't lift it. The SMTP fallback is working in prod: on 2026-10-02 between 19:10 and 20:17 UTC, 12 of 12 blocked sends were delivered with 0 failures. To save money, we're **deferring** a static egress IP and adding a free tripwire instead. ## Changes - `infrastructure/modules/monitoring.py`: `create_email_delivery_observability()` adds: - A log-based metric `email/postmark_smtp_fallback_failures`, plus an alert policy ("Email - Postmark SMTP fallback failing (emails being lost)") that fires on any failed fallback send in karaoke-backend **or** karaoke-decide. - The filter matches **both** `jsonPayload.message` (gen logs structured JSON) and `textPayload` (decide logs plain text). - `docs/archive/2026-10-02-postmark-ip-block-options.md`: full write-up covering the evidence, Postmark's answer, how the fallback works, the alert, and the static-IP options. It includes **measured costs**: reusing the existing `github-runners-nat` with a reserved IP is ~$4–7/mo. It also has Pulumi steps, NAT sizing, and how to verify the reserved IP is used. - `docs/TROUBLESHOOTING.md` and `infrastructure/monitoring/README.md` link to it. ## Testing / rollout - Filter validated against real logs: - 0 failures, which is correct. - Positive control on the gen JSON branch: 12/12 real fallback sends matched. - Positive control on the decide text branch: 5/5 real lines matched. - `pulumi preview` showed: +2 to create, 274 unchanged. - **Already applied locally with `pulumi up`** (repo rule). The first apply hit the log-metric propagation race, and re-running succeeded; that gotcha is now documented in the code. The alert is live, enabled, and has the email channel attached. - Cost: $0 now. Alert policies cost $0.35/mo from 2027-09-01. Local CodeRabbit review: 2 findings (Cloud NAT sizing, reserved-IP verification in the options doc), both addressed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
9.2 KiB
Monitoring Infrastructure
This directory contains Cloud Monitoring dashboard and alerting configurations for the karaoke-gen backend.
🎯 Quick Start: Debug a Failed Job
Bookmark this URL:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22karaoke-backend%22;duration=PT6H?project=nomadkaraoke
How to investigate a job:
-
Open the bookmarked URL (Logs Explorer)
-
Add your job ID to the query. In the query box, add a third line with your job ID:
resource.type="cloud_run_revision" resource.labels.service_name="karaoke-backend" "[job:YOUR_JOB_ID"(Replace
YOUR_JOB_IDwith the actual ID, e.g."[job:8f03f0ac") -
Click "Run query" or press
Shift+Enter -
Enable "Stream logs" (toggle at top right) for real-time updates
What you'll see:
WORKER_START- When each worker (audio/lyrics/screens/video) beginsWORKER_END status=success/error- When it finishes with duration- All log messages from that job's processing
- Any error tracebacks
Example queries:
All logs for job 8f03f0ac:
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
"[job:8f03f0ac"
Only errors for a job:
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
"[job:8f03f0ac"
severity>=ERROR
Only worker start/end events:
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
"[job:8f03f0ac"
("WORKER_START" OR "WORKER_END")
Dashboard
The dashboard.json file defines a Cloud Monitoring dashboard with the following panels:
Overview Metrics
- Jobs Created (Last Hour): Count of jobs created
- Cloud Run Request Count: Request rate to the backend
- Cloud Run Error Rate: Percentage of non-2xx responses
Performance Metrics
- Request Latency (p95): 95th percentile response time
- Cloud Run Instance Count: Active container instances
- CPU Utilization: CPU usage across instances
- Memory Utilization: Memory usage across instances
Cloud Tasks
- Queue Depth: Number of tasks waiting in each queue
- Execution Rate: Task processing rate per queue
Logs
- Worker Logs: Searchable by
job_id - Error Logs: All ERROR-level logs from the backend
Importing the Dashboard
To import the dashboard into Cloud Monitoring:
# Using gcloud CLI
gcloud monitoring dashboards create --config-from-file=infrastructure/monitoring/dashboard.json --project=nomadkaraoke
Or manually:
- Go to Cloud Console → Monitoring → Dashboards
- Click "Create Dashboard"
- Click "JSON Editor" and paste contents of
dashboard.json
Alert Policies
Alert policies are defined in infrastructure/__main__.py as Pulumi resources:
High Error Rate
- Condition: >10% of requests return errors for 5+ minutes
- Severity: Warning
- Auto-close: 1 hour after resolution
Queue Backlog
- Condition: >50 tasks in any queue for 10+ minutes
- Severity: Warning
- Auto-close: 30 minutes after resolution
High Memory Usage
- Condition: >85% memory utilization for 5+ minutes
- Severity: Warning
- Auto-close: 30 minutes after resolution
Service Unavailable
- Condition: No healthy Cloud Run instances for 5+ minutes
- Severity: Critical
- Auto-close: 10 minutes after resolution
Email - Postmark SMTP fallback failing (emails being lost)
- Defined in:
modules/monitoring.py→create_email_delivery_observability() - Condition: any failed Postmark SMTP-fallback send in karaoke-backend or karaoke-decide in a 10-minute window. Counted by the log-based metric
email/postmark_smtp_fallback_failures. - Gotcha: the filter matches both
jsonPayload.message(karaoke-backend logs structured JSON) andtextPayload(karaoke-decide logs plain text). A filter on only one field silently misses the other service. - Meaning: the Postmark API was edge-blocked (HTML 403) and the SMTP fallback failed, so users aren't getting magic links or order emails.
- Action / options:
docs/archive/2026-10-02-postmark-ip-block-options.md(static egress IP via the existing Cloud NAT, ~$4–7/mo) - Auto-close: 60 minutes after resolution
Notification Channels
Notification channels are managed as Pulumi IaC in
infrastructure/modules/monitoring.py — see
create_notification_channels() and the notification_channels=... arg
attached to every alert policy.
Current channel: one email channel (alerts-email) delivering to the
project owner.
Gotcha: Pulumi-created email channels don't auto-send the verification email
When an email channel is created through the Cloud Console, Cloud Monitoring
automatically sends a verification link to the address. When it's created
through the API or Pulumi, that auto-send does not happen — the channel
lands in verificationStatus=UNVERIFIED and stays there silently. Alerts will
fire and resolve but no email gets delivered.
After pulumi up creates a new email channel, trigger the verification email
explicitly:
unset GOOGLE_APPLICATION_CREDENTIALS # need admin creds, not claude-readonly
CHANNEL_ID=$(gcloud alpha monitoring channels list --project=nomadkaraoke \
--filter='type=email AND displayName="Andrew (email)"' \
--format='value(name.basename())')
curl -sS -X POST -d '{}' \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
"https://monitoring.googleapis.com/v3/projects/nomadkaraoke/notificationChannels/${CHANNEL_ID}:sendVerificationCode"
The email arrives within a minute or two. Subject is something like
"Verify your email address for Google Cloud Alerting" and the body contains
a code in the form G-NNNNNN plus a clickable link. Either path completes
the verification — clicking the link is simplest, or POST the code to
:verify:
curl -sS -X POST -d '{"code":"G-NNNNNN"}' \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
"https://monitoring.googleapis.com/v3/projects/nomadkaraoke/notificationChannels/${CHANNEL_ID}:verify"
Confirm afterwards:
curl -sS -H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://monitoring.googleapis.com/v3/projects/nomadkaraoke/notificationChannels/${CHANNEL_ID}" \
| python3 -c "import json,sys; print(json.load(sys.stdin).get('verificationStatus','(missing)'))"
Expected output: VERIFIED. Until that's true, gcloud alpha monitoring channels list --format='table(...,verificationStatus,...)' returns an
empty verificationStatus column (which is itself a tell — verified
channels populate the field).
Adding a second channel (e.g. Discord)
Discord supports Slack-compatible webhooks: append /slack to a Discord
webhook URL and Discord parses Slack-formatted payloads. To add a Discord
channel alongside email:
-
Create a webhook in Discord (Server settings → Integrations → Webhooks).
-
Store the URL in Secret Manager (the project already has
discord-alert-webhookfor the error-monitor service). -
In
monitoring.py, add a second entry tocreate_notification_channels():channels["discord"] = gcp.monitoring.NotificationChannel( "alerts-discord", display_name="Alerts (Discord)", type="slack", sensitive_labels=gcp.monitoring.NotificationChannelSensitiveLabelsArgs( auth_token=discord_webhook_url + "/slack", ), enabled=True, ) -
pulumi up. All alert policies pick up the new channel automatically because they iteratechannels.values().
Log-Based Metrics
The structured logging in backend/services/structured_logging.py outputs metrics as JSON log entries. You can create log-based metrics to track:
- Go to Cloud Console → Logging → Log-based metrics
- Create new metric with filter:
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
jsonPayload.metric_name="jobs_total"
- Set extraction field to
jsonPayload.metric_value
Repeat for other metrics:
worker_invocations_totaljob_stage_duration_secondsexternal_api_calls_totalexternal_api_duration_secondsgcs_operations_total
Useful Log Queries
Find all logs for a specific job:
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
jsonPayload.job_id="YOUR_JOB_ID"
Find worker errors:
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
textPayload:"WORKER_END" AND textPayload:"status=error"
Find slow workers (>5 min):
resource.type="cloud_run_revision"
resource.labels.service_name="karaoke-backend"
textPayload:"WORKER_END" AND textPayload:"duration="
Then add duration filter in the UI.
Cloud Trace Integration
Structured logs include trace correlation fields:
logging.googleapis.com/trace: Links to Cloud Tracelogging.googleapis.com/spanId: Current span ID
To view correlated traces:
- Find a log entry in Cloud Logging
- Click the "View in Trace" link in the log entry
- See the full distributed trace across services