feat(alerts): route non-signal Discord posts to #noise (v0.278.8) (#1186)

## Summary
Andrew asked for #alerts to carry only real problems. Routine and
informational posts now go to the new **#noise** channel. The GitHub org
PR webhook was already moved to #noise separately.

| Message | Before | After |
|---|---|---|
| Error monitor: New Error Pattern / Spike / Incident | #alerts |
#alerts |
| Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** |
| Error monitor: 📊 Daily Error Digest | #alerts | **#noise** |
| Nightly Backup Report (clean run) | #alerts | **#noise** |
| Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED")
| #alerts | #alerts |
| ✅ E2E Daily Test PASSED | #alerts | **#noise** |
| 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise**
|
| 🩹 self-heal PR needs CI approval click / could not open | #alerts |
#alerts |
| E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish
completeness SHADOW, Gemini unavailable, OAuth credential alerts,
quarterly DR drill reminder | #alerts | #alerts (unchanged) |

## How the noise webhook is resolved
- **Backend / error monitor and the backup function** read it at
runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager
secret `discord-noise-webhook` (the name can be overridden with
`DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed,
because both service accounts already have project-wide
`secretAccessor`.
- **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL ||
secrets.DISCORD_ALERT_WEBHOOK_URL`.
- **Nothing is silently dropped.** If the noise webhook can't be
resolved, or a post to it fails, the message goes to #alerts instead. In
the workflow this uses `curl -fsS` with a retry on the alert webhook.
- The error monitor now rate-limits each channel separately (10 per run
each), so noise can't use up the alert budget during an incident.

## Deploy notes
- **Backup Cloud Function:** CI doesn't deploy its source. Run
`infrastructure/functions/backup_to_aws/deploy.sh` after merge.
- **Error monitor job:** runs `karaoke-backend:latest`, so it picks up
the change once CI pushes the image.
- **No `pulumi up` needed.**

## Testing
- `tests/unit/services/error_monitor`: 376 passed. New tests cover noise
vs alert routing, fallback when the noise webhook is missing or fails,
separate rate limits, and secret resolution.
- `backend/tests` (excluding emulator): 6418 passed, 36 skipped.
- `infrastructure/functions/backup_to_aws`: 60 passed, including the new
`test_discord_alert.py` and routing tests in `test_main.py`. These were
run in a scratch venv because `functions_framework` isn't in the conda
env, which was already the case before this change.
- `actionlint` on `e2e-daily.yml`: no new findings. The only findings
are info-level shellcheck notes that were already there.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-10-19 02:46:50 +00:00
1 parent 290f927175
commit a6d4163d2e
14 files changed
+554 -57

No files matched your search

+20 -6
View File
@@ -411,22 +411,31 @@ jobs:
if: always() && steps.pr.outputs.unchanged != 'true'
run: |
PR="${{ steps.pr.outputs.pr_url }}"
# #alerts = needs a human; #noise = informational (auto-merge handles it).
# Noise falls back to the alert webhook if its secret is unset.
ALERT_WEBHOOK="${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
NOISE_WEBHOOK="${{ secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL }}"
if [ -n "$PR" ] && [ "${{ steps.pr.outputs.ci_needs_approval }}" = "true" ]; then
TITLE="🩹 E2E self-healed Stripe — PR needs a CI approval click"
DESC="LLM fallback healed a Stripe Checkout change. The auto-merge PR's CI run awaits approval (open the PR → Checks → Approve and run): $PR"
COLOR=16776960
WEBHOOK="$ALERT_WEBHOOK"
elif [ -n "$PR" ]; then
TITLE="🩹 E2E self-healed a Stripe Checkout change"
DESC="Stripe changed its checkout page; the LLM fallback got through and the purchase was verified. Auto-merge PR: $PR"
COLOR=16776960
WEBHOOK="$NOISE_WEBHOOK"
else
TITLE="⚠️ E2E self-heal PR could not be opened"
DESC="The LLM fallback healed a Stripe step but opening the PR failed — see the run."
COLOR=15105570
WEBHOOK="$ALERT_WEBHOOK"
fi
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"$TITLE\", \"description\": \"$DESC\n\n**Run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": $COLOR}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
PAYLOAD="{\"embeds\": [{\"title\": \"$TITLE\", \"description\": \"$DESC\n\n**Run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": $COLOR}]}"
# A failed #noise post retries on #alerts so it is never silently lost.
curl -fsS -H "Content-Type: application/json" -d "$PAYLOAD" "$WEBHOOK" \
|| { [ "$WEBHOOK" != "$ALERT_WEBHOOK" ] && curl -s -H "Content-Type: application/json" -d "$PAYLOAD" "$ALERT_WEBHOOK"; } \
|| true
# =========================================================================
# Stage 2: Happy Path (uses credit from Stage 1)
@@ -600,12 +609,17 @@ jobs:
steps:
- name: Discord notification (success)
# Passes are not signal: they go to #noise (fallback: #alerts if the
# noise secret is unset, so nothing is ever silently dropped).
if: needs.e2e-happy-path.result == 'success'
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
ALERT_WEBHOOK="${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
NOISE_WEBHOOK="${{ secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL }}"
PAYLOAD="{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}"
curl -fsS -H "Content-Type: application/json" -d "$PAYLOAD" "$NOISE_WEBHOOK" \
|| { [ "$NOISE_WEBHOOK" != "$ALERT_WEBHOOK" ] && curl -s -H "Content-Type: application/json" -d "$PAYLOAD" "$ALERT_WEBHOOK"; } \
|| true
- name: Send email notification
if: github.event_name == 'schedule'