mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
feat(alerts): route non-signal Discord posts to #noise (v0.278.8) (#1186)
## Summary Andrew asked for #alerts to carry only real problems. Routine and informational posts now go to the new **#noise** channel. The GitHub org PR webhook was already moved to #noise separately. | Message | Before | After | |---|---|---| | Error monitor: New Error Pattern / Spike / Incident | #alerts | #alerts | | Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** | | Error monitor: 📊 Daily Error Digest | #alerts | **#noise** | | Nightly Backup Report (clean run) | #alerts | **#noise** | | Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED") | #alerts | #alerts | | ✅ E2E Daily Test PASSED | #alerts | **#noise** | | 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise** | | 🩹 self-heal PR needs CI approval click / could not open | #alerts | #alerts | | E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish completeness SHADOW, Gemini unavailable, OAuth credential alerts, quarterly DR drill reminder | #alerts | #alerts (unchanged) | ## How the noise webhook is resolved - **Backend / error monitor and the backup function** read it at runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager secret `discord-noise-webhook` (the name can be overridden with `DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed, because both service accounts already have project-wide `secretAccessor`. - **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL`. - **Nothing is silently dropped.** If the noise webhook can't be resolved, or a post to it fails, the message goes to #alerts instead. In the workflow this uses `curl -fsS` with a retry on the alert webhook. - The error monitor now rate-limits each channel separately (10 per run each), so noise can't use up the alert budget during an incident. ## Deploy notes - **Backup Cloud Function:** CI doesn't deploy its source. Run `infrastructure/functions/backup_to_aws/deploy.sh` after merge. - **Error monitor job:** runs `karaoke-backend:latest`, so it picks up the change once CI pushes the image. - **No `pulumi up` needed.** ## Testing - `tests/unit/services/error_monitor`: 376 passed. New tests cover noise vs alert routing, fallback when the noise webhook is missing or fails, separate rate limits, and secret resolution. - `backend/tests` (excluding emulator): 6418 passed, 36 skipped. - `infrastructure/functions/backup_to_aws`: 60 passed, including the new `test_discord_alert.py` and routing tests in `test_main.py`. These were run in a scratch venv because `functions_framework` isn't in the conda env, which was already the case before this change. - `actionlint` on `e2e-daily.yml`: no new findings. The only findings are info-level shellcheck notes that were already there. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
290f927175
commit
a6d4163d2e
14 files changed
+554
-57
No files matched your search
@@ -411,22 +411,31 @@ jobs:
|
||||
if: always() && steps.pr.outputs.unchanged != 'true'
|
||||
run: |
|
||||
PR="${{ steps.pr.outputs.pr_url }}"
|
||||
# #alerts = needs a human; #noise = informational (auto-merge handles it).
|
||||
# Noise falls back to the alert webhook if its secret is unset.
|
||||
ALERT_WEBHOOK="${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
NOISE_WEBHOOK="${{ secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
if [ -n "$PR" ] && [ "${{ steps.pr.outputs.ci_needs_approval }}" = "true" ]; then
|
||||
TITLE="🩹 E2E self-healed Stripe — PR needs a CI approval click"
|
||||
DESC="LLM fallback healed a Stripe Checkout change. The auto-merge PR's CI run awaits approval (open the PR → Checks → Approve and run): $PR"
|
||||
COLOR=16776960
|
||||
WEBHOOK="$ALERT_WEBHOOK"
|
||||
elif [ -n "$PR" ]; then
|
||||
TITLE="🩹 E2E self-healed a Stripe Checkout change"
|
||||
DESC="Stripe changed its checkout page; the LLM fallback got through and the purchase was verified. Auto-merge PR: $PR"
|
||||
COLOR=16776960
|
||||
WEBHOOK="$NOISE_WEBHOOK"
|
||||
else
|
||||
TITLE="⚠️ E2E self-heal PR could not be opened"
|
||||
DESC="The LLM fallback healed a Stripe step but opening the PR failed — see the run."
|
||||
COLOR=15105570
|
||||
WEBHOOK="$ALERT_WEBHOOK"
|
||||
fi
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"$TITLE\", \"description\": \"$DESC\n\n**Run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": $COLOR}]}" \
|
||||
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
PAYLOAD="{\"embeds\": [{\"title\": \"$TITLE\", \"description\": \"$DESC\n\n**Run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": $COLOR}]}"
|
||||
# A failed #noise post retries on #alerts so it is never silently lost.
|
||||
curl -fsS -H "Content-Type: application/json" -d "$PAYLOAD" "$WEBHOOK" \
|
||||
|| { [ "$WEBHOOK" != "$ALERT_WEBHOOK" ] && curl -s -H "Content-Type: application/json" -d "$PAYLOAD" "$ALERT_WEBHOOK"; } \
|
||||
|| true
|
||||
|
||||
# =========================================================================
|
||||
# Stage 2: Happy Path (uses credit from Stage 1)
|
||||
@@ -600,12 +609,17 @@ jobs:
|
||||
|
||||
steps:
|
||||
- name: Discord notification (success)
|
||||
# Passes are not signal: they go to #noise (fallback: #alerts if the
|
||||
# noise secret is unset, so nothing is ever silently dropped).
|
||||
if: needs.e2e-happy-path.result == 'success'
|
||||
run: |
|
||||
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}" \
|
||||
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
ALERT_WEBHOOK="${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
NOISE_WEBHOOK="${{ secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
PAYLOAD="{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}"
|
||||
curl -fsS -H "Content-Type: application/json" -d "$PAYLOAD" "$NOISE_WEBHOOK" \
|
||||
|| { [ "$NOISE_WEBHOOK" != "$ALERT_WEBHOOK" ] && curl -s -H "Content-Type: application/json" -d "$PAYLOAD" "$ALERT_WEBHOOK"; } \
|
||||
|| true
|
||||
|
||||
- name: Send email notification
|
||||
if: github.event_name == 'schedule'
|
||||
|
||||
Reference in new issue
Block a user