Files
karaoke-gen/tests
Andrew Beveridge ☄️andClaude Opus 5.5 70b2f4279a feat(alerts): collapse error bursts, suppress link previews, de-noise #alerts (v0.278.9) (#1188)
Andrew's goal: "I'd like to make sure that our alerts channel only has
alerts in it that are actionable and important, or are just signal and
not noise." This follows #1186 (#noise routing).

## Changes
1. **Burst collapse** (`error_monitor/monitor.py`, `discord.py`,
`config.py`). When a run has more than 3 new patterns for one service,
including leftover un-alerted ones, the monitor sends ONE #alerts
message. It has the count, total occurrences, the top 5 by count,
first/last seen, and a `scripts/query-error-patterns.py --service X
--status new` pointer. All of those patterns are marked alerted. The
grouping is deterministic, since LLM grouping is off in prod. The
10-messages-per-run cap still applies, and unsent patterns are picked up
again next run. Evidence: on 2026-10-07/08, one mb-refresh run produced
73 separate alerts.
2. **No link previews.** Every error-monitor post (`_post`) and every
`ops_alerts` post now sets `flags: 4` (SUPPRESS_EMBEDS), through a new
`suppress_embeds` kwarg on `DiscordNotificationService.post_message`.
The YouTube "New upload" post keeps its embed.
3. **Loop watchdog.** `EVENT_LOOP_STALL in progress` is now always
WARNING. The `ended` line keeps the existing ERROR rule. In the last two
weeks there were 24 "in progress" pattern alerts on top of 15 "ended"
ones. I checked `infrastructure/modules/monitoring.py`
`EVENT_LOOP_STALL_LOG_FILTER`: it keys on `"EVENT_LOOP_STALL ended"` +
`severity>=ERROR`, so this needs no Pulumi change.
4. **Frontend chunk-load errors** (`frontend_ingestion.py`,
`firestore_adapter.py`). Reports matching `isChunkLoadError` collapse
into one pattern per build with `alert_min_hourly_count=10`. The monitor
holds that pattern un-alerted until more than 10 arrive within an hour,
which means a broken deploy. Separately, Safari/Firefox stacks now get
the message prepended before hashing (a pattern was named
`@https://gen.nomadkaraoke.com`). Chrome hashes are unchanged. Existing
Safari/Firefox patterns will re-key once, and burst collapse groups
them.
5. **known_issues.** Added a service-scoped ignore for
`github-runner-manager` "no available instance" aborts during scale-up
(seen 09-20, 09-23, 09-27, 10-08). The same message from other services
still alerts.
6. **divebar-mirror schedulers.** Added `attempt_deadline="600s"` on
both `divebar-mirror-daily` and `divebar-mirror-refresh`; the function
timeout is 540s. With the 180s default, Cloud Scheduler retried syncs
that were still running, and the overlapping runs hit OOM (4x in 30
days).

## ⚠️ Pulumi
`infrastructure/modules/divebar_mirror.py` changed, so **this needs
`pulumi up` by Andrew before merge**. Auto-merge is NOT enabled.

## Testing
- New and updated unit tests in
`tests/unit/services/error_monitor/{test_monitor,test_discord,test_frontend_ingestion,test_firestore_adapter,test_known_issues}.py`,
`backend/tests/{test_loop_watchdog,test_discord_service,test_ops_alerts}.py`,
and `tests/unit/test_divebar_mirror_scheduler.py` (an AST guard:
deadline > function timeout).
- `pytest backend/tests --ignore=backend/tests/emulator`: 6421 passed,
36 skipped.
- `tests/unit/services/error_monitor`: all pass.
- Local CodeRabbit review (`--committed`) completed with 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 05:13:13 +00:00
..