mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
Andrew's goal: "I'd like to make sure that our alerts channel only has alerts in it that are actionable and important, or are just signal and not noise." This follows #1186 (#noise routing). ## Changes 1. **Burst collapse** (`error_monitor/monitor.py`, `discord.py`, `config.py`). When a run has more than 3 new patterns for one service, including leftover un-alerted ones, the monitor sends ONE #alerts message. It has the count, total occurrences, the top 5 by count, first/last seen, and a `scripts/query-error-patterns.py --service X --status new` pointer. All of those patterns are marked alerted. The grouping is deterministic, since LLM grouping is off in prod. The 10-messages-per-run cap still applies, and unsent patterns are picked up again next run. Evidence: on 2026-10-07/08, one mb-refresh run produced 73 separate alerts. 2. **No link previews.** Every error-monitor post (`_post`) and every `ops_alerts` post now sets `flags: 4` (SUPPRESS_EMBEDS), through a new `suppress_embeds` kwarg on `DiscordNotificationService.post_message`. The YouTube "New upload" post keeps its embed. 3. **Loop watchdog.** `EVENT_LOOP_STALL in progress` is now always WARNING. The `ended` line keeps the existing ERROR rule. In the last two weeks there were 24 "in progress" pattern alerts on top of 15 "ended" ones. I checked `infrastructure/modules/monitoring.py` `EVENT_LOOP_STALL_LOG_FILTER`: it keys on `"EVENT_LOOP_STALL ended"` + `severity>=ERROR`, so this needs no Pulumi change. 4. **Frontend chunk-load errors** (`frontend_ingestion.py`, `firestore_adapter.py`). Reports matching `isChunkLoadError` collapse into one pattern per build with `alert_min_hourly_count=10`. The monitor holds that pattern un-alerted until more than 10 arrive within an hour, which means a broken deploy. Separately, Safari/Firefox stacks now get the message prepended before hashing (a pattern was named `@https://gen.nomadkaraoke.com`). Chrome hashes are unchanged. Existing Safari/Firefox patterns will re-key once, and burst collapse groups them. 5. **known_issues.** Added a service-scoped ignore for `github-runner-manager` "no available instance" aborts during scale-up (seen 09-20, 09-23, 09-27, 10-08). The same message from other services still alerts. 6. **divebar-mirror schedulers.** Added `attempt_deadline="600s"` on both `divebar-mirror-daily` and `divebar-mirror-refresh`; the function timeout is 540s. With the 180s default, Cloud Scheduler retried syncs that were still running, and the overlapping runs hit OOM (4x in 30 days). ## ⚠️ Pulumi `infrastructure/modules/divebar_mirror.py` changed, so **this needs `pulumi up` by Andrew before merge**. Auto-merge is NOT enabled. ## Testing - New and updated unit tests in `tests/unit/services/error_monitor/{test_monitor,test_discord,test_frontend_ingestion,test_firestore_adapter,test_known_issues}.py`, `backend/tests/{test_loop_watchdog,test_discord_service,test_ops_alerts}.py`, and `tests/unit/test_divebar_mirror_scheduler.py` (an AST guard: deadline > function timeout). - `pytest backend/tests --ignore=backend/tests/emulator`: 6421 passed, 36 skipped. - `tests/unit/services/error_monitor`: all pass. - Local CodeRabbit review (`--committed`) completed with 0 findings. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>