
Andrew Beveridge ☄️andClaude Opus 4.6
9a5c9d497a
feat: production error monitor — log collection, LLM analysis, Discord alerts (#713)
## Summary
- Add a production error monitoring service that runs as a Cloud Run Job
every 15 minutes
- Queries Cloud Logging across all NK services (Cloud Run, Jobs,
Functions, GCE VMs), normalizes/deduplicates errors, uses Gemini Flash
for incident grouping, sends Discord alerts, and auto-resolves silent
patterns
- Updates all `/prod-*` slash commands to use pre-computed Firestore
patterns as primary data source
## Architecture
Based on the proven Aquarius error bot system, adapted for Nomad
Karaoke's multi-service architecture (20+ resources across 4 resource
types).
**New backend modules** (`backend/services/error_monitor/`):
- `config.py` — constants, monitored services, env var overrides
- `normalizer.py` — regex normalization + SHA-256 pattern hashing
- `known_issues.py` — 8 ignore patterns for infrastructure noise
- `firestore_adapter.py` — CRUD for `error_patterns`, `error_incidents`,
`discord_alerts`
- `discord.py` — webhook client with 5 alert formatters (new pattern,
incident, spike, auto-resolved, daily digest)
- `llm_analysis.py` — Gemini Flash incident grouping + near-duplicate
detection
- `monitor.py` — orchestrator entry point (`python -m
backend.services.error_monitor.monitor`)
**New infrastructure** (`infrastructure/modules/error_monitor.py`):
- Cloud Run Job (`nomad-error-monitor`) sharing karaoke-backend image
- Cloud Scheduler: every 15 min monitor + 08:00 UTC daily digest
- Scheduler service account with `roles/run.invoker`
- Backend SA gets `roles/logging.viewer`
**New CLI scripts:**
- `scripts/query-error-patterns.py` — query patterns from Firestore
- `scripts/resolve-error-pattern.py` — mark patterns as fixed with PR
reference
## Testing
- 310 unit tests across 7 test modules (all passing)
- Covers: config, normalizer, known issues, Firestore adapter, Discord
formatting, LLM analysis, monitor orchestrator
- P0/P1 caller-callee contract tests for monitor integration points
- All tests use mocked external dependencies (no GCP calls)
## Review
- [x] CodeRabbit CLI review completed locally (1 cycle, 3 issues fixed)
- [x] Test review completed (P0/P1 gaps addressed)
- [x] Documentation updated (README, ARCHITECTURE)
- [x] Version bumped (0.168.2 → 0.169.0)
## Design Spec
`docs/archive/2026-04-12-error-monitor-design.md`
@coderabbitai ignore
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 01:02:08 -04:00
..
2025-12-29 23:22:05 -05:00
2025-12-31 14:15:51 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 04:06:05 -05:00
2025-12-28 12:05:27 -05:00
2025-12-29 15:25:50 -05:00
2025-12-29 13:32:00 -05:00
2025-12-29 23:31:27 -05:00
2025-12-30 00:36:25 -05:00
2025-12-30 01:39:30 -05:00
2025-12-30 15:00:42 -05:00
2025-12-30 20:05:28 -05:00
2025-12-30 22:50:16 -05:00
2025-12-30 17:24:52 -05:00
2025-12-31 16:13:46 -05:00
2025-12-31 00:06:57 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2026-01-19 16:53:34 +00:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2026-01-19 16:53:34 +00:00
2026-01-19 16:53:34 +00:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2026-01-19 16:53:34 +00:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2025-12-28 02:43:02 -05:00
2026-01-01 15:42:24 -05:00
2026-01-04 11:07:16 -05:00
2026-01-07 23:10:35 -05:00
2026-01-08 15:30:53 -05:00
2026-01-08 00:53:17 -05:00
2026-01-08 16:39:56 -05:00
2026-01-19 00:29:10 -08:00
2026-01-09 19:11:50 -05:00
2026-01-19 00:29:10 -08:00
2026-01-21 20:42:26 -08:00
2026-01-23 03:25:35 +00:00
2026-01-26 18:45:04 -08:00
2026-02-02 22:48:57 -08:00
2026-02-10 14:21:14 -05:00
2026-02-24 18:07:57 -05:00
2026-02-28 21:43:18 +00:00
2026-03-01 02:58:50 -05:00
2026-03-01 02:58:50 -05:00
2026-03-01 02:58:50 -05:00
2026-03-01 02:58:50 -05:00
2026-03-01 02:58:50 -05:00
2026-03-01 02:58:50 -05:00
2026-03-01 23:35:54 -05:00
2026-03-02 18:54:38 -05:00
2026-03-03 14:31:26 -05:00
2026-03-03 20:19:26 +00:00
2026-03-04 15:16:28 -05:00
2026-03-04 11:44:23 -05:00
2026-03-04 21:04:36 -05:00
2026-03-04 18:47:08 -05:00
2026-03-05 00:40:09 -05:00
2026-03-05 13:12:04 -05:00
2026-03-06 19:06:51 -05:00
2026-03-07 02:23:38 -05:00
2026-03-08 00:23:42 -05:00
2026-03-09 11:59:04 -04:00
2026-03-08 20:31:13 -04:00
2026-03-08 20:31:13 -04:00
2026-03-08 20:31:13 -04:00
2026-03-08 20:31:13 -04:00
2026-03-09 11:59:04 -04:00
2026-03-08 22:57:03 -04:00
2026-03-08 04:40:35 -04:00
2026-03-08 04:40:35 -04:00
2026-03-08 04:40:35 -04:00
2026-03-09 18:04:13 -04:00
2026-03-10 01:16:43 -04:00
2026-03-15 16:19:59 -04:00
2026-03-19 16:15:01 -04:00
2026-03-21 00:04:26 -04:00
2026-03-21 01:51:44 -04:00
2026-03-21 16:15:52 +00:00
2026-03-21 16:15:52 +00:00
2026-03-21 16:15:52 +00:00
2026-03-22 22:20:55 -04:00
2026-03-26 17:37:59 +00:00
2026-03-25 14:36:03 -04:00
2026-03-25 22:36:31 +00:00
2026-03-26 15:13:51 -04:00
2026-03-26 15:13:51 -04:00
2026-03-27 14:44:27 -04:00
2026-03-27 14:44:27 -04:00
2026-03-31 03:32:01 +00:00
2026-03-27 12:14:32 -04:00
2026-03-31 03:32:01 +00:00
2026-03-31 03:32:01 +00:00
2026-03-31 21:44:53 +00:00
2026-04-03 21:50:53 -04:00
2026-04-04 17:39:18 -04:00
2026-04-04 17:39:18 -04:00
2026-04-05 19:17:43 -04:00
2026-04-05 22:36:03 -04:00
2026-04-06 19:05:13 -04:00
2026-04-09 19:02:12 -04:00
2026-04-13 01:02:08 -04:00
2026-04-13 01:02:08 -04:00