Files
karaoke-gen/backend/services/error_monitor/config.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 a6d4163d2e feat(alerts): route non-signal Discord posts to #noise (v0.278.8) (#1186)
## Summary
Andrew asked for #alerts to carry only real problems. Routine and
informational posts now go to the new **#noise** channel. The GitHub org
PR webhook was already moved to #noise separately.

| Message | Before | After |
|---|---|---|
| Error monitor: New Error Pattern / Spike / Incident | #alerts |
#alerts |
| Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** |
| Error monitor: 📊 Daily Error Digest | #alerts | **#noise** |
| Nightly Backup Report (clean run) | #alerts | **#noise** |
| Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED")
| #alerts | #alerts |
| ✅ E2E Daily Test PASSED | #alerts | **#noise** |
| 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise**
|
| 🩹 self-heal PR needs CI approval click / could not open | #alerts |
#alerts |
| E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish
completeness SHADOW, Gemini unavailable, OAuth credential alerts,
quarterly DR drill reminder | #alerts | #alerts (unchanged) |

## How the noise webhook is resolved
- **Backend / error monitor and the backup function** read it at
runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager
secret `discord-noise-webhook` (the name can be overridden with
`DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed,
because both service accounts already have project-wide
`secretAccessor`.
- **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL ||
secrets.DISCORD_ALERT_WEBHOOK_URL`.
- **Nothing is silently dropped.** If the noise webhook can't be
resolved, or a post to it fails, the message goes to #alerts instead. In
the workflow this uses `curl -fsS` with a retry on the alert webhook.
- The error monitor now rate-limits each channel separately (10 per run
each), so noise can't use up the alert budget during an incident.

## Deploy notes
- **Backup Cloud Function:** CI doesn't deploy its source. Run
`infrastructure/functions/backup_to_aws/deploy.sh` after merge.
- **Error monitor job:** runs `karaoke-backend:latest`, so it picks up
the change once CI pushes the image.
- **No `pulumi up` needed.**

## Testing
- `tests/unit/services/error_monitor`: 376 passed. New tests cover noise
vs alert routing, fallback when the noise webhook is missing or fails,
separate rate limits, and secret resolution.
- `backend/tests` (excluding emulator): 6418 passed, 36 skipped.
- `infrastructure/functions/backup_to_aws`: 60 passed, including the new
`test_discord_alert.py` and routing tests in `test_main.py`. These were
run in a scratch venv because `functions_framework` isn't in the conda
env, which was already the case before this change.
- `actionlint` on `e2e-daily.yml`: no new findings. The only findings
are info-level shellcheck notes that were already there.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:46:50 +00:00

197 lines
7.5 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Configuration for the production error monitoring service.
All constants, service lists, and environment-driven configuration
functions used across the error monitor modules.
"""
import os
# ---------------------------------------------------------------------------
# GCP project settings
# ---------------------------------------------------------------------------
GCP_PROJECT: str = os.environ.get("GCP_PROJECT", "nomadkaraoke")
GCP_REGION: str = os.environ.get("GCP_REGION", "us-central1")
# ---------------------------------------------------------------------------
# Log querying settings
# ---------------------------------------------------------------------------
#: How far back to look for errors on each run (minutes). Must equal the
#: scheduler interval (ErrorMonitorConfig.MONITOR_SCHEDULE, hourly since
#: 2026-09-26) so consecutive windows tile without gaps or double counting.
LOOKBACK_MINUTES: int = 60
#: Window (minutes) assumed for ``rolling_counts`` entries written before
#: entries recorded their own ``window_minutes`` (i.e. the 15-min era).
LEGACY_ROLLING_WINDOW_MINUTES: int = 15
#: Maximum number of log entries to fetch per resource type per run (scaled
#: 4x with the 15 -> 60 min window to keep the same per-minute capacity).
MAX_LOG_ENTRIES: int = 2000
# ---------------------------------------------------------------------------
# Monitored services
# ---------------------------------------------------------------------------
MONITORED_CLOUD_RUN_SERVICES: list[str] = [
"karaoke-backend",
"karaoke-decide",
"audio-separator",
# Gen2 Cloud Functions appear as Cloud Run services in logs
"gdrive-validator",
"github-runner-manager",
"backup-to-aws",
"divebar-mirror",
"kn-data-sync",
"divebar-lookup",
"encoding-worker-idle-shutdown",
]
MONITORED_CLOUD_RUN_JOBS: list[str] = [
"video-encoding-job",
"lyrics-transcription-job",
"audio-separation-job",
"audio-download-job",
# karaoke-decide weekly MusicBrainz dump -> BigQuery refresh
"mb-refresh",
# karaoke-decide ListenBrainz statistics dump -> BigQuery refresh
"lb-refresh",
]
# Gen2 Cloud Functions log as cloud_run_revision, not cloud_function.
# Their names are in MONITORED_CLOUD_RUN_SERVICES above.
MONITORED_CLOUD_FUNCTIONS: list[str] = []
MONITORED_GCE_INSTANCES: list[str] = [
"encoding-worker-a",
"encoding-worker-b",
"flacfetch-vm",
"divebar-sync-vm",
]
# ---------------------------------------------------------------------------
# Service dependency map
# Human-readable description of which services depend on which workers.
# Used as LLM context when generating root-cause analysis.
# ---------------------------------------------------------------------------
SERVICE_DEPENDENCY_MAP: str = """\
karaoke-backend (main API)
→ video-encoding-job (Cloud Run Job, triggered per order)
→ lyrics-transcription-job (Cloud Run Job, triggered after audio-sep)
→ audio-separation-job (Cloud Run Job, runs on Cloud Run GPU)
→ audio-download-job (Cloud Run Job, uses flacfetch-vm)
→ gdrive-validator (Cloud Function, validates output files)
→ encoding-worker-a/b (GCE VMs, long-running encoding workers)
karaoke-decide (catalog / discovery API)
→ kn_data_sync (Cloud Function, syncs KaraokeNerds catalog)
→ divebar_mirror (Cloud Function, mirrors Divebar catalog)
→ divebar_lookup (Cloud Function, single-song lookup)
audio-separator (Cloud Run service, GPU stem separation)
→ audio-separation-job (Cloud Run Job, launched by audio-separator)
flacfetch-vm (GCE VM)
→ audio-download-job (initiated by karaoke-backend)
divebar-sync-vm (GCE VM)
→ divebar_mirror (Cloud Function, reads from this VM)
Shared infrastructure:
→ runner_manager (Cloud Function, manages encoding worker lifecycle)
→ encoding_worker_idle (Cloud Function, shuts down idle encoding workers)
→ backup_to_aws (Cloud Function, daily GCS → S3 backup)
"""
# ---------------------------------------------------------------------------
# LLM analysis settings
# ---------------------------------------------------------------------------
LLM_ANALYSIS_MODEL: str = os.environ.get("LLM_ANALYSIS_MODEL", "gemini-3.8-flash")
#: Minimum number of active error patterns required before triggering LLM analysis.
MIN_PATTERNS_FOR_ANALYSIS: int = 3
# ---------------------------------------------------------------------------
# Spike detection
# ---------------------------------------------------------------------------
#: An error count is a spike if it exceeds SPIKE_MULTIPLIER × baseline average.
SPIKE_MULTIPLIER: float = 5.0
#: Minimum absolute error count before spike detection fires (avoids noise).
SPIKE_MIN_COUNT: int = 5
# ---------------------------------------------------------------------------
# Auto-resolution settings
# ---------------------------------------------------------------------------
#: Multiply LOOKBACK_MINUTES by this to get auto-resolve window in minutes.
AUTO_RESOLVE_MULTIPLIER: int = 8
#: Minimum hours before a pattern can be auto-resolved.
AUTO_RESOLVE_MIN_HOURS: int = 6
#: Maximum hours before a pattern is force-resolved even if still active.
AUTO_RESOLVE_MAX_HOURS: int = 168 # 1 week
#: Fallback resolve window (hours) if baseline data is unavailable.
AUTO_RESOLVE_FALLBACK_HOURS: int = 48
# ---------------------------------------------------------------------------
# Discord settings
# ---------------------------------------------------------------------------
#: Maximum Discord messages to send per monitor run (rate-limit guard).
MAX_DISCORD_MESSAGES_PER_RUN: int = 10
#: Discord API hard limit on message length.
DISCORD_MAX_MESSAGE_LENGTH: int = 2000
# ---------------------------------------------------------------------------
# Pattern tracking settings
# ---------------------------------------------------------------------------
#: Maximum number of active error patterns to track in Firestore.
MAX_ACTIVE_PATTERNS: int = 500
#: Rolling window used when computing error baselines.
ROLLING_WINDOW_DAYS: int = 7
#: Maximum length of a normalized error message stored in Firestore.
MAX_NORMALIZED_MESSAGE_LENGTH: int = 200
# ---------------------------------------------------------------------------
# Helper functions
# ---------------------------------------------------------------------------
def get_llm_enabled() -> bool:
"""Return True if LLM analysis is enabled via the LLM_ANALYSIS_ENABLED env var."""
return os.environ.get("LLM_ANALYSIS_ENABLED", "false").strip().lower() == "true"
def get_discord_webhook_secret_name() -> str:
"""Return the Secret Manager secret name for the Discord webhook URL."""
return os.environ.get("DISCORD_WEBHOOK_SECRET_NAME", "error-monitor-discord-webhook")
def get_discord_noise_webhook_secret_name() -> str:
"""Return the Secret Manager secret name for the #noise Discord webhook.
#alerts is reserved for real problems (new patterns, spikes, incidents);
informational posts (auto-resolved summaries, the daily digest) go to #noise.
"""
return os.environ.get("DISCORD_NOISE_WEBHOOK_SECRET_NAME", "discord-noise-webhook")
def get_digest_mode() -> bool:
"""Return True if the monitor should run in digest mode (batch alerts instead of immediate).
In digest mode all alerts are bundled into a single Discord message sent at
the end of the run, rather than sending one message per error pattern.
"""
return os.environ.get("DIGEST_MODE", "false").strip().lower() == "true"