mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Andrew asked for #alerts to carry only real problems. Routine and informational posts now go to the new **#noise** channel. The GitHub org PR webhook was already moved to #noise separately. | Message | Before | After | |---|---|---| | Error monitor: New Error Pattern / Spike / Incident | #alerts | #alerts | | Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** | | Error monitor: 📊 Daily Error Digest | #alerts | **#noise** | | Nightly Backup Report (clean run) | #alerts | **#noise** | | Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED") | #alerts | #alerts | | ✅ E2E Daily Test PASSED | #alerts | **#noise** | | 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise** | | 🩹 self-heal PR needs CI approval click / could not open | #alerts | #alerts | | E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish completeness SHADOW, Gemini unavailable, OAuth credential alerts, quarterly DR drill reminder | #alerts | #alerts (unchanged) | ## How the noise webhook is resolved - **Backend / error monitor and the backup function** read it at runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager secret `discord-noise-webhook` (the name can be overridden with `DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed, because both service accounts already have project-wide `secretAccessor`. - **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL`. - **Nothing is silently dropped.** If the noise webhook can't be resolved, or a post to it fails, the message goes to #alerts instead. In the workflow this uses `curl -fsS` with a retry on the alert webhook. - The error monitor now rate-limits each channel separately (10 per run each), so noise can't use up the alert budget during an incident. ## Deploy notes - **Backup Cloud Function:** CI doesn't deploy its source. Run `infrastructure/functions/backup_to_aws/deploy.sh` after merge. - **Error monitor job:** runs `karaoke-backend:latest`, so it picks up the change once CI pushes the image. - **No `pulumi up` needed.** ## Testing - `tests/unit/services/error_monitor`: 376 passed. New tests cover noise vs alert routing, fallback when the noise webhook is missing or fails, separate rate limits, and secret resolution. - `backend/tests` (excluding emulator): 6418 passed, 36 skipped. - `infrastructure/functions/backup_to_aws`: 60 passed, including the new `test_discord_alert.py` and routing tests in `test_main.py`. These were run in a scratch venv because `functions_framework` isn't in the conda env, which was already the case before this change. - `actionlint` on `e2e-daily.yml`: no new findings. The only findings are info-level shellcheck notes that were already there. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
197 lines
7.5 KiB
Python
197 lines
7.5 KiB
Python
"""Configuration for the production error monitoring service.
|
||
|
||
All constants, service lists, and environment-driven configuration
|
||
functions used across the error monitor modules.
|
||
"""
|
||
|
||
import os
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# GCP project settings
|
||
# ---------------------------------------------------------------------------
|
||
|
||
GCP_PROJECT: str = os.environ.get("GCP_PROJECT", "nomadkaraoke")
|
||
GCP_REGION: str = os.environ.get("GCP_REGION", "us-central1")
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Log querying settings
|
||
# ---------------------------------------------------------------------------
|
||
|
||
#: How far back to look for errors on each run (minutes). Must equal the
|
||
#: scheduler interval (ErrorMonitorConfig.MONITOR_SCHEDULE, hourly since
|
||
#: 2026-09-26) so consecutive windows tile without gaps or double counting.
|
||
LOOKBACK_MINUTES: int = 60
|
||
|
||
#: Window (minutes) assumed for ``rolling_counts`` entries written before
|
||
#: entries recorded their own ``window_minutes`` (i.e. the 15-min era).
|
||
LEGACY_ROLLING_WINDOW_MINUTES: int = 15
|
||
|
||
#: Maximum number of log entries to fetch per resource type per run (scaled
|
||
#: 4x with the 15 -> 60 min window to keep the same per-minute capacity).
|
||
MAX_LOG_ENTRIES: int = 2000
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Monitored services
|
||
# ---------------------------------------------------------------------------
|
||
|
||
MONITORED_CLOUD_RUN_SERVICES: list[str] = [
|
||
"karaoke-backend",
|
||
"karaoke-decide",
|
||
"audio-separator",
|
||
# Gen2 Cloud Functions appear as Cloud Run services in logs
|
||
"gdrive-validator",
|
||
"github-runner-manager",
|
||
"backup-to-aws",
|
||
"divebar-mirror",
|
||
"kn-data-sync",
|
||
"divebar-lookup",
|
||
"encoding-worker-idle-shutdown",
|
||
]
|
||
|
||
MONITORED_CLOUD_RUN_JOBS: list[str] = [
|
||
"video-encoding-job",
|
||
"lyrics-transcription-job",
|
||
"audio-separation-job",
|
||
"audio-download-job",
|
||
# karaoke-decide weekly MusicBrainz dump -> BigQuery refresh
|
||
"mb-refresh",
|
||
# karaoke-decide ListenBrainz statistics dump -> BigQuery refresh
|
||
"lb-refresh",
|
||
]
|
||
|
||
# Gen2 Cloud Functions log as cloud_run_revision, not cloud_function.
|
||
# Their names are in MONITORED_CLOUD_RUN_SERVICES above.
|
||
MONITORED_CLOUD_FUNCTIONS: list[str] = []
|
||
|
||
MONITORED_GCE_INSTANCES: list[str] = [
|
||
"encoding-worker-a",
|
||
"encoding-worker-b",
|
||
"flacfetch-vm",
|
||
"divebar-sync-vm",
|
||
]
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Service dependency map
|
||
# Human-readable description of which services depend on which workers.
|
||
# Used as LLM context when generating root-cause analysis.
|
||
# ---------------------------------------------------------------------------
|
||
|
||
SERVICE_DEPENDENCY_MAP: str = """\
|
||
karaoke-backend (main API)
|
||
→ video-encoding-job (Cloud Run Job, triggered per order)
|
||
→ lyrics-transcription-job (Cloud Run Job, triggered after audio-sep)
|
||
→ audio-separation-job (Cloud Run Job, runs on Cloud Run GPU)
|
||
→ audio-download-job (Cloud Run Job, uses flacfetch-vm)
|
||
→ gdrive-validator (Cloud Function, validates output files)
|
||
→ encoding-worker-a/b (GCE VMs, long-running encoding workers)
|
||
|
||
karaoke-decide (catalog / discovery API)
|
||
→ kn_data_sync (Cloud Function, syncs KaraokeNerds catalog)
|
||
→ divebar_mirror (Cloud Function, mirrors Divebar catalog)
|
||
→ divebar_lookup (Cloud Function, single-song lookup)
|
||
|
||
audio-separator (Cloud Run service, GPU stem separation)
|
||
→ audio-separation-job (Cloud Run Job, launched by audio-separator)
|
||
|
||
flacfetch-vm (GCE VM)
|
||
→ audio-download-job (initiated by karaoke-backend)
|
||
|
||
divebar-sync-vm (GCE VM)
|
||
→ divebar_mirror (Cloud Function, reads from this VM)
|
||
|
||
Shared infrastructure:
|
||
→ runner_manager (Cloud Function, manages encoding worker lifecycle)
|
||
→ encoding_worker_idle (Cloud Function, shuts down idle encoding workers)
|
||
→ backup_to_aws (Cloud Function, daily GCS → S3 backup)
|
||
"""
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# LLM analysis settings
|
||
# ---------------------------------------------------------------------------
|
||
|
||
LLM_ANALYSIS_MODEL: str = os.environ.get("LLM_ANALYSIS_MODEL", "gemini-3.8-flash")
|
||
|
||
#: Minimum number of active error patterns required before triggering LLM analysis.
|
||
MIN_PATTERNS_FOR_ANALYSIS: int = 3
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Spike detection
|
||
# ---------------------------------------------------------------------------
|
||
|
||
#: An error count is a spike if it exceeds SPIKE_MULTIPLIER × baseline average.
|
||
SPIKE_MULTIPLIER: float = 5.0
|
||
|
||
#: Minimum absolute error count before spike detection fires (avoids noise).
|
||
SPIKE_MIN_COUNT: int = 5
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Auto-resolution settings
|
||
# ---------------------------------------------------------------------------
|
||
|
||
#: Multiply LOOKBACK_MINUTES by this to get auto-resolve window in minutes.
|
||
AUTO_RESOLVE_MULTIPLIER: int = 8
|
||
|
||
#: Minimum hours before a pattern can be auto-resolved.
|
||
AUTO_RESOLVE_MIN_HOURS: int = 6
|
||
|
||
#: Maximum hours before a pattern is force-resolved even if still active.
|
||
AUTO_RESOLVE_MAX_HOURS: int = 168 # 1 week
|
||
|
||
#: Fallback resolve window (hours) if baseline data is unavailable.
|
||
AUTO_RESOLVE_FALLBACK_HOURS: int = 48
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Discord settings
|
||
# ---------------------------------------------------------------------------
|
||
|
||
#: Maximum Discord messages to send per monitor run (rate-limit guard).
|
||
MAX_DISCORD_MESSAGES_PER_RUN: int = 10
|
||
|
||
#: Discord API hard limit on message length.
|
||
DISCORD_MAX_MESSAGE_LENGTH: int = 2000
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Pattern tracking settings
|
||
# ---------------------------------------------------------------------------
|
||
|
||
#: Maximum number of active error patterns to track in Firestore.
|
||
MAX_ACTIVE_PATTERNS: int = 500
|
||
|
||
#: Rolling window used when computing error baselines.
|
||
ROLLING_WINDOW_DAYS: int = 7
|
||
|
||
#: Maximum length of a normalized error message stored in Firestore.
|
||
MAX_NORMALIZED_MESSAGE_LENGTH: int = 200
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Helper functions
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def get_llm_enabled() -> bool:
|
||
"""Return True if LLM analysis is enabled via the LLM_ANALYSIS_ENABLED env var."""
|
||
return os.environ.get("LLM_ANALYSIS_ENABLED", "false").strip().lower() == "true"
|
||
|
||
|
||
def get_discord_webhook_secret_name() -> str:
|
||
"""Return the Secret Manager secret name for the Discord webhook URL."""
|
||
return os.environ.get("DISCORD_WEBHOOK_SECRET_NAME", "error-monitor-discord-webhook")
|
||
|
||
|
||
def get_discord_noise_webhook_secret_name() -> str:
|
||
"""Return the Secret Manager secret name for the #noise Discord webhook.
|
||
|
||
#alerts is reserved for real problems (new patterns, spikes, incidents);
|
||
informational posts (auto-resolved summaries, the daily digest) go to #noise.
|
||
"""
|
||
return os.environ.get("DISCORD_NOISE_WEBHOOK_SECRET_NAME", "discord-noise-webhook")
|
||
|
||
|
||
def get_digest_mode() -> bool:
|
||
"""Return True if the monitor should run in digest mode (batch alerts instead of immediate).
|
||
|
||
In digest mode all alerts are bundled into a single Discord message sent at
|
||
the end of the run, rather than sending one message per error pattern.
|
||
"""
|
||
return os.environ.get("DIGEST_MODE", "false").strip().lower() == "true"
|