2 Commits
Author SHA1 Message Date
Andrew Beveridge ☄️andClaude Opus 4.8 e806616408 fix: idempotent session revoke, error-pattern script crash, mobile header overlap (#798)
## Summary
Three independent fixes bundled together:

1. **`revoke_session` idempotency** — stops the recurring `Error
revoking session` prod error on `karaoke-backend`. Firestore
`document.update()` raises `NotFound` (404) when a session doc is
already gone (logged out / expired / cleaned up); logout against an
already-absent session now succeeds instead of erroring.
2. **`query-error-patterns.py` crash** — `last_seen`/`first_seen` are
stored as ISO-8601 strings in Firestore, but the sort key and `--hours`
filter did `float(value)`, raising `ValueError` and breaking the
`--json` path that `/prod-investigate` relies on. Added a `_to_epoch()`
helper handling datetime objects, ISO strings, and numeric epochs.
3. **Mobile header overlap** — `AppHeader` was `position: fixed` and
each consumer cleared it with a hardcoded top padding. On narrow
viewports the header wraps to 3 rows (149px on a 412px Pixel), exceeding
the 112px offset, hiding the "Create Karaoke Video" heading and the top
of the form behind it (worse across locales with longer button labels).
Switched `AppHeader` to `position: sticky` (matching the already-working
jobs-page header) so content always clears it at any height, and removed
the now-redundant magic-number offsets.

## Changes
- `backend/services/user_service.py` — catch
`google_exceptions.NotFound` in `revoke_session`, return `True`
- `backend/tests/test_revoke_session.py` — new unit tests (existing /
missing / unexpected-error paths)
- `scripts/query-error-patterns.py` — `_to_epoch()` helper for sort +
`--hours` filter
- `frontend/components/app-header.tsx` — `fixed` → `sticky`
- `frontend/app/[locale]/app/page.tsx`,
`frontend/app/[locale]/app/referrals/page.tsx` — drop hardcoded top
padding
- `pyproject.toml` — bump to 0.174.17

## Testing
- [x] Backend tests pass locally (47 passed incl. 3 new `revoke_session`
tests)
- [x] `query-error-patterns.py --json` and `--hours` verified against
live Firestore
- [x] Mobile header fix verified in Playwright at 412px (Pixel) and
1280px — overlap eliminated (title moves from 12px behind the header to
a clear gap below it)
- [x] CodeRabbit CLI review: 0 findings

## Review
- [x] Local CodeRabbit review completed
- [x] Review feedback addressed (none)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 23:24:21 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 9a5c9d497a feat: production error monitor — log collection, LLM analysis, Discord alerts (#713)
## Summary

- Add a production error monitoring service that runs as a Cloud Run Job
every 15 minutes
- Queries Cloud Logging across all NK services (Cloud Run, Jobs,
Functions, GCE VMs), normalizes/deduplicates errors, uses Gemini Flash
for incident grouping, sends Discord alerts, and auto-resolves silent
patterns
- Updates all `/prod-*` slash commands to use pre-computed Firestore
patterns as primary data source

## Architecture

Based on the proven Aquarius error bot system, adapted for Nomad
Karaoke's multi-service architecture (20+ resources across 4 resource
types).

**New backend modules** (`backend/services/error_monitor/`):
- `config.py` — constants, monitored services, env var overrides
- `normalizer.py` — regex normalization + SHA-256 pattern hashing
- `known_issues.py` — 8 ignore patterns for infrastructure noise
- `firestore_adapter.py` — CRUD for `error_patterns`, `error_incidents`,
`discord_alerts`
- `discord.py` — webhook client with 5 alert formatters (new pattern,
incident, spike, auto-resolved, daily digest)
- `llm_analysis.py` — Gemini Flash incident grouping + near-duplicate
detection
- `monitor.py` — orchestrator entry point (`python -m
backend.services.error_monitor.monitor`)

**New infrastructure** (`infrastructure/modules/error_monitor.py`):
- Cloud Run Job (`nomad-error-monitor`) sharing karaoke-backend image
- Cloud Scheduler: every 15 min monitor + 08:00 UTC daily digest
- Scheduler service account with `roles/run.invoker`
- Backend SA gets `roles/logging.viewer`

**New CLI scripts:**
- `scripts/query-error-patterns.py` — query patterns from Firestore
- `scripts/resolve-error-pattern.py` — mark patterns as fixed with PR
reference

## Testing

- 310 unit tests across 7 test modules (all passing)
- Covers: config, normalizer, known issues, Firestore adapter, Discord
formatting, LLM analysis, monitor orchestrator
- P0/P1 caller-callee contract tests for monitor integration points
- All tests use mocked external dependencies (no GCP calls)

## Review

- [x] CodeRabbit CLI review completed locally (1 cycle, 3 issues fixed)
- [x] Test review completed (P0/P1 gaps addressed)
- [x] Documentation updated (README, ARCHITECTURE)
- [x] Version bumped (0.168.2 → 0.169.0)

## Design Spec

`docs/archive/2026-04-12-error-monitor-design.md`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 01:02:08 -04:00