mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415
## What changed
**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.
**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.
I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.
**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.
**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.
**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.
## Testing
- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.
## Not verified
- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).
## Known leftovers
- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.
## For Andrew (infra and cost, not changed here)
- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
137 lines
5.0 KiB
Python
137 lines
5.0 KiB
Python
"""POST /api/client-errors — ingest frontend crash reports.
|
|
|
|
Unauthenticated (logged-out users' crashes still matter). Deduped and alerted
|
|
via the existing error-monitor pipeline — see
|
|
``backend/services/error_monitor``.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import time
|
|
from typing import Optional
|
|
|
|
from fastapi import APIRouter, Request, HTTPException
|
|
from pydantic import BaseModel, Field, field_validator
|
|
|
|
from backend.services.error_monitor.firestore_adapter import ErrorPatternsAdapter
|
|
from backend.services.error_monitor.frontend_ingestion import (
|
|
FrontendErrorReport,
|
|
RateLimiter,
|
|
build_pattern_data,
|
|
is_bot_user_agent,
|
|
is_host_app_injection_error,
|
|
is_opaque_script_error,
|
|
sanitize_url,
|
|
)
|
|
|
|
logger = logging.getLogger(__name__)
|
|
router = APIRouter(prefix="/client-errors", tags=["client-errors"])
|
|
|
|
_limiter = RateLimiter(max_per_minute=60)
|
|
_adapter_singleton: ErrorPatternsAdapter | None = None
|
|
|
|
|
|
def _get_adapter() -> ErrorPatternsAdapter:
|
|
global _adapter_singleton
|
|
if _adapter_singleton is None:
|
|
_adapter_singleton = ErrorPatternsAdapter()
|
|
return _adapter_singleton
|
|
|
|
|
|
class ClientErrorPayload(BaseModel):
|
|
message: str = Field(..., min_length=1, max_length=4000)
|
|
stack: Optional[str] = Field(None, max_length=1_000_000)
|
|
url: str = Field("", max_length=2048)
|
|
user_agent: str = Field("", max_length=1024)
|
|
release: str = Field("", max_length=64)
|
|
user_email: Optional[str] = Field(None, max_length=320)
|
|
viewport: Optional[dict] = None
|
|
locale: str = Field("en", max_length=10)
|
|
source: str = Field("unknown", max_length=64)
|
|
# optional free-form extra for debugging. Capped server-side.
|
|
extra: Optional[dict] = None
|
|
|
|
@field_validator("message", "url", "user_agent", "locale", "source")
|
|
@classmethod
|
|
def strip(cls, v: str) -> str:
|
|
return (v or "").strip()
|
|
|
|
|
|
class ClientErrorResponse(BaseModel):
|
|
pattern_id: str
|
|
is_new: bool
|
|
|
|
|
|
@router.post("", response_model=ClientErrorResponse, status_code=202)
|
|
def report_client_error(payload: ClientErrorPayload, request: Request) -> ClientErrorResponse:
|
|
client_ip = request.client.host if request.client else "unknown"
|
|
if not _limiter.allow(client_ip, time.monotonic()):
|
|
raise HTTPException(status_code=429, detail="too many reports")
|
|
|
|
# Crawler/bot crashes are pure alert noise (bots run headless Chrome and trip
|
|
# DOM errors no real user hits). Accept the report so the client doesn't retry,
|
|
# but don't persist/alert on it.
|
|
if is_bot_user_agent(payload.user_agent):
|
|
# Log the bot UA (not sensitive) + sanitized URL (query/token stripped)
|
|
# for visibility into what's being dropped, without persisting/alerting.
|
|
logger.info(
|
|
"frontend_crash_ignored_bot ua=%r url=%s source=%s",
|
|
payload.user_agent[:120],
|
|
sanitize_url(payload.url),
|
|
payload.source,
|
|
)
|
|
return ClientErrorResponse(pattern_id="bot-ignored", is_new=False)
|
|
|
|
# Opaque cross-origin "Script error." — no file/line, not our code.
|
|
if is_opaque_script_error(payload.message, payload.source, payload.extra):
|
|
logger.info(
|
|
"frontend_crash_ignored_opaque_script_error url=%s source=%s",
|
|
sanitize_url(payload.url),
|
|
payload.source,
|
|
)
|
|
return ClientErrorResponse(pattern_id="opaque-script-error-ignored", is_new=False)
|
|
|
|
# A script the in-app browser's host app injected (Zalo), not our code.
|
|
if is_host_app_injection_error(payload.message, payload.user_agent):
|
|
logger.info(
|
|
"frontend_crash_ignored_host_app_injection url=%s source=%s",
|
|
sanitize_url(payload.url),
|
|
payload.source,
|
|
)
|
|
return ClientErrorResponse(pattern_id="host-app-injection-ignored", is_new=False)
|
|
|
|
report = FrontendErrorReport(
|
|
message=payload.message,
|
|
stack=payload.stack,
|
|
url=payload.url,
|
|
user_agent=payload.user_agent,
|
|
release=payload.release,
|
|
user_email=payload.user_email,
|
|
viewport=payload.viewport,
|
|
locale=payload.locale,
|
|
extra=payload.extra,
|
|
)
|
|
pattern_data = build_pattern_data(report)
|
|
|
|
try:
|
|
result = _get_adapter().upsert_pattern(pattern_data)
|
|
except Exception: # pragma: no cover - Firestore transient errors
|
|
logger.exception("Failed to upsert frontend error pattern")
|
|
raise HTTPException(status_code=503, detail="storage unavailable")
|
|
|
|
logger.info(
|
|
"frontend_crash_reported pattern_id=%s is_new=%s source=%s",
|
|
result.pattern_id,
|
|
result.is_new,
|
|
payload.source,
|
|
# Full client diagnostics + breadcrumbs, searchable in Cloud Logging
|
|
# (the alert sample only carries a trimmed summary).
|
|
extra={
|
|
"error_pattern_id": result.pattern_id,
|
|
"client_url": sanitize_url(payload.url),
|
|
"client_release": payload.release,
|
|
"client_extra": payload.extra,
|
|
},
|
|
)
|
|
return ClientErrorResponse(pattern_id=result.pattern_id, is_new=result.is_new)
|