Files
Andrew Beveridge ☄️andClaude Opus 5.5 70bdd0e693 fix(alerts): start-review 409, per-cause crash patterns, one alert per job failure (v0.290.1) (#1243)
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

## What changed

**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.

**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.

I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.

**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.

**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.

**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.

## Testing

- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.

## Not verified

- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).

## Known leftovers

- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.

## For Andrew (infra and cost, not changed here)

- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:04:04 +00:00

174 lines
6.6 KiB
Python

"""API tests for POST /api/client-errors."""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from backend.api.routes.client_errors import router as client_errors_router
@pytest.fixture
def client(monkeypatch):
# reset the module-level limiter between tests
import backend.api.routes.client_errors as mod
mod._limiter = mod.RateLimiter(max_per_minute=60)
fake_adapter = MagicMock()
fake_adapter.upsert_pattern.return_value = MagicMock(
pattern_id="deadbeef", is_new=True, previous_status=None
)
monkeypatch.setattr(mod, "_get_adapter", lambda: fake_adapter)
app = FastAPI()
app.include_router(client_errors_router, prefix="/api")
client = TestClient(app)
client.fake_adapter = fake_adapter # type: ignore[attr-defined]
return client
def _payload(**overrides):
base = {
"message": "TypeError: Cannot read properties of null",
"stack": "TypeError: Cannot read properties of null\n at x (app.js:1:2)",
"url": "https://gen.nomadkaraoke.com/en/app/jobs/#/abc/review",
"user_agent": "Mozilla/5.0 (Android 14; Mobile; rv:138.0) Gecko/138.0 Firefox/138.0",
"release": "abc1234",
"user_email": None,
"viewport": {"w": 412, "h": 915},
"locale": "en",
"source": "window.onerror",
}
base.update(overrides)
return base
def test_accepts_valid_report_and_upserts(client):
resp = client.post("/api/client-errors", json=_payload())
assert resp.status_code == 202
assert resp.json()["pattern_id"] == "deadbeef"
client.fake_adapter.upsert_pattern.assert_called_once()
(pd,), _ = client.fake_adapter.upsert_pattern.call_args
assert pd.service == "frontend"
assert pd.resource_type == "browser"
def test_bot_report_is_accepted_but_not_persisted(client):
bot_ua = (
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; "
"bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/136.0.0.0 Safari/537.36"
)
resp = client.post("/api/client-errors", json=_payload(user_agent=bot_ua))
# Accepted (so the bot won't retry) but never upserted / alerted.
assert resp.status_code == 202
assert resp.json()["is_new"] is False
client.fake_adapter.upsert_pattern.assert_not_called()
@pytest.mark.parametrize("message", ["Script error.", "Error: Script error.", "script error"])
def test_opaque_script_error_is_accepted_but_not_persisted(client, message):
# Opaque cross-origin error (older cached bundles still report it) — the
# "stack" is only our own handler's frame, so it's pure alert noise.
resp = client.post(
"/api/client-errors",
json=_payload(
message=message,
stack=f"Error: {message}\n@https://gen.nomadkaraoke.com/_next/static/chunks/x.js:6:874",
source="window.onerror",
extra={"filename": "", "lineno": 0, "colno": 0},
),
)
assert resp.status_code == 202
assert resp.json()["is_new"] is False
client.fake_adapter.upsert_pattern.assert_not_called()
def test_located_script_error_message_is_persisted(client):
# A genuine `throw new Error("Script error.")` from our own code has a real
# file/line on the ErrorEvent — actionable, so it must still be recorded.
resp = client.post(
"/api/client-errors",
json=_payload(
message="Error: Script error.",
stack="Error: Script error.\n at foo (https://gen.nomadkaraoke.com/_next/static/chunks/y.js:3:120)",
source="window.onerror",
extra={"filename": "https://gen.nomadkaraoke.com/_next/static/chunks/y.js", "lineno": 3, "colno": 120},
),
)
assert resp.status_code == 202
client.fake_adapter.upsert_pattern.assert_called_once()
def test_message_merely_mentioning_script_error_is_persisted(client):
resp = client.post("/api/client-errors", json=_payload(message="Script error in chunk loader"))
assert resp.status_code == 202
client.fake_adapter.upsert_pattern.assert_called_once()
def test_rejects_missing_message(client):
resp = client.post("/api/client-errors", json=_payload(message=""))
assert resp.status_code == 422
def test_caps_oversized_payload(client):
huge = "x" * 200_000
resp = client.post("/api/client-errors", json=_payload(stack=huge))
# server accepts but truncates — sample_message must be <= MAX
assert resp.status_code == 202
(pd,), _ = client.fake_adapter.upsert_pattern.call_args
from backend.services.error_monitor.frontend_ingestion import (
MAX_SAMPLE_MESSAGE_CHARS,
)
assert len(pd.sample_message) <= MAX_SAMPLE_MESSAGE_CHARS
def test_rate_limit_returns_429(client, monkeypatch):
import backend.api.routes.client_errors as mod
mod._limiter = mod.RateLimiter(max_per_minute=2)
assert client.post("/api/client-errors", json=_payload()).status_code == 202
assert client.post("/api/client-errors", json=_payload()).status_code == 202
r3 = client.post("/api/client-errors", json=_payload())
assert r3.status_code == 429
def test_strips_sensitive_query_string(client):
resp = client.post(
"/api/client-errors",
json=_payload(url="https://gen.nomadkaraoke.com/en/app/jobs/?token=SECRET#/abc/review"),
)
assert resp.status_code == 202
(pd,), _ = client.fake_adapter.upsert_pattern.call_args
assert "SECRET" not in pd.sample_message
assert "token=" not in pd.sample_message
ZALO_UA = (
"Mozilla/5.0 (Linux; Android 16; X Build/Y;) AppleWebKit/537.36 (KHTML, like Gecko) "
"Version/4.0 Chrome/156.0.8078.47 Mobile Safari/537.36 Zalo android/261002903 ZaloTheme/light"
)
ZALO_MESSAGE = "ReferenceError: zaloJSV2 is not defined"
def test_zalo_host_app_injection_is_accepted_but_not_persisted(client):
resp = client.post(
"/api/client-errors",
json=_payload(message=ZALO_MESSAGE, stack=f"{ZALO_MESSAGE}\n at <anonymous>:1:2", user_agent=ZALO_UA),
)
assert resp.status_code == 202
assert resp.json()["is_new"] is False
client.fake_adapter.upsert_pattern.assert_not_called()
def test_real_errors_inside_zalo_are_persisted(client):
resp = client.post("/api/client-errors", json=_payload(user_agent=ZALO_UA))
assert resp.status_code == 202
client.fake_adapter.upsert_pattern.assert_called_once()
def test_zalo_message_from_another_browser_is_persisted(client):
resp = client.post("/api/client-errors", json=_payload(message=ZALO_MESSAGE))
assert resp.status_code == 202
client.fake_adapter.upsert_pattern.assert_called_once()