Files
karaoke-gen/backend/services/error_monitor/normalizer.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 1344a086da fix: worker Cloud Run Jobs never dispatch video/render into the API process (v0.278.12) (#1192)
## Problem: a regression from #1191, caught by #alerts triage
At 07:03 UTC on 2026-10-19, two new freeze alerts fired: **52s with 29
requests affected** (culprit
`firestore_service.append_log_to_subcollection`, during CDG packaging)
and **81s with 33 requests** (culprit `dropbox_service.upload_file`).
The stacks show `video_worker.generate_video` running **inside
karaoke-backend**.

Cause: in today's daily E2E run (job 5a5e5181), #1191's park/resume
worked as designed (parked 06:55:04, resumed 06:57:50). But the resume
runs in `audio-separation-job`, which doesn't set
`USE_CLOUD_RUN_JOBS_FOR_VIDEO`. So `trigger_video_worker` fell back to
Cloud Tasks, which runs it as a `BackgroundTask` on the API, putting the
whole video worker on the API's event loop. The same latent bug already
applied to the audio worker's auto-approval → render trigger.

## Fix
- `backend/config.py`: inside a Cloud Run Job (`CLOUD_RUN_JOB` is set by
the platform), `use_cloud_run_jobs_for_video` and
`use_cloud_run_jobs_for_render` default to **true**. An explicit env
value still wins. This covers every worker job (audio, lyrics, download,
...) with no infra change, through the normal image deploy.
- Error monitor normalizer: `requests_affected=N` →
`requests_affected=<N>`, so a stall culprit is one alert pattern instead
of one per request count.

## Testing
- New `backend/tests/test_config_cloud_run_job_dispatch.py`: inside a
job → true; outside → unchanged; explicit env wins.
- New normalizer test.
- `backend/tests` + `tests/unit/services/error_monitor`: 6871 passed.
- Local CodeRabbit review: 0 findings.

## Verify after deploy
The next parked-then-resumed job logs `Started Cloud Run Job
video-encoding-job` from audio-separation-job, and there are no
`EVENT_LOOP_STALL` lines on karaoke-backend with `video_worker` in the
stack.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 13:17:49 -04:00

153 lines
5.6 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Message normalizer for the production error monitoring service.
Replaces variable parts of error messages (IDs, timestamps, paths, etc.)
with stable placeholders so the same logical error always produces the same
pattern hash, regardless of which specific resource or timestamp was involved.
Normalizer rules are applied in order — more specific patterns first.
"""
import hashlib
import re
from backend.services.error_monitor.config import MAX_NORMALIZED_MESSAGE_LENGTH
# ---------------------------------------------------------------------------
# Compiled regex patterns (applied in order — more specific first)
# ---------------------------------------------------------------------------
_NORMALIZERS: list[tuple[re.Pattern, str]] = [
# 1. GCS paths: gs://bucket/path/to/object → gs://<BUCKET>/<PATH>
# Must come before Firestore and URL normalization.
(
re.compile(r"gs://([^/\s]+)/([^\s]+)"),
r"gs://<BUCKET>/<PATH>",
),
# 1b. GCS bucket root (no object path): gs://bucket/ or gs://bucket
# Uses a lookahead so the slash/end-of-string is not consumed, and uses
# a character class that excludes '<' to avoid re-matching placeholders.
(
re.compile(r"gs://([^/<\s]+)(?=[/\s]|$)"),
r"gs://<BUCKET>",
),
# 2. Firestore document paths (collection/doc/…/doc — at least 2 segments)
# Negative lookbehind for '.' prevents matching inside domain names (URLs).
# Negative lookbehind for '/' prevents matching inside already-replaced paths.
(
re.compile(
r"(?<![./])\b([a-zA-Z_][a-zA-Z0-9_-]*/[a-zA-Z0-9_-]+(?:/[a-zA-Z0-9_-]+){1,})\b"
),
r"<DOC_PATH>",
),
# 3. UUIDs (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
(
re.compile(
r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b"
),
"<ID>",
),
# 4. ISO 8601 timestamps (2024-01-15T10:30:45Z / …+05:30 / ….123Z)
(
re.compile(
r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})"
),
"<TS>",
),
# 5. Epoch timestamps: 10+ digit number optionally followed by fractional seconds
(
re.compile(r"\b\d{10,}(?:\.\d+)?\b"),
"<EPOCH>",
),
# 6. Email addresses
(
re.compile(r"\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b"),
"<EMAIL>",
),
# 7. IP addresses (IPv4) — must come before large-number replacement
(
re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b"),
"<IP>",
),
# 8. URLs — strip to domain only (https://host/path → https://host)
# Handles http and https; preserves the scheme+domain for context.
(
re.compile(r"https?://([A-Za-z0-9.\-]+(:\d+)?)/[^\s]*"),
r"https://\1",
),
# 9. Job IDs in paths (/jobs/<alphanumeric>)
(
re.compile(r"/jobs/[A-Za-z0-9]+"),
"/jobs/<ID>",
),
# 9b. Preview render job IDs: preview_<jobid>_<renderid> (underscore-joined
# hex runs). The generic hex rule (11) is \b-anchored and '_' is a word
# char, so it can't reach these — leaving every preview render of the
# same job as a distinct pattern (N alerts for one incident). Collapse
# the whole token so they map to a single pattern.
(
re.compile(r"\bpreview_[0-9a-fA-F]{6,}(?:_[0-9a-fA-F]{6,})+\b"),
"preview_<ID>",
),
# 10. Firebase UIDs (20–28 mixed-case alphanumeric characters)
# Firebase Auth UIDs are typically 28 chars; allow 20-28 for flexibility.
# Must contain both letters and digits (to avoid replacing plain words).
(
re.compile(r"\b(?=[A-Za-z0-9]{20,28}\b)(?=[A-Za-z0-9]*[A-Z])(?=[A-Za-z0-9]*[a-z])(?=[A-Za-z0-9]*\d)[A-Za-z0-9]{20,28}\b"),
"<UID>",
),
# 11. Hex/alphanumeric IDs: 8+ chars containing both letters and digits
# (e.g. git SHAs, Cloud Run revision IDs, job IDs)
# Uses a lookahead to require at least one digit AND one letter.
(
re.compile(r"\b(?=[a-fA-F0-9]*[a-fA-F][a-fA-F0-9]*\d|[a-fA-F0-9]*\d[a-fA-F0-9]*[a-fA-F])[a-fA-F0-9]{8,}\b"),
"<ID>",
),
# 12. Request counts in loop-stall lines (EVENT_LOOP_STALL ... requests_affected=29):
# small numbers that would otherwise make every stall a "new" pattern.
(
re.compile(r"\brequests_affected=\d+"),
"requests_affected=<N>",
),
# 13. Large numbers (4+ digits) — must come after epoch/IP/UUID replacements
(
re.compile(r"\b\d{4,}\b"),
"<NUM>",
),
]
def normalize_message(message: str) -> str:
"""Normalize a raw log message by replacing variable parts with placeholders.
Applies each normalizer rule in order (more specific patterns first) and
truncates the result to MAX_NORMALIZED_MESSAGE_LENGTH characters.
Args:
message: Raw log message string.
Returns:
Normalized message string, at most MAX_NORMALIZED_MESSAGE_LENGTH chars.
"""
if not message:
return ""
result = message
for pattern, replacement in _NORMALIZERS:
result = pattern.sub(replacement, result)
return result[:MAX_NORMALIZED_MESSAGE_LENGTH]
def compute_pattern_hash(service: str, normalized_message: str) -> str:
"""Compute a stable SHA-256 fingerprint for an error pattern.
Args:
service: The originating service name (e.g. "karaoke-backend").
normalized_message: The output of normalize_message().
Returns:
64-character lowercase hex SHA-256 digest.
"""
payload = f"{service}:{normalized_message}"
return hashlib.sha256(payload.encode("utf-8")).hexdigest()