fix: alert noise + loop stalls + kn-data-sync OOM from #alerts triage (v0.282.1) (#1210)

## Summary
Fixes for the open and noise groups from the 2026-10-22 #alerts triage.

**Noise that should not page:**
- `stripe_service.py`: "Invalid webhook signature" now logs WARNING. All
42 hits on 10-21 came from `facebookexternalhit` via Cloudflare with a
malformed header. Every real Stripe delivery since 10-14 got a 200.
(#alerts 1562375740685815953)
- `error_monitor/known_issues.py`: `startup_probe` also matches Cloud
Run's "Default STARTUP **TCP** probe failed" wording.
(1562224513276706933)
- `frontend/lib/crash-reporter.ts`: a `removeChild`/`insertBefore`
NotFoundError is benign **only while the page is machine-translated**,
i.e. `<html>` has `translated-ltr`/`translated-rtl`. In the 10-21 crash,
Chrome had translated `/en/app` into Spanish ("Crea un video karaoke" is
not our string), and the customer's job completed and was downloaded.
(1562541569238638725)

**Event-loop stalls** (continues the WIP freeze item in BACKLOG):
- `review.py` audio-editor routes: `apply_edit`, `get_metadata_from_gcs`
and the GCS `download_file`/`upload_file` calls now go through
`asyncio.to_thread`. They were stalling the loop about 10s per edit (job
330d796c, 10-19). (1561861805695967363)
- `internal.py`: `process_conversion_emails` is imported at module top.
The lazy import inside the hourly cron's `async def` wrote a `.pyc` on
the loop, stalling it up to 15.3s, 4× on 10-20/21. (1562224514933334047)
- `backend/Dockerfile`: `python -m compileall` over `/app/backend` and
`/app/karaoke_gen`, so no lazy import compiles at runtime.
- `test_no_blocking_async_routes.py`: `BLOCKING_CALLS` now includes
`apply_edit`, `get_metadata_from_gcs`, `upload_file` and
`download_file`. Two more routes in `review.py` were caught and wrapped:
the preview ASS upload and the join-audio upload.

**kn-data-sync** (`infrastructure/modules/kn_data_sync.py`):
- Memory goes from 512M to 1Gi. The 10-22 full sync OOMed at 489 MiB on
a retry. 1Gi also gives gen2 more CPU, and the cost is about $0 within
the free tier.
- `attempt_deadline="320s"` on both schedulers. The default 180s gave up
on a run that took 184.8s and succeeded, then retried it, which meant 3
full KN pulls that day. (1562752965629190217, 1562752969395535933)

## Testing
- `make test`: 6603 passed. 8 failed in `test_emulator_integration.py`,
the known local-only failure (another session's stack holds :8080); they
fail the same way on unmodified main, and CI's emulator job is
authoritative.
- Targeted runs: error monitor, audio-edit routes, Stripe/webhook,
conversion emails and the blocking guard all pass (710 tests). Frontend
crash-reporter jest: 17 passed, with new translated / not-translated
cases.
- The local CodeRabbit CLI failed with "Invalid token", so the bot
review is left on for this PR.

## Deploy note
The `infrastructure/` change (kn-data-sync) needs `pulumi up` before
merge, per the repo rule. That needs Andrew's `/gcp-admin` grant.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_015AvGQEptp3Pg9RHhKGZ5KB

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved responsiveness during audio editing and review submissions,
reducing the chance that file and audio processing delays other
requests.
* Reduced false crash reports on pages affected by machine translation.
  * Improved recognition of startup probe failures in error monitoring.
  * Clarified logging for malformed payment webhook signatures.
* **Reliability**
* Increased resources and allowed more time for scheduled data sync
jobs.
  * Reduced delays to API requests during background processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Alerts
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562375740685815953
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224513276706933
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562541569238638725
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1561861805695967363
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224514933334047
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752965629190217
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752969395535933

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-10-22 19:26:43 +00:00
1 parent e4dad6e3c5
commit f6d846234d
13 files changed
+146 -13

No files matched your search

+4
View File
@@ -36,6 +36,10 @@ RUN if [ -n "$MODEL_DIR" ]; then \
pip install --no-cache-dir -e .; \
fi
# Precompile bytecode so lazy imports never write .pyc at runtime: a cold
# import that compiles + writes on the event loop has stalled requests 10-15s.
RUN python -m compileall -q /app/backend /app/karaoke_gen
# Set environment variables
ENV PYTHONUNBUFFERED=1
ENV PORT=8080
+1 -1
View File
@@ -27,6 +27,7 @@ from backend.workers.video_worker import generate_video
from backend.workers.render_video_worker import process_render_video
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult, UserType
from backend.services.conversion_emails import process_conversion_emails
from backend.services.job_manager import JobManager, PREP_PHASE_STATUSES
from backend.utils.loop_bridge import run_in_body_thread, run_on_loop
from backend.services.tracing import (
@@ -668,7 +669,6 @@ def process_stale_reviews_endpoint(
# Free-to-paid conversion emails ride on this hourly cron (no-op unless
# CONVERSION_EMAILS_ENABLED); see backend/services/conversion_emails.py
try:
from backend.services.conversion_emails import process_conversion_emails
conversion_result = await asyncio.to_thread(process_conversion_emails)
logger.info(f"CONVERSION_EMAILS complete: {conversion_result}")
except Exception as e:
+8 -7
View File
@@ -2053,7 +2053,7 @@ async def generate_preview_video(
# Upload ASS to GCS
with create_span("upload-ass-to-gcs") as upload_ass_span:
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
storage.upload_file(ass_path, ass_gcs_path)
await asyncio.to_thread(storage.upload_file, ass_path, ass_gcs_path)
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
# A stale error marker from an earlier failed attempt at
@@ -3074,13 +3074,14 @@ async def apply_audio_edit(
if existing_stack:
duration_before = existing_stack[-1].get('duration_seconds', 0)
else:
before_metadata = edit_service.get_metadata_from_gcs(current_gcs_path)
before_metadata = await asyncio.to_thread(edit_service.get_metadata_from_gcs, current_gcs_path)
duration_before = before_metadata.duration_seconds
output_gcs_path = f"jobs/{job_id}/audio_edit/edit_{edit_id}.flac"
# Apply the edit
metadata, result_path = edit_service.apply_edit(
# Apply the edit (ffmpeg + GCS download/upload: off the event loop)
metadata, result_path = await asyncio.to_thread(
edit_service.apply_edit,
input_gcs_path=current_gcs_path,
operation=operation,
params=params,
@@ -3285,7 +3286,7 @@ async def upload_audio_for_join(
tmp_path = tmp.name
try:
storage.upload_file(tmp_path, gcs_path)
await asyncio.to_thread(storage.upload_file, tmp_path, gcs_path)
# Get metadata
metadata = edit_service.get_metadata(tmp_path)
@@ -3394,8 +3395,8 @@ async def submit_audio_edit(
# Download and re-upload to canonical path
with tempfile.TemporaryDirectory() as temp_dir:
local_path = os.path.join(temp_dir, "edited.flac")
storage.download_file(current_edited_path, local_path)
storage.upload_file(local_path, edited_input_path)
await asyncio.to_thread(storage.download_file, current_edited_path, local_path)
await asyncio.to_thread(storage.upload_file, local_path, edited_input_path)
# Update job to use edited audio as input
job_manager.update_job(job_id, {
@@ -38,7 +38,7 @@ class IgnoreReason:
_IGNORE_PATTERNS: list[tuple[str, re.Pattern, str]] = [
(
"startup_probe",
re.compile(r"startup probe failed", re.IGNORECASE),
re.compile(r"startup (?:tcp |http |grpc )?probe failed", re.IGNORECASE),
"Cloud Run cold start — startup probe failure is transient and expected",
),
(
+10 -1
View File
@@ -19,6 +19,9 @@ from backend.services.fake_workers import fake_workers_enabled
logger = logging.getLogger(__name__)
# Stripe's SignatureVerificationError message when the header isn't "t=...,v1=..." at all.
UNPARSEABLE_SIGNATURE_HEADER = "Unable to extract timestamp and signatures from header"
# Credit packages available for purchase
CREDIT_PACKAGES = {
@@ -327,7 +330,13 @@ class StripeService:
event_dict = json.loads(str(event))
return True, event_dict, "Webhook verified"
except stripe.error.SignatureVerificationError as e:
logger.error(f"Invalid webhook signature: {e}")
if UNPARSEABLE_SIGNATURE_HEADER in str(e):
# Scanners/link-preview bots POST garbage headers; Stripe never sends one.
logger.warning(f"Invalid webhook signature: {e}")
else:
# A well-formed header that doesn't match means a wrong webhook secret or a
# tampered payload: every real delivery would fail, so keep it page-worthy.
logger.error(f"Invalid webhook signature: {e}")
return False, None, "Invalid signature"
except Exception as e:
logger.error(f"Error verifying webhook: {e}")
@@ -217,6 +217,12 @@ BLOCKING_CALLS = {
"get_tenant_config", # GCS read on a cache miss (tenant middleware, up to 1.2s)
"append_log_to_subcollection", # job-log write; on the loop it is offloaded, see log_write_offload
"append_worker_log",
# 2026-10-22 #alerts triage: audio-editor routes ran ffmpeg + GCS transfers on the loop
# (storage_service.upload_file stalled 10.3s per edit).
"apply_edit",
"get_metadata_from_gcs",
"upload_file",
"download_file",
}
_BLOCKING_MODULE_CALLS = {("requests", "get"), ("requests", "post"), ("requests", "put"),
("requests", "delete"), ("requests", "request"), ("time", "sleep")}
+27
View File
@@ -210,3 +210,30 @@ class TestStripeServiceMetadata:
metadata = call_kwargs['metadata']
assert len(metadata['notes']) == 500
class TestWebhookSignatureLogLevels:
"""Scanner garbage logs WARNING; a well-formed but wrong signature stays ERROR (pages)."""
@pytest.fixture
def stripe_service(self):
with patch.dict(os.environ, {
'STRIPE_SECRET_KEY': 'sk_test_fake',
'STRIPE_WEBHOOK_SECRET': 'whsec_test_fake',
}):
from backend.services.stripe_service import StripeService
return StripeService()
def test_unparseable_header_logs_warning(self, stripe_service, caplog):
with caplog.at_level('WARNING', logger='backend.services.stripe_service'):
ok, event, _ = stripe_service.verify_webhook_signature(b'{}', 'garbage')
assert not ok and event is None
levels = [r.levelname for r in caplog.records if 'Invalid webhook signature' in r.getMessage()]
assert levels == ['WARNING']
def test_wrong_signature_logs_error(self, stripe_service, caplog):
with caplog.at_level('WARNING', logger='backend.services.stripe_service'):
ok, event, _ = stripe_service.verify_webhook_signature(b'{}', 't=1700000000,v1=deadbeef')
assert not ok and event is None
levels = [r.levelname for r in caplog.records if 'Invalid webhook signature' in r.getMessage()]
assert levels == ['ERROR']
+8
View File
@@ -6,6 +6,14 @@ Key insights for future AI agents working on this codebase.
---
## A lazy import on the loop can stall it for 15s: the image had no precompiled bytecode (Oct 2026, v0.282.1)
The hourly `process-stale-reviews` cron stalled the API loop 10–15s, 4 times in 2 days. The culprit was `internal.py:670`: a lazy `from backend.services.conversion_emails import ...` inside an `async def`. The stall stack ended in `importlib … _write_atomic`. The backend Dockerfile never ran `compileall`, so the first import on each instance compiled the module and wrote a `.pyc` on the event loop, on a CPU-throttled instance.
**Fix:** import at module top, and `RUN python -m compileall` over `/app/backend` and `/app/karaoke_gen` in `backend/Dockerfile`. decide had the same lesson for its cold start (decide #167).
**Also in this round:** the audio-editor routes ran ffmpeg and GCS transfers on the loop (10s per edit). `BLOCKING_CALLS` in `test_no_blocking_async_routes.py` now includes `apply_edit`, `get_metadata_from_gcs`, `upload_file` and `download_file`.
## A stateful fake backend finds bugs that static mocks can't (Oct 2026, v0.275.0)
The first run of the local-stack suite (real API on the emulators, fake workers) found that every job confirmed at "This song is longer than expected" would stall forever. `confirm-duration` moved the job to `separating_stage1`, which can only move on to `separating_stage2`, and the lyrics worker only advances jobs from `PREP_PHASE_STATUSES`. No unit test could see it: each one checked its own half (the route asserted it set `SEPARATING_STAGE1`, and the screens gate tests asserted the prep statuses). Only walking a job across the boundary did. No production job had confirmed a duration yet.
@@ -22,6 +22,44 @@ describe('isBenignError', () => {
expect(isBenignError(abort)).toBe(true)
})
describe('translate-induced DOM conflicts', () => {
const removeChild = new Error(
"Failed to execute 'removeChild' on 'Node': The node to be removed is not a child of this node."
)
removeChild.name = 'NotFoundError'
const insertBefore = new Error(
"Failed to execute 'insertBefore' on 'Node': The node before which the new node is to be inserted is not a child of this node."
)
insertBefore.name = 'NotFoundError'
afterEach(() => {
document.documentElement.className = ''
})
it('is benign while the page is machine-translated', () => {
document.documentElement.className = 'translated-ltr'
expect(isBenignError(removeChild)).toBe(true)
expect(isBenignError(insertBefore)).toBe(true)
document.documentElement.className = 'dark translated-rtl'
expect(isBenignError(removeChild)).toBe(true)
})
it('is benign when translator markup is present without the <html> class', () => {
const font = document.createElement('font')
font.setAttribute('style', 'vertical-align: inherit;')
document.body.appendChild(font)
try {
expect(isBenignError(removeChild)).toBe(true)
} finally {
font.remove()
}
})
it('is still reported when the page is not translated', () => {
expect(isBenignError(removeChild)).toBe(false)
expect(isBenignError(insertBefore)).toBe(false)
})
})
it('does not suppress real errors', () => {
expect(isBenignError(new Error('Cannot read properties of undefined'))).toBe(false)
expect(
+18
View File
@@ -121,12 +121,30 @@ const BENIGN_MESSAGE_PATTERNS = [
/resizeobserver loop/i,
]
/**
* Browser page translation (Chrome/Edge "Translate this page") rewrites the text
* nodes React owns, so React's next commit throws NotFoundError from removeChild /
* insertBefore. The crash is caused by the translator, not our code, and only
* counts as benign while the page is actually translated (Chrome marks <html>
* with a `translated-ltr` / `translated-rtl` class).
*/
const TRANSLATE_DOM_CONFLICT = /Failed to execute '(removeChild|insertBefore)' on 'Node'/
function isPageMachineTranslated(): boolean {
if (typeof document === 'undefined') return false
if (/\btranslated-(ltr|rtl)\b/.test(document.documentElement.className)) return true
// Translators that don't mark <html> (e.g. Edge) still wrap the text they rewrite in
// <font style="vertical-align: inherit">, which our app never renders.
return document.querySelector('font[style*="vertical-align: inherit"]') !== null
}
export function isBenignError(e: unknown): boolean {
if (e instanceof DOMException && e.name === 'AbortError') return true
// Some browsers surface the same media abort as a plain Error.
if (e instanceof Error && e.name === 'AbortError') return true
const message =
e instanceof Error ? e.message : typeof e === 'string' ? e : ''
if (TRANSLATE_DOM_CONFLICT.test(message) && isPageMachineTranslated()) return true
return BENIGN_MESSAGE_PATTERNS.some((re) => re.test(message))
}
+12 -2
View File
@@ -160,8 +160,12 @@ def create_kn_data_sync_resources(all_secrets: dict) -> dict:
),
),
service_config=cloudfunctionsv2.FunctionServiceConfigArgs(
available_memory="512M",
timeout_seconds=300, # 5 minutes
# 512M OOMed on the full sync (2026-10-22). The CPU has to be set with it:
# the function was pinned at 0.333 vCPU, which caps memory at 512Mi.
# A full vCPU also keeps the run well under the scheduler's attempt deadline.
available_memory="1Gi",
available_cpu="1",
timeout_seconds=540, # 9 minutes; the full sync took up to ~185s at 0.33 vCPU
min_instance_count=0,
max_instance_count=1,
service_account_email=sa.email,
@@ -211,6 +215,9 @@ def create_kn_data_sync_resources(all_secrets: dict) -> dict:
region=REGION,
schedule="30 4 * * *",
time_zone="America/New_York",
# Above the function's 540s timeout, so the scheduler never gives up on
# (and retries) a sync that is still running; the 180s default did.
attempt_deadline="600s",
http_target=cloudscheduler.JobHttpTargetArgs(
uri=function.url,
http_method="POST",
@@ -237,6 +244,9 @@ def create_kn_data_sync_resources(all_secrets: dict) -> dict:
region=REGION,
schedule="5 3 * * *",
time_zone="America/New_York",
# Above the function's 540s timeout, so the scheduler never gives up on
# (and retries) a sync that is still running; the 180s default did.
attempt_deadline="600s",
http_target=cloudscheduler.JobHttpTargetArgs(
uri=function.url,
http_method="POST",
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.282.0"
version = "0.282.1"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
@@ -43,6 +43,18 @@ class TestStartupProbePattern:
assert result is not None
assert result.pattern_name == "startup_probe"
def test_default_startup_tcp_probe(self):
"""Cloud Run's own wording names the probe type (seen 2026-10-20)."""
from backend.services.error_monitor.known_issues import should_ignore
msg = (
'Default STARTUP TCP probe failed 1 time consecutively for container '
'"karaoke-backend-1" on port 8080. The instance was not started.'
)
result = should_ignore("karaoke-backend", msg)
assert result is not None
assert result.pattern_name == "startup_probe"
def test_startup_probe_uppercase(self):
from backend.services.error_monitor.known_issues import should_ignore